That was it! can't believe I didn't see those rivets.
Still had to pry some...gently. There are also a bunch of sping-clip-thingies in the groove that it goes into; couple of them broke, but most let go ok. should be ok to put it back in place.
and yes, there were a few hidden pop-rivets in the skin underneath this trim, as I expected.