They may have Vulkem or some other sealer at the seams - especially if they are in the shower area. You may want to try running a putty knife between the seams to loosen them up. Also make sure the whole rivet is drilled out - including the stem which sometimes stays put after the head pops off. We just removed the entire lower skin in preparation for re-doing the floors, lots & lots of rivets! It only takes a couple to keep it from moving...also, any "walls" in the way?
Yes, you do need to access the U-channel in order to replace the plywood and reattach it properly. The panels should end at a rib - the flat ones go the entire length of the trailer though. We just removed a 12'x3' panel inside our Safari...it's a long one! The overlap is only about 3/4-1" at the seams.