The meter on the charge controller is showing you the voltage being put out by the panels. Even though they are called 12 volt
panels they are capable of more than that under the right conditions. It doesn't harm the panels to make power that is not being used however the voltage is also present at your battery connections. You should at least put an insulated cover on the positive and negative battery cables. A piece of old bicycle inner tube taped securely. That will prevent a short between cables, if they touch. It will also prevent voltage being put on the trailer, which could also result in a short. Solar panels can be shorted without harm but the controller might not like it. If you could disconnect the positive lead from the back of the controller that would stop it. If they left enough slack in the wires you could remove the mounting screws, gently pull it away from the wall and disconnect the PV+ wire. That wire will have voltage on it, but could be taped. You could install a switch on the positive wire so that next time you remove the battery, you simply flip the switch to turn the controller off. Hope that makes sense.