I'm nervous to give suggestions without seeing it, and knowing what kind of primer (like Latex or oil-based), a lot or a little, etc.
That said - I've used paint-cleaning clay on my cars and Airstream with great success to clean all kinds of foreign matter off the finish (paint overspray, bugs, tar, sap, etc.) The clay 'picks' off the contaminants and leaves the surface clean, with no swirl marks, etc. The surface is first sprayed with a mist of a special solution, and then the clay is gently used in a polishing type motion. I would test a small out of the way area first to see the results (probably good advice for anything you use). It doesn't take long or much force to get stuff removed using the clay approch, in my experience.
There are several brands on the market that you will find if you search on 'paint cleaning clay' on the internet - or I can tell you the stuff I use (is it ok to post brand name info on these forums?).
I'm sure you'll get other suggestions too - so you can choose the approach that suits you best!
Good luck on the outcome