One of the best places we have done some primative camping is Assateague Island - on the Md side.
The campground is the
Assateague Island National Seashore - not the State Campground.
The campground is very primative, providing only water (not at your site), pit toilets, and cold showers only. Dump stations are available. Bugs can be a problem - but you should be ok in May.
There are some cool things to see in the visitor centor, and some of the hikes are great.
Here is the link to the website:
Assateague Island National Seashore - Camping At Assateague (U.S. National Park Service)
Hope this helps!