I like Nicholaston Farm on the Gower, tents were separated from Caravans last time I went and it felt like proper camping. You can walk out of the bottom of the campsite to explore some of the best coastline in South Wales.
As you are in Swansea though OP it might be a bit of a busman's holiday.