Okay, so I think that the answer to this is that it's totally normal and that the pressure relief valve is operating normally but I'd feel pretty dumb if something was broken.
So, I filled the LP tanks at the local U-Haul when it was around 50 degrees out. Then later, when it was warmer and they'd been sitting in the sun for a while, I noticed that the valve was letting gas out. It's coming out of what I would describe as the back of the valve, or the side opposite of where the hose from the regulator screws in. I came home from work the other afternoon and it was doing it again. It's not constant and seems to happen after the tanks have spent some time in the sun.
After some research I'm assuming that it's something called the pressure relief valve, that the gas has expanded and that the valve is operating properly. Then again, my first boss always used to say, "assumptions are the mothers of all f*ckups" so I figured that I'd ask the experts.