YES, it can be too cold to polish properly. You didn't say which type or make of polish compound that you had but, Nuvite says that the outside temp should be above the 60 degrees range before using it's product.
Several years ago, I tried to strech my polishing efforts into the month of Oct and, found first hand, when its cold out..you might as well give it up..The polishing compound doesn't work..It had gotten down to frost~!
Think Spring..it's only a few weeks away...
WBCCI 5292 AIR 807
New England Unit