LaVonne, check that your windows have the proper weatherstripping. It should be a "double flared" type that looks like this
. (Picture from Inland RV website).
I'm not sure if having the wrong weatherstripping would cause the problem you are talking about but if you are concerned about leaks it would be a good idea to get the right stuff in place.
I definitely wouldn't just caulk the gap shut. Sounds like that might cause more trouble in the long run. If you post a picture of the problem, someone on this board may be able to offer the exact solution.