I had similar, turned out the rotary switch was shot AND I also found green corrosion under the wire nuts.
My fix was to take a piece of 14 AWG wire and cut off insulation every few inches and on the hot wires going to light bases I cut off corrosion, removed insulation and wrapped the light wire around the 14 AWG and soldered the on and taped them with UL approved electrical tape thus no more wire nuts at all in that fixture and should never be a corrosion problem again.
The rotary switch was removed and a shallow rough box was obtained at Home Depot.
Carlon 1-Gang 8 cu. in. Old Work Flanged Shallow Box-B108R-UPC at The Home Depot
I screwed this with self tapping screws to headliner right next to the light after drilling 1/4" holes in side of box for incoming wires from light fixture.
I hooked the new 14AWG to the top screw and the AS Issued "hot" wire to lower screw on a 120V commercial 15 amp wall light switch.
I put a white switch cover over on and it looks like a standard wall switch except on the ceiling instead of the wall.
Thusly I have a light switch that will never get overloaded, all joining wires in fixture are soldered so no more corrosion. Total cost of conversion was about $2.50.
Obviously I don't have the 2, 4 or 6 bulb option any longer but I never used less than 6 anyway and the LEDs I replaced them with can go wide open for a small fraction of the power used in one of the issue bulbs.
Working down the ceiling line I as the rotary switches go down they will be replaced as above and I will remove all the wire nuts and solder every ceiling connection as I do not trust wire nuts.