While pre-cooling my refrigerator before a trip earlier this month, I was able to run my refrigerator (by itself) for a good 48 hours without the house batteries dropping below 75% on the indicator, with the solar panel working, with the thermostat set to "3" and my refrigerator thermometer showing 38°F.
One important thing, the refrigerator automatically switches to 120vAC mode if the inverter is set to auto/invert. To keep the refrigerator operating on 12vDC, you must set the inverter to "Off" or "Charge Only."
So, you could be seeing a drain from the inverter (which also uses power in its own right) as well as the refrigerator if you had it set to "Auto/Invert." Plus, the refrigerator itself uses more power in 120vAC mode than in 12vDC mode.
If memory serves, there have been a couple of folks who simply unplugged the 120vAC cord of their fridge (which requires delving into the recesses of your galley cabinet) in favor of operating on 12v
only and saving electricity. I've given the idea some thought, too, but was too lazy (so far).