My Victron solar controller is connected to the positive and negative busbars with a circuit breaker on the positive leg. Here are pictures to show how mine is connected. If you post a picture of yours, we may be able to help.
I would start by cleaning the connections on the batteries. Your Victron is going through bulk, absorption and float so it seems to be working. Then check all the connections between the controller and the batteries. I don't remember anyone else having a problem with their Victron Solar Controller.
2018 GT27Q, 73 nights 12,177 miles, 600W Solar - 110kWh, PD4655L, 6V Batteries, Blue Ox, 16" tires, 2019 F250 PSD
Sold: 2017 FC25FB, 316 nights 40,150 miles, Propride, 400W Solar - 200kWh in 216 days
2013 Casita SD17 89 nights 16,200 miles