Ha, I feel this in my bones. I once tracked every time we had to look up a house rule and found it ate like 40 minutes a night. My group still didn't care. They said the arguing was part of the fun. Guess some people play games for the rules and some play for the yelling.
Isn't that just how it goes with some groups? In my experience the chatter and the silly debates are half the fun, and trying to cut that out can feel like you're killing the mood, even when you just want to get through a turn. Your mileage may vary, but I've learned to pick my battles and save the rule checks for the big stuff.
Keeping a small timer on the table changed things for our group. We set it for two minutes when a rules question comes up, and when it dings we either ask the GM for a quick call or house rule it and move on. It stopped us from falling down a rabbit hole over tiny stuff, and the chatter still happens during breaks and between turns. The best part is nobody feels like the fun police, since the timer is the bad guy instead of me. Have you tried something like that, or does your group do better without any structure?