There's a couple nice solutions to the modifier problem. My favorite that works for the whole party is to have a central list of (de)buffs that all the players can see and get used to checking. A whiteboard works best but a piece of paper in the center of the room is just fine. Any time someone brings up a buff just write it down!

Fleeting conditional modifiers (like flanking) are the kind of thing everyone needs to be conscious of.