Part of the issue with the garrisons is that time spent in garrisons and the reward for doing so scales linearly per character you intend to have viable at max level. When something is so easy to do, it's hard to not spend a bunch of time doing it on your alts so they get gear and lots of extra gold. If you exploit the garrison to the maximum it might take 15-20 minutes, but if you have 4 characters....

I now either mine or herb on each character, never both, sometimes neither, and will probably be tearing out two of my three barns since keeping all three stocked is time consuming. I check followers no more than once a day unless I have the rare mostly free weekend, and may break down and get an addon to at least make the clicking process go faster. I've stopped doing the garrison pet battle since the XP hypernerf and only do Erris. I check salvage once a week. I may cut down further, just because the Garrison sucked way too much time when I was getting maximum benefit, but if I'm going to keep anything, it's the followers. The gold haul from the inn is significant.

I finally got the one Harrison Jones daily I was missing, and have now experienced his incredible OPness. I'm doing those dailies on my alts whenever possible to cut down leveling time. 675 Harrison takes a follower from 90-100 in one level 100 XP mission, and the follower will contribute. It's amazing!