A familiar, a special mount, or an animal companion are more than just some dog you happen to have, or a horse that you ride from Point A to Point B. They are one of your closest relationships, your friend, your faithful companion you've been with through thick and thin, and a dear ally. You spend a lot of time with them (most of this happens as off-screen assumed roleplaying in a normal game).

Your cohort is also not 'some guy who brushes your horse'. He's your dear follower, a close friend, somebody who has devoted their life to your example. You presumably also spend a lot of time with them off-screen.

The penalty is supposed to reflect the relative difficultly of having 'two best friends' travelling with you all the time.