It's not a technicality, it's just how full attacks work. If you aren't using the two-weapon fighting special attack, you don't take any penalty for switching between two weapons. You only get the penalty if you use the offhand weapon to get an extra attack. If simply wielding an extra weapon imposed the penalty, nobody would ever wear gauntlets.

History lesson! Back before the 3.5 update, characters used to be either right-handed or left-handed, and you could take a feat to become ambidextrous. Any attack made with your non-dominant hand took a -4 penalty. This rule was removed in the 3.5 update, and all characters are now functionally ambidextrous.