Why is "average" so frequently taken as a severe insult? [RANT]

[Iruka]

That's when this usually comes up, at least for me. If someone asks "why aren't you calling me a genius right now?" or something to that effect, and they only seem decently smart to me, I'll answer "because you only seem decently smart to me."

Then I get yelled at for "insulting" them.

In what context does this stuff come up? I hear similar stuff when someone knows some exotic bit of knowledge or comes up with an at least clever solution for some problem. It is never meant completely serious, just celebrating their own success a bit.
In a joking tone "because you only seem decently smart to me" would probably be acceptable, in a dead serious tone it would quickly ruin the mood.