I think it's partly because it's something I pretty much never do.
In fact, it occurs to me I'm generally fine calling my best friend, because I'm used to talking to him over the phone.
Come to think of it, I guess it's insecurities. If there's any insecurity or worry or whatever I might have in connection with the person in question, it will automatically be worse over the phone than IRL.