I assumed he did not want to pay in the first place and was more "annoyed" than actually "afraid".
If you want "afraid" in, just assume he is somewhat broke and does not want to pay the little gold he has, but he is afraid people in the town find out about his financial situation. That'd explain anything we saw?