Right. They might not actually axe you right there unless other factors are in play(ie, if you were dumb enough to cast it non-stilled/silent, from where they can see you), but their attitude is going to be pretty darned bad.

I would have no problem dropping them a step lower on the diplomacy table to represent that.