Also I don't care whether or not Motti technically outranked him, he was disdainful of Vader's force powers, and a practical demonstration was the best way to bring him back in line.
Vader was there as the Emperor's personal commissar, outside the normal chain of command. I think he only listened to Tarkin as long as he did because he was on board Tarkin's space station. Everyone else's rank was irrelevant.
On the other hand, in Empire Strikes Back, I believe he had formal command over his own fleet, which is why he can just up and murder his Admirals and then promote new ones.