I think outright banning something is the lazy solution. In a world where teleport (telepork?) spells exist there would logically be precautions that were commonly taken to avoid their abuse.
If you have access to 4th level spells you are a badass. You've earned the right to waltz in to an encampment of goblin bandits, take their golden idol and teleport out before they knew what hit them.
If you go up against an opponent with as much experience as you however, you should expect them to be prepared for Scry/Teleport shenanigans. In fact, my players eventually figured out that many NPCs were so paranoid of scry-n-die attacks that they would often take elaborate precautions against them, simultaneously leaving large gaps in other areas of their defense.
Details like this, I felt, made the world feel a lot more "real."