It isn't necessarily unrealistic depending on the tournament format.

There are several forms of single elimination formats.

The first is a fixed bracket. Examples include the tennis grand slams and the U.S. college basketball tournament. In these, the bracket is created before the tournament begins. These can be seeded or unseeded. In a fixed bracket tournament, half of the competitors aren't in your half of the draw. So, even without seeding, there is a 50-50 chance that you won't face the most powerful opponent (assuming neither of you lose beforehand) until the finals.

The second is reseeding after every round. The examples that comes to mind are the National Hockey League playoffs and, to some extent, the National Football League playoffs. In a reseeding tournament, the competitors are reseeded after every round. So, if you are seeded the lowest, you'll have to face off against the highest remaining seed every round. This format is toughest for low seeds.

The third is a random draw to determine matchups after every round. The FA Cup in England does this.

Normally, you will face tougher opponents as you advance. Remember, weaker competitors are being eliminated every round. In addition, in a seeded tournament, if you have a good seed you'll naturally face tougher opponents (assuming the favorites win). Conversely, if you are seeded poorly, you'll likely have an easier second round than the first round (unless it is a reseeded tournament).

In a totally random seeded fixed tournament, you'll still see enemies get progressively tougher in general, and there's a 50-50 chance that the most powerful opponent will not be in your half of the bracket.

Even in a random draw, it is certainly plausible that due to random chance that you avoid drawing the toughest opponent until the final. Again, enemies will, in general, get progressively tougher as weaker enemies are eliminated by fellow competitors.