Could you paste the subclass into your post? Chunks of it were cut off when I tried viewing it in homebrewery.
Fiend Hunter
I would have expected Intelligence (Investigation) in there somewhere.
Lesser Totem Tattoo
Gaining a further tattoo at 5th doesn't sit well with me. It's off schedule for when rangers get subclass features, which makes this harder to keep track of and remember. It feels inelegant.
Summon Spell Dampening
While I recognize that this is a good homage to the prestige class ability, I can't imagine ever using it if I were to play a Totemic Demonslayer. The activation taking an action is a steep cost and the effects are extremely specific and unlikely to occur in the average adventure. There aren't a lot of conjuration spells that foes are likely to cast and fiends summoning other fiends will only happen if you're playing an extremely specific type of game.
I think this needs to be easier to use and to have broader effects when it is. Another option would be to move it up to be the 15th level feature and have it produce an
antimagic field instead.
Greater Totem Tattoo
Like I mentioned with the lesser, gaining subclass features on non-subclass feature levels feels inelegant.
Tattoos
Not allowing these to ever be changed reads as combative. I'm aware that was the wording of the original prestige class, but it feels out of line with the current design philosophy for 5e.
Only being able to use each tattoo once per long rest is incredibly restrictive, too. For comparison, a Horizon Walker of 3rd level can do 1d8 extra force damage at the cost of their bonus action, each and every turn. A 3rd level Totemic Demonslayer with a snake tattoo can deal an extra 1d8 + double Strength modifier damage for three rounds
per day. Unless there's only a handful of round of combat in a single adventuring day, the Horizon Walker comes out well ahead.
Lesser
Allosaurus
Taking a bonus action to activate a tattoo conflicts with this tattoo taking a bonus action to use its secondary attack.
Do you need both hands free to make use of the secondary attack? Or can you use the hand you're grappling with to make that attack?
Ape
Taking a bonus action to activate a tattoo conflicts with this tattoo taking a bonus action to rend.
Baboon
This is a utility feature. It's a really cool utility feature. I'd want to take it, too, but I don't think I would. Please don't make players choose between cool utility abilities or combat capabilities.
Badger
What.
It's been a while since I've seen a feature that is explicitly better than another one, but Badger is literally a better option than Sea Turtle, making Sea Turtle a trap option. Badger is also far and away the best option here, because it takes the primary schtick of the barbarian and adds it to the ranger. I suspect that Badger is the only option players would choose, were they to play a Totemic Demonslayer.
Either bring everything else up to this power level or get rid of this entirely. I'd suggest the latter.
Boar
This is awkward to activate. The player has to know they're likely going to drop to 0 hit points.
If they do, this makes them nearly unkillable. Like Badger, this is giving me a sour taste in my mouth of encroaching too much on the barbarian's territory.
Crocodile
This is cool as well! But it's also a feature that would be considered fluff in 5e, which ultimately means it's not worth taking when compared to any other option.
Deinonychus
Taking a bonus action to activate a tattoo conflicts with this tattoo taking a bonus action to use.
It also is the third tattoo I've read that seems to be a variation on the theme of "use a bonus action to make an unarmed strike that does more damage." Could those tattoos all be consolidated into one? They feel duplicative.
Dimetrodon
This is mostly utility as well. There's not much point in taking it when compared with the other options.
Sea Turtle
As explored above, a trap option when compared with Badger.
Snake
Here's the fourth "make an unarmed strike" tattoo. Oddly, this one doesn't mention a bonus action. Which means we finally see one of these that doesn't conflict with the activation cost for tattoos. And probably means this is the best option of the four.
Greater
Ankylosaurus
This is still worse than a Badger tattoo.
Giant Wasp
This requires you to be fighting fiends to ever be used. For that alone, it's unlikely to be picked. It's got one rare and slightly cheesy use case: A Totemic Demonslayer who uses a bow could use this to coat all of their arrows. If they were fighting fiends, this would suddenly become really useful.
Phanaton
Again, it feels bad as a player to have to choose between utility and combat features. This is a utility feature and has quite specific use cases for when it comes in handy, which make it unlikely to be picked.
Tabaxi
This is nigh useless. Any creature whose resistances can be beaten by silvered weapons can also be beaten by magic weapons. In addition, natural attacks aren't a thing in 5e. Assuming you mean unarmed strikes, a ranger has very little ability to make good unarmed attacks, even with the lesser tattoos this subclass adds.
Triceratops
This is a small damage spike. It's decent, but nothing spectacular.
Tyrannosaurus
Also a small damage spike, but lasts longer. Probably more useful.
Whirlwind
This outclasses Phanaton in every way except for duration. But it's a utility ability that emulates fly for a tenth of the duration and you only get it at 15th level at the earliest. This feels bad.
I think breaking away from copying the prestige class exactly into 5e would help this a lot. There are a lot of aspects of 3.5 design philosophy in this that don't fit in well in 5e. I'm left with the sense that if I choose this as my ranger subclass I wouldn't be getting much out of it other than a couple new ways to make unarmed strikes and a couple options for miniature damage spikes if I made it to really high levels.
The other impression I got was a sense of "I'm a ranger who actually wanted to play a barbarian" and that comes entirely from the Badger tattoo.