Dice-Based Rage Damage

[Galithar]

So, I was once in a game where we replaced the Barbarian's rage damage bonus with +1d4 damage (it would have upgraded to +1d6 at 9th level and +1d8 at 13th level, but we finished the campaign before hitting 7th). It felt really good - while it was undoubtedly stronger than the default Barbarian, I didn't feel more powerful than any of my party members (a Celestial Patron bladelock and an Inquisitive Rogue), which is always a plus. The die doubled on crits like normal - since I was playing a half-orc, my crits were really beefy (3d12+2d4+stat) damage, but they didn't come up as often as, say, the Rogue's sneak attack or the Warlock's Hex-boosted Agonizing Blast-equivalent +1 crossbow bolts¹.

So, my main question - I really love Brutal Critical despite it being a weak class feature. I like how it encourages you to use a 1d12 weapon instead of a 2d6 weapon. I don't, however, like how it only comes online at level 9. So, my main question: if I used the rage die houserule but then replaced the die with a weapon die on a crit (instead of doubling it, so you'd end up with 3d12 damage instead of 2d12+2d4 damage), would it break anything? If it does, would shoving that upgrade back a few levels (maybe to 3rd level?) make everything shake out more nicely?

¹ The DM gave him a magic crossbow that gave him the "use Charisma for attack and damage" benefit from Hex Warrior, because he wanted his character to use a heavy crossbow and it's not like that's any more powerful in a real sense than just giving him a Wand of the War Mage or Rod of the Pact Keeper.

It's definitely a power boost, but being on critical only I don't think it's going to outclass anyone by that much. It only happens 5% of the time, and it's definitely no worse then a Paladin (or Bladelock) that gets to know they Crit before spending a big spell slot on a smite.

I think it would feel good and may contemplate allowing my players to use this in my games.

Edit: The "Rage Die" is more added power then letting it turn into a weapon die on a Crit. If you and your table were happy with that part, I don't see any reason why it would be bad to use this addition.