It's not hard, maybe, but annoying. I remember that while playing a priest-type Dunmer, I had to spend a lot of money on training Long Blade because otherwise I was one skill short of advancing in House Redoran. Another character of mine, an Argonian spear fighter in medium armor, couldn't really advance in Fighters' Guild at all because neither Spear nor Medium Armor are associated skills there.