I know this feeling. Just jump into a channel that has alot of people in it and ask if anyone is free to play a game of league. That is just about the only way to get people going in mumble.
Later on when you get everyone on your friends list you can often just spam invites to people who are free and then set up a channel in mumble that you join together for your game.
Edit: Zed'd