Other downsides: you don't get to share new stuff. You don't get to show them some of your favourite stuff and see how they react to seeing them for the first time ever. You don't get to share that part of you with them. It's something they already have at the time you meet.
And similarly, you won't get to learn about their stuff. It's awesome to be able to watch any episode of a show and remember that your significant other made you discover it. It's nice to both like a show, but in comparison I would definitely say that you get more out of not knowing the same stuff.