Have it apply a -10 penalty to Concentration checks. Or a -5 if you think that's too harsh, but concentration's a joke past low levels anyway.

Really, in general I've grown annoyed at the condition options as a DM for how so many of them mess up martials but don't really affect casters. I'm in favor of expanding negative conditions for casters, dazzled seems as good a place to do that as any.