Your DM could simply rule that as you are magically bound to your familiar, the effects target both of you.
That's what I would do.
Plus probably a secondary nasty story effect (some results of draws might free your Familiar from your service resulting in a NPC villian, loss of access to the spell etc until you can find a way to fix it - likely by completing a quest).
I find these threads weird. Its like people play in games without a DM or something. I know it's theory craft, but hey.