You're not losing XP or gaining it (or losing or gaining anything).
The familiar does. He gets the benefits (or penalties) of the draw.
You're not losing XP or gaining it (or losing or gaining anything).
The familiar does. He gets the benefits (or penalties) of the draw.