Are you using the two weapon fighting rules or iterative?
If you are using a full attack to two weapon fight then the penalties apply. If you are merely using iterative and changing which weapon deals damage then you are not engaging in two weapon fighting and the penalties do not apply.