It depends a little bit on the party comp, but ill usually put both points into my damage/armor stat. If I'm using a shield, ill split between the damage stat and con, because shields need con to equip, and on my rogue I do the same, but for wits instead of con. I only really drop points in memory once I start running out of spellbook space. If I'm not using a shield, I ignore con entirely. The armor stats from gear are just so much more important to survivability because they block status effects. If I run out of armor and start taking HP damage, that character is going to get CC's to uselessness anyway.
If I'm playing, say, a battlemage that uses melee and magic abilities, ill split between both damage stats.