It's still lower than the regular skin temperature of +25°C, and definitely lower than the regular body temperature of +37°C, so I'd just be losing heat that way. On top of that, a quick test shows that I'm warmer beneath my shirt than the room around me, so no, I will no remove my fleece jacket.