You quickly take stock of what could have caused him to ignore that damage, and quickly realize that you are dealing with a warlock of at least moderate talent. He likely has a spell or two at his disposal in the form of an invocation. Furthermore, his eldritch blast can be quite deadly.

Your missile flies at him and impacts squarely in his chest. He grunts from the impact but does not appear to be that badly injured from it.

You advance right next to him.

"Boy, that was a mistake." He says. He begins to cast an invocation at you, providing you an opportunity to attack.

Spoiler
Show
full 3 damage was dealt, just not a major chunk of his hp from that.

AoO for you, if you can take it. Also roll a will save in case the invocation goes off.