"Ugh. You are clinging. And if not specifically to annoy me, then why?"
He's not that empathetic even when he's trying to be, and right now he is not trying.
Other options:
no subject
"Ugh. You are clinging. And if not specifically to annoy me, then why?"
He's not that empathetic even when he's trying to be, and right now he is not trying.