... while she felt unloved and abandoned because he never explained anything to her and she was just a child?
When he returned and immediately shamed and disciplined her?
When he has not once that we've seen tried to explain this to her in any detail?
Don't get me wrong, I see the factors that make this less egregious, but he's still accountable for his actions and severely in the negative here. As someone else has said, many characters/the plot just shrugging this off only makes things seem worse because of the dissonance.
Slightly aside, I didn't get the impression from anywhere that he left to try to help annie. He felt he couldn't face her after Surma died and then was researching for the court then went to find the psychopomps and thought he could reach Surma through the antenna. He doesn't mention trying to help her. Even Donnie says "what you just saw doesn't excuse anything he did".