Brinnie, for example, knows they're 'shifted'.
Yes. But its not specified what shifted actually means.
So it does not really tell us anything.

And the Norns. If it wasn't for Kat's time manipulations, Annie would have died.
This does not say anything about Annie not being the "real" Annie either.
So because someone, at some point saved her life, she is suddenly not "real?"
That does not make any sense.