The only reason the memory charm exists is that the Oracle doesn't want the legit stuff he says on top of what he's officially asked about (and paid for) to be remembered by his clients. It doesn't make much sense to then dismiss those things he says off the record as "not really prophecies".