I think "uphill both ways" is supposed to imply that the reminiscer is going senile.