In old school dungeon crawls, yes it matters. One of the constraints on your crawl is the supplies you have, and the threat of wandering monsters (which give no treasure, thus little xp) is a counter-pressure on being TOO cautious.

For an open-table game, tracking time on the macro-level matters, as characters may be recovering from wounds, training, etc., and making sure you only use characters that are actually available helps build the idea of the game as a consistent "world".