I like Siege as a dependent series of hotspots; depending on how the morale is doing and the breaches around the map the PCs may have more or less to do. Heroes of Battle has decent rules for morale and morale checks. As it stands though, I find the raw victory point calculations a bit of a dull overall calculation (one that the PCs almost certainly win). Rather use those for solving individual phases of the siege; e.g. do the walls fall, can the breaches be contained, does the army morale suffice, do the mercenaries and outside helpers desert or even switch sides, etc. Then use those minor outcomes to sum up the overall outcome and the minor outcomes can in part be solved with simple mathematical comparison of the defender and attacker strength. But yeah, the giant encounter in particular is weird in that you're essentially going deep into enemy territory and yet for some reason the main enemy forces seem to be nowhere to be found! At least quick response units/reserves should head there ASAP - you don't leave your artillery unprotected.

And yeah, tracking the party travel times within the town walls also makes it more dynamic; the giants and the dragon alike have plenty of time to do damage as the PCs get there. This makes things more tense and rewards the party actually making haste with fast movement options (Owls, other mounts, magical transportation, etc. - of course, living transports can tire). Same with Skather, though that's more of a special case in that Skather can't actually kill anyone, just incapacitate due to the Str damage not being lethal and him never having a chance to actually CdG targets in enemy territory.