The thing is, a good portion of the bow itself moves forward upon firing, and the bolt's mass relative to that of the bow is higher than a bullet's mass relative to that of a gun. So when a crossbow is fired a lot of the total mass moves forward, causing more recoil than you'd expect. And if by "siege crossbow" you mean "ballista", those can have a lot of jump when they're fired.