I have seen some ledgers etc from the 15th century, from mid-sized Danish manor houses (that is relatively rich knights, but not someone who are above "knight" in the hiearchy). They tend to have 4-10 men at arms (they might be a couple of guys with crossbows, a "gunner" using a blackpowder gun of some kind, and a few guys with pole weapons). But this is at their castle, I doubt they take all of them everywhere.... Then they might have one or more sons, an unlanded brother/uncle/cousin etc, who might or might not have an armed servant. In addition to any normal staff (cook, stable hands etc) which might of course take up arms if need be. In case of siege you might also include gathered peasants with weapons to defend the castle.

But you do see 'knights' or squires (landed lords who have not been knighted) with no retinue of specifically "armed" men, apart from a page/manservant (which might not be fully armoured etc). But you also see richer nobles where their retniue would be much larger, and who would never travel around with less than 20-30 more or less armed men (though they wouldn't be wearing armour or war weapons in everyday life, just as the tank commander is doing his shopping in a tank).

Saxon housecarls fulfilled the role of both bodyguard and household troops, so kinda meet your requirements. A minor thane might only have 1 or 2 housecarls in his personal retinue, whereas King Harold had between 2,000 - 2,500 at the Battle of Stamford Bridge in 1066.
The housecarls in the sense of Stamford Bridge also included Harolds vasals and their housecarls, so it might be misleading to see that figure as a "how many people he had at his home/retinue". That would be more akin to 400-500 men. He would of course most of the time travel around with fewer than even this number (especially if it is peacetime and he is going for a short visit/errand).