To elaborate on the castle question, a castle is a defensive structure, made to protect some point of strategic importance (resources, population, river crossing, trade route). Often times, a town is made adjacent to the castle, created to supply the castle with supplies and a work force (at least, this is the norm for England and France).

The bulk of the town will be within a town wall, with the market as the center of the town. The wall often abuts the castle, using the castle as part of the wall. Outside the wall, you will have all the farming and other such activities, although these farms will typically be fairly close to the wall so that the farmers can run into the defenses of the town wall in case of attack.

It is important to note that, often times, if the town was under siege, the townsfolk would retreat into the castle for safety, which means that the castle would typically be large enough to house the population of the surrounding town (albeit not very comfortably… think London Underground during the Blitz). Also, during a siege, many of the townsfolk would form a militia to help defend the castle, so they aren't completely useless during a siege.