Adamantine clothing is actually a pretty terrible idea on account of clothing wearing out and eventually rotting away. This includes adamantine clothing. Armour however, lasts forever, and nothing short of atom-smashing it or dropping it into a source tile for the magma sea will destory it.

Besides. No matter how much cloth you have, every caravan will ALWAYS bring a dozen or so bins of it. Far easier than cultivating it ourselves.