The saying "mad as a hatter" refers to the peculiar feature common at some point of time among the hatters: the mercury that was used in production of some hats damaged their nervous systems, and made them… slightly mad.
Lewis Carroll used this fact to create a character for his novels about Alice: the Mad Hatter.
For no matter what the world, men who deal in headwear are men to be trusted above any others. – "Hatter M", Frank Beddor & Liz Cavalier