Possibly a throwback to Biblical times, when tax collectors (called “publicans”, or collectors of public revenue) were considered enemies of the people of Israel.  They worked for the government (Roman Empire) and often overcharged and extorted money from the Jews.  For that reason, they were considered "unclean" and treated as heathens amongst the Jewish people.  Oftentimes, they were made to live far away from populated areas, like lepers.

Just a theory...

Regards,
Scott Perry