I bet you have mold in the AC Duct Work. Using a Good high quality anti allergen filter element will help. So will cleaning the duct work. I think after you leave, the humid air condenses inside the cool ductwork leaving moisture which facilitates the mold. If there was a way to keep the blower running once the AC was turned off and until the duct work reached ambient temps, I bet that would help a lot.
Also, what kind of insulation or barrier did you use between the sheetrock and outside metal walls of the building?
Also, what kind of insulation or barrier did you use between the sheetrock and outside metal walls of the building?