TLD badges will do very little to detect DU exposure, whether in a drawer, or worn properly. Depleted uranium is 40% less radioactive than naturally occuring uranium, and the radiation exposure is very small. The danger of DU comes from ingestion (similar to lead poisoning, but worse), and radiation exposure of the internal tissues after ingestion (which a TLD can‘t measure). DU rounds, upon impacting their target, can partially vapourize, allowing small amounts of uranium to be breathed in. For more info, see:
http://www.gulflink.osd.mil/faq_17apr.htm
That being said, if a TLD badge was issued to you, sticking it in a drawer isn‘t the best idea. The military is trying to evaluate a risk to our soldiers (and you personally), and if it is not worn properly, the risk analysis can be skewed in an unfortunate "zero risk" direction.