I'm just guessing but it probably started during the 1960s-1970s when people started using the term gay and that whole civil rights thing. It probably comes from the skeletons in the closet phrase that's been around since the 19th century.
Maybe they started saying it because they found a guy in a closet with some "South American rent boy"