Inside a Wall Street 'secret society' party

Christmas Party Horn

That infamous Wall Street party in 2012 was a "private party," not a meeting of a secret society, billionaire investor Wilbur Ross says.

Fine eating, drinking, chanting in drag outfits and mocking politicians are some of the things that happened at a Wall Street secret organization party in January 2012- at least that's what financial journalist Kevin Roose says in his new book "Young Money."

Roose describes his experience crashing the Wall Street "secret society" party, where the elite of the financial industry, including billionaire investor Wilbur Ross, got loose and joked about bank bailouts-something that has stirred critics.

(Read more: Five tycoons who mind the wealth gap )

At the annual secret Kappa Beta Phi party, new members - dressed in costumes - are invited to join the fraternity's exclusive rank, Roose says.

During an appearance Friday on CNBC's " Squawk Box ," Ross talked about the gathering. "It's a little bit of a misnomer to call it a secret meeting. It's a private organization and it was having a private party," Ross said.

"What some of the people outside were complaining about the theory that this was all people who had been bailed out by the government and now they were making fun, well I certainly didn't get any bailout money," Ross said. "Chris Williams, my predecessor as the head of the organization who runs one of the larger minority investment firms in the country, he didn't get a bailout. My successors ... didn't get a bailout. So I think there's been a little bit of hyperbole introduced into the equation."

(Read more: 10 richest neighborhoods in US )

And about the jokes by partygoers, Ross added: "It is true they were spoofing each other for a couple of hours, and when you have that kind of environment, unfortunately some people say some things that they probably wish they hadn't. But I wonder how many people have never said anything that they would like to retract, I think very few."

-By CNBC's Silvana Ordoñez. Follow her on Twitter @newsdumonde

JefferiesAnd the Best Place to Work in the global financial markets 2018 is...

Register for HITC Business News