The oft-outspoken TV personality stopped by the New York Stock Exchange on Thursday with a message to critics on the Internet.
"Fans have this perception that they can say anything to you because you are a celebrity," Barkley told CNBC. "The only reason I do no social media whatsoever [is because] I don't feel the need to talk to all these nitwits out there...If people have something bad to say, they can just say it on the Internet and it'll just go into the air."
Barkley was asked about a viral email said to have been sent from Knicks owner James Dolan to an angry fan. Dolan, in his response, questioned if the fan was an alcoholic and suggested that he should "start rooting for the Nets because the Knicks don't want you."
While Barkley doesn't agree with the negative statements made by Dolan, he is said he is well aware of the consequences that a potentially negative two-way dialogue with an angry fan could present.
As a result, Barkley has made a conscious decision to avoid social media all together.
"Unfortunately we have given all these trolls the opportunity to voice their opinion [on social media]. Just because you watch Grey's Anatomy doesn't mean you can do the operation," he said.