Tim Sherwood was sacked as Spurs boss last week after his 18-month contract was terminated after less than six months in charge at White Hart Lane.
The Englishman led the North London club to sixth in the Premier League, but was given the axe by chairman Daniel Levy just two days after their 3-0 win against Aston Villa on the final day of the season.
Sherwood reportedly fell out with Brazilian midfielder Sandro during their time working together, while the club were forced to deny reports the manager had punched a player following the defeat to Liverpool back in March.
Speaking to talkSPORT, the former Spurs boss admitted there were players who he thought were not pulling their weight and he claimed he was not afforded the chance of moving the unnamed squad members out.
He said, "I never had no bust-ups with [Daniel Levy], and I don't know if he took exception to anything I said [in interviews], but if he did, he knows Tim Sherwood. He's known Tim Sherwood for eight years previously, and if he thought I was going to change, then he was mistaken wasn't he.
"I think [Spurs] is bigger than any player or individual that goes in there and I think that sometimes it's right to act on behalf of the club when you think that there are players thinking they're doing the club a favour by turning out for them.
"A club like Tottenham should not have those players around the place, and I was trying sieve those players out and I would have done if I had longer to do it, and the club would have benefited from that."
Meanwhile, Brighton & Hove Albion boss Oscar Garcia resigned as the Seagulls' coach after defeat to Derby County in the Championship playoff semi-final last week, and Sherwood's name has come up in connection with the vacant post.
However, the former Blackburn Rovers midfielder has insisted he has not held any discussions with the Championship side.
"I have not spoken [to Brighton]," he added. "I have not been offered the job. I've heard a lot of rumours, but I'm happy to speak to any clubs. I want to stay in management.
"I want to see if the club suits me. What I don't want to do is go into a situation where in five months' time I'm out of job again, that's not what I'm looking to obviously."