Big Sam took over at Upton Park following their relegation under the hapless Avram Grant back in 2011. He promptly rebuilt the club, bouncing back to the Premier League at the first attempt.
The Hammers went on to finish 10th, 13th and 12th in the next three years in the top flight, an achievement Allardyce said he was very proud of.
He had helped West Ham make their best ever start to a Premier League campaign last season as the club sat fourth at the turn of the new year.
But just three wins in the rest of the season saw them drop like a stone down to 12th and ultimately cost the former Newcastle United boss his job.
Many fans had grown fed up with Allardyce's methods and philosophy by the end and co-owners David Sullivan and David Gold elected to replace him with Slaven Bilic.
Hammers fans' Allardyce dislike was a 'misconception'
But when it was put to Allardyce by Fantasy Football Show presenter Max Rushden that Sullivan had claimed he 'never really had the love of the fans', Big Sam disagreed.
"I think its a little bit of a misconception, I don't think it's a Sam Allardyce thing," the 61-year-old told the Fantasy Football Show.
"West Ham fans, historically, when things aren't going the way they think (they should be) give their managers a bit of stick and I accept that, I don't expect that things will be rosy all the time."
"I never had a problem outside West Ham, walking around Canary Wharf in London with a West Ham fan coming and showing disapproval to me."
"There's always somebody and I think with social media today a very small minority of fans can create a big fuss about what the manager is or isn't doing but I always felt the vast majority appreciated where I took them," he added.
The bitter medicine West Ham needed but let's all move on
Big Sam was in many ways the bitter medicine the Hammers had needed for years.
But after doing a first class job overall, West Ham were all better and many fans felt with his contract coming to an end in May, it was a natural parting of the ways for both him and the club.
There should be no controversy or animosity as he did what was asked of him while Sullivan and Gold also backed him at every opportunity – let's not forget the huge amount of money they spent on his three most expensive signings Andy Carroll, Matt Jarvis and Modibo Maiga.
With Bilic now in charge and getting off to a great start by beating title hopefuls Arsenal 2-0 on the opening day and Allardyce admitting he has no shortage of job offers if and when he decides to get back into management, both parties should now just move on.