Big Sam is now in charge at Sunderland but spent several years with the Hammers, where he was even booed after victory due to issues supporters had with the manager's prefered style of play.
And Sullivan says a change was definitely needed over the summer, despite stating that Allardyce generally did well.
Sullivan is quoted as saying by the Chronicle: "Unfortunately with Sam, although he did a good job, the [West Ham] supporters never warmed to him.
"Nobody ever sang his name. Maybe ‘cos Sam swaggered a bit. Maybe it didn’t help.
"When we were booed off after winning, it was the weirdest thing. I felt the same. That’s one of the reasons we knew we had to make a change."
West Ham face Manchester City this weekend sitting sixth in the Premier League table. Sunderland, meanwhile, are 19th but could find themselves in a much healthier position if they beat Bournemouth during their relegation six-pointer this weekend.
Allardyce was never a fit when it came to the West Ham fans but he should gel better with the Sunderland faithful.
It's hard to read exactly how Big Sam feels about the relegation battle. Anything is possible but the odds are generally against Sunderland when it comes to the fight for survival - a good January transfer window would help a lot.