Allardyce was in charge of West Ham for four years from 2011 until the end of the 2014-15 season, when his contract was not renewed.
Sullivan has explained why the club made the decision not to renew the contract of the former Newcastle United boss, who is without a managerial job at the moment.
The West Ham co-chairman has also revealed that Allardyce told him last summer that he had the worst job in the world.
Sullivan told The London Evening Standard: “We had a meeting the previous summer and Sam said to us, ‘I have got the worst job in the world.’
“And we said, ‘What do you mean Sam? What’s wrong with West Ham?’ And he said, ‘It’s not West Ham, it’s being a football manager. It is the most demanding, tiring, horrible job.’
“And I thought,‘This man needs a break if that’s how you feel earning £3-£3.5million a year’. So by the end of the season, he needed a break.”
Following the departure of Allardyce at the end of the 2014-15 campaign, West Ham appointed Slaven Bilic as their new manager.
Much is expected of Bilic, who is also a former West Ham player, but the London outfit have not started the 2015-16 campaign well.
Following a 2-0 victory over Arsenal away from home at the Emirates Stadium on the opening weekend of the Premier League season, the Hammers went down 2-1 at home to Leicester City and 4-3 to Bournemouth.