The devastating late defeat to relegation rivals Rotherham left Millwall six points from safety in the Sky Bet Championship table and with an inferior goal difference.
Holloway's men now face another tough trip north to second-placed Middlesbrough on Tuesday night, and the charismatic boss, who was quick to defend the club's fans after the recent defeat to Leeds United, acknowledged that some fans may now want to see him go.
Talking after Saturday's defeat to Kari Arnason's 85th minute header after Millwall had taken a half-time lead through Martyn Woolford, Holloway told The BBC: "If the supporters had the button they would probably have pushed it."
Though the ex-Crystal Palace, Blackpool and Leicester City boss also gave a bullish assessment of Millwall's chances of escaping the drop.
With 12 games left Holloway believes there are enough points still up for grabs and cited last season's great escape, when Millwall were bottom of the table with nine games remaining, as an example of how the club should not give up.
Holloway continued to praise the Millwall fans, despite an outbreak of disorder towards the end of the game at the New York Stadium. Having defended the Police treatment of Millwall fans in arrangements for the trip to Elland Road last month, Holloway claimed he didn't see the trouble in the away end following Arnason's late winner.
However, Holloway was grateful that the fans vocally criticised him rather than the players "Results are what it is all about," he said "I'm not getting them then the fans turn on me, I'm glad they're doing that and not on the team."