The Black Cats have just avoided relegation in most their recent campaigns, and Fletcher says their frequent narrow escapes were playing on the players' minds.
However, after the side picked up their first win of the season against Newcastle United, the Scotland international striker says that confidence is flooding back into the team.
"There was maybe a lack of belief [with us] at the start of the season,” he said. “We just manage to survive every season, and we were going into this one thinking: ‘Is it going to happen again?’ But football is a lot to do with confidence and a lot of the boys can build from this.”
Sunderland's win over Newcastle has lifted some of the pressure that was on the squad, but they cannot afford to rest on their laurels.
The Wearsiders are still fighting a relegation battle, and they remain in the drop-zone as things stand, after their terrible start.
The signs have been promising under Allardyce so far, though, and optimism is flooding back to the Stadium of Light for the future.
Sunderland's next match comes against Everton at the weekend, and a win in that fixture could push them out of the bottom three.