Joe Dodoo and Mauro Zarate goals saw the game level at 1-1 at half-time, yet it took extra-time to separate the Premier League's highest scorers.
Andy King headed home the winner three minutes before the 120 was up, to send Claudio Ranieri's side through to the fourth round.
Here are five things from a West Ham perspective which we learned from the game...
1) Andy Carroll might not suit West Ham's new style under Bilic
Yes it was only his first start since January. And yes the Hammers barely gave him any decent service. But Andy Carroll just did not fit with the way West Ham have been and were trying to play last night. Slaven Bilic's side are playing rapid counter attacking football to great effect but last night the big striker just seemed to slow everything down when he was on the ball.
The rustiness you can forgive him for, but on the odd occasion he did control the ball and bring one of the midfielders into play he didn't have the pace to keep up with the attack and get into a position to be a threat. It is still very early days for the ex-England man under Bilic. But just like Carroll was deemed surplus to requirements by Brendan Rodgers when he was bringing in a new philosophy at Liverpool, the big Geordie may soon be getting a sense of deja vu.
The Hammers are far more effective with the tireless running, hounding and chasing of Diafra Sakho up front – not to mention his solid goal record. First team opportunities will prove hard to come by for the ex-Newcastle United striker with everyone fit. He is a great weapon to have, but an expensive sub for a club like West Ham.
2) This is why Jose Mourinho let Victor Moses go on loan again
After fantastic performances against Newcastle United and City, where he also scored, this game probably showed why Jose Mourinho was happy to let Moses leave Chelsea on loan once again. While the powerful and pacy winger was always trying his best to create or score a goal, he often lost the ball and could not get into his rhythm.
That lack of consistency is why West Ham were able to sign him and why he hasn't yet cracked it in West London. The Blues handed him a new four-year deal before letting him join Bilic's East London revolution which underlines how talented he is. But that consistency will be the key as to whether he can really make it at the top level. For West Ham, though, he will prove a valuable asset no doubt and the odd bad game like this one will be tolerated as Hammers fans are a knowledgeable bunch and know a player – and a trier – when they see one.
3) West Ham can cope without Ogbonna but not without Reid too
West Ham showed in the last two games against Newcastle and Manchester City they can cope at the back without new signing Angelo Ogbonna. The Italian international has made an impressive start to life in East London and slotted in well alongside star defender Winston Reid, forging a solid partnership.
At Man City in the second half it was a real backs to the wall job and man of the match Reid was outstanding in keeping Manuel Pellegrini's side at bay. The New Zealand international was rested against Leicester, though, and it really showed as the Foxes had the Hammers defence in knots for long periods. It was only because of goalkeeper Adrian's brilliance that Leicester weren't out of sight and the Hammers will need to keep Reid and Ogbonna fit for as much of the season as possible if they are to achieve anything.
4) Bilic fast running out of ways to keep his squad happy
Going into the new season West Ham had the prospect of European football and with the Premier League, Capital One Cup and FA Cup Bilic was going to need a big squad. He got just that with Hammers co-owners David Gold and David Sullivan pushing the boat out to bring in 12 players over the summer. But having crashed out of Europe at the qualifying stage and now being dumped out of the League Cup at the first hurdle, not only are the club's realistic hopes of silverware dwindling fast but it is also giving the manager less opportunities to use all of his players.
The former Croatia boss will now have a lot of unhappy players not playing week in week out which will need to be handled carefully to ensure disharmony does not spread through the camp. Nobody can complain if the team is winning every week, the problems start once the form dips and certain players feel they should be starting or being used more frequently.
5) Early exit could help Hammers give a fitting farewell to the Boleyn Ground
It may be clutching at straws but the flip side of going out of the league cup so early on could be that West Ham are free to concentrate on the league. It is the Hammers' last ever season at their beloved Boleyn Ground home of 112 years and now they are free to focus solely on winning as many home games as possible to make it a fitting farewell. Bilic's side have already showed just how lethal they can be with four eye-catching wins already this season. If they can keep up that form now they do not have any extra midweek cup games, they could find themselves secure by Christmas. Then they will ready to really kick on and enjoy matches and entertain the fans in the last ever games at Upton Park and who knows end on a high with a concerted push in the FA Cup.