The Bhoys were expected to win the encounter with ease, and it looked like they would do so when Virgil van Dijk opened the scoring in the 18th minute with a wonderful free kick.
However, Celtic were reduced to 10 men before the hour mark when Marley Watkins was adjudged to have been brought down inside the Bhoys' penalty area by Scotland international goalkeeper Craig Gordon.
Greg Tansey stepped up to take the resulting penalty and made no mistake from 12 yards.
The match went into extra time, and Inverness took the lead through Edward Ofere, only for Manchester City-owned striker John Guidetti to score the equaliser.
There was yet more drama to come at Hampden Park in Glasgow, though, as David Raven scored the third and winning goal for Inverness in the 117th minute.
Following the 3-2 defeat, Deila gave credit to Inverness and turned his attention to winning the Scottish Premiership.
"We give credit to Inverness and start concentrating on a big game in the league on Wednesday,” the Norwegian told BBC Sport. “The most important thing is the league.
"You get disappointment in football and you need to deal with failure. Today was a big day and we were close.”
Celtic are currently five points clear at the top of the Premiership table.