Tottenham have now lost three games in a row.
Late goals from Mauro Icardi and Matias Vecino cancelled out Christian Eriksen's deflected opener.
Inter were determined to kick on after their equaliser and pushed Tottenham onto the back foot after Icardi's sensational volleyed equaliser.
Tottenham's next Champions League game is a daunting one against Barcelona, who beat PSV Eindhoven 4-0.
In some ways, to lose so narrowly was even more crushing for Tottenham, who had appeared to be cruising to an impressive away victory.
Vertonghen's motions at the final whistle said it all, with his head in his hands, wondering where it all went wrong.
Tottenham now have lost three games in succession in all competitions and next week's EFL Cup game with Watford suddenly looks quite important.