Andros Townsend's first-half penalty appeared set to send Spurs through to round five on Saturday.
However, Ulloa levelled in the 83rd minute with an emphatic finish, before Schlupp's volley squirmed under stand-in goalkeeper Michel Vorm in second-half injury time.
England international Townsend was afforded his opportunity from the spot when Liam Moore brought Roberto Soldado tumbling in the penalty area, while the visitors felt they should have had a first-half penalty of their own when Andrej Kramaric went to ground under a challenge from Vorm.
Spurs subsequently spurned chances through Erik Lamela and Paulinho, before being punished as Leicester turned things around in the closing stages.
Tottenham enjoyed the majority of possession in the early stages, but it was Leicester who engineered the first clear-cut opening.
Jeffrey Schlupp's centre from the left picked out Kramaric - making his first start for the club – and the Croatian was denied by a superb point-blank save from Vorm.
Vorm was called into action again when the lively Schlupp burst through the Spurs defence and hit a right-footed shot from the edge of the box that forced the goalkeeper to parry over the crossbar.
Those missed opportunities proved costly for Leicester in the 19th minute when Moore was penalised for clumsily felling Soldado in the penalty area.
Townsend slotted the resulting penalty to the right of Leicester debutant Mark Schwarzer - who at the age of 42 became the club's oldest player - and Moore's afternoon worsened when he limped off with an ankle injury.
Tottenham turned the screw in search of a second, but there was a moment of controversy shortly before the break when Leicester were denied a penalty.
Kramaric broke free and went to ground under a challenge from Vorm, only to be booked for simulation by referee Robert Madley.
Tottenham threatened first after the restart as Lamela - back in the side after a calf complaint - fired just over from 20 yards in the 53rd minute.
Paulinho could also have made things more comfortable for the hosts when he shot wide after Lamela's clever cut-back from the byline.
Lamela was proving a constant threat in the second half and the Argentine forward brought the best out of Schwarzer with a bending left-footed shot.
Spurs' profligacy was punished in the 83rd minute when they failed to clear their lines at a corner and Ulloa controlled from Schlupp's ball back into the area before sweeping his shot past a helpless Vorm.
There was still another twist to come, though, as Danny Simpson's excellent cross was side-footed home by Schlupp in stoppage time - much to the delight of the travelling faithful.
