The Colombians believed they had clinched a place when striker Diego Cabrera scored in the first minute of injury-time to make it 3-3 in Wednesday's Group Four match in Montevideo, the third time they had come from behind.
But Alvaro Navarro scored the winner for the Uruguayans, his second of the match, with the last kick of the game after Diego De Souza's shot had taken a deflection.
De Souza had given Defensor a 3-2 lead in the 82nd minute.
Defensor took second place in Group Four with eight points from six games, one ahead of Independiente.
Brazilian champions Sao Paulo, already qualified, took top spot by beating America of Colombia 2-1 at the Morumbi.
Cruzeiro, another Brazilian side, made sure of their place by beating Ecuadorean champions Deportivo Quito 2-0 in Belo Horizonte to top Group Five with 13 points.
Leo Fortunate and Wagner scored in the opening half hour for the twice former champions.
Tempers frayed in the second half and Deportivo Quito's Mauricio Donosa was sent off for a foul on Fabricio, who was also dismissed after the pair had clashed.
The visitors also had Tilson Minda dismissed in the 56th minute for a second bookable offence after he punched the ball into the net.
Argentina's Estudiantes followed Cruzeiro through by drawing 0-0 away to Bolivia's Universitario de Sucre.
