Fourth-tier Sportivo had Francisco Rosa sent off in the third minute when he conceded a penalty that Cesar Carranza converted for Ferro but the visitors from Vicente Lopez held on for another 74 minutes before letting in another goal.
Caballero wrapped up Ferro's victory with goals in the 78th and 83rd minutes to send the second-tier club into the second round of the cup competition.
Also on Thursday, Estudiantes Caseros stunned second-tier Sportivo Belgrano 1-0 thanks to Cristian Yassogna's goal.
Meanwhile on Wednesday, Aldosivi defeated nine-man Acassuso 2-1 with goals from Leonardo Fredes and Juan Pablo de Hoyos.
Aldosivi already led 2-0 when Acasusso had both Emiliano Pedreira and Sergio Salomone sent off in the 66th minute.
But despite losing two players with just under half an hour to play, Acasusso got themselves back into the contest when Maximiliano Plante scored with 13 minutes remaining, although the visitors were unable to find an equaliser and Aldosivi progressed.
