Stephen Keshi's men headed into the game needing to at least match the result of Group A rivals Congo to make sure of their place in the finals in Equatorial Guinea.
But they were undone by a clinical South Africa side and saw their spot in the tournament taken away by Congo, who beat Sudan 1-0 to reach the finals for the first time since 2000.
A double from Bournemouth striker Tokelo Rantie gave South Africa control of the contest, however, Sone Aluko pulled one back in the 68th minute to offer hope to the home fans at the Akwa Ibom Stadium.
The game tilted further in Nigeria's favour when Reneilwe Letsholonyane was sent off for a second bookable offence and Aluko scored again in stoppage time to give the hosts hope.
Despite their numerical advantage, Nigeria could not find the extra goal they required to progress as South Africa held on to complete an unbeaten qualification campaign.
Nigeria started the better of the two sides and forward Ahmed Musa twice went close to giving the hosts the lead, hitting the crossbar before South Africa goalkeeper Darren Keet bravely denied the CSKA Moscow man one-on-one.
And Musa's profligacy in front of goal was punished by Rantie in the 43rd minute as he worked his way past defender Kenneth Omeruo and fired beyond Vincent Enyeama.
Rantie continued to serve as South Africa's most dangerous attacking threat and just two minutes after the half-time interval he doubled his tally, leaving Nigeria with a mountain to climb.
Once again Omeruo was at fault as he lost possession to the 24-year-old, who proceeded to skip past a couple of defenders and power home.
Aluko responded, though, slotting in on the rebound after Ikechukwu Uche's curling effort had struck the post before Letsholonyane was shown a second yellow for a cynical foul on Juwon Oshaniwa four minutes later.
Keshi's side flung bodies forward in search of a dramatic turnaround and Aluko levelled matters with a low drive into the bottom left corner to deny South Africa a first competitive win over Nigeria.
However, they could not find the crucial third goal, confirming an elimination that will surely lead to speculation over the future of their coach, who only returned to his post earlier this month.
Ella Toone: Why isn't the midfielder playing for Manchester United and England?
'David Beckham was desperate to play in my squad, but I didn't want to bring him along and waste the FA’s money: I had to pick the strongest players': Great Britain coach reflects on Becks snub as 'one of the toughest decisions' of his career
