Shane Long gave Ireland a dream start by capitalising on an error from Branislav Ivanovic, but Serbia, 38 places higher than their opponents in the FIFA world rankings, showed their class in the second half to turn the friendly in Dublin on its head.
James McCarthy's own goal ended a superb incisive move after 48 minutes and Filip Djordjevic's close-range effort on the hour mark completed the turnaround.
The game was the first in which Ireland have conceded under O'Neill in his three games in charge, but he will take encouragement from the first half in particular, when his side would have had more than one goal but for a poor offside decision and wayward finishing from Long.
Ireland should have had an early lead when, Glenn Whelan, captain in the absence of John O'Shea, saw his volley touched in by Wes Hoolahan, but the flag went up for offside.
Replays showed the Norwich City midfielder was unfortunate, but O'Neill's men did get in front after eight minutes when Serbia captain Ivanovic woefully underhit a backpass, leaving Long to coolly lift in his 11th international goal.
Serbia - led by the first time by acting coach Ljubinko Drulovic - could have been level soon after when Milan Bisevac was left unmarked for a corner, but he headed straight at David Forde.
The visitors started to impose themselves and Ivanovic would have atoned for his error after 20 minutes, but McCarthy cleared his powerful header off the line.
It was Ireland's turn to threaten from a corner soon after as Marc Wilson met a deep corner before heading wide.
And Ireland should have added a second after 29 minutes. Hoolahan released Long following a quick break, but with just the goalkeeper to beat, he shot straight at Vladimir Stojkovic, who was also in the way of the rebound.
Twente's Dusan Tadic almost caught Forde out just before half-time with a stunning curling effort when a cross was more likely, but the goalkeeper got back to palm behind.
Ireland conceded their first goal of the O'Neill era just after half-time through a sweeping move that saw Tadic find Aleksandar Kolarov, whose square ball came off the unfortunate McCarthy before trickling in.
In response, Hoolahan again put Long in the clear with a superb pass but the Hull City striker elected to shoot early and put the ball wide.
The turnaround was complete on the hour mark when Seamus Coleman's sloppy pass allowed Nemanja Matic to feed Tadic on the left. Zoran Tosic could not make a proper connection with the cross, but Djordjevic calmly controlled and slotted in from close range.
Substitute David Meyler had a great chance to equaliser from a corner, but thumped his header over.
Kolarov would have made the scoreline more flattering with a thunderous low strike from long range, but Forde capped an impressive individual performance by palming away.
