Ireland went in front in the eighth minute when Shane Long punished Branislav Ivanovic's backpass, and they would have been ahead even earlier but for a questionable offside decision.
But Serbia showed their class with two goals in the opening 15 minutes of the second half to turn the game around.
First, Ireland were cut open by Dusan Tadic's incisive ball and Aleksandar Kolarov's cross was turned into his own net by James McCarthy - the first goal conceded in the O'Neill era - before Filip Djordjevic's close-range finish completed the comeback.
However, despite losing for the first time as Ireland manager, O'Neill felt that the chances created by his team provided plenty of signs for encouragement.
He told Sky Sports: "I was pleased with a lot of aspects of the game. We started off very, very brightly, had a goal disallowed and then we got a really good goal, could have made it 2-0 and at international level that would have been big for us.
"We started a little bit hesitantly in the second half but had moments in the game against a very difficult side - there was lots of the game I was very pleased with."
Long's lifted finish was his 11th international goal but he could, and maybe should, have had two more after Wes Hoolahan put him clean through twice in the game.
On the first occasion, Vladimir Stojkovic saved a weak shot and the rebound, while the chance that fell to the Hull City striker in the second half was fired wide.
And while O'Neill felt that the 27-year-old should have taken one of his other chances, he instead chose to focus on the positives.
"I thought it was a terrific goal he (Long) scored and that would set him on his way," O'Neill said.
"With that confidence, I probably expected him to take at least one of the other two chances he had. The one in the first half to make it 2-0 would have been a big moment for us.
"That aside, it can happen, but as I said earlier, (there are) plenty of positives there to take out of the game."
