O'Neill's side had taken the lead through Martin Paterson in the 14th minute before Aurelien Joachim and Stefano Bensi struck for the hosts.
And although Gareth McAuley netted an equaliser with eight minutes to play, Mathias Janisch scored the winner three minutes from time.
O'Neill was furious with the effort shown by his players in the match and criticised their work rate.
"The performance was unacceptable," O'Neill said.
"The energy and the work rate was not at the level required.
"It's okay lifting it for Portugal and Russia but we needed that tonight. When we're down gears we're average."
The defeat leaves Northern Ireland level with Luxembourg on six points from their eight Group F matches.
And O'Neill admitted his side got what their performance deserved on Tuesday.
"In these games when we're expected to win, we need to bring the basic ingredients to win a football match," he said.
"We didn't manage the game well when we were on top and we didn't deserve anything from the game.
"Luxembourg were the better team on the night."
