Italy forward Mario Balotelli struck twice to sink the three-time European champions, who have never beaten their southern European rivals in any eight of their tournament meetings.
"The Italy hurdle proved to high for us today," said 34-year-old forward Miroslav Klose. "Maybe in the back of the head we were thinking that we have never beaten them in a tournament. It will take some time to digest this defeat."
Germany, the only team to go into the semi-finals with four straight victories through the group stage and quarter-final, had been determined to snap their bad run against Italy that also included a bitter loss at the 2006 World Cup semi-final in front of a home crowd.
Instead of setting up a final against holders Spain, who beat Portugal on penalties on Wednesday, the Germans will return home after playing their fourth straight major tournament semi-final but again leaving empty-handed.
"It is always the same thing against Italy," German football association president Wolfgang Niersbach told reporters. "At the moment it is a feeling of despair."
Low said lapses in concentration proved to be their undoing.
"Twice we were careless and twice we paid the price. After that it was difficult to turn the match against a very good team," a visibly shocked Low told reporters.
"We conceded the first goal after a cross that should never have happened and we lost our concentration. After that we let in another from a long ball where we should have closed the spaces better."
Low, who led Germany to the Euro 2008 final and a third place at the 2010 World Cup, said other than those two concentration lapses his team fought "bravely" in an effort to get back into the match.
"The air is very thin in the semi-finals and moments of carelessness can decide a game at this level and this is what happened to us," he said.
"I think we are all very disappointed but football always continues and my players will find motivation to go after new titles."
