The hosts were in superb form in the first half, dictating play with their quick passing game as Italy defended in numbers.
Germany forward Miroslav Klose's 16th-minute goal looked to have handed them their first win over Italy in 16 years in the same Dortmund stadium where they lost to them in extra time in the 2006 World Cup semi-finals.
The Italians, with goalkeeper Gianluigi Buffon and Antonio Cassano back in the team, and Brazil-born midfielder Thiago Motta in his first start after playing for Brazil's Under-23 side, improved in the second half.
Substitutes Marco Borriello and Alberto Aquilani added some pace and missed a good chance each before Rossi beat the offside trap to put the ball past keeper Manuel Neuer in the 81st minute.
"We had good pace at times but we could not keep it up for 90 minutes as we had planned," Germany coach Joachim Loew told ARD television.
"We knew that if we wanted to beat Italy we had to keep up the tempo. So I am not really satisfied with the result because we did not invest enough for that second goal."
Both teams are on track to qualify for Euro 2012 with Germany topping Group A and Italy in control of Group C.
TRUST REPAID
The hosts started strongly and Klose, languishing on the Bayern Munich bench this season with one goal in nine league games, repaid Loew's trust when the Germans sliced open the tight Italian defence with a dazzling nine-pass combination.
Real Madrid playmaker Mesut Ozil then found Thomas Muller in the box with a back heel pass and he cut the ball unselfishly for his Bayern team-mate to score his 59th goal in 106 appearances, nine off Gerd Muller's all-time German record.
Klose could have doubled their lead minutes later but was denied by Buffon from close-range as the Italians slowly moved upfield, pressing the Germans in their own half.
The four-times World Cup winners levelled in the 81st minute when some sloppy defending by teenager Mario Goetze allowed Daniele De Rossi to pick out Rossi who saw his first shot from 14 metres saved before slotting in on the rebound.
"The Italians defended well but if we were a bit more patient we could have won it," Germany midfielder Bastian Schweinsteiger said. "It is a shame that we conceded that goal."
