The Premier League title rivals' first meeting since City crushed United 6-1 at Old Trafford in October lived up to expectations with City fighting back after going 3-0 down, but ultimately failing to hold on to the trophy they won last May.
In contrast to United's nail-biter, Chelsea, who won the cup three times in the four seasons before City's triumph last May, beat Portsmouth 4-0 at Stamford Bridge in a replay of the 2010 final which they won 1-0.
Juan Mata put Chelsea ahead just after half-time and three of Chelsea's goals came in the dying minutes. Ramires scored in the 85th and 87th minutes before Frank Lampard added a fourth in the fourth minute of stoppage-time against the Championship side.
Sunderland, the third Premier League team in action, also made the last 32 with a relatively easy 2-0 win at Championship side Peterborough United with Seb Larsson opening the scoring direct from a free-kick and James McClean adding the other.
There was one shock with sleeping giant Sheffield Wednesday, three times FA Cup winners in the past but now playing in League One, beating Championship promotion hopefuls West Ham United 1-0 at Hillsborough with Chris O'Grady scoring the late winner. Sam Baldock missed a penalty for West Ham early in the second half.
The third round will be completed when Arsenal face Leeds United at Emirates Stadium on Monday.
UNITED COASTING
All eyes were focused on the Etihad Stadium where City had been unbeaten since losing to Everton in December 2010.
By half-time that record was in jeopardy with United, the 11-times record FA Cup winners, 3-0 ahead and seemingly coasting into the fourth round.
Rooney headed them in front after 10 minutes, two minutes before City captain Kompany was controversially sent off for a challenge on Nani when he appeared to make fair contact with the ball, although referee Chris Foy ruled he had slid into the tackle with his feet off the ground.
Danny Welbeck scored with an acrobatic volley on the turn after 30 minutes to put United 2-0 ahead before Rooney made it 3-0 with another header, scoring on the rebound when Costel Pantilimon initially saved his penalty.
But City, victors over United in last season's semi-final, were in no mood to surrender meekly their grip on the trophy, and stormed back after the break with Aleksandar Kolarov scoring from a free-kick after 48 minutes.
Midfielder Paul Scholes, who came out of retirement on Sunday to rejoin United until the end of the season, featured as a 59th-minute substitute but his loose pass enabled City to break for Sergio Aguero to make it 3-2 after 65 minutes and set up a dramatic finale.
Rooney told ITV: "We knew it was going to be a difficult game today. Obviously going 3-0 up, we maybe took our foot off the gas a little bit, City got two goals and we had to make sure we saw the game out which we managed to do so we are delighted."
