Wednesday appeared to be on course for a Cup upset following Atdhe Nuhiu's composed 15th-minute opener and the South Yorkshire side battled valiantly to protect their lead in the wake of sustained second-half pressure.
However, James Milner celebrated his birthday with the equalising goal in the 66th minute and then broke Wednesday hearts with a close-range winner in time added on.
That came just a few moments after Wednesday goalkeeper Chris Kirkland had denied David Silva with a fine save and Gray was disappointed to lose in such cruel fashion.
"It's a sickener, once we got our noses in front we knew we could frustrate them and hit them on the break," he told BBC Sport. "They had to put David Silva and Samir Nasri on and they had to chase the game.
"We made a talented team play to the max.
"We deserved the lead and Manchester City ended up giving us the ultimate respect.
"I always say have a look at what players ended the game, rather than started it."
