Wednesday appeared to be on course for a Cup upset following Atdhe Nuhiu's composed 15th-minute opener and the South Yorkshire side battled valiantly to protect their lead in the wake of sustained second-half pressure.
However, James Milner celebrated his birthday with the equalising goal in the 66th minute and then broke Wednesday hearts with a close-range winner in time added on.
That came just a few moments after Wednesday goalkeeper Chris Kirkland had denied David Silva with a fine save and Gray was disappointed to lose in such cruel fashion.
"It's a sickener, once we got our noses in front we knew we could frustrate them and hit them on the break," he told BBC Sport. "They had to put David Silva and Samir Nasri on and they had to chase the game.
"We made a talented team play to the max.
"We deserved the lead and Manchester City ended up giving us the ultimate respect.
"I always say have a look at what players ended the game, rather than started it."
Bizarre Jose Mourinho statement aims to pre-empt punishment after Fenerbahce boss clashes with Galatasaray counterpart
'Andy Robertson has a decision to make as a player, where you either roll your sleeves up and give someone great competition or you accept decisions and say that your time is done.' Alan Shearer on the conundrum facing Arne Slot's Liverpool
