The Ligue 1 leaders were beaten 5-4 on penalties at Stade des Alpes after surrendering a lead on three occasions to draw 3-3 after extra-time.
Andre-Pierre Gignac scored twice, but lowly Grenoble levelled with goals from Mourad Nasrallah and Fares Hachi to take the game into extra-time.
Ayew came off the bench to restore Marseille's lead, but Selim Bengriba struck in time added on to force a shootout which Grenoble won when Paul Cattier tipped Florian Thauvin's spot-kick over the crossbar.
Marseille coach Marcelo Bielsa took the blame for his side's shock exit, but Ayew knows the players did not do enough to avoid an unlikely defeat.
He told the club's official website: "We have not done enough. This is unacceptable, unacceptable. I must think of something else and focus on the championship.
"The first period was not very good, we were not quite conquerors, we were not the first on the ball and we were losing the second balls.
"Grenoble showed that with will and determination, we could do great things. We are disappointed but we will do everything to reverse the trend."
