Welsh international midfielder Ramsey fired home in the second minute of time added on in a disappointing game at the Stade Velodrome to leave Arsenal well placed to reach the knockout stage.
Arsenal have seven points from three games in Group F, one more than Marseille who conceded for the first time in this season's competition. Olympiakos have three with Borussia Dortmund on one.
"We left it very late because we had a difficult start, we lost many balls in the first half," Arsenal coach Arsene Wenger, who was back on the bench after a two-match Champions League suspension, told a news conference.
"I don't think Marseille were dangerous in the second half. In fairness they defended well but we were rewarded because we kept going.
"We made a little step forward but we have to be humble because we are a team that has to continue to grow. The next game at home against Marseille will be very important."
Both sides, who have been struggling in their domestic leagues this season, failed to create many chances. The hosts wasted their best opportunity when Lucho Gonzalez met Mathieu Valbuena's low cross but could only direct his effort wide across goal in the first half.
Arsenal went close after the break when Theo Walcott slipped through but was denied by the legs of keeper Steve Mandanda who was then beaten by Ramsey at the death having just kept out Robin van Persie's low drive.
"I'm angry, but I've calmed down a little bit. It is a huge disappointment," Marseille coach Didier Deschamps told reporters.
"It's infuriating, especially since we had been warned two minutes before the goal with Van Persie's chance."
The Gunners had made a shaky start with Alex Song being shown a yellow card after only four minutes for an ill-timed challenged on Loic Remy.
In the 15th minute, right-back Carl Jenkinson appeared to handle a cross in the penalty area but referee Damir Skomina waved play on, with Andre Ayew booked for complaining.
"The penalty was obvious. Too bad the referee did not see it, it could have changed a lot of things," said Deschamps.
The hosts then had a let off when Souleymane Diawara cleared Van Persie's header off his line as Arsenal increased the pressure.
The visitors, however, lacked composure at the back.
Remy saw off Per Mertesacker and Jenkinson in the box, only for the France striker's right-footed angled attempt to be deflected just wide.
Remy then beat the offside trap but a superb tackle by Laurent Koscielny denied him a shooting opportunity.
Jenkinson, who seemed to struggle in the first half, was replaced just after the hour mark by Johan Djourou as defensive injury problems keep piling up for Wenger.
Arsenal were offered a great opportunity to take the lead in the 64th minute when Nicolas Nkoulou lost out to Walcott, only for Mandanda to save from the England winger with his left foot.
There was nothing the France keeper could do, though, when Ramsey was well posit
