The defending Serie A champions were knocked out of the competition in the group stages last term and a repeat looked on the cards just after the hour-mark on Tuesday.
Juve - who had already lost two of three Group A matches - trailed 2-1 when Delvin N'Dinga headed Giannis Maniatis' cross in to silence the home fans.
Alberto Botia had earlier levelled for the Greek giants, cancelling out Andrea Pirlo's free-kick in his 100th appearance in Europe's premier club competition.
But Juve produced a spirited comeback as Roberto's unfortunate 65th-minute own goal was followed by Paul Pogba's winner a minute later.
Arturo Vidal's failure to score a late penalty meant Massimiliano Allegri's side remain third in the group, behind Olympiacos by virtue of the Greek side's two goals at Juventus Stadium, while both sit three points adrift of leaders Atletico Madrid - who won 2-0 at Malmo.
Juve were brighter in the opening stages, but were limited to efforts from distance - Claudio Marchisio with the best opportunity as his fizzing strike was smartly palmed over by Roberto.
It was a strike from outside the box that gave the hosts the lead, though, as Pirlo netted his second free-kick in four days.
The 35-year-old - who opened the scoring from a set-piece for Juve in their 2-0 win at Empoli on Saturday - curled into the top-left corner from 25 yards in trademark fashion in the 21st minute.
But Olympiacos quickly drew level after Gianluigi Buffon turned an Ibrahim Afellay cross behind for a corner instead of catching it - something he could have done comfortably.
Buffon's unorthodox piece of goalkeeping cost Juve as Botia beat Giorgio Chiellini and saw his glancing header hit the net from Alejandro Dominguez's corner.
Juve could have gone ahead again from another Pirlo free-kick, but Leonardo Bonucci headed over, and while the visitors remained a threat on the counter-attack, the chances kept coming for Allegri's men.
Alvaro Morata first poked wide of the onrushing Roberto before Marchisio somehow missed the target from close range after Kwadwo Asamoah's excellent cross from the left.
Juve turned the screw after the break but Vidal's attempted square for a free Carlos Tevez was blocked, while the Chilean also nodded wide.
Olympiacos then took the lead in the 61st minute as a Dominguez free-kick was hooked back into the box by captain Maniatis, with N'Dinga reacting quickest to direct it into the top-right corner.
The goal could have floored Juve, but they showed impressive resolve and drew level when Botia's inadvertent flick-on hit Fernando Llorente, and the ball ricocheted off the post and in via the unwitting Roberto's shin.
Pogba then crowned his 100th Juventus appearance with his first Champions League goal as his attempted back-heel hit Botia and came back to him, the Frenchman lashing home from the edge of the penalty area.
Llorente - who made a significant contribution off the bench - had a header blocked before Tevez turned wide, while the Spaniard also fired over in the closing stages.
Vidal had the chance to add gloss to the scoreline in the 95th minute, after Tevez was brought down in the area, but saw his penalty superbly kept out by Roberto. Three goals proved enough for Juve, though, as they claimed an important victory.
