Julian Palmieri's sensational volley from the edge of the penalty area – his first goal since November 2012 – put Bastia in front for the first time in the second half, before he added a second in the final minute as they climbed out of the bottom three.
Lucas Moura opened the scoring for PSG in the 10th minute, before Adrien Rabiot added a second 10 minutes later, with both taking advantage of slack Bastia defending to go two goals ahead.
Any hopes of a routine victory were dashed by two defensive lapses as Ryad Boudebouz converted a penalty after Gregory van der Wiel had handled, and Francois Modesto headed home from a corner.
Palmieri's sensational 25-yard strike completed an unlikely turnaround after 56 minutes, with Nicolas Douchez given no chance as the ball arrowed into the top corner.
Thiago Silva had PSG's best chance to get back into the game with 21 minutes remaining, but his effort struck the crossbar and - after Palmieri extended Bastia's lead - Laurent Blanc's side were left to reflect on a failure to capitalise on Marseille's 2-1 loss at Montpellier on Friday.
The result opened the door for Saint-Etienne to climb to third with a 2-1 win at Reims later in the day.
Yohan Mollo opened the scoring a minute before the break, and Romain Hamouma doubled the advantage three minutes after the players had re-emerged.
A 90th-minute Renaud Couhade own goal meant a tense finish for the visitors, but Antoine Conte's red card soon after helped to calm any nerves as Saint-Etienne held on.
At the other end of the table, Evian climbed out of the bottom three with a 1-1 draw at home to Rennes, lifting them above Lens - 2-0 losers at Guingamp - to 17th.
In the day's remaining fixtures, Lille triumphed 1-0 at home to basement club Caen, who are now without a win in 10 matches in all competitions, while Nice beat Lorient 3-1.
