Costil saved three of Reims' seven spot-kicks - from Grejohn Kyei, Antoine Devaux and Omenuke MFulu - to spare the blushes of Sylvain Armand and Habib Habibou, who missed their attempts for Rennes.
Norwegian Andres Konradsen was then able to convert the crucial penalty for the home side.
Rennes will face Monaco in the last 16, despite failing to end their winless run, which now stretches to six matches as the tie ended 1-1 after extra time.
Philippe Montanier's side had taken the lead in the 29th minute when Armand scored from the penalty spot.
But, despite missing four regulars due to the Africa Cup of Nations, Jean-Luc Vasseur's Reims equalised 20 minutes from time through Kiey.
Reims held on to send the game to extra time and when a winner could not be found, spot kicks were required to separate the teams.
'People will remember us because we got to No.1 in the world rankings and achieved great things - but our biggest regret is that we didn’t win anything': Ex-Everton star Kevin Mirallas opens up on 'honour' of being part of nation's golden generation
‘I’m sure we’d have finished in the top four or five if he hadn’t joined Chelsea in January. Him leaving damaged the team, but we couldn’t stand in his way’: Alan Curbishley reflects on failed Champions League dream after selling key player
