Costil saved three of Reims' seven spot-kicks - from Grejohn Kyei, Antoine Devaux and Omenuke MFulu - to spare the blushes of Sylvain Armand and Habib Habibou, who missed their attempts for Rennes.
Norwegian Andres Konradsen was then able to convert the crucial penalty for the home side.
Rennes will face Monaco in the last 16, despite failing to end their winless run, which now stretches to six matches as the tie ended 1-1 after extra time.
Philippe Montanier's side had taken the lead in the 29th minute when Armand scored from the penalty spot.
But, despite missing four regulars due to the Africa Cup of Nations, Jean-Luc Vasseur's Reims equalised 20 minutes from time through Kiey.
Reims held on to send the game to extra time and when a winner could not be found, spot kicks were required to separate the teams.
