Trailing 1-0 from the first leg, Besiktas levelled the tie when Tolgay Arslan struck a fine effort into the top left-hand corner from just outside the box.
Besiktas looked the most likely to win the tie in extra time, but Liverpool held on for penalties only for defender Dejan Lovren to send the 10th kick high and wide.
It was an unhappy return to the Ataturk Stadium for Liverpool, who lifted the Champions League trophy at the same venue 10 years ago after a dramatic comeback and shootout win over Milan.
Besiktas were joined in the last 16 by defending champions Sevilla, who triumphed 3-2 at Borussia Monchengladbach to seal a 4-2 aggregate success.
There were ugly scenes in two matches, where play was stopped at Feyenoord's clash with Roma and Dynamo Kiev's meeting with Guingamp.
The first leg between Feyenoord and Roma had been played among a backdrop of crowd disturbances, and the players were taken from the field during the second half at the De Kuip Stadium due to more trouble.
After a delay of around 15 minutes, play resumed and Roma prevailed 2-1 on the night after a 1-1 draw in Italy last week.
In Kiev, crowd trouble led to a brief delay, the home side overturning a 2-1 first-leg deficit by winning 3-1 and advancing 4-3 on aggregate.
Fredy Guarin's stunning effort sealed a 1-0 win for Inter and progression after a 3-3 draw with Celtic in the first leg, while Fiorentina defeated Tottenham 2-0 to advance 3-1 overall.
Zenit defeated PSV 3-0 to add to their 1-0 first-leg triumph, while Ajax progressed by the same aggregate scoreline against Legia Warsaw.
Everton's 3-1 win over Young Boys saw them comfortably through 7-2 overall, while Napoli defeated Trabzonspor 1-0 to make it 5-0 over the two legs.
Italian success continued as Torino edged a tight tie with Athletic Bilbao 5-4 on aggregate, while Club Brugge's 3-0 win over AaB saw them advance 6-1 and Wolfsburg's goalless draw at Sporting Lisbon saw them through 2-0.
Elsewhere, Dnipro drew 2-2 with Olympiacos to advance 4-2, Dinamo Moscow overcame Anderlecht 3-1 after a goalless first leg and Villarreal's 3-1 win in Salzburg sealed a 5-2 aggregate success.
