Vitor Saba's 88th-minute strike levelled the scores at 2-2 to take the match to a shootout, where the Algerians held their nerve to prevail, Toufik Zerara converting the decisive effort.
CAF Champions League winners Setif suffered an early setback when Romeo Castelen opened the scoring in the fifth minute, before Western Sydney hit the woodwork twice.
However, Setif were in control after making a bright start to the second half in Marrakech.
Daniel Mullen inadvertently restored parity when he sent the ball beyond his own goalkeeper in the 50th minute, before Abdelmalek Ziaya put the African side in front seven minutes later.
The result means Western Sydney have failed to record a victory in any competition other than the AFC Champions League - which they won – so far this season, while Setif end their disappointing tournament with the consolation of finishing fifth.
Setif made five changes following their shock defeat at the hands of tournament surprise package Auckland City on Saturday, while Western Sydney – without captain Nikolai Topor-Stanley after he was sent off in a case of mistaken identity against Cruz Azul – made seven alterations.
Nevertheless, it was the Australians who struck the early blow – Castelen rifling home from 20 yards after being teed up by Labinot Haliti in the fifth minute.
Western Sydney should have been two up 12 minutes later when Haliti headed against the crossbar from a corner, before Mullen sent the ball against the left-hand post from the rebound.
Haliti continued to prove a thorn in the side of the Setif defence, firing over from a Seyi Adeleke cross, but Akram Djahnit narrowly cleared the crossbar with an acrobatic effort at the other end soon after as the Algerians chased a leveller.
And Setif were back on terms five minutes into the second half – Mullen knocking the ball into his own net as he failed to deal with a dangerous cross from the right wing.
Ziaya headed over as Setif looked to build on their momentum, and Sid Ali Lamri had Dean Bouzanis at full stretch with a powerful strike.
It was not long before Setif had their second, however, with Ziaya lifting over Bouzanis after controlling a 57th-minute Ahmed Gasmi cross in the centre of the penalty area.
Amine Megateli made a vital interception when Kearyn Baccus played a dangerous low ball across the six-yard box on the hour mark, before goalscorer Ziaya tested Bouzanis with a powerful free-kick from distance.
Just as victory looked a sure thing for Setif, a Saba strike caught a hand on the edge of the area a minute from time, before the Brazilian found the net from the resulting free-kick.
A drawn-out penalty shoot-out ensued, which saw both Mullen and Ziaya miss their respective efforts, but Zerara found the target with Setif's eighth spot-kick to seal the win.
‘I don’t think Liverpool would look at Ollie Watkins, a striker isn’t a pressing issue for them – it’s Arsenal who need one’ Former Reds star explains why his old club don’t need an out-and-out forward this summer
‘He’ll still be playing at 40 at a good level because he’s in such good shape and looks after himself so well. He does everything to be a top professional’: Ex-Liverpool coach insists Mohamed Salah has plenty more miles in the tank
