Rayo's hopes for a top-half finish took a blow in Vallecas as goals from Mikel San Jose, Oscar de Marcos and Ander Herrera sealed Bilbao's spot in next season's UEFA Champions League.
Jemez's side still sit ninth but can be overtaken by a host of sides over the weekend after their first home defeat since February.
And the manager slammed his side's defensive efforts from set-pieces, the source of all three goals.
"We competed well, but we lacked fluency," Jemez said.
"And against a team like Athletic, you can't afford to do that.
"The worst part of our game was our defending at set pieces – it was shameful, horrible.
"The first two goals were our fault, there's no doubt about that. They were much better than us – no complaints about that."
‘Newcastle’s board said I was over the hill, so I left and played 108 consecutive games and won the European Cup – not a bad way to answer that accusation!’: Ex-Magpies defender reflects on career highlight after proving former side wrong
‘Parma viewed him as the perfect replacement for Adrian Mutu, who’d left to sign for Chelsea. But Manchester United’s interest was much more concrete’: Cristiano Ronaldo's 2003 transfer details revealed by former Sporting director of football
