Tripolis trumped Mainz 3-1 in Greece to claim a 3-2 aggregate victory in the third round of Europa League qualifying, as the Bundesliga club failed to reach the group stage of the competition for the third straight attempt.
Mainz had previously lost to Romania's Gaz Metan Medias in qualifying for the 2011-12 Europa League, while they fell to Sevilla in the 2005-06 campaign.
But Thursday's loss appeared even harder to take for Mainz, as they were leading on away goals in Tripoli until Pablo Mazza struck his second goal of the game for the hosts with just four minutes remaining.
"This is pure disappointment but it's our own fault," Baumgartlinger said.
"We can't just give away the game like that."
Mainz won the first leg 1-0 last week thanks to Shinji Okazaki's goal but Tripolis levelled the tie in the 30th minute of the second leg when Pablo De Blasis' cross looped over goalkeeper Loris Karius and into the net.
But the German visitors regained the aggregate lead just nine minutes later when Koo Ja-cheol evaded three defenders in the box to score.
Mainz were still on top when Mazza scored his first of the game in the 68th minute after a rapid counter-attack.
But in the 86th minute, Mainz failed to defend a long ball, which was flicked on by Khalifa Sankare to Mazza, who converted from a tight angle, and suddenly Kasper Hjulmand's side were eliminated.
Hjulmand replaced long-time Mainz coach Thomas Tuchel at the end of last season but the former Nordsjaelland boss is not the only change from 2013-14 with key attackers Maxim Choupo-Moting and Nicolai Muller departing for Schalke and Hamburg respectively.
Choupo-Moting and Muller scored 19 goals between them in the Bundesliga last term.
"We knew that the beginning of the season was going to be hard because of all the departures," midfielder Johannes Geis said.
