Klopp's side had lost their previous five Bundesliga matches before hosting Lucien Favre's Gladbach on Sunday, with Werder Bremen's win over Stuttgart on Saturday seeing Dortmund drop to the bottom of the table.
Dortmund were somewhere close to being back to their best with their general play, dominating their previously unbeaten visitors from start to finish.
But with the likes of Marco Reus wasting a host of chances, Dortmund only claimed victory due to an incredible own-goal from Christoph Kramer, who lobbed his own goalkeeper from inside the centre-circle.
"I can't find the words for it [to describe his relief]," Klopp said afterwards. "It takes much pressure off of us.
"In this situation the team played an extraordinary game. In the first half we played outstanding football.
"Defensively we did not let the strengths of Gladbach come to the fore and then we scored a very strange goal.
"We have been given a bunch of chances and then a crazy goal must come to make it perfect.
"At the end we were a little shaky, but we won. It's a great feeling."
Despite earning Dortmund the victory, Klopp admitted sympathy for Kramer, although he is sure that the own-goal will only go down as a minor blot on the World Cup winner's copybook.
"He is an exceptional player who has made a terrific development," he added. "It [the own-goal] will remain only a small side note as a curiosity in his career.
"For us, it was a very important three points. If it had to be this way, then Christoph Kramer inadvertently now has a place in our history."
