Klopp's side had lost their previous five Bundesliga matches before hosting Lucien Favre's Gladbach on Sunday, with Werder Bremen's win over Stuttgart on Saturday seeing Dortmund drop to the bottom of the table.
Dortmund were somewhere close to being back to their best with their general play, dominating their previously unbeaten visitors from start to finish.
But with the likes of Marco Reus wasting a host of chances, Dortmund only claimed victory due to an incredible own-goal from Christoph Kramer, who lobbed his own goalkeeper from inside the centre-circle.
"I can't find the words for it [to describe his relief]," Klopp said afterwards. "It takes much pressure off of us.
"In this situation the team played an extraordinary game. In the first half we played outstanding football.
"Defensively we did not let the strengths of Gladbach come to the fore and then we scored a very strange goal.
"We have been given a bunch of chances and then a crazy goal must come to make it perfect.
"At the end we were a little shaky, but we won. It's a great feeling."
Despite earning Dortmund the victory, Klopp admitted sympathy for Kramer, although he is sure that the own-goal will only go down as a minor blot on the World Cup winner's copybook.
"He is an exceptional player who has made a terrific development," he added. "It [the own-goal] will remain only a small side note as a curiosity in his career.
"For us, it was a very important three points. If it had to be this way, then Christoph Kramer inadvertently now has a place in our history."
‘I don’t think Liverpool would look at Ollie Watkins, a striker isn’t a pressing issue for them – it’s Arsenal who need one’ Former Reds star explains why his old club don’t need an out-and-out forward this summer
‘He’ll still be playing at 40 at a good level because he’s in such good shape and looks after himself so well. He does everything to be a top professional’: Ex-Liverpool coach insists Mohamed Salah has plenty more miles in the tank
