After failing to break down their stubborn opponents over 90 minutes, PSG nicked an advantage through Adrien Rabiot in the first minute of stoppage of time at the Parc des Princes before Zlatan Ibrahimovic sealed the hosts' victory after 92 minutes.
PSG had 24 shots to Guingamp's seven as the capital club lacked the finishing touch in front of goal before the late rally secured an important win and Cavani was simply relieved to get the points as the reigning Ligue 1 champions continued their stuttering start to the 2013/14 season.
"It was a difficult match, very tough," the Uruguayan forward said.
"I think we attacked quite a lot but maybe we were a little bit messy sometimes and this gave us a lack of confidence or solidity in front of the goal.
"But eventually we managed to score and we were able to get three very important points for us.
"As the minutes tick by and we still weren't able to score, I thought it was necessary for me to drop back a bit to help win the ball.
"This is something important that we showed today and eventually it served us well for the victory."
PSG started the Ligue 1 season with a pair of come-from-behind draws, while their past two games have seen them score late winners against Nantes and Guingamp.
Despite their sluggish start under new head coach Laurent Blanc, PSG have moved up to second in the French league standings, just one point behind leaders Marseille, who have a game in hand.
'Obviously Messi is the best player in history and being compared to him means that you're doing things right, but I try to be myself': Barcelona star responds to comparisons with Argentine legend
'Darwin Nunez would be gold for Arsenal': Liverpool legend claims Gunners should have signed Uruguayan striker in January
