Guarin played a starring role and won the penalty that Xherdan Shaqiri converted in the second minute before Maximiliano Moralez's leveller after 27 minutes.
The visitors regained the lead through Guarin's low curling effort before half-time and Atalanta were reduced to 10 men after the break when Yohan Benalouane received two bookings in a minute.
Just after the hour mark, Guarin scored an even more spectacular goal and then set up Rodrigo Palacio, prompting Mancini's tribute.
He told the club's official website: "Fredy is a great player and it was strange to see him not performing to the best of his abilities earlier in the season.
"He's a very good midfielder and this will help him gain confidence and relax on the pitch.
"It's not vital for him to score, but of course it helps. We put our faith in him and he's repaying us with some great form."
The victory in Bergamo - Inter's first since Mancini's first stint in charge of the club - was a second successive league win and Mancini hopes the result becomes "a big step forward".
"We’re delighted because we've come to a tough ground and won," he said.
"I'm pleased with the performance, though there were still a few too many mistakes that made life difficult for us.
"We had some problems after going 0-1 up, but then after they equalised we took control of the game again.
"Of course, we're in a better position having won two games on the spin, but there's still work to do. We're not out of the woods yet, but we hope this is a big step forward for us."
