Barcelona forward Neymar stole the headlines with a hat-trick at Soccer City in Johannesburg on Wednesday, while Oscar and Fernandinho were also on the scoresheet, the latter netting with a superb 30-yard drive.
Scolari, however, admits he is surprised that South Africa are not going to be at the FIFA World Cup, which starts in Brazil later this year, and paid credit to the home side's performance.
"The score doesn't tell the whole story of the game," he said.
"South Africa played very well, they moved the ball very well, too. It's a surprise to me that they are not at the World Cup, looking at the way they played.
"The difference between the two teams was just that we have some very special players."
Despite praising South Africa, Scolari believes Brazil are in good shape ahead of the World Cup and feels there are just one or two issues to resolve before the marquee tournament.
He added: "Normally, any coach would say that every area of the team needs improving, because we always want to improve everywhere.
"But we were very happy with what we saw. There are only some small things to address now and they will be worked on in training.
"I was very happy with what I saw. There were a lot of times when the players were all working for the team, not just for themselves.
"It is only then that the special players can take advantage as the guys who work hard make spaces for them."
Scolari thinks facing South Africa was particularly relevant considering that Brazil will face African opposition in the World Cup in the form of Group A rivals Cameroon.
"Obviously that game (against Cameroon) will be much more important and we will be playing for three points," he continued. "I've seen many games in African football, especially in South Africa, and I am impressed.
"I'll see the video of Cameroon against Portugal (on Wednesday)."
Brazil have now lost just once in 19 matches and face further friendlies with Panama and Serbia ahead of the World Cup.
