Neymar, Dani Alves, Hulk and Willian struck for the FIFA World Cup hosts as they cruised to victory in a friendly.
But Scolari, who was angry after a training session on Sunday, said his side still had a lot of work to do, and believes they would have been in trouble against a better opponent.
"I asked my players to put pressure on Panama during the first 15 or 20 minutes. However, I felt the connection between our defence and midfield simply wasn't there in the early stages," Scolari said.
"We were not connecting the first and the second ball. Then we started losing the ball and playing completely disorganised football.
"After those initial 20 minutes we found our pace and started to improve a lot. We were playing more compacted and reacted to the problems posed by our opponent.
"We must remember that it's been three months since this group played together and I think that showed tonight.
"However, I admit that our tactics didn't work in the first 20 minutes and we could got in trouble if we were facing a better opponent in the World Cup."
Scolari said Brazil were 'still missing something' from the team they were at their successful Confederations Cup campaign last year.
The 65-year-old also said he expected Neymar, who completed 90 minutes, to feature against Serbia on Friday as the 22-year-old looks to build match fitness.
"Neymar wasn't playing much for Barcelona at the end of the season because he was injured," Scolari said.
"Today I kept him playing for the 90 minutes - even after the yellow card - because he needs to play and get match fitness. He is lacking pace at the moment.
"He will be playing Friday unless something happens in the next three days.
"Neymar is different from the others. He improvises so well. Every single coach in the world would praise Neymar and would not be surprised to see something amazing and new from him."
