The 25-year-old Atletico Madrid striker caused controversy this week when he voiced his desire to play for Spain instead of the country of his birth.
However Scolari, who is preparing his Brazil team for friendlies against Honduras and Chile, denied he was upset by the striker's choice, as had been widely reported.
"I ask you not to put words in my mouth because I've already seen a lot of people saying a lot of words that I never said," Scolari said.
"I said (Costa's) not selected and that's my only job.
"Now, if I express while smiling, laughing or frowning or whatever, it is just how I look.
"And you can't judge how I look - you can only judge my words."
Brazil travel to Florida for their match against Honduras on November 16 before going to Toronto to play Chile four days later.
The friendlies are a chance for Scolari to test out players he will be selecting for the World Cup in Brazil next year.
"We have to give it our all because the opposition have nothing to lose," he said.
"We're going to keep playing within our spirit, the spirit the players have, and think that this game is worth three points."
