Niklas Bendtner's headed double dragged the Danes level at 2-2 but Silvestre Varela's 87th-minute strike dashed their hopes of a point following an opening win over the Netherlands.
"It's a horrible feeling to lose so close to the end," Olsen told a news conference. "I was proud against Holland and proud of the way we fought back."
Portugal, who lost their first game to Germany, went 2-0 up through Pepe and Helder Postiga in the first half, after Denmark had started brightly but failed to capitalise.
"In the first 10 minutes we got into the game very well, we had a few corners but the quality was lacking a bit," he mused.
Cristiano Ronaldo had another poor game for Paulo Bento's Portugal side and Olsen said his side were always in the match.
"The Portuguese took over the game without being that dangerous actually," he said.
"Maybe they were a bit better, they had a few free runs... that was our mistake. I don't think they had the big chances."
Injuries to Niki Zimling and Dennis Rommedahl hurt the Danes and the pair may now miss the game with Germany on Sunday.
"Our rhythm broke. Zimling is very important for our game and that was a turning point," Olsen said.
"We have a few injuries, but we'll pick it up before the game against Germany. We deserved a point."
