Niko Kovac's side stunned the home crowd early on in Sao Paulo, as Ivica Olic's 11th-minute cross was turned past Julio Cesar by Brazil full-back Marcelo.
Brazil levelled just before the half-hour mark through Neymar, but Croatia held firm in the face of pressure from the hosts until 19 minutes from time, when referee Yuichi Nishimura controversially gave a penalty when Fred went to ground after minimal contact from Lovren.
Neymar duly converted the spot kick, before Oscar made in 3-1 in the closing stages.
And Lovren could not hide his displeasure with Nishimura's penalty calls, claiming Croatia were effectively playing against "12 players" due to the Japanese's decisions.
"What can I say? I'm sad, I want to cry. Everyone saw that," he told AFP.
"It's a scandal for FIFA. They talk about respect, they have meetings before the World Cup, FIFA people talk to us, and in the end, what happens?
"It's better they give the World Cup to Brazil straight away.
"I think we played well, and showed that we have the qualities to match Brazil, but not against 12 players.
"We now have to keep our heads up. I am sure than we can go through, but I still don't understand how the referee could have given that (penalty).
"We saw the pictures in the dressing room after the game and everyone is still asking themselves that question.
"(You should) ask the referee (about the penalty). Well, if you can because he doesn't speak English.
"Do you find it normal a referee in a World Cup doesn't speak English?"
