The Qatar-owned club were leading the quarter-final, second leg 2-1 as the game went into stoppage time before Dortmund, who were held to a 0-0 draw in last week's first leg in Spain, equalised and then grabbed a 93rd-minute winner.
Replays showed several players from the Bundesliga team in offside positions in the build-up to the third goal and the normally restrained Malaga coach Manuel Pellegrini slammed the match officials in his post-match news conference.
The club's director general, Vicente Casado, told reporters on Wednesday before flying back to Spain with the squad that Malaga would file a written complaint with football's European governing body UEFA about the refereeing.
"On this occasion we could not, or they did not want us to, get through," Pellegrini said. "After we went 2-1 ahead there was no refereeing," the Chilean added, although Eliseu's 82nd minute goal for the Spaniards also appeared to be offside.
"They forced us back with elbows and shoves. There were two sendings off that were not given, a double offside in the third goal which should not have counted.
"They said this was the best team in Europe and by the end they were hoofing long balls forward. We are leaving with very bitter feelings."
Casado said the performance of Scottish referee Craig Thomson and his officials had not been up to the level of a Champions League quarter-final.
"We are outraged," he said."Not only Malaga lost but football too. Our complaint is not directed at UEFA but at the refereeing that was absolutely not at the right level."
Winger Joaquin, who put Malaga ahead in the 25th minute, and club owner Sheikh Abdullah Al Thani, a member of the Qatar royal family, were even more outspoken.
In a series of tweets on his official Twitter feed soon after the final whistle, Al Thani blamed the defeat on "racism" and called on UEFA to launch an inquiry.
UEFA General Secretary Gianni Infantino on Wednesday dismissed Al Thani's claim, telling reporters: "UEFA is not taking any racist action against any club.
"Now depending on what he has said or not said... I can understand if you lose a match in the 93rd minute then the emotions come up and you say things you have not really thought through and should not say," Infantino said at the Soccerex European forum in Manchester.
"I don't know how there could be any racist abuse from UEFA. How can he blame UEFA for racism when they [Malaga] lost the game from two late goals?"
Infantino added: "We will analyse his comments and the UEFA disciplinary inspectors will act if they have to act depending on what has been said."
DEEPLY UNPOPULAR
Joaquin, who scored malaga's first goal, suggested UEFA President Michel Platini may have had something to do with the defeat.
Platini is deeply unpopular in Malaga after UEFA slapped a ban on the club from competing in continental competition from next season due to delays in payments to creditors.
"We suspect Platini and all the rest of them who are involved," Joaqu
