The Argentina international was dismissed in the 77th minute of United's 2-1 quarter-final defeat at home to Arsenal after reacting angrily to being booked for diving.
Di Maria clasped at the back of Oliver's top, earning a second booking and his marching orders.
"In Spain he knows that he doesn't touch the referee, but that is also in his emotion," he is quoted as saying by BBC Sport. "I've already spoken with him, he knows my opinion but also I have to see on the video.
"I think he's touched the referee and that's forbidden in every country, so he has no excuses."
Having seen Wayne Rooney restore parity after Nacho Monreal had given Arsenal a first-half lead, United old boy Danny Welbeck settled the tie by pouncing on a weak Antonio Valencia backpass shortly after the hour mark.
And Van Gaal believes his team have nobody to blame but themselves for falling short in their most realistic bid for silverware.
"I said in advance that [a top-four Premier League finish] is a priority but as a manager, and as a player and club, you want to win titles and now we cannot win a title," he added.
"I've said in the dressing room we gave [the game] away by ourselves and that is the biggest disappointment. In the second half, we had the first chance also then we gave it away again.
"We don't lose from the opponent, we lost from ourselves and it was very disappointing. You play this match to win; to beat the opponent, we wanted to be in the semi-final."
'Will Cristiano Ronaldo and Neymar extend their contracts in Saudi Arabia? We play a facilitating job, but it’s down to the players and the clubs to decide' Saudi Pro League chief reveals how talks will work for out-of-contract duo
Nations League: Who are the ITV commentators and pundits for England vs Republic of Ireland?
