Van Gaal's side looked to be cruising to victory when Ander Herrera put them 3-1 up after 57 minutes at the King Power Stadium.
However, Rafael da Silva then gave away a controversial penalty that David Nugent tucked away, before Esteban Cambiasso brought Leicester level just seven minutes after Herrera's goal.
Jamie Vardy sensationally put Nigel Pearson's team in front for the first time with 11 minutes remaining, and Leonardo Ulloa scored his second of the game from the spot shortly after to wrap up a remarkable turnaround.
United youngster Tyler Blackett was dismissed for bringing Vardy down for the second penalty, while Jonny Evans left the stadium on crutches on what ended up being a disastrous day for Van Gaal.
"I have been 25 years in this profession and have already had another experience like that," commented the Dutchman.
"With Barcelona, we were three goals ahead with a quarter to play and we lost 3-4, so that was also with a top club.
"You never expect it when you are two goals ahead for the second time - you have to kill the game, you have to keep possession, but we could not do that. We gave it away with penalties."
However, Van Gaal refused to lay the blame purely on Leicester's first spot-kick, when Rafael appeared to be pushed off the ball by Vardy before getting up and nudging the striker over in the area.
"At that time, it was still 3-2, so we could still have killed the game," he added. "We kept possession for the first 10 minutes of the second half and scored a goal from that, but we failed to carry on like that.
"We created a lot of chances and scored some superb goals but you have to do it for 90 minutes and not 60.
"After the win against QPR [United beat the Londoners 4-0 last week], to come and lose here is disappointing, as we gave it away. But I am convinced we shall improve that."
