Chris Smalling, on as a substitute for the injured Phil Jones, opened the scoring within a minute of coming on and struck again after Danny Ings had deservedly drawn the visitors level.
Robin van Persie's late penalty sealed the points, but Van Gaal was less than impressed by the display, even as his side moved up to third.
"The first half was not a good performance, they pressed us and we could not beat that. We could not keep the ball and they were better," explained the Dutchman.
"I want to give all the credit to Burnley, but we scored out of set-plays and we train for that also, so I'm happy we were at 2-1 at half-time because I could re-organise my team a little bit.
"The second half was better, not a lot better but a little bit better and we could benefit more from the weaknesses of the opponent.
"I have said also to the players when you win such matches mostly it means we have a good team, so we have to forget this match I think because it was not the best match.
"For the process it was not a good match [but] I don't think every match is not attractive and we are not playing attacking football - I don't agree with you - but today you are right."
