Ander Herrera broke the deadlock against the run of play after 28 minutes, before Ki Sung-yueng restored parity two minutes later at the Liberty Stadium.
United put Swansea under sustained pressure for long periods of the second half, but were eventually undone by a Jonjo Shelvey strike that deflected in off Bafetimbi Gomis in the 73rd minute.
"It's not so important if you deserve more because in football only goals count and they scored the second goal," he said.
"Of course you can say that we dominated the game, but only the goals count.
"We were the unlucky team today."
Asked whether United created enough chances, Van Gaal added: "I think so, especially in the second half.
"I think also in the first half we were the dominant team and in the second half even more.
"We were not so lucky today."
The result represents United first defeat in five league matches.
