The home side dominated the majority of possession at St Mary's Stadium, but Jonjo Shelvey's superb late strike saw Swansea take all three points back to south Wales.
Southampton's day was made worse when Ryan Bertrand was shown a straight red card for a dangerous challenge on Modou Barrow a minute from time.
"I'm disappointed, we were a little bit unlucky because we did a lot of good things and played [a] very good first half," Koeman told Sky Sports.
"We had some good chances but you have to score and if you don't score you see what happened in the second half.
"We had less control but okay, it was a very good goal from them. They came for one point and they got three but we have to accept it.
"We lost a bit of domination in the game, but even with 10 players we had good chances to make the draw.
"In my opinion it isn't a red card, he [Bertrand] touches the ball. Maximum yellow card, not a red card."
