Villa looked set to claim a 0-0 draw at home to Garry Monk's men, only for Bafetimbi Gomis to strike three minutes from time to give Swansea all three points.
Villa suffered a similar fate in Sherwood's first match in charge against Stoke City last month, when Victor Moses netted at the death.
And Sherwood is starting to become concerned by Villa switching off in the latter stages of matches.
"That's the second time it's happened now," he said. "It happened in my first game against Stoke - although that [the Stoke winner] was a little bit later.
"It's disappointing to lose in that manner. I think we deserved a point.
"They were better than us in the first period. I thought after the break we looked the most likely to score.
"Losing so late makes it even more disappointing.
"There comes a point in the game when, if you look like not winning it, you make sure you don't lose it. That's twice it's happened now. We have to try and address that.
"It's difficult to not be down-hearted, but when I took over we were in a hole, there were no smiles and morale was low.
"The spirit is a lot better in the camp and we have to make sure we don't take this defeat too hard."
