A calamitous mistake from captain Lee Cattermole let in Gabriel Agbonlahor for the winning goal in the 15th minute, and it was a lead the visitors held until the final whistle to end Sunderland's run of four Premier League games without defeat.
The hosts were unable to muster much of a response, although Fabio Borini saw a goal correctly ruled out for offside.
After the match, which keeps Poyet's side bottom of the table, four points adrift of safety, the Uruguayan struggled to mask his frustration.
"I'm getting a little bit fed up," he told BBC Sport.
"It is difficult to take. Football is unpredictable but when you start well and have chances you have to take them.
"Unfortunately the mistake was against us and after that we had too many nerves and too much tension.
"I don't know why they were nervous and we were not organised enough. People were making individual decision, but not thinking as a team and that disappoints me.
"The weakest people look for excuses and it is all down to us to correct it. We miss quality to play at this level and do what we need to do when we are in control of the game."
