The visitors moved ahead in just the fifth minute on the opening day of the Premier League season at The Hawthorns thanks to Lee Cattermole's 25-yard thunderbolt.
However, Sunderland felt strongly aggrieved three minutes before half-time when Swarbrick adjudged Valentin Roberge to have tugged down Victor Anichebe in the area and Saido Berahino confidently levelled with the resulting spot-kick.
Sunderland were staring down the barrel when Berahino put the hosts ahead in the 74th minute, but Sebastian Larsson earned Sunderland a point in the closing stages.
Despite seeing his side claim a point, Poyet was left baffled by the penalty decision.
"With the incident I was quite disappointed," he told Sky Sports.
"I was not expecting to come here and even try to speak about a referee's decision.
"[It was] soft. We are not playing football to give soft decisions. Everybody in the stadium thought he was going to give a yellow card to give Anichebe [for simulation].
"After that I don't know what happened, but it is not the way I wanted to start the Premier League.
"I don't want to talk about referees. I want to talk about football."
Though disappointed not to win, Poyet credited his team for their second-half showing.
"We went for it and had nothing to lose. We were much better in the second half and let us see if we can maintain it." he said.
'Will Cristiano Ronaldo and Neymar extend their contracts in Saudi Arabia? We play a facilitating job, but it’s down to the players and the clubs to decide' Saudi Pro League chief reveals how talks will work for out-of-contract duo
Nations League: Who are the ITV commentators and pundits for England vs Republic of Ireland?
