After taking the lead at the Britannia Stadium on Saturday through Andy Carroll in the fifth minute, West Ham were undone as Peter Odemwingie's brace and one from Marko Arnautovic sealed all three points for the hosts.
It is the fifth time this season that West Ham have fallen to defeat after taking the lead in the Premier League, more than any other side, but Allardyce felt his players deserved at least a point.
Marc Muniesa and Marc Wilson both handled in the area, but the incidents went unpunished by Pawson late on and the West Ham boss was left confused that neither had been spotted.
"I think that in the cold light of day we have to blame ourselves, there is a big part to blame the referee for," Allardyce said.
"Of that there is no doubt in my mind. I have reviewed the situation and watched a blatant handball (from Muniesa) that stops Carroll chesting a ball down and trying to volley it.
"The ball is diverted away by his left arm. It's arm to ball, not ball to arm. The referee is in a fantastic position to see it. I'm totally and utterly baffled."
Referee Pawson was not the only person Allardyce had harsh words for, with his defenders also feeling the wrath of the 59-year-old.
"It's our fault for the poor defending has cost us," he added. "On two occasions we just have to defend correctly.
"We were in the ascendancy, in control looking for a goal to get in front, then we fall asleep and I don't know where our defending went."
Defeat sees West Ham drop into the bottom half of the table, with Stoke climbing above the London club.
