Romelu Lukaku scored the only goal of the match in the final 10 minutes as he smashed home from Leighton Baines' cross on his comeback from injury, but Allardyce was not impressed with referee Jonathan Moss in the first half.
With 20 minutes gone Kevin Nolan looked set to break into the Everton penalty area, but he was hauled down by Barry on the edge of the area with just Tim Howard to beat.
Referee Moss waved play on as Nolan angrily protested and Allardyce believes that the decision to ignore West Ham's claims and keep the red card in his pocket was a key moment in the match.
"There is no doubt in my mind," Allardyce said. "Gareth Barry at the time, from my position, a bit behind the ref's position, there was no obscured view of Barry pulling Kevin Nolan down.
"That was a key moment in the game which would have meant Kevin was perhaps scoring another goal. He probably would have put that away.
"By looking at it, it is stopping a goal-scoring opportunity, so it's a red card. But we didn't get anything, so I'm really disappointed with that. I can't control that."
The defeat saw West Ham's unbeaten run in the Premier League end at five matches and Allardyce was left ruing his wingers' inability to make the big difference in the final third.
"The opportunities come away from home by exploiting the space the opposition leave," he added. "You know how Everton play, you know they push balls to full-backs.
"You have to be patient and when you get that possession you have to strike quick down the left and right-back positions. When we got down there on early on, Stewart (Downing) didn't produce the ball for Matt Taylor when one-on-one.
"That continued throughout the game. In wide areas the quality of cross, had it been better, with Nolan in the goalscoring form he is in and (Andy) Carroll on, with us in a good frame of mind, we just didn't find the right ball today unfortunately.
"We didn't create enough from what we had in the final third. Here we are, in this league, it is cruel. We didn't do enough at their end. We did a sterling job at defending, but it wasn't enough."
