Charlie Adam and Steven Nzonzi grabbed the goals for Mark Hughes side, but the visitors were aggrieved as Brown was dismissed for a challenge on the Scottish midfielder.
Contact appeared minimal, but referee Kevin Friend nevertheless sent him off for dangerous play and Stoke went on to win.
"It was a key moment," Poyet said.
"It was a moment when we were trying to come back, to press. Then we were down to 10 men without explanation.
"I'd like to know why. I’d like someone to call me and explain. I’m sure there’s no explanation, so no-one will call me.
"You will see the action of Wes Brown from four different angles, and from every angle it looks less (of a foul). I'm just analysing. It wasn't even a foul. The referee took three seconds to give it.
"I had to put on a centre-half and take off a player like (Emanuele) Giaccherini. It was a different game. It doesn't mean that if it was 11 against 11 we would have won the game, but I'd like the referee’s association to apologise to me."
Sunderland have lost five successive away league games and failed to score in their last four. With just one points from a possible 18 on their travels in 2013-14, Poyet knows it is something he must address.
"I don’t know (what happens) away from home,” said the 46-year-old. "We scored two own-goals and (conceded) a penalty against Swansea, we had two sending offs against Hull, and another one today. We need to change.
"I'm going to analyse the game. I'm not going to analyse the red card again because it was very (clear) for me. We need to improve, keep getting better, make sure we don't concede."
Manchester United report: Antony's future has already been decided after new comments from CEO
‘Slot hasn’t come in and changed everything, making it all about himself – it’s been about the players he’s inherited, getting the best out of a system he likes’: Liverpool manager impressing former players with personality and tactical approach
