Sunderland were 2-1 down at the Stadium of Light when Altidore netted, but Atkinson had already blown for a foul on the United States international by Bacary Sagna outside the area.
Di Canio, who was later sent off for protesting against perceived time-wasting by Arsenal, was also angry that Sagna was not sent off for preventing a clear goalscoring opportunity, given that Altidore's shot did end up in the net.
"This is a big mistake, it's obvious," Di Canio told BBC Sport. "This is not Paolo Di Canio's opinion. The referee has to wait and see how the action finishes.
"When it is body to body with Jozy Altidore and Bacary Sagna, he has to wait until how it finishes and not stop the game. It was a clear big, big mistake.
"I didn't protest because a mistake can happen. But he has to realise that he made a mistake which can cost us three points."
The Italian's dismissal to the stands came when he exhorted Atkinson to make Arsenal's physiotherapist hurry up during an injury stoppage.
"You know that if you lose two minutes (to an injury stoppage) they are never going to replace them," said Di Canio.
"(Atkinson) came to me and said if you keep going with your manner I will send you up to the stand. I said to him 'if you want to complete a perfect job, you send me off.' He took it seriously and sent me off. Next time I will not invite the referee to send me off."
Altidore also felt that his strike should have been allowed.
"I was just tussling back and forth with Bacary Sagna but I thought the referee was going to let us play," he said.
"I got the better of him and then finished it off but I didn't hear the whistle and it was brought back.
"The keeper didn't either as he played 100 per cent and I think Kieran Gibbs tried to clear it too. It was a good time to score and it would have been a good reward for us."
