Jurgen Klinsmann's men went into the clash at Stadion Letzigrund in Zurich in need of a confidence boost having lost four of their last five games - including last week's 3-2 reverse to Denmark.
And they looked to be on their way after Brek Shea opened the scoring just before half-time, the Orlando City man bending home an excellent free-kick.
But the game turned in the 66th minute when Toronto striker Altidore was dismissed.
Altidore was shown a straight red card for aiming abusive language towards referee Luca Banti following an innocuous foul.
Vladimir Petkovic's Switzerland pushed on to try to capitalise on their numerical superiority, and the pressure duly told as Valentin Stocker prodded home an 80th-minute equaliser from close range.
Both sides struggled to create chances in the early going but USA would have taken the lead had Michael Bradley been more accurate when presented with a simple chance in the 27th minute.
The midfielder shot over the crossbar from point-blank range with the goal gaping following good work from Alejandro Bedoya to get to the byline and pull the ball back.
Bedoya then raced into the box to meet a left-wing cross from Shea, but could only direct his header straight at Switzerland goalkeeper Roman Burki.
And the visitors were almost punish for their profligacy in the 39th minute, Switzerland forward Admir Mehmedi unfortunate not to make turn home Xherdan Shaqiri's ball across the face of goal.
Yet USA got the goal their play perhaps merited on the stroke of half-time as Shea netted a superb left-footed curling free-kick that left an unsighted Burki helpless.
Gyasi Zardes went close to doubling the visitors' advantage as he headed wide after outjumping Burki to a high ball shortly after the restart.
However, the momentum shifted in favour of Switzerland when Altidore was given his marching orders for a petulant show of dissent.
Pajtim Kasami sent a free header over the bar at the far post, but Switzerland eventually made the extra man count, Stocker levelling with a simple finish after USA failed to deal with a ball into the box.
