The West Ham striker was dismissed by referee Howard Webb during Saturday's game at Upton Park - which the home side won 2-0 - after being adjudged to have swung an arm into the face of Chico Flores.
The Swansea defender went down theatrically clutching his face when replays showed Carroll's arm to have brushed the top of his head, but the FA announced on Tuesday that West Ham's appeal had been turned down.
Carroll will now miss the trip to Aston Villa this weekend as well as the home matches with Norwich City and Southampton - vital games in West Ham's bid to avoid relegation from the Premier League.
An FA statement said: "West Ham United's Andy Carroll will serve a three-match suspension with immediate effect after his claim of wrongful dismissal was rejected by an Independent Regulatory Commission.
"As stated in the Rules and Regulations of The FA, an Independent Regulatory Commission must apply the test of whether the decision to dismiss a player was 'an obvious error'.
"The decision of an Independent Regulatory Commission in relation to a claim of wrongful dismissal is final and binding on all parties and is not subject to appeal."
Tuesday's news comes as a huge blow to both player and club, with Carroll setting up Kevin Nolan for both goals earlier in the Swansea game, in what was only his second start since returning from a long-term foot injury.
Carroll appeared to hit out at the decision on Twitter, as he posted: "DISGRACEFUL!!!"
If the FA deem Carroll's tweet to be aimed at them then it could lead to further punishment for the former Newcastle United and Liverpool striker.
