The Sweden international was shown a second yellow card in the 60th minute of Sunday's 1-0 defeat at Molineux, after hauling down Rajiv van La Parra as the winger sped towards the right-hand side of the area.
As referee Simon Hooper brandished the card in Olsson's direction, the defender laid a hand on the official and gave him a slight shove, before walking from the field visibly angry.
On Monday, the FA issued a statement announcing that Olsson had been charged.
"Norwich City's Martin Olsson has been charged by the FA in relation to an incident which occurred during the game against Wolverhampton Wanderers on Sunday August 10, 2014," it read.
"It is alleged that Olsson's action in making physical contact with the referee immediately prior to being shown a red card and for his behaviour as he left the field of play amounted to improper conduct, which occurred outside the jurisdiction of the match officials.
"Olsson has until 6pm on August 14, 2014 to respond to the charge."
