Pioli, who had been linked in the media with the vacant AS Roma job, led Chievo to mid-table security last term playing an attractive brand of football.
"I am very proud to be here," the 45-year-old said in a club statement. "When a big club like Palermo come knocking at the door you can't listen to other offers."
Palermo finished eighth in the league last season and qualified for the Europa League after finishing runners-up to Inter Milan in the Italian Cup.
The club parted company with coach Rossi on Wednesday for the second time in four months in a typically bizarre move from eccentric club owner Maurizio Zamparini.
Pioli's decision to take the Palermo job means only two names remain in the frame for the Roma role, according to Italian newspaper reports.
Barcelona B coach Luis Enrique is viewed as favourite but Roma have confirmed they have talked to Olympique Marseille boss Didier Deschamps.
Vincenzo Montella, who was interim boss at Roma but failed to secure Champions League football when they finished sixth, has been linked in the media with Palermo's Sicilian rivals Catania after coach Diego Simeone left by mutual consent.
