The former Yugoslavia international had been at the club for 15 months, having been appointed when Cesare Prandelli left to take charge of Italy, but grew increasingly unpopular with the club's fans over recent performances.
"Sinisa Mihajlovic, who has carried out the role with dignity and professionalism, is leaving... with the club's hope that his career can continue with increasing satisfaction and deserved success," Fiorentina said in a statement.
Fiorentina, who are 13th in Serie A with 12 points from 10 games following Sunday's 1-0 defeat to Chievo, said Rossi would be in charge until the end of next season.
Mihajlovic was jeered by supporters during a 1-0 win over Genoa 10 days ago and fans have never been happy with what they saw as the team's unadventurous style under his leadership despite a respectable ninth-place finish last season.
Rossi was twice coach of Palermo last season. He was dismissed following a 7-0 home defeat by Udinese but rehired a month later, then left again by mutual consent at the end of the season.
The 51-year-old is one of Serie A's most seasoned campaigners having coached at Genoa, Pescara, Lecce, Atalanta and Lazio.
Fiorentina became the sixth team to change coaches this season following Cagliari, Palermo, Inter Milan, Bologna and Cesena.
