The former Real Madrid and Brazil defender took charge in June 2013 and led the club to fifth place in the Turkish top flight last season.
However, Carlos has been unable to replicate that success this campaign and makes his departure with Sivasspor sitting second bottom in the league following Saturday's 2-0 home defeat to Istanbul Basaksehir.
Revealing the news on his Instagram profile, Carlos said: "Thank you very much for all this affection and requests to stay, but it is best for the club that I leave.
"I leave my heart with all of you. Thank you very, very, very much Sivas and Sivasspor."
Carlos had previously enjoyed a spell as an assistant to Guus Hiddink at Russian side Anzhi.
Sivasspor start life without Carlos in Tuesday's Turkish Cup clash with Gaziantep BB before visiting Kasimpasa next Sunday.
