"Our approach and that of Joern Andersen did not agree anymore as the coach was developing in a different direction," club manager Christian Heidel told reporters.
The club also sacked assistant coach Juergen Kramny and appointed under-19 coach Thomas Tuchel as Andersen's replacement.
Andersen joined Mainz in 2008, steering them to promotion in his first season in charge.
"I regret the decision of the club but our visions were different," Andersen told reporters.
Mainz lost 2-1 to lowly Luebeck and were eliminated in the first round of the Cup.
Bizarre Jose Mourinho statement aims to pre-empt punishment after Fenerbahce boss clashes with Galatasaray counterpart
'Andy Robertson has a decision to make as a player, where you either roll your sleeves up and give someone great competition or you accept decisions and say that your time is done.' Alan Shearer on the conundrum facing Arne Slot's Liverpool
