The Italian took over in 2010 and won the Asian Cup a year later but oversaw a disappointing campaign at the World Cup in Brazil.
Despite being tipped as ones to watch in Group C, Japan finished bottom of a pool that also included Colombia, Greece and Ivory Coast - picking up just one point in the process.
That came courtesy of a goalless draw against Greece - with that result sandwiched by a 2-1 defeat to Ivory Coast and a 4-1 hammering from Colombia.
Confirming his intention to leave on Wednesday, the former Milan and Juventus coach said: "I am disappointed and not satisfied with the results.
"I really wanted to take us forward and qualify for the second round and I am so disappointed," he was quoted as saying by the Kyodo News agency. "But I picked the team, decided on the tactics and how we play and I want to take full responsibility.
"I told the players and staff that I was happy and proud to be their coach. Over the last four years I have put my heart and soul into helping this team grow."
The first major test for Zaccheroni's successor will be defending Japan's title at the Asian Cup next year.
Japan are in Group D for the competition, to be held in Australia, alongside Jordan, Iraq and Palestine.
