The Italian took over in 2010 and won the Asian Cup a year later but oversaw a disappointing campaign at the World Cup in Brazil.
Despite being tipped as ones to watch in Group C, Japan finished bottom of a pool that also included Colombia, Greece and Ivory Coast - picking up just one point in the process.
That came courtesy of a goalless draw against Greece - with that result sandwiched by a 2-1 defeat to Ivory Coast and a 4-1 hammering from Colombia.
Confirming his intention to leave on Wednesday, the former Milan and Juventus coach said: "I am disappointed and not satisfied with the results.
"I really wanted to take us forward and qualify for the second round and I am so disappointed," he was quoted as saying by the Kyodo News agency. "But I picked the team, decided on the tactics and how we play and I want to take full responsibility.
"I told the players and staff that I was happy and proud to be their coach. Over the last four years I have put my heart and soul into helping this team grow."
The first major test for Zaccheroni's successor will be defending Japan's title at the Asian Cup next year.
Japan are in Group D for the competition, to be held in Australia, alongside Jordan, Iraq and Palestine.
'I would love to relive walking out at Wembley in the FA Cup final with West Ham when I was captain': Gilly Flaherty reflects on sensational career
'When Manchester City play it's hard not to be biased as they are my club and I love it when they win - but it frustrates me even more when we get beat' Steph Houghton on the problems of punditry and remaining neutral
