Yoshito Okubo struck a brilliant late winner for Japan in the first minute of second-half stoppage time to salvage the win for his side, soon after Lubambo Musonda had equalised with an incredible strike.
Japan had looked on track to win 3-2 after Shinji Kagawa and Keisuke Honda both scored in the space of two second-half minutes at Raymond James Stadium in Tampa, where Zambia led 2-0 in the opening half-hour.
But in the 89th minute, Musonda burst forward before unleashing a thunderous strike over Shusaku Nishikawa and, two minutes later, Okubo showed fine control to bring down a high pass and then power his shot into the net.
The win was Japan's fifth in a row but coach Alberto Zaccheroni will no doubt be concerned by his team's poor start and how they struggled with Zambia's physicality, plus their continued reliance on attacking midfield pair Honda and Kagawa for goals.
Zaccheroni made five changes to his starting line-up from Monday's 3-1 win over Costa Rica with Eiji Kawashima, Masato Morishige, Toshihiro Aoyama, Okubo and Yuya Osako dropping to the bench.
Nishikawa was given the gloves, Yuto Nagatomo started at left-back, Yasuhito Endo was installed in midfield and Shinji Okazaki overcame a cold to take up position on the right wing.
Yoichiro Kakitani started as the lead striker, which has been a problematic position for Zaccheroni, after scoring as a substitute against Costa Rica.
Zambia took a few minutes to settle but Japan were unable to make the most of their early possession and were made to pay in the 10th minute.
A cross from the right was flicked on by the outstretched boot of Felix Katongo and his brother Christopher got in between Nishikawa and Atsuto Uchida to head the ball into the net - the captain's first goal for Zambia in 364 days.
In the 18th minute, a clash of heads between Okazaki and Zambia goalkeeper Toaster Nsabata and play was stopped for five minutes as both players received treatment.
Zambia's early goal and the clash between Okazaki and Nsabata seemed to affect Japan, who struggled to string passes together and looked generally rusty, which led to the Africans doubling their lead just before the half-hour mark.
After Felix Katongo's powerful shot was deflected for a corner, Chisamba Lungu drove his set-piece to the top of the box, where Sinkala blasted the ball across his body and into the net.
But Japan dragged themselves back into the contest in the 40th minute after Shinji Kagawa's cross struck Emmanuel Mbola's hand and Honda converted the spot-kick.
At half-time, and with Kakitani having struggled to provide much threat up front, Zaccheroni replaced the Cerezo Osaka man with Okubo but Zambia continued to have the better goalscoring opportunities in the second half.
Japan struggled with Zambia's pressing and the Africans' power in the forward third.
In the 54th minute, Christopher Katongo should have claimed his second goal when he wasted a free header from his brother's cross, while Zambia's captain also fired a long-range shot just wide two minutes later.
The wastefulness of Patrice Beaumelle's men came back to the haunt them in the final 20 minutes.
In the 74th minute, Kagawa cut in from the left wing and his attempted cross evaded Okubo on the edge of the six-yard box and fizzed into the net.
Straight from the restart, Japan regained possession and attacked, with Morishige spinning in the area before crossing for Honda to stab the ball into the net to make it 3-2.
All that was left was a dramatic finale with the two best goals of the game thanks to Musonda and Okubo, with the latter scoring his first goal for Japan since 2008.
