Japan led 1-0 at half-time at the Itaipava Arena Pernambuco thanks to Keisuke Honda but Ivory Coast claimed victory with two goals in as many minutes just after the hour-mark.
Full-back Serge Aurier created both goals with impressive crosses from the right with Wilfried Bony and Gervinho taking advantage, although the latter benefited from poor goalkeeping from Japan's Eiji Kawashima.
Saturday's game was the first time Ivory Coast have won their opening fixture at a FIFA World Cup as the African nation look to claim a maiden berth in the knockout stages in their third consecutive appearance at the global tournament.
The shock in the two starting line-ups was the absence of captain Didier Drogba from Ivory Coast's XI, after he failed to recover from a thigh strain suffered at training on Wednesday.
Bony took Drogba's spot up front, while coach Sabri Lamouchi made two other changes to the XI that began Ivory Coast's 2-1 win over El Salvador with Kolo Toure - who had struggled with malaria before the tournament - and Max Gradel being replaced by Didier Zokora and Yaya Toure.
Japan coach Alberto Zaccheroni recalled Makoto Hasebe after the captain recovered from a knee injury to make just his third appearance for club or country this year.
Hasebe started as one of two holding midfielders alongside Hotaru Yamaguchi, while Yuya Osako got the nod in Japan's most troublesome position - centre forward.
Of the four strikers in Zaccheroni's squad, Osako was the only one yet to have scored for Japan in 2014 but started at the point of their attack ahead of Shinji Okazaki, Yoshito Okubo and Yoichiro Kakitani.
Okazaki started on the right wing, while Okubo and Kakitani were named on the bench.
Ivory Coast had 59.7 per cent possession in the first half with the Africans starting stronger, as Bony received the ball in the box before his shot was deflected past the post in the sixth minute.
But Japan edged their way into the game and in the 16th minute they hit the front.
Yuto Nagatomo collected a throw-in and fed Honda on the edge of the box, and the Milan man slipped away from Toure and fired a thunderous strike into the top corner.
Atsuto Uchida could have made it 2-0 five minutes later but the full-back's powerful drive was parried by Boubacar Barry, before Ivory Coast's goalkeeper was almost chipped by Honda.
While Ivory Coast had more of the ball, they generally struggled to break down Japan's well-organised defence and their best opportunities came on the counter-attack.
After Toure and Arthur Boka just missed the target with free-kicks, Gervinho got free on the break in the 34th minute only for a scrambling Maya Yoshida to block his shot with a sliding challenge.
Honda slalomed through the opposition defence soon after only to watch his deflected shot loop over the bar, while Ivory Coast had a couple of half chances to Salomon Kalou and Boka but were unable to beat Kawashima.
After a frantic first half, the second started slightly slower but in the 57th minute Ivory Coast had a penalty claim ignored after Toure's drive into the box ended with a stumble following Yoshida's sliding challenge.
Drogba was brought on in the 62nd minute for Serey Die and almost created a goal immediately with a back-heel to Gervinho before Japan's defence scrambled the ball clear.
But the Ivory Coast fans did not need to wait long for an equaliser with Bony getting between Yoshida and Masato Morishige to nod Aurier's fine cross past Kawashima in the 64th minute.
A cross from Aurier set up Ivory Coast's second goal soon after but Kawashima would have been unhappy with his effort, failing to stop Gervinho's header despite getting two hands to the ball at the near post.
Kawashima did better in the 82nd minute, parrying Drogba's free-kick away, while the Galatasaray striker had a shot deflected past the post soon after.
But it mattered little as Ivory Coast moved into second in Group C behind Colombia ahead of their clash on Thursday, while Japan face Greece on the same day.
