The Premier League outfit took the lead early in the first half when Fernando Torres brilliantly finished off a flowing move, but they were pegged back shortly after the break by a long-distance Franck Ribery strike.
Prior to the game going to extra time, Jose Mourinho's men lost Ramires to a second bookable offence, but they were back in front when Eden Hazard's effort deceived Manuel Neuer to restore their advantage.
However, a tap-in from substitute Javi Martinez at the death sent the game to penalties, where Neuer's save from Romelu Lukaku's weak effort handed Bayern the trophy.
With doubts over the fitness of Martinez and Bastian Schweinsteiger, Bayern boss Guardiola elected to start right-back Philipp Lahm in a holding midfield position.
Mourinho, meanwhile, handed Torres a return to action after the striker had been benched for Monday's 0-0 Premier League draw with Manchester United at Old Trafford.
And it did not take long for the Spaniard to reward his manager's faith, as he got on the scoresheet with just eight minutes on the clock.
Hazard was the catalyst, with the Belgian taking out two opponents with a jinking run before playing out wide to Andre Schurrle, who pulled back for Torres to smash the ball home.
Ribery almost provided a reply courtesy of a brilliant exchange with Mario Mandzukic at the edge of the Chelsea box, but he could not have expected Cech to somehow keep out his low curling effort with fantastic dive down to his left.
And he fired over from a similar position 10 minutes before the half-time whistle as Bayern were frustrated in their efforts to restore parity.
However, it did not take long for the newly named UEFA Best Player in Europe's quality to shine through in the second period.
The Chelsea defence appeared to have the 30-year-old under control as he ambled toward the box two minutes after the restart but, from nothing, the winger produced a clever shift onto his right before thrashing past Cech at his near post to level.
The goal understandably buoyed a Bayern side reliant on possession and forced Chelsea into attempting to replicate the sort of rearguard performance familiar throughout their victorious 2011-12 Champions League campaign.
But Mourinho's men were the recipients of the best opportunities of the second half, with Branislav Ivanovic heading onto the bar and David Luiz nodding straight at Neuer after Oscar had dithered when through on goal.
Unfortunately for the Portuguese boss, his side's hopes would receive a sizeable blow with just four minutes of normal time remaining as Ramires' poor challenge on substitute Mario Gotze brought him a second yellow card and ended his participation.
But incredibly, it was Chelsea who took the lead three minutes into added time.
Hazard's well-hit shot from the edge of the box looked destined for the safety of Neuer's gloves but the Germany international somehow misjudged the bounce and allowed the ball through his hands into the back of the net.
A series of fabulous saves from Cech in the second half of added time - from Manduzkic, Toni Kroos, Xherdan Shaqiri and Ribery - looked to have clinched the trophy for Chelsea, but there was still time for more drama.
After 15 minutes of seemingly unrelenting pressure, substitute Martinez was the recipient of a fortunate bounce in the Chelsea area with just seconds remaining and tapped the ball past Cech to send the game to penalties.
Much like they had been throughout a fabulous contest, the two teams were faultless from the spot, but with Lukaku looking set to send the shootout to sudden-death the Belgian produced a weak effort that Neuer kept out down to his left.
