Klinsmann was told after a morning meeting with board members Uli Hoeness, Rummenigge and Karl Hopfner. Assistant coaches Martin Vasquez and Nick Theslof were also sacked.
Hoeness said the coach had been surprised and "even shocked."
"Naturally, I am very disappointed at the moment," Klinsmann said in a statement. "We have set the foundation for the future. I still believe the team can win the championship."
Bayern are third in the Bundesliga, three points behind leaders VfL Wolfsburg, after losing 1-0 at home to Schalke 04 on Saturday and are in danger of winning no silverware this season.
The defeat by Schalke piled the pressure on the former Germany coach, with thousands of fans chanting "Klinsmann out".
HEYNCKES RETURN
Heynckes, 63, coached Bayern from 1987-91 and won two championships.
He also enjoyed a brief spell at Real Madrid, winning the Champions League in 1998, but his most recent Bundesliga job at Borussia Moenchengladbach ended midway through the 2007 season after his team had gone 13 matches without a win.
"We needed this solution," said Rummenigge. "We trust Jupp, with his calmness and experience to steer things in the right direction."
Rummenigge ruled out giving Heynckes a long-term deal.
"I am doing this for Bayern because they were the stepping stone for my international career and because of my friendship with Uli Hoeness," Heynckes told the online edition of the Rheinische Post newspaper.
Klinsmann, 44, joined Bayern in July and said his goals for the season were to retain the Bundesliga and the German Cup and to achieve success in Europe.
"I want to make every player a bit better after every game," he said.
Bayern, though, were destroyed 4-0 by Barcelona in the Champions League quarter-finals before drawing the return 1-1 in Munich.
They went out of the German Cup in the last eight, losing 4-2 to Bayer Leverkusen.
Bayern's defence has also been a far cry from last season's rock-solid backline.
The job was Klinsmann's first club assignment after the former international striker, who also played for Bayern, steered Germany to third place in the 2006 World Cup on home soil.
