A host of clubs have been investigated by UEFA in recent months after possible breaches of FFP regulations, including Liverpool, Inter and Roma.
Manchester City and Paris Saint-Germain, meanwhile, were both fined and hit with sanctions by the governing body in May for breaches of the regulations.
Rummenigge believes fines and sanctions are not the right way forward, though, and says self-regulation between the clubs is the best way for FFP to be implemented.
"Basically I'm not a friend of penalties as there are many examples in politics that show sanctions are not getting the intended results," he told Goal.
"I'm a person who likes to convince. We've got a meeting at UEFA in Geneva on which the president of PSG and the chairman of Man City will participate, and also clubs will participate who have well implemented the guidelines of FFP.
"At the end of the day we need to create a community. We're all in the same boat and we need to row together for the benefit of football.
"I can remember a conversation with [UEFA president Michel] Platini and me with Roman Abramovich, the owner of Chelsea and he uttered very positive [words] about FFP. He had to spend hundreds of millions of euros every year to equalise the deficits of the club and to keep the balance sheets in good conditions.
"That cannot be in the interest of an investor.
"We heard the same from Silvio Berlusconi, owner of Milan, and Massimo Moratti, former owner of Inter.
"Actually I've heard nobody commenting negative about FFP.
"It's about us and especially about Platini to convince these persons to walk alongside to this path and not to try to go their own way."
