The London organisers are searching for a tenant for the main stadium after the Olympics and had promised to maintain the track as an athletics legacy for the city.
Premier League clubs Tottenham Hotspur, bidding with entertainment giants AEG, have been shortlisted along with a rival West Ham United consortium as preferred tenants for the 516 million pound venue in east London.
On Wednesday Tottenham said if they beat West Ham to the long-term tenancy of the stadium in Stratford they would consider demolishing the arena and building a new stadium. They are also not eager to keep the athletics track.
"I know it is a hotly debated topic. The position of the IOC is clear: this is not our business," Rogge told reporters.
He said any decision after the Olympics rested solely with those in charge of the project.
"The IOC has absolutely no say on what will happen with the London stadium (after the Games). We would favour as outsiders a solution with a track legacy. But... don't expect the IOC to intervene forcefully."
London's Olympic stadium will seat 80,000 people during the 16 days of the Games but will then be scaled down to a 25,000-seater.
