Barry Hearn, whose club is close to the £500 million stadium in a previously rundown part of east London, has long been against a bigger side like second tier West Ham United moving into the venue and his patch after 2012.
A football tenant for the stadium, which will retain its athletics track after the Games and will host the 2017 World Championships, seems the most likely.
"The more I look at the stadium, the more and more convinced I am is that it is not fit for football purposes," the businessman told Britain's Sky Sports News.
"I have to say after my tour of the stadium, wonderful stadium though it is, I don't think it is good for football."
His club were involved in a legal challenge against West Ham taking over the stadium and although an initial deal collapsed, the Hammers are still widely expected to eventually call the arena their home once capacity is cut from 80,000 to 60,000.
A new process has started to find tenants for the stadium.
Hearn, whose small club are still interested despite his reservations, said he hoped the goalposts would not change again before the bid process concludes next month.
"No one has actually taken responsibility or admitted they have built the "wrong" stadium. They didn't listen to advice five years ago when everyone was telling them what kind of stadium can work," he added.
"We have to face these problems and be honest. Let's use some common sense otherwise it is going to be a horrendous white elephant. We have seen that happen around the world, let's hope it does not happen to London."
‘The pure simplicity of the way Slot has managed the squad is probably the biggest thing I could say about him. It’s not broken, so let’s get on with it’: Liverpool legend full of admiration for Jurgen Klopp's successor at Anfield
‘I trained at Spurs and thought they’d give me a chance. But I received a letter thanking me but saying they didn’t think I was good enough – I was gutted’: How Tottenham missed out on signing Wales legend John Toshack
