Johansson, 82, who was UEFA's president for 17 years and a former FIFA vice-president, fought and lost a bitter 1998 FIFA presidential election to Blatter, who has remained in power ever since.
Blatter, who will be 76 next month, secured a fourth term unopposed last year and has said he will retire in 2015.
However the Swede told the BBC's World Service: "It cannot be that only one man should be dictating and taking all the decisions about world football.
"There is not much more to do than to get rid of the man in question."
Johansson said that Michel Platini, who succeeded him as UEFA president in 2007, should replace Blatter.
"He is, to me, closer to what I was looking for," Johansson explained.
"He has tried to be fair, he's open-minded, he allows discussions and he obeys decisions made by the majority."
FIFA has been hit by a wave of corruption allegations over the last 18 months which have ended the footballing careers of a number of high-ranking officials including Qatari Mohammed Bin Hammam, who pulled out of the FIFA presidential election last year after being accused of bribery, allowing Blatter to be re-elected unopposed.
‘The pure simplicity of the way Slot has managed the squad is probably the biggest thing I could say about him. It’s not broken, so let’s get on with it’: Liverpool legend full of admiration for Jurgen Klopp's successor at Anfield
‘I trained at Spurs and thought they’d give me a chance. But I received a letter thanking me but saying they didn’t think I was good enough – I was gutted’: How Tottenham missed out on signing Wales legend John Toshack
