Johnson returned for a second spell in charge of the Somerset club in January 2012 and led them to promotion to the Championship in May 2013 as they won the League One play-off final.
However, Yeovil were immediately relegated back to England's third tier and, with the club now sitting bottom of League One, they have now decided to part company with the former Latvia, Bristol City and Peterborough United boss.
Terry Skiverton - who Johnson replaced in 2012 - will take over as acting manager with Darren Way serving as his assistant.
"The decision to relieve Gary of his management duties is the hardest I have been involved in during my 23 years at Huish Park," Yeovil chairman John Fry told the club's official website.
"I cannot speak too highly of the man who in the past brought so much success to our club.
"We part company on good terms and I would like personally to wish him well for the future.
"Unfortunately, ours is a business that depends on good results, and at this stage of the season the facts speak for themselves.
"Following our relegation from the Championship we are now bottom of League One.
"After considerable discussion regarding the best way forward, the club's directors decided to terminate Gary's contract and to pursue alternative managerial options."
