Dutchman Stevens, who will be replaced until the end of the season by youth team coach Jens Keller, was in his second stint at Schalke after taking over in September 2011.
Despite leading the side into the knockout stages of this season's Champions League, Stevens paid the price for a disappointing run of no wins and four defeats in their last six league games and is leaving at the start of the winter break.
"This was a very hard decision for everyone involved given the merits the 59-year-old has earned with the club," Schalke said in a statement.
Stevens had been a crowd favourite ever since leading the success-starved Ruhr valley club to a 1997 UEFA Cup victory and back-to-back German Cup wins from 2001 in his first stint at the club that ended nine years ago.
This season, Schalke started off as Bundesliga leaders Bayern Munich's main title rivals but with a busy schedule, the squad has started to show signs of fatigue as the burden of European and domestic campaigns took its toll on the players.
Schalke picked up 20 points in their opening nine matches but added just five more in their next eight games.
"We sat down this morning and had a discussion with Huub Stevens, we analysed the situation together and came to the conclusion together that it made sense to end the collaboration immediately," said Schalke sports director Horst Heldt.
"The Bundesliga is our daily bread and there we were well behind expectations," he said.
Keller will lead training on Monday and will be in charge for their German Cup round of 16 tie against Mainz on Tuesday. He had a brief spell in charge of VfB Stuttgart in 2010.
"Obviously I was surprised," said Keller. "But obviously it is not something you turn down, being part of one of the big European leagues.
"We need to talk to the boys now, take away the fear they show in their game. I want them to see the Cup tie as a final and to end the year on a positive note," said Keller.
