Schaff, who was the longest-serving active coach in the league, would not be in charge for the last game of the season against Nuremberg on Saturday, the club said in a statement.
The 52-year-old led Werder to the 2004 German league and cup double and two more German Cup wins but the club struggled this season and only secured Bundesliga safety last week.
"In the past few days we sat down and analysed our sporting situation and came to the conclusion that we needed to dare and make a new start," Werder sports director Thomas Eichin said.
"After securing our Bundesliga spot we jointly agreed that parting ways was the best thing for a new start."
Schaaf's ties with Werder stretch back to 1972 when he joined as an 11-year-old before embarking on a 17-year playing career at the northern club.
He took over as head coach on May 9, 1999 after coaching the club's youth teams for a decade.
"I had an extraordinary time here, coupled with many positive events and great successes," said Schaaf, who opted not to coach the team for the last match of the season.
"I want to thank everyone who supported me and accompanied me on this path."
Assistant coaches Wolfgang Rolff and Matthias Hoenerbach will replace him for the season finale.
