The 23-year-old moved to White Hart Lane in January 2013 but has found his first-team opportunities limited at Spurs.
Having made just six starts in the Premier League this season, the German has been linked with a return to the Veltins Arena.
Despite Spurs manager Tim Sherwood hinting that Holtby could be allowed to leave, Keller said it was unlikely Schalke could finance such a deal.
"We can't really imagine such a transfer financially," he told Bild.
"Talks have not yet reached an advanced stage."
Keller did add that Holtby would represent a boost for his squad, with midfielder Jermaine Jones set to leave the club in the current transfer window.
"Lewis has already played as a defensive midfielder under me," Keller added. "He can play there and is tactically very well-schooled."
Holtby originally moved to Schalke from Alemannia Aachen before being loaned to Bochum and Mainz during his time at the club.
