The 23-year-old moved to White Hart Lane in January 2013 but has found his first-team opportunities limited at Spurs.
Having made just six starts in the Premier League this season, the German has been linked with a return to the Veltins Arena.
Despite Spurs manager Tim Sherwood hinting that Holtby could be allowed to leave, Keller said it was unlikely Schalke could finance such a deal.
"We can't really imagine such a transfer financially," he told Bild.
"Talks have not yet reached an advanced stage."
Keller did add that Holtby would represent a boost for his squad, with midfielder Jermaine Jones set to leave the club in the current transfer window.
"Lewis has already played as a defensive midfielder under me," Keller added. "He can play there and is tactically very well-schooled."
Holtby originally moved to Schalke from Alemannia Aachen before being loaned to Bochum and Mainz during his time at the club.
'People will remember us because we got to No.1 in the world rankings and achieved great things - but our biggest regret is that we didn’t win anything': Ex-Everton star Kevin Mirallas opens up on 'honour' of being part of nation's golden generation
‘I’m sure we’d have finished in the top four or five if he hadn’t joined Chelsea in January. Him leaving damaged the team, but we couldn’t stand in his way’: Alan Curbishley reflects on failed Champions League dream after selling key player
