The 23-year-old moved to White Hart Lane in January 2013 but has found his first-team opportunities limited at Spurs.
Having made just six starts in the Premier League this season, the German has been linked with a return to the Veltins Arena.
Despite Spurs manager Tim Sherwood hinting that Holtby could be allowed to leave, Keller said it was unlikely Schalke could finance such a deal.
"We can't really imagine such a transfer financially," he told Bild.
"Talks have not yet reached an advanced stage."
Keller did add that Holtby would represent a boost for his squad, with midfielder Jermaine Jones set to leave the club in the current transfer window.
"Lewis has already played as a defensive midfielder under me," Keller added. "He can play there and is tactically very well-schooled."
Holtby originally moved to Schalke from Alemannia Aachen before being loaned to Bochum and Mainz during his time at the club.
‘The pure simplicity of the way Slot has managed the squad is probably the biggest thing I could say about him. It’s not broken, so let’s get on with it’: Liverpool legend full of admiration for Jurgen Klopp's successor at Anfield
‘I trained at Spurs and thought they’d give me a chance. But I received a letter thanking me but saying they didn’t think I was good enough – I was gutted’: How Tottenham missed out on signing Wales legend John Toshack
