The Football Association had suggested last week that any supporters heard using the term would face legal action and banning orders.
But Cameron insists that there is a distinction to be made between fans using the term as a way of describing themselves and being used to cause offence.
"There's a difference between Spurs fans self-describing themselves as Yids and someone calling someone a Yid as an insult," he told the Jewish Chronicle.
"You have to be motivated by hate. Hate speech should be prosecuted — but only when it's motivated by hate."
However, Peter Herbert of the Society of Black Lawyers has hit out at the prime minister's comments.
He told Sky Sports News: "David Cameron, luckily, is not in charge of the Crown Prosecution Service or the Metropolitan Police, therefore these decisions are not his to make.
"These comments are like saying that I actually condone anti-semitism or racism. It's no better than that, and it doesn't get any better just because you're the prime minister saying it."
Tottenham supporters have often referred to themselves as the "Yid Army" in reference to the club's connections to the local Jewish community, and chants containing the word are frequently heard from sections of their support during matches.
The club is currently canvassing the opinions of its supporters to gauge their opinion on the use of the term.
