QPR defender Ferdinand no longer speaks to Terry or Ashley Cole after the ugly incident involving his brother Anton.
Terry shouted the words "f****** black c***" towards Anton Ferdinand during a league game in 2011, while Cole supported the 33-year-old in court.
Rio Ferdinand said he would never forgive Terry for the pain he put his family through.
"For me, the biggest idiot will always be John Terry," Ferdinand writes in his autobiography, an extract of which was published in The Sun on Sunday.
"As England captain and my centre-back partner he could have saved everyone a lot of pain by admitting immediately he used the word in the heat of the moment, but was no racist.
"I think that's probably what happened and what the truth is. Anton and I would've accepted that – instead he never gave us a chance.
"I've never actually spoken to John about the case. I no longer talk to him, but even three years later I find it impossible to forgive or forget the pain he put us through."
Terry was cleared of racially abusing Anton in court, but the FA found him guilty of "using abusive and/or insulting words and/or behaviour".
Rio Ferdinand said his friendship with Cole ended when the left-back decided to support Terry.
"Ashley Cole had long been a friend of mine. We've known each other since we were kids," Ferdinand said.
"But for me it ended the day he decided to go to court in support of John Terry.
"Anton rang me and my head nearly blew off. He was betraying Anton, who he'd also known for years."
