The 33-year-old Parisian, who is black, was denied access to a train prior to the UEFA Champions League clash at the Parc des Princes last month.
The incident caused outrage and, after Chelsea suspended several supporters who were allegedly involved, the Premier League side offered Souleymane the chance to be a guest for Wednesday's return leg against PSG.
However, after the lawyer for the French-Mauritanian dismissed suggestions he would turn up at Stamford Bridge, Souleymane accused Chelsea of trying to "buy" him.
"I won't go. They can't buy me with a little piece of paper. I'm not a child," he told French radio station RTL.
"I don't want to sit in that stadium next to those people who pushed me.
"I still hear the voices of those people who pushed me because of the colour of my skin. I can't go to work anymore.
"I live with racism but it's the first time I've had to go to a doctor to ask for pills to calm myself down."
'I’d probably pick Cole Palmer alongside Bellingham and Rice in there at the moment. For me, Declan Rice is good enough to play that central midfield role on his own' Thomas Tuchel advised to DROP Manchester City star by England legend
Manchester United staff threatened with sacking for leaked information... in a leaked email
