The Juventus player was dismissed for a horrendous tackle as the frustrated South Americans slumped to their fifth successive defeat, conceding 14 goals in the process.
Vidal, also red-carded in last month's World Cup qualifying defeat in Ecuador, was given his marching orders for a two-footed lunging challenge from behind in the 79th minute.
Serbia were already leading 3-0 with goals by Lazar Markovic, Filip Djordjevic and Filip Djuricic when Vidal lost his head. Angelo Henriquez scored Chile's goal in the 87th minute.
"They've asked me to step aside, I'm no longer the coach of the Chilean national team," Borghi told Chilean reporters after the match in the Swiss town of St Gallen.
"The evaluation is not good, I had a meeting with the directors and they asked me to step aside."
The 48-year-old Argentine had taken over in February last year after the resignation of his compatriot Marcelo Bielsa, who had become a national hero after leading Chile to the last 16 of the 2010 World Cup.
Chile have slumped to sixth in the nine-team South American World Cup qualifying group with 12 points from nine games.
Sinisa Mihajlovic's team went ahead when Markovic took advantage of sloppy marking to score with a header in the 21st minute.
Djordjevic extended Serbia's lead two minutes after the break and Djuricic fired the third before the hour.
"We feel we are to blame for what happened today and in the previous games," Barcelona forward Alexis Sanchez, who has not scored for Chile for 16 months, told reporters.
"We didn't give 100 percent on the field of play.
"He trusted us, the players, and I personally think we let him down. We should have given a lot more."
Borghi's reign began badly with a quarter-final elimination at the hands of Venezuela in last year's Copa America.
They briefly topped the South American qualifying group after wins against Peru, Bolivia, Paraguay and Venezuela, but have plummeted down the table and are now out of the qualifying positions.
Borghi, who made his name as coach of an exciting and highly successful Colo Colo side, inherited one of the finest generations of players Chile has ever produced but struggled to emerge from Bielsa's shadow.
He was also plagued by indiscipline and at one point last year banned five players, including Vidal, for "inappropriate behaviour" after they turned up late for training.
