Moyes was relieved of his managerial duties by United last month, just 10 months into a six-year deal at Old Trafford, with the fallen Premier League champions languishing in seventh spot.
Club legend Ryan Giggs was appointed interim manager, though Dutchman Louis van Gaal is expected to take over in the coming days.
The axing ended Neville's eight-year association with Moyes, having followed the Scot from Everton to Manchester at the start of the season.
Speaking a day after United's 3-1 victory over Hull City at Old Trafford, Neville admitted he was hurt by Moyes' departure.
"At a club that should be challenging for the title, to finish sixth or seventh is obviously not good enough," Neville said at BBC's World Cup launch.
"And to see a man who, for the last eight years, has put so much faith and confidence in me, giving me the platform either to play or coach, lose his job was probably one of the toughest days I've had in football."
Neville, whose future remains up in the air, added: "Everyone connected to the club will take responsibility because we win together and lose together.
"It was really disappointing when David lost his job because I class him as a friend as well as a manager.
"As a player, I had never experienced anything like that with a manager before in my career - maybe once with England, when Kevin Keegan lost his job."
Moyes' demise came on the back of a disastrous record at Old Trafford that saw the club lose a record seven league games on home soil throughout the campaign.
United played their final home match on Tuesday, which was followed by a lap of honour, a moment Neville had pictured differently before the season kicked off last year.
"Walking around the pitch was probably the emptiest I've ever felt on a football field," said the 37-year-old.
"I took the job fully expecting to be walking around either challenging for the title or parading the trophy because that's what this club demands. We've got to make sure we are doing that next season."
