The A-League side took the lead in the 24th minute through Nikita Rukavytsya, but Shoma Doi levelled on the hour mark for Kashima before Mu Kanazaki's 92nd-minute winner settled the game.
That defeat, coupled with Seoul's draw with Guangzhou Evergrande, leaves Western Sydney bottom of Group H with five points, one adrift of Seoul and Kashima, who meet in the final matchday next month.
Western Sydney will need a draw in that game and a victory over group winners Guangzhou in order to progress.
Al Hilal, last year's runners-up, are through from Group C following a 2-1 win over Lokomotiv Tashkent, with Al Sadd also qualifying for the last 16 alongside them courtesy of a 1-0 win against Foolad.
Nasaf remain very much in contention for a place in the knockout stages thanks to Artur Geworkyan's brace that gave them a 2-1 victory at Group D rivals Tractor Sazi.
Late goals from Ko Cha-won and Kaio gave Suwon Bluewings a 2-1 success away from home against Urawa Reds that sends them top of Group G courtesy of Beijing Guoan's 1-0 defeat to Brisbane Roar.
Suwon and Beijing are level on 10 points - three points ahead of Brisbane - and will qualify with a draw in their final group encounter in South Korea.
