Goals from Steve Sidwell and Dimitar Berbatov either side of half-time were enough to secure victory for Fulham after Mohamed Diame's early opener.
Nolan was sent off just before the break for a clash with Fernando Amorebieta off the ball with the scores at 1-1.
And the Upton Park boss was in no doubt that Nolan's dismissal was the turning point in what became a miserable afternoon for his side.
"We got off to a great start with one goal from Mo Diame and we looked like we were going to get another on," Allardyce said.
"I think that gave us the confidence to go on and test Fulham's back four, but when you get a man sent off then it gets very difficult indeed.
"So he's responsible for us losing the game more than anyone else, I don't think there's any doubt about that.
"I just don't understand because what he's done has come straight after what he did at Liverpool (where he was sent off for a lunge on Jordan Henderson) and I'm going to have to find out what's wrong with him.
"There's certainly something wrong with his mentality at the minute because at his age and his experience and as a player in the many years I've known him I've never seen this reaction.
"You can expect it from somebody starting out as a youngster finding his way in the game, finding out what happens in the Premier League, the block-offs the pulls, the pinches, but not Kevin.
Allardyce also refused to confirm whether he will consider Nolan's future as club captain.
He added: "Time will tell and we will deal with the situation internally as we always will, but my thoughts are now about the players that are available to play, not Kevin Nolan, who's not available."
