The England international has failed to make an appearance for the club this season, having signed from Liverpool in June, but is back in training following a foot injury.
Sullivan has admitted that if the club knew Carroll was going to spend so much time out injured, they "would not have signed him", and Allardyce said he "understands" why Sullivan made such remarks.
Allardyce witnessed his side held 0-0 at home by rock-bottom Sunderland on Saturday and believes Carroll's absence is proving a major factor in West Ham's poor form.
"He (Sullivan) said, 'Only at the end of the season will we be able to judge whether Andy Carroll is a good investment'," Allardyce explained.
"(Given) the size of that investment I can understand why he's said that.
"It's not (Carroll's) fault he's injured, he's the difference between us winning more games and us not.
"If you look at our win ratio with Andy Carroll (compared) to without him, it was pretty obvious (in the close season) we should sign him.
"It's like Manchester City not having (Sergio) Aguero or (Alvaro) Negredo."
West Ham were under pressure for large parts of the game, with the draw leaving them 17th in the table - a point above the relegation zone.
Allardyce believes a lack of belief, as well as "injuries and suspensions" has caused their form to dip in recent weeks.
"You wouldn't say (the performance was) 'one of those things'. I would look at the nervous tension that I have to relieve around the place at the minute," he continued.
"This game was no bigger than the Fulham game two weeks ago. Both teams needed a win and we played a fantastic game and won comfortably 3-0.
"Today we didn't see that. I've picked up a lot more injuries and suspensions since then, so my squad has got thinner.”
