Levy parted company with head coach Tim Sherwood earlier this week, having only appointed the former England midfielder last December.
Nine managers have come and gone under Levy and Allen, who was employed as a coach and interim boss at White Hart Lane, has called the Tottenham chairman's knowledge of football into question.
He told BBC Radio 5 live: "He's a businessman, an incredible businessman,
"It's a very expensive squad and someone has to be answerable for that,
"It's so bitterly disappointing the squad has finished sixth this season and the squad of players in all honesty, are not competitive with the top four teams.
"A decision was made that a new direction had to be found and I just wonder what that direction is."
Southampton manager Mauricio Pochettino is the favourite to replace Sherwood, with Ajax coach Frank de Boer also reported to be a contender.
