Marseille led 2-0 just after the hour-mark on Friday, when Andre-Pierre Gignac tapped in from Benjamin Mendy's parried shot, but Bielsa's men could not resist Caen's comeback as the visitors surged to victory.
Nicolas Seube pulled a goal back from a corner soon after Gignac's effort, while Emiliano Sala struck on the counter-attack in the 70th minute to level the match before Nicolas Benezet's powerful drive from the edge of the area clinched the three points for Caen with three minutes remaining.
Bielsa, who selected four attacking midfielders behind striker Michy Batshuayi, conceded he got his line-up wrong as Marseille gave up a lead for the third straight game.
"We were not secure defensively in either half. We were vulnerable to their attacks in both halves," the Argentine coach said.
"The very attacking system that I put in place created an imbalance that made it harder for us to win the ball back. Caen counter-attacked us a lot.
"The number of players we had in the opposition half during attacks did not help us improve our recovery of the ball and weakened us defensively. I take responsibility for the fact that the team was not balanced. I can't make the players responsible for the situation that I am describing."
Bielsa insisted, however, that he will not give up his attacking credo.
"My way of playing is about attacking but that does not necessarily mean defending badly," he said.
"There is undoubtedly a balance that is needed but that I am not able to get across to my team. But these ideas will not be abandoned because I think they are good."
Caen coach Patrice Garande revealed he had targeted Bielsa's one defensive midfielder - Gilbert Imbula - to limit the amount of possession available to the likes of Batshuayi, Andre Ayew and Florian Thauvin.
"When we saw the Marseille line-up, we tried to put someone on Imbula all the time to make it harder for them to get from back to front," Garande said.
