Cristiano Ronaldo struck in the third minute to get the visitors off to a perfect start at Estadio de Vallecas, before Karim Benzema headed in his third goal of the week after a good cross from Gareth Bale.
The Welsh winger was integral again for their third as he set Ronaldo up for his second of the game, but two Jonathan Viera penalties saw Rayo's confidence grow.
Diego Lopez was forced into action numerous times in the final 15 minutes, but Real managed to hold on for a 3-2 triumph and Ancelotti believes his players are in need of a change in attitude.
He said: "I think like it happened against Sevilla when we were winning 3-0 and later on the match was reopened again.
"Today, it's been the second time. It cannot be by coincidence. I have to think we have to change the attitude we have sometimes on the pitch because we think easily we are greater than them.
"It is not possible for a team with such quality to reopen a 3-0 match. It has happened twice but I am pretty sure it won't happen again."
It was suggested to the Italian that his side had become lazy, but he refused to openly criticise his players, instead insisting such debates will be kept private.
"No, I cannot talk about too much," he added. "I usually speak about those things on the dressing room, here I am talking about things in general because twice after our advantage at the beginning the laziness on the pitch has come in and that's not good for us."
