Oscar Trejo's 67th-minute penalty proved to be enough for Rayo to triumph, with the result leaving the home side in eighth position in the table, boasting 21 points from their opening 13 matches.
Rayo took the lead in just the fifth minute of the contest, with Santi Comesana volleying past Thibaut Courtois from inside the box to spark jubilant scenes inside the stadium.
Real Madrid turned the match around late in the space of four minutes, though, with Luka Modric converting from the penalty spot in the 37th minute after Francisco Garcia had tangled with Marco Asensio inside the box.
Los Blancos then took the lead in the 41st minute through Eder Militao, who headed a corner from Asensio into the back of the net.
Rayo answered back three minutes later, though, levelling it at 2-2 courtesy of a powerful strike from Alvaro Garcia, who had the most of a loose ball inside the box.
Trejo's spot kick, which came after a Carvajal handball, proved to be the difference between the two sides 23 minutes from time.
Los Blancos goalkeeper Courtois saved the initial penalty, but Carvajal was penalised for running into the area too soon, which led to a retake, with Trejo scoring to make it 3-2.
Real Madrid's best chance of a leveller came in the final exchanges, with Asensio picking out Rodrygo, but the Brazil international fired over the crossbar, as Ancelotti's side lost for the first time in La Liga this season.
Los Blancos will be bidding to return to winning ways when they take on Cadiz on Thursday, while Rayo will be in action against Celta Vigo on the same night.