Los Blancos entered the match knowing that a draw or a win would have moved them above Barcelona to the top of the division, but they were overcome by Quique Setien's side.
Karim Benzema's 60th-minute penalty cancelled out an opener from Yeremi Pino, but Gerard Moreno netted a penalty of his own in the 63rd minute to decide the contest.
Villarreal made a strong start to the match, with Alex Baena lifting one over the Real Madrid crossbar inside the first minute before Francis Coquelin hit the post for the home side four minutes later.
Benzema had Real Madrid's first sight of goal in the 14th minute, but he fired wide of the post from the edge of the Villarreal box before Eder Militao missed the target seven minutes later.
Pino was denied by Thibaut Courtois down the other end, before Baena and Gerard Moreno had opportunities in the period that followed, with Villarreal very much on the front foot during a strong first-half performance.
Pepe Reina did brilliantly to keep out a strike from Vinicius Junior in the 40th minute, but Villarreal had the final chance of the half, with Dani Parejo seeing his attempt kept out by Courtois.
Villarreal took the lead early in the second half, though, with Pino's strike from inside the box finding the back of the net despite the best efforts of Ferland Mendy, who had given the ball away earlier in the move.
Benzema missed the target with a low strike in the 49th minute, as Los Blancos looked to level, and the score became 1-1 on the hour when Benzema scored from the penalty spot after a Juan Foyth handball.
Villarreal had a penalty of their own minutes later, though, with David Alaba penalised for a handball, and Gerard Moreno stepped forward to make it 2-1 to the home side.
Toni Kroos fired over the home side's crossbar in the 73rd minute before Rodrygo passed up a good opportunity four minutes later, with Villarreal keeping hold of their lead.
Marco Asensio was introduced late on as Real Madrid chased a leveller, but Villarreal came close to a third in the final minute through Samuel Chukwueze, who brought a smart save from Courtois, and the Yellow Submarine closed out the match to secure a vital three points.
There was one more piece of drama late on, though, as Arnaut Danjuma missed an open goal after Courtois had gone forward for a corner.
Next up for Real Madrid is a clash with Valencia in the semi-finals of the Spanish Super Cup, while Villarreal will be back in La Liga action against Celta Vigo on January 13.