Villarreal looked to be heading for the quarter-finals of the competition when Etienne Capoue and Samuel Chukwueze struck in the first period, but goals from Vinicius Junior and Eder Militao levelled the scores before Dani Ceballos won it for Los Blancos late on.
Villarreal took the lead in just the fourth minute of the contest, with Capoue firing the ball past Thibaut Courtois in the Real Madrid goal after meeting a flick from Gerard Moreno.
The home side's forward did brilliantly to shield the ball and then pick out his teammate, and it was a stunning effort from Capoue into the top corner of the net.
There were two chances in the period that followed, one for each side, with Federico Valverde firing over for Real Madrid after meeting a low cross from Ferland Mendy, before Yeremy Pino failed to hit the target from inside the away side's box after smart work from Capoue.
The Yellow Submarine came close to doubling their lead in the 32nd minute through Alex Baena, but the attacker's goal-bound strike was brilliantly blocked by Nacho.
Real Madrid had two big chances of their own as the first period continued to gather pace, with Mendy firing straight at Filip Jorgensen before Rodrygo saw a goal-bound effort blocked by Pau Torres.
Villarreal doubled their lead just before the break, though, with Gerard Moreno again providing the assist, this time finding Chukwueze, who lifted the ball past Courtois for 2-0.
Real Madrid pulled a goal back in the 57th minute of the contest, with Vinicius firing the ball past Jorgensen to set up a grandstand final 30 minutes.
Villarreal almost had a brilliant opportunity to score a third in the 65th minute when Pino raced into a dangerous position, but Antonio Rudiger was across to make an excellent challenge.
Real Madrid managed to level the scores in the 69th minute when Militao poked the ball into the back of the net from close range after Karim Benzema had seen his header saved by Jorgensen.
Los Blancos then took a 3-2 lead in the 86th minute, with Ceballos finding the bottom corner with a brilliant strike after being found by Marco Asensio, and the effort proved enough to send Carlo Ancelotti's side into the quarter-finals of the competition.
Real Madrid will be back in La Liga action against Athletic Bilbao on Sunday night, while Villarreal will continue their league campaign at home to Girona earlier in the afternoon.