Ivan Perisic cancelled out a first-half effort from Daizen Maeda to leave the two teams level at the end of 120 minutes, but Croatia won it on penalties, with Japan missing three of their four spot kicks.
Croatia, who were the runners-up in Russia four years ago, will now take on either Brazil or South Korea in the last eight of the competition.
Japan had a golden chance to take the lead in the third minute of the contest when Shogo Taniguchi found himself free inside the Croatian box, but the 31-year-old headed over the crossbar.
Dominik Livakovic then had a nervy moment between the Croatia sticks moments later, with Maeda closing down the stopper, who looked nervous with the ball at his feet.
Croatia had a big chance of their own in the eighth minute when Perisic broke into the Japan box after a mistake from Takehiro Tomiyasu, but the attacker's effort was kept out by Shuichi Gonda.
Japan had another opportunity in the 14th minute when Junya Ito delivered an excellent cross into the Croatian box, but Maeda could not make contact under pressure from Dejan Lovren.
Josko Gvardiol had a half-chance for Croatia in the 24th minute after a free kick from Luka Modric, but the in-demand centre-back scooped his effort over the crossbar, before Bruno Petkovic failed to make the most of a brilliant position inside the Japan box, with Andrej Kramaric free at the far post.
Daichi Kamada scooped the ball over the Croatian crossbar in the 41st minute, but Japan managed to make the breakthrough in the 43rd minute when Maeda smashed the ball home from close range after a well-worked corner had fallen kindly for the striker.
Mateo Kovacic missed the crossbar with a strike early in the second period, but Japan continued to cause problems down the other end, with Hajime Moriyasu's side threatening a second goal on the counter-attack.
It was Croatia that struck in the 55th minute, though, with Perisic planting a brilliant header into the bottom corner after meeting a cross from Josip Juranovic.
Wataru Endo tested the Croatian goalkeeper with a strike from distance just before the hour, before Modric's long-range effort was kept out by Gonda down the other end of the field.
Ante Budimir then headed just wide of the Japan post in the 67th minute, before Perisic had one deflected wide of the target in the 77th minute, with Croatia looking the more likely.
Neither side could find a second before the end normal time, with the match heading for extra-time.
The best chance of extra-time came to Kaoru Mitoma, but his long-range strike was kept out by the Croatian goalkeeper, before Lovro Majer sent one wide of the post in the final seconds, and a penalty shootout was therefore needed to separate the two sides.
Takumi Minamino saw his first effort kept out by Livakovic, allowing Croatia to take a 1-0 lead, with Nikola Vlasic finding the back of the net.
Mitoma was also denied by the Croatian goalkeeper, before Marcelo Brozovic scored to make it 2-0 to Croatia.
Japan managed to get off the mark through Takuma Asano, and Livaja then hit the post for Croatia, before Yoshida saw his attempt kept out, handing the European side the chance to win it.
Mario Pasalic stepped up for Croatia, finding the back of the net with a composed penalty to send his side into the last eight.