Ilkay Gundogan's 33rd-minute penalty saw Germany lead in the Group E contest, but Japan turned it around in the second half, levelling through Ritsu Doan in the 75th minute before Takuma Asano won it eight minutes later.
Germany, who were eliminated in the group stage of the 2018 World Cup, will be bidding to bounce back when they take on Spain on Sunday, while Japan are next in action against Costa Rica on the same day.
Japan had the ball in the back of the Germany net in the eighth minute of the contest when Daizen Maeda tapped home from close range, but the offside flag was correctly raised.
Untidy work from Gundogan in the middle of the park had seen Japan break clear, and a brilliant cross from Junya Ito was converted, but he was clearly ahead of the final defender.
Germany's first half-chance came in the 16th minute when Antonio Rudiger met a corner from the left, but the centre-back's header was always dropping wide of the opposition's goal.
Gundogan fired one into the arms of the Japan goalkeeper in the 28th minute, before the Manchester City midfielder had a goal-bound strike blocked shortly after, with Germany putting the pressure on.
Hansi Flick's side were awarded a penalty in the 31st minute when David Raum has brought to the ground by Shuichi Gonda, who bundled into the back of the defender, and Gundogan stepped forward to put his team ahead in the 33rd minute, sending the Japan stopper the wrong way from the penalty spot.
Joshua Kimmich struck over the Japan crossbar late in the first period, before Jamal Musiala fired high of the target from outside the box, with Germany threatening a second before the interval.
Kai Havertz turned a low Serge Gnabry cross into the back of the net from close range in first-half stoppage time, but the offside flag prevented the Chelsea attacker from getting his name on the scoresheet.
Maeda then headed narrowly wide of the German post moments later, as both sides had opportunities to register before the half-time whistle arrived.
Gnabry struck over the Japan crossbar early in the second half after smart work from Thomas Muller, before Musiala hit the stands in the 51st minute after showing brilliant feet inside the box.
Gundogan had a brilliant chance to score a second just before the hour when he burst towards the edge of the box, but the midfielder's low effort clipped the outside of the post.
Excellent goalkeeping from Gonda presented Germany from scoring 20 minutes from time, with the stopper making a string of excellent saves in quick succession, most impressively from Gnabry on two occasions.
Manuel Neuer then expertly kept out Ito down the other end before Hiroki Sakai somehow fired wide of the post from the rebound.
Japan levelled the scores in the 75th minute, though, when Doan fired home from close range after Neuer could only palm a low cross into the penalty box.
Hajime Moriyasu's side then scored a second in the 83rd minute, with Asano meeting a long ball from Ko Itakura before holding off the challenge of Nico Schlotterbeck and finishing high into the net.
Germany's best chance of a leveller came in the 95th minute through Leon Goretzka, but the midfielder placed his effort just wide of the post, with Japan holding on to secure a famous victory in Qatar, surviving a late chance for Niklas Sule, who had a close-range effort blocked.
Flick's side are now facing a must-win game against Spain, while Japan will be dreaming of reaching the knockout round of the competition for the fourth time.