Hajime Moriyasu's side entered the match knowing that a victory would put them in a brilliant position to reach the last-16 stage of the competition, having overcome Germany in their tournament opener.
However, Costa Rica, who lost 7-0 to Spain in their first match in the competition, secured all three points courtesy of a second-half effort from Keysher Fuller.
The result has left Japan in second spot in Group E behind Spain, while Costa Rica are third, also boasting three points from their first two matches.
Japan were on the front foot in the early exchanges, with Ritsu Doan firing wide of the Costa Rica post in the third minute during a strong start from Moriyasu's side.
Costa Rica managed to settle as the match progressed, though, with Joel Campbell causing problems in the final third of the field, and Luis Fernando Suarez's side were showing improvement from their disastrous start to the tournament against Spain, which saw them concede seven goals without reply.
Campbell placed an effort over the Japan crossbar in the 35th minute, but it was a disappointing first half from both sides, with neither doing enough to deserve to be ahead at the half-time break.
There was not a single shot on target from either side in the opening half, but Costa Rica goalkeeper Keylor Navas had to make a smart stop to keep out a strike from Hidemasa Morita shortly after the restart.
Navas then had to keep out an effort from Takuma Asano, who again made an impact off the bench.
Yuki Soma had a brilliant chance to again test the Costa Rican goalkeeper in the 63rd minute from a free kick just outside the box, but the midfielder's effort was over the crossbar.
Japan had another excellent free-kick opportunity in the 72nd minute, which Daichi Kamada struck, but the Costa Rican wall stood strong to make the block.
Costa Rica then took the lead in the 80th minute of the contest when Fuller managed to lift the ball into the top corner despite the best efforts of Shuichi Gonda, who got fingertips to the ball.
Japan had a golden chance to level the scores in the 88th minute when Kamada found himself free inside the box, but Navas managed to make the save before gathering following a scramble, with Costa Rica surviving what was a very dangerous moment to pick up a huge three points.
The goal came from Costa Rica's first shot on target in the entire tournament, and it has left Japan in a difficult spot ahead of their final match against Spain.