Giovanni van Bronckhorst's side needed to win by at least five goals to pip the Alfred Schreuder's men to third place, but strikes from Steven Berghuis, Mohammed Kudus and Francisco Conceicao ended any slim hopes of a remarkable success story.
James Tavernier had briefly given Rangers hope from the penalty spot, but they ultimately bowed out of Group A and became the first Scottish side in history to lose all six of their group-stage games in Europe's top tournament.
Despite the roar of the Ibrox crowd, it was the visitors who struck first after just four minutes, as Kudus laid the ball off for Berghuis to finish with a delicate, yet possibly mishit, effort past Allan McGregor on the edge of the area.
However, Ajax then had to rely on ex-Rangers defender to bail them out of trouble just a few minutes later, as Calvin Bassey did brilliantly to clear Fashion Sakala's header away from danger, but replays showed that the Zambian's effort may not have been goal-bound after all.
Nevertheless, the hosts had responded to their early setback, but Ajax remained tight in the defensive third and soon began to see more of the ball in the Rangers half, and the Gers' momentum would soon take another hit.
After 29 minutes, Owen Wijndal managed to keep the ball in play on the stretch and pick out Kudus in acres of space in the box, and the striker paused for a second before letting fly, with his effort meeting the head of Leon King and deflecting into the back of the net.
It could have been three for the Dutch champions only moments later, but McGregor got a firm hand to Steven Bergwijn's effort after the ex-Tottenham Hotspur man got in behind, and Ajax were then dealt a blow as left-back Wijndal limped off the field after falling into the advertising boards.
Schreuder's side nevertheless got to half time with their 2-0 lead intact, and they briefly thought they had made it three through Kudus soon after the restart, but VAR and offside came to Rangers' rescue as the Ajax man had gone a fraction too early.
A Tavernier free kick from long range fell comfortably into the arms of Remko Pasveer as Rangers continued to struggle for inspiration, and the 38-year-old goalkeeper then had to be alert to spectacularly tip Scott Arfield's half-volley over the top.
The hosts were not giving up the ghost heading into the dying embers, but in an effort that summed up their evening and Champions League campaign as a whole, Alfredo Morelos missed an open goal after robbing Pasveer of the ball.
However, Rangers were then awarded a spot kick in the 87th minute as Rabbi Matondo burst forward before being felled by Edson Alvarez, and Tavernier calmly slotted home down the middle to reduce the deficit.
Any hopes that Van Bronckhorst's side had of claiming their first European points were soon dashed, though, as substitute Conceicao burst down the left before cutting inside and firing home his side's third in front of the away end with one minute remaining.