Albert Gudmundsson, who scored a hat-trick in Iceland's 4-1 win over Israel last weekend, scored a wonderful opener on the half-hour mark to put the visitors in front at the break.
However, Ukraine valiantly fought back in the second half, with Viktor Tsyhankov's 54th-minute equaliser followed by a dramatic 84th-minute winner from Chelsea winger Mykhaylo Mudryk.
Serhiy Rebrov's men have successful advanced to their fourth consecutive European Championships and will join Belgium, Slovakia and Romania in Group E at this summer's tournament, facing the latter in their first match on June 17.
In front of over 30,000 raucous supporters, Ukraine made a bright start to proceedings but were unable to create any clear-cut chances, with all three of their shots inside the opening 25 minutes fizzing wide.
Iceland relied on the counter-attack to make their mark in the final third, and they were not too far away from opening the scoring when Jon Dagur Thorsteinsson fired a testing shot towards the far corner, but Real Madrid goalkeeper Andriy Lunin was at full stretch to keep the scores level.
However, the visitors eventually made the breakthrough on the half-hour mark in stunning fashion when Gudmundsson weaved his way past a couple of Ukraine defenders into a central position on the edge of the area, before placing a pinpoint finish into the bottom-right corner.
Ukraine responded well, and after Hakon Valdimarsson made a superb save to keep out a powerful effort from Heorhiy Sudakov, the hosts thought they had restored parity in the 39th minute, but Roman Yaremchuk was ruled offside by VAR after slotting the ball into the back of the net.
Another second-half comeback - just like their 2-1 win over Bosnia-Herzegovina last weekend in which they scored twice in the final five minutes to set up Tuesday's contest with Iceland - was required from Ukraine if they were to progress to Euro 2024.
Rebrov's half-time team talk certainly did the trick, as his Ukraine side came out of the blocks after the break and netted an equaliser just nine minutes after the restart when Tsyhankov cut inside from the right and bent a brilliant left-footed strike from the edge of the box into the far corner.
Both teams made changes for the final half-hour, with Arsenal's Oleksandr Zinchenko brought on for Ukraine at the expense of Ruslan Malinovskyi, while Iceland duo Orri Oskarsson and Kolbeinn Finnsson replaced Andri Gudjohnsen and Gudmundur Thorarinsson.
Ukraine looked the most likely to snatch victory in the closing stages; Yaremchuk fired a right-footed shot agonisingly wide from close range before Sudakov had an effort towards the top-right corner saved by Thorsteinsson.
Iceland still posed a threat on the counter-attack, though, and almost regained the lead when Thorsteinsson cut inside and shifted the ball on to his right foot before watching his strike arrowing towards the top corner expertly saved by the fingertips of Lunin.
However, it was Ukraine who had the cutting edge in the final third, and Mudryk was the man to fire his nation to Euro 2024 six minutes from time when he whipped a first-time strike with his right foot from around 20 yards out into the bottom left-corner.
Iceland were unable to respond in the closing stages and Age Hareide's side miss out on qualifying for just their second European Championships after their first appearance in 2016.
As for Ukraine, who have now won 14 of their last 20 Euro qualifiers, they held on to win a playoff match at the third time of asking - having previously fallen at this hurdle in 2000 and 2016 - to secure their spot at this summer's tournament in Germany.