Ryan Kent's deflected effort appeared to have earned Rangers a draw in Serbia but a stoppage-time penalty saw Red Star win on the night, however, Gers go through with a 4-2 win on aggregate.
However, the Scottish champions owe a huge amount to goalkeeper Allan McGregor, who made a number of great saves to ensure his side's safe passage to the last eight.
After Mirko Ivanic gave the home side an early lead, McGregor got down well to push the goalscorer's left-footed strike past the post moments later.
McGregor also did brilliantly to palm away Cristiano Piccini's curling effort and just after the break he kept out an awkward corner from the left which unexpectantly dropped towards goal.
Shortly after Kent's equaliser, McGregor made a great double save - first to deny Guelor Kanga and then Ben Nabouhane on the follow-up.
Nabouhane scored the stoppage-time penalty but it proved to be a consolation as Giovanni van Bronckhorst's men advanced to the next stage.
Van Bronckhorst, whose side could face the likes of Barcelona, Braga and Atalanta in the last eight, went with the same starting line-up that won 3-0 at Ibrox seven days earlier, suggesting he saw scope for more goals.
The Serbian champions also went into the game with the knowledge that they could cause Rangers problems after having three goals disallowed for close offside decisions in Glasgow as well as hitting the bar and having a penalty saved by McGregor.
Alfredo Morelos had an ambitious effort saved for Rangers before Red Star got the early goal they craved.
The hosts worked a two-against-one from a short corner and the resulting cross bounced in the Rangers box before Ivanic had time to chest down and fire home.
The goalscorer soon had another good chance after Red Star cut through the heart of the Gers defence but McGregor made an excellent stop.
McGregor saved Ohi Omoijuanfo's header before Morelos was inches wide from a difficult chance.
Gers soon had a let-off midway through the half after Leon Balogun and Calvin Bassey got in each other's way from a long ball. Milan Pavkov scuffed his effort wide with only McGregor to beat.
McGregor was warned about time-wasting from the resulting goal-kick but Rangers settled down and Morelos had a weak shot saved when Kent looked to be in a better position.
McGregor kept Rangers two ahead with an excellent diving stop from Cristiano Piccini in the final minute of the half.
Rangers withstood some pressure early in the second half. McGregor had to look lively to stop a corner which bounced at his near post and soon made a more comfortable save from Ivanic.
Red Star had a great chance when Omoijuanfo held off Bassey and tried to set up Pavkov for an open goal but got the pass wrong. McGregor soon saved from Omoijuanfo before Rangers got the goal that settled matters.
Kent continued his run after passing to Glen Kamara on the halfway line and the Finland midfielder's flick sent the winger on his way. The chasing Piccini pulled his hamstring which gave Kent a free run to the box and his shot took a deflection and looped over the goalkeeper.
Ivanic missed a good chance and McGregor made another few saves but Rangers looked comfortable even after losing Morelos and John Lundstram to what looked minor injuries.
Substitute Kemar Roofe had a chance to put Rangers ahead but saw his shot saved at the near post.
Red Star were given a slight glimmer of hope when Kamara stretched at a loose ball in the box and caught Filippo Falco to allow Nabouhane to score.
There were some angry scenes after the game when some Red Star players appeared to take exception to Connor Goldson's celebrations but matters were calmed.