Alfredo Morelos first half strike and Bonke Innocent's sending off before the interval looked to be sending the Gers into the next round before the Swedes produced a stunning turnaround at Ibrox.
Two goals from striker Antonio Colak early in the second half gave Malmo a victory which should embarrass the Gerrard's men, whose hopes of reaching the lucrative group stages were ended.
Saturday's 1-0 defeat by Dundee United in the Premiership at Tannadice on Saturday means Rangers have lost three games in a row.
They will drop down to face either Kairat Almaty of Kazakhstan or Armenian champions Alashkert in the Europa League play-offs.
The constant din inside a packed Ibrox reached a crescendo when Kent slung in a cross from the left and Morelos knocked the ball past Malmo keeper Johan Dahlin with what looked like his back or shoulder.
The visitors came out of their shell towards the end of the first half, but seconds before the interval Innocent was shown a second yellow card by referee Slavko Vincic for a foul on Gers defender Connor Goldson, a decision which looked harsh.
Malmo took the game to Rangers at the start of the second half and Colak, who missed a first-half header, was clinical in the 53rd minute when set up by Veljko Birmancevic, firing past Allan McGregor from the angle.
And four minutes later he gathered a throw in, drifted past Leon Balogun and drilled in a low drive from 16 yards.
Malmo defended desperately under siege.
Morelos had a shot blocked then he headed wide and there were several more attempts which failed before Gerrard's side ran out of inspiration, hope and then time before the boos rang round Ibrox.