Jack Vale sent the Championship side ahead before Pablo Fornals levelled late in the first period.
West Ham looked to have won it when Michail Antonio made it 2-1 in the 78th minute, but Ben Brereton Diaz sent the match to penalties with a late equaliser, and the visitors incredibly triumphed 10-9 in the shootout.
Blackburn were bright in the early exchanges, with Dilan Markanday and John Buckley both having attempts inside the first four minutes, and the Championship side managed to make the breakthrough in the sixth minute.
A brilliant pass from Tayo Edun released Vale, and the 21-year-old scored his first goal for the club through the legs of Alphonse Areola.
West Ham improved in the period that followed, with Nayef Aguerd heading a Manuel Lanzini cross wide of the post, before Fornals and Antonio both had attempts.
Vladimir Coufal was next to try his luck in the 30th minute, missing the target from outside the box, before Emerson Palmieri fired over the Blackburn crossbar two minutes later.
The Hammers were pushing, though, and their reward eventually came in the 39th minute through Fornals, with the Spaniard levelling the scores with a smart finish into the far corner after pouncing on a loose ball.
Lanzini had a chance to send the home side ahead in the 42nd minute, but the attacker's effort from outside the box was too high.
Antonio hit the Blackburn post in the 55th minute, but it was not all one-way traffic, with Blackburn looking dangerous when they had the chance to break.
The opportunities were coming for David Moyes's side, though, with Nayef Aguerd heading wide of the post in the 63rd minute, before Antonio had a powerful strike blocked.
Diaz fired just wide of the West Ham goal shortly after, before Angelo Ogbonna missed the target with a header in the 73rd minute.
The Hammers scored a second in the 78th minute through Antonio, who fired one into the corner after the ball had dropped kindly for the forward.
There was still plenty of time for Blackburn to respond, though, and the leveller came in the 88th minute when Diaz cut in from the left and fired into the back of the net, with the match heading for a penalty shootout.
Incredibly, both sides scored their opening nine spot kicks, but Ogbonna's miss proved to be crucial, with Blackburn progressing to the fourth round courtesy of a 10-9 victory in the shootout.