Charlie Austin headers saw the Championship side lead twice in the first half, with replies coming from Lucas Digne and then Andros Townsend shortly after the break.
The subsequent shoot-out saw every penalty scored up to goalkeeper Seny Dieng saving Tom Davies' effort in sudden death at 7-7 and Jimmy Dunne then converted to send the hosts through to the last 16.
There was another victory for a second-tier outfit against top-flight opposition as Stoke won 3-1 at Watford.
Sam Clucas and Josh Tymon both stuck late on after Watford's Ashley Fletcher had cancelled out Nick Powell's opener.
Holders Manchester City made comfortable progress in the end as they won 6-1 at the Etihad Stadium against League One Wycombe, who took a shock lead in the 22nd minute through Brandon Hanlan.
Kevin De Bruyne equalised seven minutes later and efforts from Riyad Mahrez and Phil Foden had City 3-1 up heading into the break. There were further goals in the second half from Ferran Torres, Mahrez and Cole Palmer.
Liverpool won 3-0 at Norwich, with Takumi Minamino scoring in each half.
After Minamino's fourth-minute finish, Christos Tzolis had the chance to equalise from the penalty spot just before half-time but saw his spot-kick saved by Caoimhin Kelleher.
Divock Origi made it 2-0 shortly after the interval and Minamino added his second in the 80th minute.
Southampton got past Sheffield United 4-2 on penalties after a 2-2 draw at Bramall Lane, with Fraser Forster saving kicks from Rhian Brewster and Oli McBurnie in the shoot-out.
Enda Stevens had put the Championship hosts 1-0 up after eight minutes before strikes from Ibrahima Diallo and Mohammed Salisu turned the game on its head. McBurnie then levelled things up in the 66th minute.
Leeds were taken to penalties as well before getting past Fulham at Craven Cottage, winning 6-5 after a 0-0 draw, with Rodrigo Muniz producing a decisive miss in the shoot-out.
Marcus Forss scored four times, including a penalty, as Brentford thrashed Oldham 7-0 at home, with Yoane Wissa netting twice and Raphael Diarra scoring an own goal.
Jay Rodriguez also registered four goals in Burnley's 4-1 home victory over Rochdale, who went in front early in the second half via Jake Beesley.
Elsewhere, Preston beat Cheltenham 4-1 at Deepdale and Sunderland were 2-0 victors at Wigan.