Tam Courts' men came into the game fresh from their morale-boosting 1-0 Premiership win over champions Rangers at Tannadice last week but could not reach those heights again against the Championship side.
Tomi Adeloye fired the Honest Men ahead in the 55th minute but with 10 minutes remaining striker Nicky Clark levelled from the spot after Ayr defender Jack Baird, booked earlier by referee Nick Walsh for a foul on Adrian Sporle, was handed another yellow and a consequent red for tangling with Tannadice defender Ryan Edwards.
The match went to extra-time and then the drama of penalties where Clark scored the decisive spot-kick for a 4-3 win.
Ayr skipper Aaron Muirhead, Sean McGinty and Nick McAllister scored but Tannadice keeper Ben Siegrist saved from Patrick Ready and Jonathan Afolabi.
Marc McNulty skied the Taysiders' first penalty over the bar but Charlie Mulgrew, Peter Pawlett and Jamie Robson also netted to take the visitors into Sunday's quarter-final draw.
Courts had resisted the temptation to give new signing McNulty a start.
The Scotland striker started on the bench after signing on at Tannadice for a second loan spell from Reading, but Darren Watson and Sporle came back into the side.
Ayr brought in Aidan McAdams, Muirhead, Michael Hewitt and Mark McKenzie but the match took some time to warm up.
In the 20th minute James Maxwell fired wide and two minutes later, Ayr's keeper McAdams kept out a low drive from Tannadice midfielder Ian Harkes.
Ayr's skipper Muirhead then picked out Adeloye in the box with a searching pass and he fired an angled shot just wide of the far post.
At the other end, McAdams thwarted Pawlett after the United midfielder had been played in by Clark.
Ayr came close to taking the lead after the break when Hewitt's cross from the right found the unmarked Maxwell but his side-footed volley was knocked over the bar by Siegrist who gathered the corner.
However, the goal did arrive when United's defence failed to deal with a long ball into the box, McKenzie slipping the ball Adeloye who knocked it into the net at the second attempt after an initial Siegrist save.
The Tannadice keeper soon had to deal with a drive from Maxwell and the Ayr midfielder then fired another attempt wide.
In between those two attempts, McNulty replaced Sporle as Courts looked for a way back.
Siegrist again proved his worth, knocking a Reading shot over the bar before tipping a close-range effort by McKenzie onto the bar and over after he had been set up by Adeloye.
United looked bereft of belief but Clark grabbed the chance to level from the spot after Baird was judged to have fouled Edwards at the back post as they jousted for a Mulgrew cross.
Ayr battled with 10 men but McNulty missed a great chance with a header in added time and passed up another opportunity at the start of extra time when he scooped a Pawlett cut-back over the bar.
Defender Liam Smith rattled the post with a long-distance piledriver before McAdams saved a Clark shot on the half turn.
After the turnaround, Pawlett had the ball in the net only to be ruled offside and then Ayr looked to sneak a late win but it went all the way to penalties and ultimately a Dundee United win.