With five minutes remaining, Villa had been on course to end their seven-game losing streak in the competition courtesy of a first-half goal from Morgan Sanson.
However, after Leander Dendoncker was sent off, Jamie Reid netted from the penalty spot before Dean Campbell scored a brilliant winner in the final minute.
Although Unai Emery made numerous changes to his starting lineup, Villa ended the match having introduced five first-team players for the closing stages, making Stevenage's fightback all the more impressive.
The League Two outfit had only missed out on taking the lead in the first half courtesy of an extremely tight offside call, and it allowed Villa to eventually go in front themselves.
A neat team move was finished off by Sanson, with the French midfielder advancing into the penalty spot and converting into the far bottom corner.
However, having failed to score a second, Villa were always susceptible to a sucker punch, one which was delivered when Dendoncker was dismissed for pulling down Campbell as he made his way into the penalty area.
Despite a lengthy VAR check, Reid held his nerve to send Robin Olsen the wrong way, and Stevenage appeared likely to accept a lucrative replay.
Instead, they netted a dramatic winner moments later, catching Villa out with a short corner to allow Campbell to lash a well-struck effort inside the near post, sparking ecstasy in the away end.
Steve Evans's side now take their place in the last 32 and will travel to Stoke City, with Stevenage booking their spot in this round of the competition for the first time since 2014.