The Hammers led on nine minutes through Lucas Paqueta but lost 2-1 in the third round tie at Villa Park in a controversial second half.
Amadou Onana equalised in the 71st minute with a close range finish following a contentious corner that sparked West Ham protests.
Referee Tim Robinson awarded the set piece after he believed that a shot from Boubacar Kamara deflected off one of the visitors' players.
However, replays show that it didn't and the call was therefore wrong, with Morgan Rogers scoring Villa's second just five minutes later.
Defeat for West Ham is their second of 2025 and comes a day after they announced Potter as the successor to Julen Lopetegui.
READ MORE ON WEST HAM
Graham Potter instantly makes history as West Ham boss during Aston Villa clash
West Ham emerge as shock contenders to sign Marcus Rashford from Manchester United
The 49-year-old was out of work since leaving Chelsea in April 2023 but has signed a two-and-a-half year deal at the London Stadium.
Potter returns to the Premier League with a club who are seven points above the relegation zone despite a summer of big spending.
Vice-chairman Karren Brady has revealed why West Ham decided to hire him despite his lengthy absence away from the touchline.
"In Graham Potter, we have recruited another very good man," she said in a column for The Sun.
Most read in Football
Plymouth confirm Wayne Rooney's replacement and explain why new boss is man to save season
Why Aston Villa are wearing a special kit against West Ham as club honour anniversary
Graham Potter instantly makes history as West Ham boss during Aston Villa clash
Moyes to make Everton return, Man City strike deal, Barcelona express interest in Rashford
"A hugely respected elite Premier League coach, and someone who was the unanimous choice of our Board to steer our project back in the right direction.
"From our early conversations with Graham, we felt the overwhelming sense that he is the right fit for West Ham United, as are we for him.
"As Graham himself has said, since leaving Chelsea 20 months ago, he has been waiting for and carefully considering the right opportunity, seeking the challenge and project that would re-ignite the feeling of excitement and anticipation that would inspire his return to top-flight management.
"We are delighted that, despite being highly sought-after by some of the biggest clubs in Europe, he has discovered that feeling with West Ham United.
"Graham understands and shares our principles - that hard work, togetherness and commitment breeds progress and success. He was attracted by our ambitious project and the steps we are taking to reach our objectives.
"And he has found common ground with the core values that our supporters and the people of east London hold dear - a family culture, a sense of belonging to a clear identity, and a dedication to hard graft and craft.
"Modern east London is a thriving, exciting hub of innovation combined with tradition. A creative centre of opportunity and chance, especially for young people.
"Graham has already spoken with Mark Noble about his desire to engage with and maximise the talent of our world-renowned Academy - something that is embedded so deeply into the fabric and soul of our club.
"As Graham has shown throughout his successful coaching career so far, he is adept at building on strong foundations and creating a culture of togetherness, spirit and trust so that, regardless of the outcome of a single 90-minute football match, everyone connected with the club feels a part of something bigger and more significant.
"Of belonging to an identity that continuously fosters pride, honour and belief. We believe that the right foundations are in place for further success in the future at West Ham United.
"It is only 18 months ago that we enjoyed European glory and, while that night in Prague will never be forgotten, there is a genuine and fierce hunger at this club to ensure that we continue to work as hard as possible to build and create more outstanding memories.
"Graham Potter shares that ambition. That is why we didn't hesitate to hand him a two-and-a-half-year contract, and that is why we firmly believe we have appointed not only a good man, but the right man for West Ham United."
Read More on talkSPORT
David Moyes heading back to Everton and Wayne Rooney tipped to join him
'Out of order' - Ferguson says Everton player 'had a pop' for not playing Young's son
Potter's first Premier League match as West Ham boss will come on Tuesday at home to Fulham.
He then faces Crystal Palace before another trip to Villa, who are now in the fourth round of the FA Cup.
Ben Jacobs reveals all about Graham Potter replacing Julen Lopetegui as West Ham manager