Sekou Mara sent the Saints ahead in the 23rd minute of the contest before Moussa Djenepo registered the home side's second five minutes later.
Southampton have lost their last six Premier League matches, but they beat Crystal Palace in the FA Cup on Saturday and have now advanced to the semi-finals of the EFL Cup, with Nathan Jones enjoying comfortably his best moment at the club since arriving as head coach in November.
Man City have dominated this competition in recent years, lifting the trophy on six occasions since 2014, but their journey in this season's League Cup has now come to an end in surprising fashion.
The away side's first real chance of the match came in the sixth minute when Cole Palmer broke towards the edge of the Southampton box before feeding Ilkay Gundogan, but the midfielder swept his effort wide of the post.
Duje Caleta-Car then came close for Southampton, with the defender attempting to pick out the bottom corner from the edge of the box, only to be denied by a smart save from Stefan Ortega.
Mara was next to try his luck for the home side, as he turned and fired wide of the Man City goal in the 21st minute, with the visitors struggling to find their rhythm.
Southampton made the breakthrough in the 23rd minute of the contest when Mara expertly picked out the back of the net with a clever finish after excellent work from Lyanco, who had read a loose pass before driving down the right and delivering a low cross.
The home side then doubled their lead in the 28th minute when Djenepo advanced towards the Man City box before finding the back of the net with an excellent finish, but there will be question marks surrounding the goalkeeping of Ortega, who was far off his line.
The expected response from Man City did not arrive in the first half, and the Citizens ended the opening 45 minutes without managing to register a shot on target.
Julian Alvarez sent one just wide of the far post early in the second half after meeting a pass from Kevin De Bruyne, who entered the field as a half-time substitute alongside Nathan Ake and Manuel Akanji.
Adam Armstrong had Southampton's first half-chance after the restart, with the forward firing wide of the near post after advancing to the edge of the area.
Erling Braut Haaland was introduced for Jack Grealish just before the hour, but it was Southampton that continued to look the most likely, with Kyle Walker-Peters swinging one wide of the post in the 65th minute.
Haaland volleyed one over the crossbar 20 minutes from time, before Che Adams had the ball in the back of the net for Southampton, but the offside flag came to Man City's rescue.
Gavin Bazunu then managed to keep out a close-range attempt from Alvarez in the 78th minute, with the Argentina international on the full stretch.
Man City could not respond in the latter stages, though, with Southampton comfortably advancing to the final four courtesy of a two-goal success at St Mary's.
Next up for Man City is a trip to an in-form Manchester United in the Premier League on Saturday, while Southampton will visit struggling Everton on the same afternoon.