Cameron Archer scored in the eighth minute after the hosts took a quick free kick and worked the ball out wide to Ryan Manning, who squared across for Archer to finish from close range.
Southampton doubled their lead in the 28th minute thanks to good work on the left flank from Kyle Walker-Peters, who drove into the box and passes across goal to Joe Aribo for an easy finish.
Foxes manager Steve Cooper saw his side half their deficit in the 65th minute after Abdul Fatawu low cross from the right of the six-yard box bounced off of Facundo Buonanotte and into the back of the net, before Jamie Vardy equalised from the penalty spot less than 10 minutes later.
Jordan Ayew scored the winner in the eighth minute of second-half stoppage time when Harry Winks played a pass from a corner to the edge of the box, and the result leaves Leicester in 13th place with nine points after eight games.
Meanwhile, Southampton remain winless and are last in 20th place with just one point from eight matches and have now conceded 18 times in the Premier League, the second most in the division.
Southampton's first goal can also be credited to Walker-Peters given he carried the ball from just inside his own half towards Leicester's penalty area before passing to Manning, who then played across to Archer.
Boss Martin might be particularly pleased with his side's second goal considering several players combined on the left flank and managed to create a one vs. one situation between Walker-Peters and Justin, who allowed his opponent to drive into the box far too easily.
However, Fatawu's introduction in the 59th minute changed the course of the game, with the winger providing an assist for Leicester's first goal of the game.
Despite Vardy converting a penalty with a quarter of the game remaining, it looked as if Southampton would hold on for a point, but somehow Ayew was left free on the edge of the box to strike home Winks' delivery from a corner deep into second-half stoppage time.
SOUTHAMPTON VS. LEICESTER CITY HIGHLIGHTS
8th min: Southampton 1-0 Leicester City (Cameron Archer)
Southampton work the ball to the left flank towards Ryan Manning, who fires a low cross into the box but the ball takes a deflection and lands at the feet of Cameron Archer just yards out in the centre of the goal.
He could hardly miss!
28th min: Southampton 2-0 Leicester City (Joe Aribo)
Southampton score their second thanks to Joe Aribo's finish into the bottom-right corner after Southampton isolate Kyle Walker-Peters against James Justin, who fails to defend the full-back's square cross into the penalty area.
A very similar goal to the first!
65th min: Southampton 2-1 Leicester City (Facundo Buonanotte)
Leicester City pull one back thanks to Facundo Buonanotte, who is in the right place at the right time to see the ball bounce off him and past Aaron Ramsdale after Abdul Fatawu drives into the penalty area from the right and fires a low cross into the box.
Leicester are back in it!
74th min: Southampton 2-2 Leicester City (Jamie Vardy)
After the VAR recommends that referee Anthony Taylor go to the monitor, Ryan Fraser is shown a straight red card for preventing Jamie Vardy from scoring thanks to a pull on the Leicester forward in the box following a save by Ramsdale, but the striker converts the penalty and gets his goal anyway!
All square now!
98th min: Southampton 2-3 Leicester City (Jordan Ayew)
Leicester have a corner in the dying seconds of the game and Harry Winks plays the ball low to the edge of the penalty area, where Jordan Ayew is completely unmarked and fires into the bottom-right corner.
Ayew has won it!
MAN OF THE MATCH - ABDUL FATAWU
Abdul Fatawu was brought on in the 59th minute but his impact was substantial despite his limited minutes, with his direct pace, skill and intensity constantly threatening Southampton's backline.
The young winger completed two dribbles, won four of his seven duels and had seven touches in the box, the most of any player on the pitch.
Fatawu also managed to produce the assist for Buonanotte's goal just minutes after coming on, driving powerfully and deliberately into the penalty area before playing a low cross into the danger zone.
SOUTHAMPTON VS. LEICESTER CITY MATCH STATS
Possession: Southampton 42%-58% Leicester City
Shots: Southampton 14-18 Leicester City
Shots on target: Southampton 7-4 Leicester City
Corners: Southampton 10-6 Leicester City
Fouls: Southampton 10-10 Leicester City
BEST STATS
WHAT NEXT?
Southampton's win on Saturday was much needed considering they take on champions Manchester City next Saturday in the Premier League, before facing Stoke City in the fourth round of the EFL Cup on October 29.
As for Leicester, they will hope to bounce back against Nottingham Forest in their next league game on Friday, but will then have the opportunity to cause an upset in the EFL Cup on October 30 when they travel to face Manchester United.