Mikel Arteta's side came back from 1-0 down to beat Fulham 2-1 in the final minutes of their game at the Emirates on Saturday.
Gabriel - who made the error that allowed the Cottagers to take the lead - scored the winner, securing the club's fourth win in as many games and sending the Arsenal faithful into a frenzy.
But following the match, beIN Sport's co-hosts Richard Keys and Andy Gray quickly became the topic of conversation after they criticised the jubilant scenes in north London.
The pair's view divided opinions, with talkSPORT's Breakfast host and Arsenal fan Laura Woods disagreeing with them and labelling them 'the celebration police'.
However, for talkSPORT's Souness and Jordan, whilst they understood Arsenal's joy in the moment, ultimately, Keys and Gray's criticism of the celebrations was spot on.
"You win, you're delighted. When you win from coming from behind then you're even more delighted, there's an element of that," Souness said on Monday's White and Jordan show.
"Rich and Andy are correct, they've won nothing, it's just another step in the right direction.
"We come back to it, their big tests [against Liverpool and Manchester City] are all ahead of them."
Jordan then expressed he agreed with Souness, and highlighted that the celebrations could actually be damaging for the ambitions of the club.
"You're celebrating mediocrity. You've won a game," the former Crystal Palace chairman said.
"For god's sake what are you going to do if you actually win something? How far do you go then, what do you do then? Is everyone going to strip butt naked and run around the pitch in laps and celebrate for a week and a half?"
He continued: "In the moment, yes. But after the game, get on with the next one. Next. Next. Next."