Alex Neil's side beat Wycombe Wanderers 2-0 to end their four-year stay in England's third tier, and the Sunderland end erupted with emotion and relief as the final whistle blew.
Gray, who was reporting on the game for talkSPORT, has always supported Sunderland, playing for their first team between 1992 and 2004 after coming through the Black Cats' academy.
The talkSPORT co-commentator knows exactly how it feels to lose a play-off final too, after having his decisive penalty saved in 1998 against Charlton Athletic.
And the relief he felt for his club after they'd finally won at Wembley shone through.
"Well I've taken a breath, wow it's special," Gray managed to say, after holding back the tears as he soaked up what just happened.
"Just to see this in front of us at Wembley. I didn't think it was ever going to happen, you just have that expectation things won't go your way.
"What a group of players. Alex Neil coming in halfway through the season, what he's achieved at this football club, my football club.
"They never deserved to be in League One in the first place, we've had to spend four years there and we've done it the hard way. And now they get the chance to celebrate with those Sunderland supporters, it's just incredible."
But whilst the former player was over the moon that his club had finally escaped League One, Gray also highlighted how key it was that the club's hierarchy doesn't now miss the chance to push on from Saturday's victory.
There has been confusion this season over who owns the Black Cats, with multiple shareholders in the mix and it being unclear just how much say each party has in the goings-on at the Stadium of Light.
And this is something Gray says needs to be fixed and not ignored just because they've been promoted.
"We're a country mile away from there [being back in the Premier League]," Gray said.
"The ownership of the club, that needs stabilising still. Who's in charge of who's coming into the club? Who's leaving?
"And the club itself need to know which direction this football club is going in because they [the fans] deserve that. How many people bring 46,000 to Wembley Stadium?
"They should not be in League One, and they're there because of mistake after mistake. It now looks like they're on the right path to success."
He added: "And if they start doing things right and they make things happen, this is what you can achieve."