The Norwegian superstar was handed the armband for the clash with Newcastle United on Saturday afternoon.
City made a number of changes for the contest with their Champions League tie with Real Madrid in mind, but the line-up still consisted of plenty of experienced heads.
Ilkay Gundogan, John Stones and Phil Foden all started the game, as did experienced goalkeeper Ederson.
However it was Haaland who was given the honour, much to the ex-Sunderland and Blackburn star's surprise.
Speaking on Gameday Live, Gray told talkSPORT: "It's a bit of a surprise to me, I have to say.
READ MORE ON FOOTBALL
'Wind your neck in' - Lewis-Skelly and 'pups' Arsenal sent warning in stunning Cundy rant
Erling Haaland makes title jibe as his dad claps back at Arsenal in social media post
"I don't see Erling Haaland as a leader - I mean, he just drifts out of a game so much albeit it seems to be him who scores the goals every week.
"John Stones has the experience and with the players on the pitch, it's really, really surprising."
Despite not being deemed as captain-material by Gray, his tally of 27 goals in 34 games this season has certainly impressed.
Explaining the decision post-match, Guardiola said: "Haaland has signed a 10 year contract, so I have the feeling about him as captain.
Most read in Football
Preston player accused of racism as incident reported in Burnley match
Jude Bellingham sent off in bust-up with LaLiga referee to leave teammates shocked
Premier League star in floods of tears after being forced off 15 minutes into clash
Diallo suffers serious injury as Man United hit with quadruple blow before Tottenham clash
"I am really happy for him.
Facing Newcastle, many sparked the debate of who was better between Haaland and Alexander Isak - the striker who has scored 19 goals in 28 games.
But despite the comparison, Gray says that there's only one winner.
"There's no comparison at the moment. That's not to say that I don't love watching Alexander Isak play, because I do. I think he's an outstanding player.
"He can certainly play for one of the top sides in the Premier League, there's no doubt about that but if you're looking at a man who has won it all, done it all, it has to be Erling Haaland."
Despite Haaland captaining the side, it was his strike partner Omar Marmoush who stole the show with a hat-trick against the Magpies.
With James McAtee striking late on, it meant that City ran out as 4-0 winners and moved back into the top four in the Premier League.
Despite the victory, fans were left worried when Haaland went down four minutes from time with what looked like a potential knee injury.
But speaking after the game, Guardiola was able to put everyone's minds at ease.
He explained: "When he went down everyone was scared but he walked off smiling and the doctor didn't tell me bad news so hopefully he is fine [to face Real Madrid on Wednesday]."
Haaland then shared an image in the dressing room of him eating steak in celebration.
Reflecting on the win, Guardiola said: "The way we played, we remembered what we were and what we have not been this season.
"We always try, but sometimes we cannot do it. I am really pleased with the pace and intensity we had up front, Nico [Gonzalez] and [Abdukodir] Khusanov were extraordinary.
Read More on talkSPORT
'Preparing for Jon Jones' - Tom Aspinall stunned by power of 400Ib World's Strongest Man
British 15-year-old, faster than Bolt and Lyles, is already breaking records
"We need to win these games to qualify for the Champions League. We have to make this kind of performance to send our people home happy.
Simon Jordan insists Newcastle's Alexander Isak has the same market value as Erling Haaland!
"[Omar]Marmoush did really well in Germany, a player with his quality and pace, I was pleased with the way he scored. Against Orient he missed chances, but today in the final third he was really good."