Referee Jarred Gillett did not hesitate to point to the spot when John Egan was adjudged to have handled Julian Alvarez's cut-back.
But when Haaland stepped up to take the penalty, his effort rattled off the post while Wes Foderingham dived the wrong way.
It sparked an X-rated chant from the Sheffield United fans, who serenaded the superstar striker with a rendition of 'you're just a s*** Billy Sharp'.
Sharp is a Sheffield United legend, having played there for eight years before leaving this summer for MLS side LA Galaxy.
His eight-year stint was his third spell with the Blades having come up through their academy.
READ MORE ON CITY
Why is Pep Guardiola not on the touchline for Man City against Sheffield United?
Man City to attempt raid of Premier League clubs for two signings worth combined £130m
It was a difficult first-half for City where they enjoyed plenty of possession against United but could not break down a resilient defence.
But Haaland had the last laugh eventually when he finally got on the scoresheet after an hour on the clock.
Jack Grealish clipped a cross toward the back post and there was the gargantuan striker, who made no mistake as he headed down and into the United net.
There were muted celebrations from the City star, who looked more relieved than anything to finally get on the scoresheet.
Most read in Football
Chelsea and Tottenham stars emerge as Man United transfer options after injury
Piers Morgan says Mikel Arteta's 'tinkering' is to blame for Arsenal setback
Maddison oozes class and Arsenal's tinkering - What we learned in Premier League
Dean Henderson undergoing Crystal Palace medical with a view to £20million switch
Juanma Lillo is standing in charge for the Premier League champions with Pep Guardiola absent.
City had been expected to run riot over the Blades, who came up from the Championship this season, but their defence was finally broken at Bramall Lane.
It looked like the Blades were going to claim an unlikely point when Jayden Bogle's deflected strike beat Ederson after an uncharacteristic error by Kyle Walker.
But Walker was bailed out by midfielder Rodri, who smashed into the top corner with his left foot as he pounced on a loose ball in the box.