The competition co-hosts kicked off the ninth edition of the women's tournament with a bang at Eden Park.
Hannah Wilkinson scored the only goal in their 1-0 win with a close-range finish from a superb New Zealand counter-attack.
The home crowed roared when the ball hit the back of the net and did the same at full-time whistle, with the players celebrating wildly on the pitch.
New Zealand sit just 26th in the world rankings but proved too much for 12th place Norway, who played with the first ever female Ballon d'Or winner - Ada Hegerberg - up front.
After such an astonishing victory, captain Riley was full of happy tears when she spoke on the momentous occasion.
READ MORE ON WOMEN'S WORLD CUP
Arsenal star McCabe 'breaks finger' before crunching Raso and Laura Woods loves the battle
Barca star Walsh is key for Lionesses and Chelsea star could be breakout name at World Cup
"I can't I'm so, so proud we have been fighting for this for so long," she told BBC Sport.
"We had a clear goal that we wanted to inspire young girls, young people around this country and around the world and I really think we did that tonight. Anything is possible!"
Hours before the match, a deadly shooting took place in Auckland that left three people, including the gunman, dead.
Riley spoke on how they wanted to bring something positive to fans after the terrible events that took place.
Most read in Football
Mahrez and Henderson set for Saudi, Newcastle agree Barnes deal, Onana to Man United
Goldbridge: Ten Hag has humiliated Maguire and Man United may have to let him go for free
Angry Pochettino runs onto pitch and confronts referee in Chelsea's win against Wrexham
Stunning referee bodycam footage shows Jesus' perfect chip for Arsenal goal
"This morning something really, really tragic happened and we wanted to bring something positive tonight.
"We thought of the victims and the first responders and they made us so proud. We wanted to just help bring something amazing today."
New Zealand could very well have scored a second against Norway but Ria Percival saw her late penalty crash against the crossbar.
However, they were able to hold on and make history in front of a 42,137 person crowd, the largest attendance ever for a football match in the country.
You can follow live commentary of the 2023 Women's World Cup on the talkSPORT network, with our coverage kicking off on Saturday with England vs Haiti on talkSPORT at 10.30am