The Citizens dominated proceedings in the first half and were two goals up at the break courtesy of two almost identical strikes from Ilkay Gundogan that were both set up from the right by Riyad Mahrez.
Gundogan had the chance to seal his hat-trick and the three points for City when he stepped up to take an 84th-minute penalty after Phil Foden was brought down by Pascal Struijk inside the area, but the German midfielder struck his effort against the post.
Just a minute later, Leeds went up the other end and pulled one back through substitute Rodrigo, setting up a nervy finish for the hosts in the closing stages.
However, Pep Guardiola's side held on for their 10th consecutive league victory, condemning Leeds to their 19th defeat of the season and Allardyce's first since taking charge of the Whites on Wednesday.
Leeds remain 17th in the table and sit just above the relegation zone on goal difference having played a game more than the bottom three, but despite the defeat, Allardyce was able to take some positives from his side's second-half performance.
"In difficult circumstances but accepting the job, you have to do the best you can. It is about being on the training ground for the last week and feeling the adrenaline rush of that and a squad of players we have to make better and as good as we can with four games left," Allardyce told BBC Match of the Day.
"It is a difficult challenge but one we have jumped at - me, Karl [Robinson] and Robbie [Keane]. We have to try to get to the position where we can try and win a football match. We take some hope out of today on the second-half performance. Not the first, but the second.
"We now need to build on that and try to get a result. I was slightly fearful, but we sorted it out for the second half and told them they were playing with too much fear.
"They needed to get on the front foot and try to challenge them. Apart from a silly penalty... and a great save by Joel [Robles] that gave us a chance to go up the other end and score. When you see Man City running down the clock for four minutes at the end, you know you have given yourself a chance. We did right until the end, but it didn't quite happen.
"Without a huge amount of knowledge, barring that from the staff that was already here, we knew we would have to make five subs today. It was disappointing that one of our subs [Pascal Struijk] gave the penalty away.
"The rest came on and kept us going. We were putting fresh legs on and adding energy which helped us create a good goal. Karl was saying, when we were making the last two, there was still 15 minutes to go. But I told him we'd take the risk. It was worth it. We came close, but not close enough."
Allardyce added: "It is the old story of taking one game at a time. We have to get better between now and next week. We have to improve our performance by 5%.
"It would be too much to ask more than that in such a short space of time. We can iron out a bit more in terms of the playing side and our possession. That is what let us down in the first half.
"It was too rush and not accurate enough. They weren't hitting the right spaces. When we did that a bit better in the second half we got more attacking options and it gave the defenders some respite."
With Nottingham Forest and Everton taking on Southampton and Brighton & Hove Albion respectively on Monday, Leeds could find themselves inside the relegation zone before welcoming Newcastle United to Elland Road next weekend.
The Whites will then prepare for a trip to West Ham United on the penultimate weekend of the season before playing host to Tottenham Hotspur on the final day.