The forward's second-half goal gave City a deserved three points at the King Power Stadium.
The champions made it three straight Premier League wins, but it was not a classic performance, with the visitors often wasteful after hitting 10 goals in their previous two games.
Silva was linked with a move away from the Etihad in the summer, but Guardiola expects big things from him this season.
He said: "I always have done, it's an incredible situation. It's not just the quality he played or the incredible runs or work ethic he has.
"We are happy he is here and in the future we will see what happens.
"He played really well in the last two games and today again. The connection with Jack (Grealish) and Joao (Cancelo) was brilliant.
"After an international break it is not one of the best places to come to play and we played really well. We created a lot, unfortunately we didn't score more goals.
"We suffered sometimes because they defended deep and had the counter-attack. Kasper Schmeichel had the quality to put the balls in the right area. It was a difficult, we knew it because always we suffer against Leicester."
Schmeichel saved Gabriel Jesus' far-post header after six minutes following some Silva trickery on the left before Kyle Walker and Ederson thwarted Harvey Barnes.
Schmeichel turned Silva's drive wide and Grealish fired over after a neat free-kick move.
But Leicester, who have now won two and lost two of their opening four games, nearly stunned City twice early in the second half.
Barnes' header hit the top of the bar from Vardy's cross before Vardy had a goal correctly ruled out for offside after rounding Ederson.
City heeded the warning and found the winner after 62 minutes.
Ferran Torres had already gone close before Cancelo's drive was blocked by Caglar Soyuncu, the ball falling into Silva's path for the forward to grab his first goal of the season from close range.
Ederson still needed to deny Ademola Lookman a debut goal, while Fernandinho and Grealish had chances to add a late second for City.
Leicester boss Brendan Rodgers said: "I feel most disappointed for the players because I thought they deserved more from the game from the opportunities they had. We had to defend but showed that resilience and mentality and we had our moments.
"In the moments we played through their pressure we looked a good side. We nearly got in with Barnes in the first half, in the second half we reinforced some tactical ideas to help us cope.
"We had good opportunities and then they get a little bit of luck with the ball falling to them.
"What is so important against Manchester City after their last few results is you stay in the game. We were right in there against a world-class team and unfortunately didn't take something from it.
"I'm proud of how the team kept fighting and how competitive they were."