Postecoglou has tasted defeat in five of his six away matches in charge and the Tony Macaroni Arena proved a difficult ground for his predecessors, with neither Brendan Rodgers nor Neil Lennon able to take three points since Livi were promoted.
Celtic have not won on their past four visits and former Partick Thistle player Penrice is aiming to make life tough for the visitors.
"I wasn't involved in the games previously when Celtic were here but I watched them and that's what it's about, you want to make it as uncomfortable as possible," the 22-year-old said.
"They are going to have a lot of fans out there and you want the fans to start getting on their back because they have not scored and are not creating chances and looking a bit nervous. That's what we want and that will be a big part on Sunday.
"Aberdeen brought a large following here and we got off to a good start and they started to get on their backs. Although it didn't end well for us, you want to turn them against them and we will try and use that to our advantage."
The last two meetings between the sides in West Lothian have ended 2-2 and there could be more goals in store on Sunday given Postecoglou's commitment to attacking football.
There were chances galore at either end as Celtic lost 4-3 to Real Betis on Thursday and Penrice is excited about the prospect of an end-to-end encounter.
"You want those kind of games, you want the fans to be excited," he said. "I think the fans will like it when it is end to end. We know we can compete and that's the big thing for us.
"First of all, from my point of view, as a defence we have to make sure we get a clean sheet and the attacking players need to go and open up those gaps that get left, and hopefully we can do that."