The Portuguese, who led Spurs from 2012-2013, believes that while the club has all the tools for success, things always go awry.
He believes it's an issue with owner Joe Lewis and chairman Daniel Levy who, despite creating a world-class opertation off the field, can't get it right on the pitch.
They have employed coaches such as Mauricio Pochettino and Jose Mourinho, but silverware remains elusive.
And Villas-Boas reckons this could create a hard situation for new head coach Conte, despite his title-winning talents.
He said: "For Antonio it will be a challenge, a great challenge.
"We have a Premier League-winning coach in Antonio, going to a club with the best stadium in England and the best training facilities in Europe or the world. But you always see that something is missing at Spurs.
"Daniel has had a coach that took the club to a Champions League final in Mauricio Pochettino and one of the most successful coaches in the world coaching his team in Jose Mourinho.
"It's up to him and Joe Lewis to decide what they want for the organisation.
"But it's a difficult organisation because of that lack of clarity."
Villas-Boas didn't want to totally discredit Levy but believes the chief can be too hard to deal with for a manager.
He added: "I don't want to turn this into a criticism of Daniel.
"Everybody has their ups and downs, strengths and weaknesses, but Daniel is not an easy person to deal with.
"There are a lot of things that have to be said face to face, each side with truthfulness.
"But sometimes they look like they don't happen that way.
"Coaches have to be judged on their performance — I agree.
"We lost, miserably, two games that led to us agreeing terms for me to leave but he's not the easiest person to deal with. Yet what is easy right now in football?"
Meanwhile, Villas-Boas has lamented how his tenure turned out in north London.
He believes he's now a forgotten man, despite turning down Paris Saint-Germain to try and build something special at White Hart Lane before being sacked after heavy back-to-back defeats to Man City and Liverpool.
He said: "It looked like a moment where we could create a great dynamic with the arrival of Franco [Baldini - Tottenham's former technical director] and it turned out to be a nightmare. To this day, I was proven right.
"Franco was not fair. A long time has passed, I don't even know where Franco is and he doesn't know where I am and that is it. Football has moved on.
"I have the greatest memory of the Tottenham fans, the ambience of White Hart Lane and the supporters.
"But for me and my history at Tottenham, it is like I don't exist any more. This is how people treat you when you are kicked out of the door.
"That is the nature of football and it is very sad. The Tottenham story was a pity.
"At the end of my first season, I didn't go to Paris Saint-Germain because I thought we were building something special at Tottenham. I thought the love was reciprocal.
"When I found out that people were actually kicking me out of the door, that is when things turned nasty."
But, Villas-Boas does conceded Conte and Tottenham could mesh well.
He added: "I'm a highly emotional and highly principled person and for me not to have renewed my contract at the end of my first season at Tottenham is something I will never forgive those people for.
"But Antonio was very successful at Chelsea and Inter Milan, he can do a good job.
"He is one of those people who puts people into the peaks of performance by motivation, by determining objectives.
"In football, you always reach stages where that link-up to the top has to be there.
"If that link-up is fine-tuned it can maybe be a good fit."