The former Gunners winger was forced to interrupt after the Spurs supporter made the bold claim after they lost 3-2 at Everton on Sunday.
Spurs have not dropped out of the top flight since 1977, when they made an immediate return from the second tier.
However, Sunday's 3-2 loss to Everton leaves Ange Postecoglou's side 15th in the Premier League after 22 matches.
Following the defeat, Spurs fan Richard slammed Postecoglou as well as taking aim at Daniel Levy and the club's hierarchy.
He told The Sunday Session: "He's (Postecoglou) not good enough.
READ MORE IN FOOTBALL
'This might be his last week' - Postecoglou sack fears as Tottenham undone by 'Moyesball'
Man United shocked by Brighton with feat only Man City have achieved before
"The job is too big for him. I've said this before on talkSPORT. He's not good enough.
"He was found out 12 months ago. It was all good when he came into Tottenham playing and for it was all good. Then he got rumbled then he got tumbled and then that's when the problem started.
"He's not good enough... but the biggest problem is above. And Mr. Levy and Joe Lewis are the biggest problem at Tottenham.
"And I hope all the Spurs fans are listening to this. Do not go to Tottenham Hotspur Stadium anymore. Hold onto your feet and let's hope. I've been a Spurs fan all my life, 47 years.
Most read in Football
Dele Alli seals return to football to link up with Premier League legend in free transfer
Amorim says he's boss of 'worst' Man United team as club record 131-year unwanted first
'Great start to an interview' - Postecoglou gives spiky response after Tottenham defeat
Postecoglou confirms Solanke blow for Tottenham ahead of semi-final second leg
"Yeah, I've been a Spurs fan. I'd love the club to get relegated."
The bold claim forced Groves to interrupt, exclaiming: "You can't say that you want your club to get relegated because as a fan you support your team through thick and thin no matter what.
"The owners come and go, managers come and go, players come and go.
"Fans are the most consistent thing there. You can't say that you want your club to get relegated just so that the owners go because it goes to say you're gonna get back."
However, Richard continued: "Listen, we've got a massive stadium. As Mr. Levy says, 'it's all beautiful, it's got all the bells and whistles'. It's got this, it's got that.
"At the end of the day the man who owns it don't put no money in. The money don't come in.
"We've underachieved for many many years and I feel sorry for all the Spurs fans out there. I'm speaking from the heart."
Spurs now find themselves just eight points above the relegation zone, compared to the nine-point gap to the top half.
Emmanuel Petit calls for Ange Postecoglou to be sacked after Arsenal beat Tottenham in the North London derby
Meanwhile, it is almost 17 years since the club's last trophy, a triumph in the League Cup in 2008.
Postecoglou could still end that drought this term, though, with Spurs leading their Carabao Cup semi-final against Liverpool following the first leg.