The 62-year-old was suspended by the League Two club last month following what it described as "serious and credible" allegations made against him.
A club statement on Friday confirmed the departure of Yems but did not allude to the allegations and said the move was made by mutual consent.
Crawley co-chairman Preston Johnson said: "We're looking forward to the next era of Crawley Town Football Club.
"We have an opportunity to build on more than 125 years of rich history and take this club to the next level.
"We're eager to partner with our players and supporters as we build a team and community that Red Devils fans can continue to be proud of - both on and off the pitch."
Yems' alleged conduct is being investigated by the Football Association, which issued a statement on the issue on Wednesday.
It read: "We are aware of allegations made against Crawley manager John Yems. We are treating the allegations extremely seriously and are currently conducting an investigation into them. We cannot comment further until that has concluded."
The Daily Mail reported that the allegations against Yems included the use of discriminatory language towards black and Asian players at the club, and segregation of the training ground.
Yems was appointed manager in December 2019 and guided the club to two consecutive 12th place finishes, the position they currently hold with one game remaining.
Assistant manager Lewis Young will continue to serve as interim manager for the club's final game of the season away to relegated Oldham on Saturday.