Twice the visitors went ahead through Vitaliy Mykolenko and Abdoulaye Doucoure, and twice Palace would respond through Eberechi Eze and Odsonne Edouard, but Gueye's 86th-minute strike proved decisive in a spectacular Selhurst contest.
Saturday's game saw the welcome return of Michael Olise to the Palace bench, while Eze - fresh from signing a new long-term contract - returned to the Eagles' first XI too.
Everton had failed to score in a league-high five top-flight games this season before facing a Palace side with an identical number of clean sheets, but Roy Hodgson's normally reliable defence was split open with just 52 seconds on the clock.
One week on from his early opener against Brighton & Hove Albion, Mykolenko saw one shot from the edge of the box blocked by Joachim Andersen, but Everton worked the ball out to the right to Jack Harrison, who sent in a delicate cross for the unmarked left-back to head home in the centre of the box.
Mykolenko's effort marked the fastest goal in the 2023-24 Premier League season so far, but Everton's joy was short-lived, as Palace were awarded a spot kick when Jarrad Branthwaite caught Eze with a trailing leg.
Eze stepped up to the 12-yard mark, took a stuttering run-up and coolly placed the ball into the bottom corner past a stagnant Jordan Pickford to draw Palace level in the fifth minute, and the frenetic end-to-end action did not cease there.
Nineteen minutes after the penalty incident, Eze once again went down in the box under the attention of Branthwaite, but instead of awarding a second spot kick, referee Samuel Barrott booked the Englishman for diving.
Eze drew another save from Pickford in injury time before the referee's whistle signalled the end of a breathless first half, and fans barely had a chance to gather their thoughts before Doucoure propelled Everton back into the ascendancy.
Four minutes after the restart, Mykolenko struck a fierce volley from the edge of the box, which rebounded off the post to Doucoure, who had kept himself onside and had the simple job of tapping home into an open goal.
The Toffees were then forced to weather an Eagles storm, and it took a brilliant save from Pickford to prevent Gueye from converting into his own net on the hour mark, and Hodgson did not hesitate to introduce a fit-again Olise as his side pressed for a leveller.
While Olise was not involved, the equaliser would arrive for Palace in the 74th minute, as a Joel Ward delivery was only cleared as far as Jefferson Lerma, who headed the ball back into the danger area for Edouard to poke home.
James Tarkowski allowed Lerma's lofted header to drop over him, seemingly thinking that Edouard would be caught offside, but the Frenchman's strike stood following a quick review as Selhurst Park roared their side on for a winner.
However, not long after coming close to scoring an own goal, Gueye seemingly came up with the decisive moment for Everton in the 86th minute, as Doucoure fizzed a ball into the box for the former Paris Saint-Germain midfielder, who created a brilliant angle for a strike with a neat first touch before finding the bottom corner on the stretch.
Palace's injury-time efforts were futile as Everton made it six wins from their last nine games in all tournaments, rising to 14th in the table in the process, just two places and one point worse off than Hodgson's side.
Crystal Palace now have two weeks to gear up for a trip to Luton Town after the international hiatus on November 25, one day before Everton welcome Manchester United to Goodison Park.