The 20-year-old has struggled for regular first-team football under Jesse Marsch so far this season, starting only four of his 19 appearances across all competitions.
Gelhardt had been linked with a possible loan move to a number of Championship teams earlier this month, including Blackburn Rovers and former club Wigan Athletic, but Sunderland have won the race for his signature.
The youngster, who is highly-rated by Leeds, will now test himself in the second tier with the Black Cats, who currently sit ninth in the table and just one point behind the playoffs.
Speaking to Sunderland's official website on his arrival at the club, Gelhardt said: "I know how big the club is and I'm really looking forward to meeting the lads and trying to help.
"I spoke with the head coach and the playing style seemed a perfect fit for me. I've watched Sunderland on the TV this season and seen some positive, attacking football and I can't wait to get involved."
Head coach Tony Mowbray added: "It is no secret that Joe had plenty of options on the table this month, so it is a testament to the club and its staff that he felt Sunderland was the best move for him.
"He has made an impact at Premier League level throughout the past 18 months and he is a really exciting signing, who fits our playing style and will complement a forward line that is already packed with talent.
"I'm sure the fans can't wait to see him in action and we are looking forward to supporting his development."
Gelhardt becomes Sunderland's second signing within the last 24 hours after the arrival of winger Isaac Lihadji from Lille for an undisclosed fee.
Both Gelhardt and Lihadji could be in a position to make their Sunderland debut when they travel to Millwall in the Championship on February 4.