Three late goals from Whittaker - in the space of 12 minutes - put Russell Martin's side through, while Daniel Williams also scored for the Swans before Rhys Shirley equalised for the League One side.
The Swans opened the scoring in the 29th minute with Williams' shot from just outside the box finding the back of the net, after taking a huge deflection off the leg of Plymouth goalkeeper Callum Burton.
Swansea's Ryan Manning received the game's first yellow card for charging into Panutche Camara and the hosts threatened for the rest of the first half and turned around with a 1-0 lead.
Just after the hour mark, 18-year-old Shirley's volley found the back of the net after a terrific cross from Adam Randell.
The game looked to be heading towards a penalty shootout, but a tremendous long-range shot from Whittaker put Swansea back ahead in the 79th minute.
And Whittaker claimed a second goal with four minutes remaining with a terrific through ball from Jamal Lowe allowing him to score Swansea's third.
And Whittaker struck again at the death with his low shot putting the result beyond doubt for the Championship side.