The Conference side fell behind to a Tommy Rowe strike in the 26th minute of their third-round replay on Tuesday, before goals from Michael Gash and Jack Byrne turned the game on its head.
Britt Assombalonga pulled the hosts level from the spot in the 74th minute after Lee Vaughan had felled the striker in the box, but Joe Lolley sealed a famous victory for Andy Thorn's men two minutes later.
Elsewhere, Joe Garner scored a hat-trick as League One side Preston North End overcame Championship club Ipswich Town 3-2 to set up a fourth-round match against Garner's former club Nottingham Forest.
Frank Nouble gave the visitors the lead at Deepdale, but a Garner double put Preston 2-1 up within 10 minutes.
David McGoldrick appeared to have sent the clash to extra time with a 76th-minute equaliser, until Garner netted a dramatic winner two minutes from time.
Holders Wigan Athletic, meanwhile, survived a potential slip-up as they scraped a 3-1 extra-time win against Milton Keynes Dons.
Luke Chadwick and Nick Powell scored either side of half-time to send the game to an additional 30 minutes, before Powell and Marc-Antoine Fortune booked a clash with Crystal Palace in the next round.
In the only all-Premier League clash of the day, Fulham cruised to a 3-0 victory over Norwich City.
Darren Bent, Ashkan Dejagah and Steve Sidwell all found the net as Fulham extended their unbeaten run against Norwich to a remarkable 15 matches in all competitions and booked a tie against Sheffield United.
In Tuesday's remaining third-round replays, Watford recorded a 2-0 victory at home against Bristol City and will face the winner of tomorrow's clash between Manchester City and Blackburn Rovers, Port Vale will host Brighton and Hove Albion after triumphing 3-2 at Plymouth Argyle, and Sheffield Wednesday face a trip to Rochdale after their 4-1 victory over non-league Macclesfield Town.
In addition to the replays, there were a further three third-round matches on Tuesday, and League Two Oxford United let a two-goal lead slip as they were held to a 2-2 draw by Championship Charlton Athletic.
Birmingham City will be relishing the opportunity to dump Swansea City out of a second cup competition this season, after a 3-0 triumph over Bristol Rovers set up a fourth-round tie against Michael Laudrup's men, whom they knocked out of the League Cup in September.
And Bournemouth face a mouth-watering match against Premier League giants and seven-time FA Cup winners Liverpool in the fourth round after storming to a 4-1 win against Burton Albion.
