With Hartlepool bottom of the Football League and having won just two of their previous 15 games in all competitions prior to Friday's tie, an upset looked likely at Victoria Park, but Jonathan Franks eased initial nerves for the League Two strugglers when he opened the scoring.
However, former Hartlepool midfielder Stephen Turnbull hauled the sides level before Marlon Harewood controversially had an effort ruled out for the hosts.
Rivers' big moment arrived in the 90th minute when he cut in from the wing and unleashed an unstoppable drive, but there was still time for even more late drama as Matthew Bates struck the crossbar as Hartlepool pushed for an equaliser.
Seventh-tier Blyth will now go into Monday's draw for the next round, when they will hope for a money-spinning tie against a Premier League side.
The deadlock was broken when Franks timed his run to perfection to head home from close range from Neil Austin's chipped cross after 32 minutes.
On his return to his old stomping ground, Turnbull curled a 25-yard free-kick up over the wall and into the top corner just before the hour as Blyth grew in stature as the game progressed.
Harewood thought he had scored after 65 minutes as he swivelled and shot from the edge of the six-yard box. The ball squirmed past goalkeeper Peter Jeffries, but he eventually clawed it away, with replays showing that a sizeable part of it had crossed the line. However, the referee waved play on.
Rivers then clinched the victory late on, before Blyth rode their luck in the final stages to progress.
