Pardew has won all four matches in charge since he took over from Neil Warnock earlier this month, and Palace have come from behind in each of their last three outings.
In an exciting first half which provided all of the game's goals, Southampton twice took the lead through Graziano Pelle and Scott Dann's own goal.
Marouane Chamakh and Yaya Sanogo found the net to cancel out those two goals, before Chamakh netted his second to give Palace the lead for the first time.
The visitors held on in the second period to book their place in the fifth round for the first time since 2010, and Pardew was thrilled with the display put on by his side.
"The players have been phenomenal," he told BBC Sport.
"I did have a chuckle this morning when I saw the think-tank in the paper when it said we had 0.02 per cent chance to win the FA Cup, that's probably gone up a bit.
"I thought both teams were great. There was some really good football played. We are disappointed with the goals we conceded but the third goal would have graced any pitch in the world.
"Yaya Sanogo was good, Marouane Chamakh was better so it gives me a welcome selection headache."
‘I don’t think Liverpool would look at Ollie Watkins, a striker isn’t a pressing issue for them – it’s Arsenal who need one’ Former Reds star explains why his old club don’t need an out-and-out forward this summer
‘He’ll still be playing at 40 at a good level because he’s in such good shape and looks after himself so well. He does everything to be a top professional’: Ex-Liverpool coach insists Mohamed Salah has plenty more miles in the tank
