Many expected the south-coast club to struggle this season with a new coach and much-changed squad but they have gelled quickly and find themselves second in the Premier League after five matches.
After replacing Mauricio Pochettino in June, the Dutch coach saw his squad lose top players like Adam Lallana, Rickie Lambert and Dejan Lovren as Southampton cashed in on an impressive 2013-14 campaign.
Three successive wins, including Saturday's 1-0 victory over Swansea City, have lifted them above every team except pacesetters Chelsea in the table, but they have the League Cup to thank for giving them impetus.
Having taken a point from two league matches Koeman picked up a first competitive win in the second round against Millwall and his side have not looked back.
Since the 2-0 win at the New Den they have scored eight goals while only conceding one, but Koeman does not want his players to get carried away with their start and has called for focus ahead of Tuesday's trip.
"Of course, it's a great start," said the Dutchman. "Maybe nobody expected this, but we believe in our ambition and our spirit.
"We have to keep fighting every day to continue with the results we've had until now."
They are likely to face a much-changed Arsenal side from the one that beat Aston Villa 3-0 on Saturday, though, given Arsene Wenger's tendencies in the competition.
The Frenchman has never won the competition but regularly uses it to blood some of his younger players, affording them much-needed match action, and this clash should be no different.
Chuba Akpom's six goals in four matches for the Under-21s may be impressive enough to earn a spot, while Hector Bellerin, Serge Gnabry and Joel Campbell could also be involved.
However, defensive shortages may force Wenger to use most of his regular back four.
Mathieu Debuchy (ankle) is definitely out of the trip while Nacho Monreal is also a doubt having missed the matches with Borussia Dortmund and Villa.
That may see Kieran Gibbs play and former Southampton academy graduate Calum Chambers, who has made eight appearances in all competitions this term, could face his old side for the first time.
Despite Wenger's rotation policy in the competition, Arsenal have not been knocked out at this stage since 2002-03, however they did need penalties to beat West Brom in the third round last season.
Southampton, who are looking for a fourth successive appearance in the fourth round, also have injury concerns with James Ward-Prowse and Maya Yoshida both picking up knocks at Swansea, but Koeman hopes to have them available.
"Prowsey and Maya, we have to wait to know exactly what happened and how big is the injury," Koeman told The Southampton Echo.
"But the doctor told me these are not big injuries and hopefully they are back on Tuesday."
