The 29-year-old has only started 16 Premier League games this term with injuries playing a part, while Mamadou Sakho and Martin Skrtel have been the preferred pairing.
He spoke of his frustration at how his personal campaign had gone in the Danish press recently and hinted at a potential exit, stating he was "no-one's third choice".
Barcelona have previously been linked with a move for the Denmark captain and national coach Olsen fears Agger could grow frustrated without regular first-team football.
"You may ask Daniel about (it)," he was quoted as saying by Bold.
"He can best gauge how it looks now and, more important, (for) next season.
"But of course he should be playing. Otherwise, I believe we will have a frustrated and irritated captain.
"I hope he will make the right choice."
