Mertens opened the scoring in the 18th minute for Belgium, who were regularly frustrated by Australia goalkeeper Mat Ryan.
But Axel Witsel put the result beyond doubt with a stunning 77th-minute strike to delight the Stade Maurice Dufrasne crowd.
Belgium were disappointing at this year's World Cup despite making it to the quarter-finals, but coach Marc Wilmots will have been encouraged by this display.
The hosts – who do not play again until their first Euro 2014 qualifier against Andorra next month – dominated the first half, but Australia were much improved after the interval.
However, after their encouraging performances in Brazil, the Socceroos were unable to produce the same quality against the fifth-ranked Belgium as they look ahead to next year's Asia Cup, which they will host in January.
Belgium defender Vincent Kompany started despite the calf scare he suffered in training on Wednesday, while Divock Origi led the line in the absence of the rested Romelu Lukaku.
Australia coach Ange Postecoglou handed debuts to Chris Herd and Trent Sainsbury in defence, while youngster Brad Smith won a spot on the bench after his international clearance came through just hours before kick-off.
Despite the visitors' bright start, Ryan had to be alert to keep out Mertens' low drive in the 12th minute.
And it was a sign of things to come as Belgium pinned their opponents back, with Kevin De Bruyne next to go close after he rattled the crossbar following Australia's sloppy concession of the ball.
The Socceroos failed to clear the danger, though, and the hosts deservedly went in front, as Mertens, at the far post, saw his scuffed shot bounce into the ground, over Ryan, and in.
Tim Cahill – Australia's all-time top scorer – almost levelled midway through the first half after Mathew Leckie surged down the right, but his strike was brilliantly saved by Thibaut Courtois.
Mertens and Origi then tested Ryan with long-range efforts before Chris Herd went off with a leg injury.
Belgium needed a last-ditch tackle from Jan Vertonghen to deny Leckie after the break, while at the other end, De Bruyne shot just wide on a superb counter-attack.
Several substitutions interrupted the flow of the game but Ryan was still at the top of his game, making himself big to deny Nacer Chadli in a dangerous one-on-one opportunity.
There was nothing the 22-year-old keeper could do with 13 minutes left, though, as Witsel was given space on the edge of the penalty area and smashed a half-volley into the top-right corner.
And that settled the contest as Australia suffered their fifth defeat in a row.
