The ex-Scottish champions, recovering from a financial crisis, needed a last-minute equaliser from Andy Little in front of 4,500 fans at Balmoor stadium on the north-east coast of Scotland.
It was Rangers' first league match outside the top flight of Scottish football.
The club went into administration in February before a takeover in June, and were demoted from the Scottish Premier League to the third division - the bottom of the country's footballing pyramid - after a meeting of Scottish Football League clubs last month.
It means Rangers will not renew league rivalries with Glasgow neighbours Celtic until at least 2015.
Rangers are clear favourites to win the third division, but manager Ally McCoist said Saturday's result was a warning to his players. "If our boys at any time thought this was going to be a cakewalk, they know where they are now," he told Sky Sports.
"Obviously them scoring late on, [I'm] probably relieved to get a point after the game. I thought we defended appallingly, to be honest."
McCoist's side got a fine start when Barry McKay put them in front with a cool finish on 26 minutes.
But Rory McAllister levelled for Peterhead just after an hour and, with eight minutes remaining, Scott McLaughlin scored with a ferocious shot from just inside the box.
Little rescued Rangers in the 90th minute, getting the final touch on the line after a header from Kevin Kyle came back off the crossbar.
