Peebles must have been in confident mood coming to Kelso, having taken 15 goals off the hosts in their last two meetings.
However, this was an evenly-matched, hard-fought encounter, with the visitors just taking the spoils.
United started the match well and were rewarded after 10 minutes when Damien McNulty picked up the ball in midfield and threaded a pass to Chris Falconer on the right wing.
The Berwick man swung a cross into the box and Peebles central defender James Young stuck a leg out to divert the ball into the net past his own keeper Darren Walker.
The home side could have made it two nine minutes later as a corner on the right was headed out by the Peebles defence, only as far as the feet of United’s Darren Bowie – only a great save by Walker, diving to his left, prevented the goal.
Peebles were starting to come back into the game and a shot from 25 yards by Daniel McAleavey whistled past the post and minutes later the same man supplied a cross which Paul Murray headed wide.
Ryan Beveridge, in the United midfield, tried his luck with a 30-yard right-footed shot which went inches over the crossbar.
Rovers equalised in the 38th minute as McAleavey again swung a dangerous cross into the box over the home defence and this time Jamie Doyle cushioned a right-foot shot past Elliott Turnbull in the Kelso goal from six yards.
The game ebbed and flowed, and the home team were reduced to 10 men in the 78th minute when United’s Liam O’Rourke was sent off after receiving his second yellow card from Edinburgh referee Michael Addy.
This did not seem to deter United, who created a golden chance two minutes later when Shane Bonnington beat two defenders before crossing on the left to find Beveridge unmarked on the six-yard line. However, instead of hitting the ball first time, the midfielder dallied and was thwarted by the Peebles defence.
The game was still on a knife-edge, but the visitors supplied the killer punch eight minutes from time as Daniel Ndiwini received the ball on the left edge of the box.
The winger managed to escape a couple of challenges before planting the ball in the bottom corner of the net past the diving Turnbull, earning his team another three points to keep them at the top of the East of Scotland Division One table.