The former world No. 1 has slipped down the rankings following a long injury lay-off, but he overcame the brave resistance of Rubin at the Rod Laver Arena.
A first-round winner over Bjorn Fratangelo, Rubin pushed the 17-time grand slam winner all the way, taking the third set to tie-break.
Earlier, Rubin stood his ground, saving numerous break points in an entertaining first set before Federer took it 7-5.
The four-time Australian winner showed his class in the second which he won 6-3.