The former world No. 1 has slipped down the rankings following a long injury lay-off, but he overcame the brave resistance of Rubin at the Rod Laver Arena.
A first-round winner over Bjorn Fratangelo, Rubin pushed the 17-time grand slam winner all the way, taking the third set to tie-break.
Earlier, Rubin stood his ground, saving numerous break points in an entertaining first set before Federer took it 7-5.
The four-time Australian winner showed his class in the second which he won 6-3.
Rubin continued to battle, and he led 5-3 in the third set, but Federer’s experience proved crucial as he attacked Rubin’s serve to see out the match on a tie-break.
Paying tribute to his opponent, Federer said: “Rubin’s a great fighter ... he's aggressive on the ball and I think he had the upper hand on the baseline.
“I think my serving kept me in the match today ... I'm happy I got through somehow.
"I wasn't feeling as good as in the beginning. I definitely got a little lucky in that third set ... so it was good to fight through."
There was also disappointment for two other Jewish players on Wednesday. Dudi Sela succumbed in four sets against Lukas Lacko of Slovakia. The Israeli won the opening set 6-2 but Lacko battled back to win 6-3, 6-2, 6-4.
Diego Schwartzman of Argentina was beaten by Steve Darcis in four sets.
In the Ladies Singles, Camila Giorgi bowed out against 12th seed Timea Bacsinszky.