Paul Linford has a fascinating post on previous examples of 'young' Speakers (I do like that adjective being widely applied to a 46 year old man):

The year 1789 is chiefly remembered for being the year of the French
Revolution. But it was also the year the Commons elected two
thirty-something Speakers who both went on to occupy Number 10 Downing
Street.

The first of these was William Grenville, who was
elected Speaker at the ripe old age of 30 and held the office only very
briefly before quitting to become Home Secretary.

In his place
was elected the 32-year-old Henry Addington, who remained in the Chair
until 1801 when he suddenly found himself elevated to the Premiership
in place of his childhood friend Pitt the Younger, who declared that
Addington was the only successor he could countenance.

In the
meantime, Grenville had gone into opposition, along with his close ally
Charles James Fox. But in 1806, he was summoned by King George III to
head up what was termed the Ministry of All Talents, though
unfortunately for him, it only lasted a year.

Even further back,
in 1715, one Spencer Compton was elected to the Commons chair at the
age of 42 - four years younger than John Bercow is now. He served as
Speaker for 12 years until 1727, when he was elevated to the House of
Lords as the 1st Earl of Wilmington. In 1742, he succeeded Sir Robert
Walpole as Prime Minister.

Bercow has said he will do nine years
in the Chair, effectively two full Parliaments plus the toe-end of this
one. That will make him 55 when he stands down - younger than Gordon
Brown was when he became Prime Minister in 2007.