When he challenged the MP over whether he meant because he was Jewish or working class, the man replied, “Both”.
Mr Bercow said his membership of the right-wing Monday Club as a student had been an “appalling decision”.
"Here I was, a Jewish boy … sidling up to racists," he said.
The former Speaker, who stepped down before the election in December, previously said that he did not believe Labour leader Jeremy Corbyn was personally antisemitic.
He was elected as a Conservative in 1997 before he became Speaker in 2009.