At the heart of the critique is the fear that Mr Livingstone's language when discussing the Jewish community, Israel and Zionism is close to classic antisemitism.
"Ken determines Jews as a religious group but does not accept Jews as an ethnicity and a people and did not respond on this other than to say that as an atheist he found this hard to comprehend. In the same way that Black, Irish, Women and LGBT groups are afforded the right to determine their own identity, many of us feel that Ken doesn't afford Jews that right."
The letter reveals that Mr Livingstone used the words Zionist, Jewish and Israeli interchangeably and did so "in a perjorative manner".
The Jewish Labour figures behind the letter, which include Liberal rabbi Danny Rich and former Limmud chair Andrew Gilbert, express concern that Mr Livinstone has aligned himself with the politics of radical Islam. "The real and more pressing issue is that of the strong perception that Ken is seeking to align himself with the politics of the Muslim Brotherhood and the Iranian regime, whilst at the same time turning a blind eye to Islamist antisemitism , mysogynism (sic) and homophobia."
The authors of the letter added: "We are concerned that this is more about infantile, far-left politics, being seen to take a stance against whatever the anti-establishment or anti-imperialism cause of the moment is."
The letter was written ahead of a meeting between the Jewish communal leadership and Ed Miliband later this month, when the issue of Mr Livingstone will be top of the agenda.
Read the letter to Ed Miliband