The CAA launched a legal challenge but on Wednesday the High Court upheld the CPS decision.
In the judgment, it was revealed the CPS halted the prosecution because it did not believe anything Mr Ali said was "threatening, either explicitly or implicitly" and therefore a prosecution under Section 5 of the Public Order Act would likely fail.
The law forbids public behaviour that causes "harassment, alarm or distress" to those nearby.
The CPS also felt Mr Ali "qualified his statements by blaming the Grenfell tragedy on Tory Party policies" and "thus merely engaged in strident criticism of the Government", Lord Justice Hickinbottom noted.
The CAA argued this was irrational, saying Mr Ali's words meant that he believed Zionists had deliberately given money to the Tories to faciliate tragedies like Grenfell, where 72 people had died in a fire days before Mr Ali spoke.
But Lord Justice Hickinbottom said the Grenfell comments had to be "viewed in the context of the address as a whole" and no other part of Mr Ali's comments alleged "a deliberate and direct intention on the part of Zionists to murder people in high-rise blocks".
He said the CPS decision to stop the prosecution was reasonable "when that sentence is looked at in context".
He wrote: "Nothing in this judgment should be taken as condoning anything Mr Ali, or others at the rally whose words are recorded in the transcript, said.
"Clearly some things that were said were intemperate and deeply offensive and distressing to others, and not simply to those in whose direction they were aimed."
A CAA spokesman said: "Our judicial review did not succeed because the court decided that the decision to drop the case could not be characterised as irrational.
"That is hardly a ringing endorsement of the CPS’ conduct. We deeply regret that once more, hate speech on our streets has gone unpunished."
