When David Lean directed Oliver Twist 65 years ago, the character of Fagin had already been long established as a popular villain. There was the serialisation and subsequent editions of Charles Dickens's novel, while the celebrated actor-manager Herbert Beerbohm Tree played the part in a successful stage version in 1905. And there had been many film adaptations. Lon Chaney was Fagin in one of several silent versions; Irving Pichel took on the role in a 1933 sound version.The George Cruikshank drawings, which accompanied the original serialisation, provided a model that made Fagin, with his long beard, hat and notorious, beaked nose, as instantly recognisable a villain as Captain Hook or Dracula.

The crude, racist stereotyping went back to the original conception of the character. When Fagin makes his first appearance, he is described as "a very old shrivelled Jew, whose villainous-looking and repulsive face was obscured by a quantity of matted red hair". He is then referred to invariably as "the Jew" as though that were the key to his behaviour.

Dickens came to regret this, explaining that, at the time, the kind of criminal he was describing invariably was a Jew, but he was so uncomfortable that he removed many of the references from a later edition. In practice, however, it was no more than a gesture, offering little practical mitigation of the racial slur. A richly dramatic caricature, Fagin lived on into the 20th century as a negative but often revived archetype of Jewishness.

Lean's 1948 adaptation presents Fagin faithfully as the duplicitous criminal of "evil conscience" that Dickens had created. It does not add racist colour that was not already there, yet at the same time gives full weight to a portrait of rare nastiness. Beneath a surface warmth lies utter viciousness. Fagin grooms his young orphans to steal. He seems to offer them sanctuary but in reality condemns them to ruin. In their joint criminal enterprise, even his accomplice Bill Sikes is the victim of Fagin's superior intellect. Sikes steals, Fagin fences; Sikes provides the brawn, Fagin the brains. Although Sikes kills Nancy, it is Fagin who puts him up to it.

In a chilling sequence in the Lean film, which culminates with Nancy's murder, Fagin asks Sikes what he would do if he discovered that the Artful Dodger had "peached" on him. "I'd smash his head in," he says.

The question of banning Oliver Twist divided even its Jewish critics

Fagin asks what if it were one or other of the boys.

"No matter who, I would do the same." It is only then that he tells Sikes that his girlfriend Nancy has turned informer. The effect is that of unleashing a savage dog.

Lean distilled Dickens's work into brilliantly cinematic images but it was the fidelity of those images to the original racist conception of Fagin that made them especially shocking in the context of the 1940s. The immediate aftermath of the Holocaust might have seemed the time to avoid such a negative stereotype, yet Lean carried on regardless of the consequences.

It wasn't that he hadn't been warned. In May 1947, the Production Code Administration, Hollywood's self-regulatory censorship body, said: "We assume, of course, that you will bear in mind the advisability of omitting from the portrayal of Fagin any elements or inference that would be offensive to any specific racial group or religion. Otherwise, of course, your picture might meet with very definite audience resistance in this country."

Not long after, make-up artist Stuart Freeborn began turning Alec Guinness into Fagin. He recalled that Lean requested two looks: one that followed the Cruikshank drawings, and another toned-down version. In the latter, Fagin "looked like Jesus Christ," remembered Freeborn. "David said: 'Forget that. It's not what we want at all'."

If Lean's instinct overrode a wider awareness, the ivory tower he occupied served only to encourage a blinkered outlook. Around that time, he explained the astonishingly favourable conditions under which the Rank Organisation financed its top producers to make whatever they wanted, how they wanted: "We can cast whatever actors we choose, and we have no interference at all in the way the film is made. No one sees the films until they are finished, and no cuts are made without the consent of the director or producer."

But surviving correspondence from the company's US distributor, Eagle-Lion, reveals that, behind the scenes, Rank was already regretting this set-up. In November 1947, Rank's publicity chief, Jock Lawrence, wrote to the head of Eagle-Lion, Robert Benjamin: "There are such problems… the Jewish one on Oliver Twist is a very serious one. It is something that I will have to show you here, rather than write them in a letter." He must have known that Lean had disregarded the Production Code Administration's advice concerning Fagin.

What made this seem all the more foolhardy was the crisis in Palestine. Lean's Oliver Twist opened in Britain a month after Israel's declaration of independence. Rarely could a première have been so badly timed.

Lawrence derived some comfort from the finished picture, which he saw only days before the première. "I was very happily surprised by the Fagin character," he wrote to Benjamin. "The film itself is so very good that the character of Fagin sinks somewhat into insignificance as compared to the whole. The fidelity of characterisation is such that I believe we have strong grounds for fighting any attacks… There is no doubt in my opinion, however, that the 'lunatic fringe' will attack the film on the basis of Fagin. But… it would not at all be justified except for the unusual length of the character's nose. However, we have it that way, and it is a truly great picture that I believe should overcome any such objections."

Lawrence believed that attacks would be inevitable when the film opened in the US. He suggested that Eagle-Lion delay the release to allow time for the Palestine situation to be settled, so that the film "might not be used by the Zionist groups for propaganda". He suggested, too, that Eagle-Lion's publicity stress the fidelity of the character to the original story. And it seemed to him "doubly important now" to organise a US visit of the child star of the film. "We can, in that way, stress the title Oliver Twist, through the character himself, taking away any attention possible from the Fagin character."

The Rank Organisation settled on a US release of September 1948. Following Lawrence's advice, it arranged a private advance screening for Jewish campaign groups. The reaction was not favourable. The Anti-Defamation League considered the characterisation to be an offensive stereotype that would be harmful in the light of existing tensions. The New York Board of Rabbis went even further, declaring it a "vehicle of blatant antisemitism" that "would play into the hands of un-American elements". It wrote to the president of the Motion Picture Producers Association of America asking that the film be banned. Rather than risk further protests that might jeopardise the company's other prestige releases, Rank postponed the release until a more opportune time.

The film's notoriety made it a magnet for further trouble. When, in February 1949, it opened in the British sector of Occupied Berlin, protesters picketed the theatre. They were mostly Jewish displaced persons, but their anger was shared by many in a city that was trying to recover from the poison of an all-too-recent-past. Berlin's mayor was one of several prominent gentiles to sign a petition that warned of the danger of "arousing antisemitism in Germany", and urged that the film be withdrawn. When the British authorities refused to intervene, the crowds were back the next evening and there were riots in which several were badly hurt. The British military government still stuck to its line that it would not impose a ban but must have been very relieved when Rank decided to withdraw the film.

The question of banning Oliver Twist was an issue that divided even its Jewish critics. The ADL stressed that at no time had it called for a ban, even though it believed events had borne out its warnings about the film's inflammatory nature. The American Council for Judaism condemned outright such calls, arguing that "opinions formed or opposition voiced after the event constituted the proper exercise of public opinion".

The film finally opened in the US in May 1951 after the Production Code Administration granted its seal of approval on the basis of cuts intended "to eliminate wherever possible the photography of the character of Fagin". The New York Times welcomed a sensible compromise: "Except in so far as the appearance of Fagin in point of time has been reduced, his motivating influence and his impact upon the story have been preserved. And that is both just to the purpose of the producers and considerate of those who might take reasonable exception to an excessive portrayal of a stereotyped Jew."

The lesson was that Fagin had to be rehabilitated. When Lionel Bart wrote his musical in 1960, he gave him the heart that was absent in both Lean's film and the novel. Reviewing the situation, Fagin finds it hard "to be really as black as they paint…" With irresistible tunes, the musical provides a lyrical redemption that makes him lovable. It rescues Fagin from the gallows that awaited him in the novel so that, memorably in the 1968 adaptation, he can dance towards a new dawn with the Artful Dodger, Jewish, exotic, other, but delightful.

Polanski's 2005 adaptation did not duck the retribution that faced Fagin but still offered a revisionist version. The novel describes Oliver's new guardian taking him to visit Fagin in prison, to show how he has received his due punishment. Polanski switches the agency for the visit from the adult to the child, who wants to express his gratitude to a man who gave him shelter.

"Fagin, you were kind to me," says Oliver. They hug and Fagin offers a final gesture when he tells him where to find his box of treasures. "It's yours, Oliver, it's yours."

What would Dickens have made of this? Essentially humanist and progressive, I think he would have understood why. But nonetheless he would have regretted the loss of a complex articulation of evil. Such was the price of too easy an acceptance of a racial stereotype.

Charles Drazin lectures on cinema at Queen Mary, University of London. He spoke about 'The Jewish Villain' last month as part of a lecture series organised by the Leo Baeck Institute in co-operation with the Wiener Library