Essaouira’s Portuguese fortications are the stuff of travel posters. The Atlantic rollers thrash against the walls of this historic Moroccan town, visited by surfers and tourists charmed by its laidback atmosphere.

Few among them know that Essaouira was once predominantly Jewish, consisting of a wealthy and cosmopolitan community.

Joseph Sebag, the self-proclaimed “last Jew” of Essaouira, will gladly chat about the fishing port’s Jewish past in his bookshop-cum-antique store where he sells Berber jewellery alongside African masks and old French crockery.

His family has lived here uninterruptedly for 250 years and his 88-year-old mother can still recall the story of almost every Jewish home.

Wearing a silver Berber bracelet and rings on his fingers, Mr Sebag receives visitors in the cavernous shop that was once the office from which his father ran a shipping insurance business.

“I am a secular Jew on the outside but orthodox within,” he says. His mother is of Berber Jewish descent and his father was a Sephardic Jew from Spain.

After seven years in the United States and Canada, where he sold computer parts, Sebag missed Essouaria so much that he returned. Morocco’s Jews have had a chequered history, but Sebag has never been deterred. “This is where my friends are, my family and my roots, and I have never regretted coming back. I feel safe in Morocco.”

Essaouira’s Jews left mostly in the late 1940s and 1950s to go mainly to Israel, and also to France, the United States and Canada. But Sebag thinks they should have stayed. Morocco was always a welcoming home, he claims, and is certainly the only place where he wants to live.

When during the Second World War the French Vichy government asked King Mohammed V how many Jews lived in Morocco, which was still a French protectorate, he replied, “I have no Jews, only Moroccans.”

Essaouira has long been a symbol of tolerance, ever since the 18th-century Sultan Mohammed Ben Abdallah founded the town then known as Mogador next to a Portuguese fort and settled a handful of Jewish trading families.

The last stop for trans-Saharan cavarans carrying spices, almonds and tiles, Mogador, along with Tangiers, became Morocco’s only port open to European trade and as such thrived for 150 years. Muslims, Jews and Christians worked comfortably side by side.

Trading families from England set up home and founded an English school, and for a while English teas and bowler hats became the fashion. England’s famous Sebag-Montefiore dynasty has roots in Mogador.

The richer Jewish families lived in grand houses in the Kasbah, while the poorer Jews from Marrakesh and the countryside lived in the Mellah, the increasingly cramped Jewish quarter. For a while, Jews outnumbered Muslims. In the oldest of the two walled Jewish cemeteries by the ocean, the layered tombs testify to generations of Jewish presence.

The houses in the run down Mellah are now occupied by Essaouira’s poorest residents, but the refurbishment of three of the town’s synagogues is a sign of a renewed interest in the town’s Jewish past. Descendants of one-time Jewish residents make nostalgic trips to discover traces of their ancestral roots, while every September pilgrims from around the world visit to pay homage to the legendary Rabbi Chaim Pinto who is buried here.