The family has now taken legal action in the United States and is calling for a complete accounting and the return of all the work the Hungarian government possesses.
Baron Herzog’s great-grandson said the subject of the stolen art was “very emotional” and that he was frustrated by the failure to retrieve it.
David de Csepel told the New York Times: "I want to see justice done. My great-grandfather was one of the most famous collectors in all of Europe.”
His lawyer Michael Shuster said Hungary has been "one of the countries that has been the most recalcitrant" about returning looted art.
Earlier in July an Austrian museum agreed to pay more than £12 million for a painting stolen from its Jewish owner by the Nazis.
The dispute over the expressionist painting by Egon Schiele was settled out of court after a decade-long struggle.