Historic England, which is tasked with protecting the country's historical environment, wrote to Westminster Council in March about its “concerns... on heritage grounds”.
But Mr Brokenshire believed the location should be viewed in a positive light as an example of “what government can do - and what can happen when it fails to act”.
He was keen to dispel some of the misconceptions he said some have about the memorial, which would be “sensitive to the site itself. The building will only take up seven per cent of the area.”
It was important “not to forget” the purpose of the educational centre, as “a means of learning. It is there to challenge our thinking and what we can do as individuals.”
During his visit, Mr Brokenshire announced plans for a time capsule at the centre, which he encouraged survivors to contribute to.
"Having their lived experiences recorded is just so important. I’m keen for that to be at the heart of our work.”
He also told survivors about his personal connection to the Holocaust. “My father in law, who was Jewish, was assisted out of Germany by Frank Foley, the Mi6 agent."
