After the first interruption, a tannoy announcement said the game would be abandoned if the racist behaviour continued.
However, the game was stopped again just before half time after discussions between the referee and Mr Southgate. At half time, many of the men left the premeses and England chose to continue the game.
They went on to win 6-0.
On Tuesday morning, Bulgarian prime minister Boyko Borissov called for Mr Mihaylov to resign because of the incident, saying he "strongly condemned" the behaviour.
Uefa president Aleksander Ceferin also spoke out, urging the "football family [to] wage war on the racists".
After the match, England striker Marcus Rashford tweeted: "Not an easy situation to play in and not one which should be happening in 2019.
"Proud we rose above it to take three points but this needs stamping out."
Another England player Harry Maguire called it "disgraceful", however Bulgaria coach Krasimir Balakov said he "did not hear" any racist chanting from his side.
The Vasil Levski Stadium was already subject to a partial closure for this match after Bulgaria were sanctioned for racist behaviour from their fans during earlier qualifiers against Kosovo and Czech Republic.
England fans who attended the match posted photos of racist stickers that had been put up in the area surrounding the stadium, including swastikas and one that read "thank you for not mixing".
