The South Americans were criticised for their heavy-handed tactics which saw six players cautioned, including Wilmar Barrios for what looked like a headbutt on Jordan Henderson.
England forged ahead via the penalty spot after Carlos Sanchez's foul on top-scorer Harry Kane, who, after a lengthy interruption following Colombian protests, which saw several players rake the penalty spot, the Spurs striker stepped up to coolly convert his shot down the centre of goal.
England fans were celebrating until the final minute of injury time, when Perry Mina rose highest to head home from a corner and force extra-time.
With no further goals, the match went to penalties. Henderson saw his spot-kick saved by David Ospina, but Colombia missed two efforts - one hitting the crossbar and the other superbly blocked by Jordan Pickford. And it was left to Eric Dier to convert and send England through.
Colombia manager Jose Pekerman said: "People in England or others should not think of Colombia players like this. It's a very competitive game.
"England have a great weapon in the air. Players fall in the box. They collide and fall. It's hurtful. You have to try and stand in the shoes of the players. They are subject to situations which shouldn't be present in football.
"These interruptions are bad. It's OK, but when there are so many fouls and interruptions that's not good. We have to find the right balance so both teams can play with the same intentions.
"We should not only look to Colombia but also England's players. In the next match England will be more careful. Today we were on the receiving end of this. It's been so obvious, too obvious."
Southgate responded by saying Colombia's tactics came as no surprise. "We didn't rise to it in general, which was brilliant. A couple of times we lost our cool.
"Today was a special night for every Englishman. The fans that were here were outnumbered but I'm delighted for them and everyone at home.
"We're trying to write our own history, and I've talked to the players about that. They write their own stories. We don't have to be bowed by the pressure of the past.
"This was special but I want us to go on. I don't want to go home yet."
While Pekerman and his players are now preparing for the long journey home, England can look forward to a quarter-final meeting against Sweden at 3pm on Saturday.