Leading 2-0 from the home leg, the Israeli champions saw their advantage erased by two first-half strikes from Wanderson in Razgrad.
Ludogorets moved in front 11 minutes after the interval, with Marcelinho making it 3-0 on the night.
Hapoel needed to score to stay in the competition and an all-important strike from Mohammed Ghadir after 61 minutes kept the Israeli champions alive.
Resilient defending kept Ludogorets at bay and their frustrations boiled over in the last minute of added time when Wanderson was sent off.
Hapoel could meet SPL champions Celtic in the playoff round following their hard-fought 1-0 victory Rosenborg.
Celtic boss Brendan Rodgers praised Israeli international Nir Bitton for a commanding performance after he was called from the bench to replace the injured Erik Sviatchenko.
Rodgers said: “Nir came in as a midfielder but played at the back as though he’d been there all his life.”