Charlie Mulgrew had put the Scots ahead from the penalty after 35 minutes after Steven Naismith had been fouled, but Israel, ranked 94 in the world and with just one win in their past 10 games, turned the game on its head with two second-half goals.
Dor Peretz hauled Israel back on level terms on 52 minutes followed great work by Taleb Twatha and Israel's hopes were further boosted when Scotland's John Souttar was sent off nine minutes later, for his second caution following a lunge on Munas Dabbur.
Israel finished the match strongly and deserved their win, although it needed an own goal from Celtic left-back Kieran Tierney to secure the win.
This was Israel's best performance of the Herzog reign and the result would have been more emphatic had it not been for a stellar performance in goal by Allan McGregor, for a Scotland team ranked 39 in the world. The Rangers' stopper produced fine stops to deny Bibras Natcho and Dia Seba, while Twatha hit the bar from four yards out.
Scotland boss Alex McLeish admitted his players would have to suffer "for a wee bit" after losing against an Israel team for whom this was only their second home win in four years. "It's doom and gloom, we lost, we know the consequence of losing," he said.
McLeish described Souttar's dismissal as "a bit soft" and remarked: "It's never easy playing with 10 against 11. Israel had one or two players in really good form and they deserved it on the night although we are disappointing with the goals we lost."
At the halfway stage of the Nations League group, Scotland, Israel and Albania all have three points, with Scotland hosting Israel in the return match next month.
TEAMS
ISRAEL: Harush, Yeini, Tibi, Ben Harush, Dasa, Peretz, Natcho, Kayal (sub: Einbinder 82), Tawatha (sub: Atar 76), Dabbur , Sahar (sub: Saba 45).
Subs not used: Haimov, Glazer, Habashi, Tzedek, Hemed, Micha, Awwad, Bitton, Kannadli
GOALS: Peretz 51, Own goal 74
BOOKINGS: Ben Harush, Natcho
SCOTLAND: McGregor, Souttar , Mulgrew (sub: McKenna 45), Tierney O'Donnell, McGinn, McDonald, Robertson, Russell (sub: Forrest 67), McGregor, Naismith (sub: McBurnie 76)
Subs not used: Gordon, Henry, Shinnie, Armstrong, McLaughlin
GOAL: Mulgrew (pen) 35
BOOKINGS: Robertson
RED CARD: Souttar 61
-----------------
MATCH STATS:
POSSESSION
Israel 56%, Scotland 44%
SHOTS
Israel 25, Scotland 7
ON TARGET
Israel 9, Scotland 2
CORNERS
Israel 6, Scotland 3
FOULS
Israel 14, Scotland 8