"I, of course, withdraw and apologise," he said.
Ms Alibhai-Brown had called on David Cameron to withdraw the whip from Mr Fabricant, who is no stranger to controversy.
He was sacked as vice-chairman of the party earlier this year for criticising the HS2 rail project and making online comments about colleague Maria Miller.
Mr Fabricant was criticised in 2007 after he described an election defeat as being like "a holocaust".