The pundit made the remark on BBC Radio 5 Live ahead of the Manchester City-West Ham game on Sunday.
Cole made the analogy on BBC Radio 5 Live during the build-up to Sunday’s Premier League game at the Etihad Stadium.
Asked if West Ham boss David Moyes had selected a defensive side against the champions, Cole replied: “You can say he has gone a bit negative. Why not?
“You’ve got to give Man City some respect otherwise you’re going to get picked off.
“Otherwise it will be a Holocaust and you don’t want that.”
The Holocaust, carried out by the Nazis during the Second World War, claimed the lives of around six million Jews.
Cole’s comments came on the first day of Hanukkah, an eight-day Jewish festival which commemorates the recovery of Jerusalem.
The former West Ham and Chelsea forward later apologised for his remarks on the same show.
He said: “I’d just like to apologise to the listeners for a totally unacceptable phrase that I used earlier in the show.
“I’m sorry if I’ve offended anybody, really and truly. Sorry.”
Last month, Bristol Rovers manager Joey Barton apologised after comparing a poor performance from his League Two side to the Holocaust.