According to our best records of the transitions from glacial to interglacial and back it is deglaciations that are rapid and glaciations that are slow. Not that temperatures cannot swing wildly during a deglaciation (e.g. The Younger Dryas) so, in that sense, cold can come on quickly, but warming is just as fast or faster. But a true slide into glacial conditions requires time. This makes sense when you think about the huge quantities of ice that must form by precipitation.