If using ADL slows down writing the file, it is by a minuscule amount and the buffer would cover tha. Unless you are shooting really long bursts, you'd never notice. This is one of those theoretical things that makes no difference in actual use.
As Stan says, there are other benefits to using ADL that do make a difference in your results. Some people like ADL, and others don't. But that decision is based on workflow and desired results, not any camera speed concerns.