Seems like a simple thing, copying files from one system to another, but sometimes it’s the little things that make us think.

Case in point, a few years back we ran into an issue when copying many large (20Gb+) files for one system to another. The system would actually run out of kernel memory after a couple of hours of copying the files and freeze up. A few tweaks and tests later we had found an acceptable number of concurrent processes and the copies went on without a hitch.

That was until recently, when the actual speed of the copy was brought into question.

Speed of the copy that’s fixed right? Not much we can do here outside of using robocopy or some third party tool, and even then we are still limited to by the fact that the transaction is buffered.  Buffered you say, what’s that?  Well in a Buffered transaction Windows keeps track of file reads and writes and buffers these transactions. Normally this works great speeding up future reads and writes to the files, but when dealing with large files the overhead of buffering can be staggering, not only in performance, but in kernel memory usage, hence the original problems with kernel pool exhaustion we experienced in the past.

Ok, so buffered transactions are slow down large file copies, how would we go about doing and unbuffered copy?  As far as I can tell there isn’t a native Windows tool that does un-buffered copy, but there is an Exchange tool ESEUTIL that can be used to accomplish this.  Just copy the ESEUTIL.EXE and ESE.DLL files from your friendly Microsoft Exchange system. The syntax for the command follows:

Eseutil /Y [Sourcefile] /D [Destinationfile]

One caveat with the Eseutil is that you can only do one file at a time and the utility does not accept wild cards so you’ll have to use the full path and file names when copying files.

So is it worth all this? You be the judge. In testing we had the following results for a 50GB  file copy  over a 1GB connection:

All in all not too shabby,  For more details  on this and other performance related issues check out  the Askperf Blog