Compression

Certain data types, such as text documents, may transfer faster if the data is compressed while in transit.  ExpeDat can apply ZLIB compression to transferring files, which typically reduces the size of compressible data types to about one-third of their original size.

Compression is activated in movedat by using the -z command line option.  For example:

movedat -z user@example.com:/path/downloadme /local/destination

Compression should only be used on files known to contain compressible data, such as text documents.  For these data types, compression may increase transfer speeds by three times or more, on top of MTP's throughput gains.

ExpeDat uses the same compression algorithm as Zip and Gzip.  Data which compresses well using those utilities will also compress well with ExpeDat.

Compression is very CPU intensive.  Data files which are already compressed, encoded, or encrypted, will not benefit from compression.  If the server or client is CPU limited, enabling compression may severely reduce performance.  If you are on a very fast network, even compressible data may move faster without compression due to the CPU overhead of compressing it.  See Tech Note 0014 for tips on deciding when to enable compression.

The best way to tell whether a given file type will benefit from compression when transferring to a particular server is to experiment with the setting both on and off to see which is faster.

Compression also uses extra memory, as network data must be buffered for the compression engine.  If your network is very fast and your latency is very high, then you may need to increase the size of the compression buffer.  The buffer should be at least twice the bandwidth-delay-product of your network path.  (Multiply the path bandwidth in bits-per second by the latency in milliseconds, then divide by 4096000.  The result is the number of kilobytes recommended.)  The default value of 16 is adequate up to a gigabit network with 75ms latency, or a 100 megabit network with 750ms latency.

The StreamSize option only affects buffers used by the movedat client.  If you find it necessary to increase this value, you should increase the server StreamSize as well.

Older servers and servers where compression is disabled will return an error if you attempt a compressed transaction.  Inline compression cannot be used with Streaming Folders or Object Handlers.

Since compression changes the amount and timing of data being sent, progress reports may be inaccurate while using compression.