Inline Compression

To facilitate inline compression of files, servedat includes a built-in "zli" object handler.  Compatible ExpeDat clients can request ZLIB compression of file data while it is in transit.  SyncDat clients do not currently support compression.

Inline compression is not available with ExpeDat Streaming Folders or other Object Handlers.

When inline compression is enabled, data is automatically compressed and decompressed as part of the transfer, so that no extra steps or third-party software are required.

Compression is very CPU intensive.  Data files which are already compressed, encoded, or encrypted, will not benefit from compression.  If the server or client is CPU limited, enabling compression may severely reduce performance.

Compression should only be used on files known to contain compressible data, such as text documents.  Audio, video, pictures, encrypted data, and already compressed archives may not benefit from compression.  If you are on a very fast network, even compressible data may move faster without compression due to the CPU overhead of compressing it.  See Tech Note 0014 for tips on deciding when to enable compression.

Compression also uses extra memory, as network data must be buffered for the compression engine.  If your network is very fast or your latency is very high, then you may need to increase the size of the compression buffer.  The buffer should be at least twice the bandwidth-delay-product of your network path.  (Multiply the path bandwidth in bits-per second by the latency in milliseconds, then divide by 4194304000.  The result is the number of megabytes recommended.)  The default value of 32 megabytes is adequate up to a gigabit network with 150ms latency, or a 100 megabit network with 1500ms latency.

The StreamSize setting only affects servedat's own buffers.  If you find it necessary to increase this value, you should increase the client buffers as well.

If you are concerned about the CPU or memory load that may be created by users requesting compression of their files, you can disable servedat compression support using the NoCompression configuration variable or the -Z command line option.

Since compression changes the amount and timing of data being sent, progress reports may be inaccurate while using compression.  Transfers which are interrupted while using compression may need to start over from the beginning, or they may need to resume without compression enabled.  See the documentation of each client for details.