Known Issues
The following issues and limitations are known to exist in this version of SyncDat or the systems which it supports. Also refer to the Limitations section. For further information about these issues, please contact DEI Technical Support.
Network Attached Storage
Transferring files which are stored on NAS devices may be severely limited by the speed of the legacy protocols used by those devices. If you must use NAS, see Tech Note 0029 for guidance on tuning NAS devices for high performance. See the Performance chapter for general guidance on storage performance.
Network attached storage mounts may enforce their own authentication and access controls which may restrict access by the SyncDat server. See Tech Note 0031 for details.
Mixed Storage Performance
When writing files to multiple storage devices, poor performance by one storage device may degrade performance for other devices. In extreme cases, a hung storage device may hang the entire host operating system. This is especially noticeable when writing to network attached storage systems of differing capabilities. For best performance, ensure that all storage devices are capable of consistently reading and writing data faster than the network. See Tech Note 0029 for guidance on tuning NAS devices for high performance.
Be wary of allowing users to access arbitrary network mounts (such as with UNC paths). To control which devices users can access, set RestrictHome and use AllowPath to limit access to approved devices. In some cases, it may be desireable to serve different storage devices from separate ExpeDat hosts.
Maximum File Sizes
Individual files transferred by SyncDat must be less than 8 exabytes each, or the maximum file size supported by the host operating system. The total size of all files transfered per client session must be less than 16 exabytes.
Memory Limitations
The maximum number of files which can be synchronized in a single syncdat session is limited by available client system memory (RAM). Two gigabytes of free RAM is typically enough for about one million files.
Special Characters
Certain characters should not be used in file and folder names because they have special meaning on some platforms. When transferring files between different platforms is is possible for a file name which is valid in one context to cause problems in another. Avoid the following characters in file names:
/ \ : * ? " ' ` ; = ( ) [ ] & | ! ~ { } @ $
For example, if you try to upload a file named "recipes:recent" to a Windows NTFS filesystem, the file or its contents may seem to disappear and servedat may log an error because Windows reserves the ':' character for Alternate Data Streams.
Maximum Path Length
SyncDat limits the total length of a file's pathname to 1024 bytes, including UTF-8 encoding. Some operating systems, particularly Windows, may impose much shorter limits on pathnames for some operations. This varies greatly depending on the operating system version, but it is generally safest to keep Windows paths below 256 characters. Network attached storage devices (NAS/SAN) may also impose shorter limits.
Some Windows systems allow you to exceed the 256 character path limit by prefixing absolute paths with "\\?\" for direct-attached paths or "\\?\UNC\" for UNC paths. Remember to use forward slashes when specifying remote paths. For example:
syncdat -l /home/me/folder user@server.example.com://?/C:/Users/Administrator/folder syncdat -l /home/me/folder user@server.example.com://?/UNC/nas_server/folder/ syncdat -l C:\MyFolder\myfile.txt user@server.example.com://?/UNC/nas_server/folder/
Using "\\?\" syntax for local paths may conflict with wildcard parsing in some Windows environments. See Windows Command Syntax for details.
As of Windows 10, version 1607, MAX_PATH limitations can be removed by editing the Windows registry or group policy. ExpeDat and SyncDat applications have adopted Microsoft's mechanism for enabling this on a per-application basis, but as of this writing only the system-wide Windows setting is effective. When the new long path behavior is enabled, pathnames up to 1024 UTF-8 bytes are supported.
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled 1
For more details about long paths in Windows, see:
https://docs.microsoft.com/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation
Windows Command Syntax
When syncdat is used from a Windows command shell, the following limitations apply.
Arguments, including paths, which contain spaces must be enclosed in double quotes. For example:
syncdat -l "C:\folder" "user@server.example.com://?/C:/Users/My Name/folder"
The character sequence \" is ambiguous in Windows. In most cases, backslash is a path delimiter. But when it appears immediately before a quote, it is treated as an escape, nullifying the effect of the quote. This can have the effect of seeming to merge two arguments into one. For example:
syncdat -l "C:\My Dir\" "user@server.example.com://?/C:/Users/My Name/My Folder" Local Application: Invalid Argument: Folder transfer requires source and destination directories
To work around this limitation, remove the trailing backslash:
syncdat -l "C:\My Dir" "user@server.example.com://?/C:/Users/My Name/My Folder"
Unicode / Non-ASCII Characters
On all platforms, non-ASCII filename characters are encoded as UTF-8 and will be preserved between platforms.
The ability of SyncDat software to display non-ASCII characters will depend upon the fonts and settings of the host system. This is particularly true for the Windows console. See the syncdat introduction for steps to improve the display of non-ASCII characters with the Windows console.
In some cases, the Windows version of SyncDat may not correctly display non-ASCII characters even when unicode fonts are available. However, SyncDat will correctly preserve unicode file name characters even when they are not displayed correctly.
Some systems allow file names to be arbitrary binary strings, which may not be valid unicode. SyncDat will attempt to preserve such file names, but they are likely to display incorrectly and may cause errors.
If either the source or destination filesystem is a network mount, support for unicode file name characters will depend on the type and configuration of the network mount.
Usernames and passwords should consist of only ASCII letters, numbers, and printable symbols. The use of other characters, such as extended unicode characters, may work in some environments but is not assured and may compromise security.
Care must be taken to ensure that any shell scripts using syncdat are able to properly handle non-ASCII characters, which may include raw-binary strings.
Windows Performance Limits
Windows has a fixed limit on the amount of memory available for buffering file and network operations. When the total amount of I/O occurring in the system exceeds this level, operations begin to fail and Windows may become unstable and require a reboot. SyncDat may report this as:
Insufficient system resources exist to complete the requested service
However, I/O memory is shared by all processes, so its lack may cause errors in any application, process, or device driver. Errors may be reported in the Windows event log, by other applications you are not using, or they may not be reported at all.
The problem may be triggered by file transfers totaling more than a hundred megabits per second, by multiple applications performing high volume I/O operations, or by the presence of third-party anti-virus, firewall, or virtualization software. The use of Virtual Machines, Network Attached Storage, Storage Area Networks, or third-party storage drivers increases I/O buffering and may make the problem worse.
Once Windows' paged pool memory has been exhausted, it is necessary to reboot Windows to prevent application crashes and other system-wide problems. The following Microsoft article explains how to improve paged pool performance, which may allow faster throughput:
If it is not possible to adjust the paged pool, or if doing so does not solve the problem, it may be necessary to disable other software, limit SyncDat's speed, or reconfigure a virtualized environment.
Windows versions prior to Windows 8 and Windows Server 2012 were particularly vulnerable to this problem. This problem has not been reported with Windows 8, Windows Server 2012, or later.
Small MTU
Network links with a Maximum Transmit Unit (MTU) smaller than the common 1500 byte minimum may cause datagrams to be fragmented (broken into multiple smaller packets) which will magnify datagram loss and may trigger firewall blocks. Fragmentation can also be caused by applying an excessive number of tunneling layers, each of which adds to the size of datagrams.
MTP disables fragmentation of its own datagrams and automatically adjusts their size to fit the network. However, VPNs and other tunnels may override this and, if their MTU does not match the network, may cause fragmentation anyway. If you are experiencing high loss rates, low throughput, or connectivity problems with a VPN or other tunnel, check that its MTU and that of other devices in the path are correctly configured.
You can test whether MTU is a problem by manually lowering MTP's maximum datagram size. Set the MaxDatagram option of servedat or syncdat to 512 bytes and then test whether performance improves significantly. If it does, you can experiment to find the value which achieves the best performance and investigate all network devices to see if any have an abnormally low MTU or if you are using too many tunnel layers.
Resuming an Upload with Compression
When resuming an interrupted upload, compression cannot be used. To resume an upload, either compression must be disabled or the upload must be restarted. This will be done automatically if the client is non-interactive.
Drag and Drop
The ExpeDat Desktop GUI does not currently support dragging remote files to a local destination. Instead, users can drag a local folder into the local browser pane to select that folder as the destination.
Dragging local files to the ExpeDat Desktop GUI will select those files in the local file browser. If the dragged files are contained in multiple folders, only the those in one of the folders will be displayed.
Listing Performance
Listing a very large directory, either locally or remotely, may temporarily reduce the performance of any file transfers occurring at the same time.
Resuming Partial Downloads
If a download is interrupted, it will be restarted from the beginning upon retry. Interrupted uploads may be resumed from the point of interruption.
File Access Conflicts
The SyncDat server takes steps to ensure that files being uploaded are not written to or read from by other transactions. However, the mechanisms for this protection are highly dependent upon the operating environment and are not perfect. In particular, files on network storage devices may be susceptible to modification by other programs while in use. Users should take care not move or modify files which may be actively transferring.
Linux SMB Upload Hangs
Linux kernel 5.5 introduced breaking changes to the behavior of CIFS/SMB volumes which may result in servedat hanging at the start of uploads. The preferred solution is to use the mount option "nobrl" to disable these changes. Alternatively, you may enable the NoFileLocks option in servedat.cf to disable file locks entirely.
Linux SMB Permission Errors
Authenticated SMB mounts from a Linux system may trigger permission errors at the end of uploads due to conflicts between Linux and SMB authentication models. If uploads to a Linux mounted SMB volume are failing at the end with Permission Denied errors, setting "noperm" in the volume mount options may resolve the conflict.
System Clock Accuracy
Severe problems may occur if the system clock of a host running SyncDat software is not accurate or changes substantially. See the "System Clocks" section for details.
Operating System Under-Reports Progress
During file transfer, some operating environments may report the size of the arriving file as being much smaller than it actually is. This is most notable with Windows Server running in an EC2 instance, but may occur on other systems as well. After the transfer has stopped, the operating system should show the correct size of the file.
Windows Symbolic Links
Symbolic Links under Windows are not replicated. Instead, their targets will be replicated.
In a unix shell environment, the syntax "path1/symlink/../path2/" would access the directory "path2" in the parent directory of the symlink target. Using such a path via servedat will instead access the directory "path2" in the parent directory of the symlink itself. In other words, the path would be resolve to "path1/path2/".
Irregular Files
Irregular or "special" files cannot be synchronized. If an irregular file (device, socket, pipe, etc.) already exists at a destination path, then the synchronization may fail.
Checkpoint Files
The server stores partial uploads using "-sv.tmp" and "-sv.met" files. See the Upload Status chapter for details.
"Lost" Transactions
If a transaction fails while the server is under heavy load and the network path is experiencing packet loss, a client may receive a "Lost Transaction" error instead of the exact error message. The exact error message will appear in the server logs.
Server Restarts
Restarting the server via SIGHUP, which is necessary to reload the configuration file, will interrupt any active transactions. Interrupted transactions will automatically retry if that option has not been disabled.
Slow Devices (CD, DVD, tape, NAS over WAN, etc.)
Targeting files on very slow filesystems or devices such as optical disks, tape drives, network attached storage across a WAN, or any other filesystem with extreme latency may result in severely reduced performance or loss of connectivity. Targeting slow filesystems on the server may affect the performance and connectivity of all clients accessing that server, even those targeting other filesystems.
Write Confirmation
SyncDat does not consider a file transfer complete until the operating system confirms that all data has been committed to storage. File transfers may seem to pause for long periods of time at the end if the storage system is slower than the network and the operating system has a large file cache. The client may report "Waiting for storage" or "Waiting for Server". When SyncDat reports a final transfer speed, it includes the time it takes to write data to storage. This may cause SyncDat to seem slower than products which claim to be done before they have finished writing your data.
See Tech Note 0023 for general guidance on storage performance. Linux may buffer particularly large amounts of file write data, which can result in extreme wait times and all around poor write performance. See Tech Note 0035 for instructions to adjust Linux write buffer sizes.
Hardware & Power Failures
Resuming a synchronization which was interrupted by a hardware or power failure is not recommended. SyncDat cannot ensure the integrity of storage which has been physically disrupted. After repairing disrupted hardware, all filesystems and files should be checked for integrity and files which were being written at the time of the disruption should be re-transferred or removed manually.
Username Case Sensitivity
Some authentication databases, notably Active Directory, may treat usernames which differ only by case as the same user. For example "Joe" and "joe" may match. Most other databases, including SyncDat's private authentication and system caching, are case sensitive. To avoid problems and inconsistencies, users should always use the correct capitalization when typing their username.
Filename Case Sensitivity
Some file systems, notably Windows and macOS native filesystems, may treat filenames which differ only by case as the same file. For example "Data" and "data" may match. Other filesystems, including UFS and EXT, are case sensitive. Network Attached Storage protocols may introduce additional complications. To avoid problems and possible data loss, always use correct capitalization when transferring files and never permit two files whose names differ only by case.
WinSock Reset
If any Windows application improperly accesses the WinSock network layer, it can disrupt networking for all applications, including SyncDat. The following error messages indicate that a WinSock reset has occurred:
NetIn Unable to receive from network NetOut Unable to send to network
If you see these errors, reboot Windows and check the system event logs for warnings about WinSock. Disable unneeded network services. For example, the "WinHTTP Web Proxy Audio-Discovery Service" has been known to cause this problem.
macOS SMB2 File Corruption
macOS may corrupt file data when writing to SMB2 network attached storage. SMB2 was the default protocol for macOS 10.9 when mounting a Windows file server in the Finder or when using an "smb://" URL. To avoid this issue under 10.9, you must mount Windows file servers using the "cifs://" URL, which forces the use of the SMB1 protocol, or use a different NAS protocol such as NFS or AFP. You can see which SMB versions your mac is using by typing the following command in a terminal:
smbutil statshares -a
SyncDat no longer supports macOS 10.9, and later macOS versions use SMB3 when it is available. To prevent macOS from accidentally using the SMB2 protocol, and avoid an unrelated SMB3 signing bug, create or edit the file /etc/nsmb.conf and place the following text at the end:
[default] client_signing=no smb_neg=smb3_only