Known Issues
The following issues and limitations are known to exist in this version of ExpeDat or the systems which it supports. For further information about these issues, please contact DEI Technical Support.
Network Attached Storage
Transferring files which are stored on NAS devices may be severely limited by the speed of the legacy protocols used by those devices. If you must use NAS, see Tech Note 0029 for guidance on tuning NAS devices for high performance. See the Performance chapter for general guidance on storage performance.
Network attached storage mounts may enforce their own authentication and access controls which may restrict access by the ExpeDat server. See Tech Note 0031 for details.
Mixed Storage Performance
When writing files to multiple storage devices, poor performance by one storage device may degrade performance for other devices. In extreme cases, a hung storage device may hang the entire host operating system. This is especially noticeable when writing to network attached storage systems of differing capabilities. For best performance, ensure that all storage devices are capable of consistently reading and writing data faster than the network. See Tech Note 0029 for guidance on tuning NAS devices for high performance.
Be wary of allowing users to access arbitrary network mounts (such as with UNC paths). To control which devices users can access, set RestrictHome and use AllowPath to limit access to approved devices. In some cases, it may be desireable to serve different storage devices from separate ExpeDat hosts.
macOS 13 Drag and Drop Bug
macOS 13 (Ventura) has a bug which prevents .app and .exe entities from being dragged onto any application, including DropDat, droplets, and ExpeDat Desktop. As a workaround, place the DropDat master, droplet, or ExpeDat Desktop in your Dock and drag droplets or files onto to that Dock icon for editing or sending. You may also drag a droplet or ExpeDat Desktop onto a DropDat Create or Edit window to load and edit the dropped item.
Maximum File Sizes
Individual files transferred by ExpeDat must be less than 8 exabytes each, or the maximum file size supported by the host operating system. The total size of all files transfered per client session must be less than 16 exabytes.
Memory Limitations
The maximum number of individual files which can be transferred one-at-a-time (as opposed to Streaming) is limited by available client system memory (RAM). Two gigabytes of free RAM is typically enough for about one million files with ExpeDat Desktop or DropDat, and about eight million with movedat. Streaming Folders uses a fixed buffer size, typically 64 megabytes, to process up to around 2 billion files.
Special Characters
Certain characters should not be used in file and folder names because they have special meaning on some platforms. When transferring files between different platforms is is possible for a file name which is valid in one context to cause problems in another. Avoid the following characters in file names:
/ \ : * ? " ' ` ; = ( ) [ ] & | ! ~ { } @ $
For example, if you try to upload a file named "recipes:recent" to a Windows NTFS filesystem, the file or its contents may seem to disappear and servedat may log an error because Windows reserves the ':' character for Alternate Data Streams.
Maximum Path Length
ExpeDat limits the total length of a file's pathname to 1024 bytes, including UTF-8 encoding. Some operating systems, particularly Windows, may impose much shorter limits on pathnames for some operations. This varies greatly depending on the operating system version, but it is generally safest to keep Windows paths below 256 characters. Network attached storage devices (NAS/SAN) may also impose shorter limits.
Some Windows systems allow you to exceed the 256 character path limit by prefixing absolute paths with "\\?\" for direct-attached paths or "\\?\UNC\" for UNC paths. Remember to use forward slashes when specifying remote paths in an ExpeDat client. Below are some examples using movedat:
movedat user@server.example.com://?/C:/Users/Administrator/ movedat user@server.example.com://?/UNC/nas_server/folder/ movedat C:\MyFolder\myfile.txt user@server.example.com://?/UNC/nas_server/folder/
Using "\\?\" syntax for local paths may conflict with wildcard parsing in some Windows environments. See Windows Command Syntax for details.
As of Windows 10, version 1607, MAX_PATH limitations can be removed by editing the Windows registry or group policy. ExpeDat and SyncDat applications have adopted Microsoft's mechanism for enabling this on a per-application basis, but as of this writing only the system-wide Windows setting is effective. When the new long path behavior is enabled, pathnames up to 1024 UTF-8 bytes are supported.
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled 1
For more details about long paths in Windows, see:
https://docs.microsoft.com/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation
Windows Command Syntax
When movedat is used from a Windows command shell, the following limitations apply.
Arguments, including paths, which contain spaces must be enclosed in double quotes. For example:
movedat "user@server.example.com://?/C:/Users/My Name/file.txt" "C:\tmp"
The character sequence \" is ambiguous in Windows. In most cases, backslash is a path delimiter. But when it appears immediately before a quote, it is treated as an escape, nullifying the effect of the quote. This can have the effect of seeming to merge two arguments into one. For example:
movedat -h "C:\My Dir\" "user@server.example.com://?/C:/Users/My Name/My Folder" Local Application: Invalid Argument: Folder transfer requires source and destination directories
To work around this limitation, remove the trailing backslash:
movedat -h "C:\My Dir" "user@server.example.com://?/C:/Users/My Name/My Folder"
In some Windows shell environments, paths starting with "\\?\" may be parsed incorrectly due to Windows globbing rules.
Unicode / Non-ASCII Characters
On all platforms, non-ASCII filename characters are encoded as UTF-8 and will be preserved between platforms.
The ability of ExpeDat software to display non-ASCII characters will depend upon the fonts and settings of the host system. This is particularly true for the Windows console. See the movedat introduction for steps to improve the display of non-ASCII characters with the Windows console.
In some cases, the Windows version of ExpeDat may not correctly display non-ASCII characters even when unicode fonts are available. However, ExpeDat will correctly preserve unicode file name characters even when they are not displayed correctly.
Some systems allow file names to be arbitrary binary strings, which may not be valid unicode. ExpeDat will attempt to preserve such file names, but they are likely to display incorrectly and may cause errors.
If either the source or destination filesystem is a network mount, support for unicode file name characters will depend on the type and configuration of the network mount.
Usernames and passwords should consist of only ASCII letters, numbers, and printable symbols. The use of other characters, such as extended unicode characters, may work in some environments but is not assured and may compromise security.
Care must be taken to ensure that any shell scripts using movedat or Object Handler scripts are able to properly handle non-ASCII characters, which may include raw-binary strings.
Windows Performance Limits
Windows has a fixed limit on the amount of memory available for buffering file and network operations. When the total amount of I/O occurring in the system exceeds this level, operations begin to fail and Windows may become unstable and require a reboot. ExpeDat may report this as:
Insufficient system resources exist to complete the requested service
However, I/O memory is shared by all processes, so its lack may cause errors in any application, process, or device driver. Errors may be reported in the Windows event log, by other applications you are not using, or they may not be reported at all.
The problem may be triggered by file transfers totaling more than a hundred megabits per second, by multiple applications performing high volume I/O operations, or by the presence of third-party anti-virus, firewall, or virtualization software. The use of Virtual Machines, Network Attached Storage, Storage Area Networks, or third-party storage drivers increases I/O buffering and may make the problem worse.
Once Windows' paged pool memory has been exhausted, it is necessary to reboot Windows to prevent application crashes and other system-wide problems. The following Microsoft article explains how to improve paged pool performance, which may allow faster throughput:
If it is not possible to adjust the paged pool, or if doing so does not solve the problem, it may be necessary to disable other software, limit ExpeDat's speed, or reconfigure a virtualized environment.
Windows versions prior to Windows 8 and Windows Server 2012 were particularly vulnerable to this problem. This problem has not been reported with Windows 8, Windows Server 2012, or later.
Small MTU
Network links with a Maximum Transmit Unit (MTU) smaller than the common 1500 byte minimum may cause datagrams to be fragmented (broken into multiple smaller packets) which will magnify datagram loss and may trigger firewall blocks. Fragmentation can also be caused by applying an excessive number of tunneling layers, each of which adds to the size of datagrams.
MTP disables fragmentation of its own datagrams and automatically adjusts their size to fit the network. However, VPNs and other tunnels may override this and, if their MTU does not match the network, may cause fragmentation anyway. If you are experiencing high loss rates, low throughput, or connectivity problems with a VPN or other tunnel, check that its MTU and that of other devices in the path are correctly configured.
You can test whether MTU is a problem by manually lowering MTP's maximum datagram size. Set the MaxDatagram option of servedat, ExpeDat Desktop, or movedat to 512 bytes and then test whether performance improves significantly. If it does, you can experiment to find the value which achieves the best performance and investigate all network devices to see if any have an abnormally low MTU or if you are using too many tunnel layers.
Resuming an Upload with Compression
When resuming an interrupted upload, compression cannot be used. To resume an upload, either compression must be disabled or the upload must be restarted. This will be done automatically if the client is non-interactive.
Drag and Drop
The ExpeDat Desktop GUI does not currently support dragging remote files to a local destination. Instead, users can drag a local folder into the local browser pane to select that folder as the destination.
Dragging local files to the ExpeDat Desktop GUI will select those files in the local file browser. If the dragged files are contained in multiple folders, only the those in one of the folders will be displayed.
Listing Performance
Listing a very large directory, either locally or remotely, may temporarily reduce the performance of any file transfers occurring at the same time.
File Access Conflicts
The ExpeDat server takes steps to ensure that files being uploaded are not written to or read from by other transactions. However, the mechanisms for this protection are highly dependent upon the operating environment and are not perfect. In particular, files on network storage devices may be susceptible to modification by other programs while in use. Users should take care not move or modify files which may be actively transferring.
Linux SMB Upload Hangs
Linux kernel 5.5 introduced breaking changes to the behavior of CIFS/SMB volumes which may result in servedat hanging at the start of uploads. The preferred solution is to use the mount option "nobrl" to disable these changes. Alternatively, you may enable the NoFileLocks option in servedat.cf to disable file locks entirely.
Linux SMB Permission Errors
Authenticated SMB mounts from a Linux system may trigger permission errors at the end of uploads due to conflicts between Linux and SMB authentication models. If uploads to a Linux mounted SMB volume are failing at the end with Permission Denied errors, setting "noperm" in the volume mount options may resolve the conflict.
Compression Performance
Compression can negatively affect performance, particularly if the data being transferred is encrypted or already compressed. Compression should only be used if the data being transferred is known to be compressible. See Tech Note 0014 for more details about compression.
Compression Statistics
Information about the progress and speed of a file transfer may not be accurate when compression is being used. The server will log the amount of data actually transferred across the network (after compression). If a compressed transfer is interrupted, the amount received may not match the amount sent due to buffering.
Streaming Folders Statistics
Information about the progress and speed of a file transfer may not be accurate when Streaming Folders is being used. The current progress may be displayed, but the total size of the folder is not known until it has been completely transmitted. If a Streaming Folders transfer is interrupted, the files actually received may not match the files reported sent due to buffering.
Resume Statistics
After resuming an interrupted transfer, clients may report amounts of data or speeds which include data transferred in a previous session. The server logs only count the data moved during the current session.
System Clock Accuracy
Various problems may occur if the system clock of a host running ExpeDat software is not accurate or changes substantially. Examples include interrupted transfers, failure to resume interrupted transfers, incorrect file time stamps, and poor performance. All ExpeDat hosts, especially servers, should enable time synchronization software and ensure that the system clock remains accurate. You can test whether the clocks of a client and server are synchronized by using mtping to query the server from the client and observing the "Clock Skew". A Clock Skew of more than a few of seconds should be corrected.
Operating System Under-Reports Progress
During file transfer, some operating environments may report the size of the arriving file as being much smaller than it actually is. This is most notable with Windows Server running in an EC2 instance, but may occur on other systems as well. After the transfer has stopped, the operating system should show the correct size of the file.
Symbolic Links
Accessing files and folders via symbolic links may produce unexpected results. In order to access the target of a symbolic link, the permissions of the link's parent folder, the link itself, the target's parent folder, and the target itself all must permit access.
Attempting to download a link which points to a folder may result in an error or a zero length file. To list a folder pointed to by a symbolic link in movedat, place an explicit "/" character after the link name. In ExpeDat Desktop, symbolic links which point to a folder may be displayed with a folder icon and a slash after the name. You may double click on these to browser the target folder. In some cases, such as with older ExpeDat servers, symbolic links to folders may look like regular files. To list a folder pointed to by an untyped symbolic link in ExpeDat Desktop, enter its path in the Remote Path field and click the remote 
refresh button.
In a unix shell environment, the syntax "path1/symlink/../path2/" would access the folder "path2" in the parent folder of the symlink target. Using such a path via servedat will instead access the folder "path2" in the parent folder of the symlink itself. In other words, the path would be resolve to "path1/path2/".
With the macOS version of ExpeDat Desktop, browsing into a local folder pointed to by a symbolic link may change the Local Path to that of the target. In that case, clicking 
may take you to the parent of the target, not the folder you were just in.
Irregular Files
Irregular or "special" files cannot be targeted as the destination of a Send. If an irregular file (device, socket, pipe, etc.) already exists at the destination path, then the upload will fail with an "Object Unavailable" error. See the movedat and ExpeDat Desktop documentation for details.
Irregular or "special" files such as devices, sockets, pipes, and Windows symbolic links, cannot be copied. If these files are included in a directory or folder being transferred, they may be ignored or they may cause the transfer to fail.
Checkpoint Files
Different formats are used to store information about partial transfers.
The server stores partial uploads using "-sv.tmp" and "-sv.met" files. See the Upload Status chapter for details.
The Windows clients store partial downloads using ".TMP" and ".CHK" files. All other clients store partial downloads using a suffix of the form "#00000000.TMP". If the NoCheck option is enabled, the movedat client will download directly to a target file.
"Lost" Transactions
If a transaction fails while the server is under heavy load and the network path is experiencing packet loss, a client may receive a "Lost Transaction" error instead of the exact error message. The exact error message will appear in the server logs.
Server Restarts
Restarting the server via SIGHUP, which is necessary to reload the configuration file, will interrupt any active transactions. Interrupted transactions will automatically retry for ExpeDat Desktop, DropDat, and for movedat if it has not been disabled.
Mass Deletion
Folders can only be deleted if they are empty. Mass deletion of a folder's contents is not yet supported.
Hierarchical Wild-Cards
Remote directories selected using the movedat wild-card option are not currently subject to hierarchical processing. To transfer the contents of such remote directories, they must be explicitly named on the command line.
ExpeDat Desktop Windows Sorting
In the Windows version of ExpeDat Desktop, the browser sort order reverts to Name-Ascending whenever a new listing is performed.
Slow Devices (CD, DVD, tape, NAS over WAN, etc.)
Targeting files on very slow filesystems or devices such as optical disks, tape drives, network attached storage across a WAN, or any other filesystem with extreme latency may result in severely reduced performance or loss of connectivity. Targeting slow filesystems on the server may affect the performance and connectivity of all clients accessing that server, even those targeting other filesystems.
Write Confirmation
ExpeDat does not consider a file transfer complete until the operating system confirms that all data has been committed to storage. File transfers may seem to pause for long periods of time at the end if the storage system is slower than the network and the operating system has a large file cache. The client may report "Waiting for storage", "Waiting for Server", or a similar message. When ExpeDat reports the final transfer speed, it includes the time it takes to write data to storage. This may cause ExpeDat to seem slower than products which claim to be done before they have finished writing your data.
See Tech Note 0023 for general guidance on storage performance. Linux may buffer particularly large amounts of file write data, which can result in extreme wait times and all around poor write performance. See Tech Note 0035 for instructions to adjust Linux write buffer sizes.
Hardware & Power Failures
Resuming a transfer which was interrupted by a hardware or power failure is not recommended. ExpeDat cannot ensure the integrity of storage which has been physically disrupted. After repairing disrupted hardware, all filesystems and files should be checked for integrity and files which were being written at the time of the disruption should be re-transferred from the beginning.
Username Case Sensitivity
Some authentication databases, notably Active Directory, may treat usernames which differ only by case as the same user. For example "Joe" and "joe" may match. Most other databases, including ExpeDat's private authentication and system caching, are case sensitive. To avoid problems and inconsistencies, users should always use the correct capitalization when typing their username.
Filename Case Sensitivity
Some file systems, notably Windows and macOS native filesystems, may treat filenames which differ only by case as the same file. For example "Data" and "data" may match. Other filesystems, including UFS and EXT, are case sensitive. Network Attached Storage protocols may introduce additional complications. To avoid problems and possible data loss, always use correct capitalization when transferring files and never permit two files whose names differ only by case.
WinSock Reset
If any Windows application improperly accesses the WinSock network layer, it can disrupt networking for all applications, including ExpeDat. The following error messages indicate that a WinSock reset has occurred:
NetIn Unable to receive from network NetOut Unable to send to network
If you see these errors, reboot Windows and check the system event logs for warnings about WinSock. Disable unneeded network services. For example, the "WinHTTP Web Proxy Audio-Discovery Service" has been known to cause this problem.
macOS SMB2 File Corruption
macOS may corrupt file data when writing to SMB2 network attached storage. SMB2 was the default protocol for macOS 10.9 when mounting a Windows file server in the Finder or when using an "smb://" URL. To avoid this issue under 10.9, you must mount Windows file servers using the "cifs://" URL, which forces the use of the SMB1 protocol, or use a different NAS protocol such as NFS or AFP. You can see which SMB versions your mac is using by typing the following command in a terminal:
smbutil statshares -a
ExpeDat no longer supports macOS 10.9, and later macOS versions use SMB3 when it is available. To prevent macOS from accidentally using the SMB2 protocol, and avoid an unrelated SMB3 signing bug, create or edit the file /etc/nsmb.conf and place the following text at the end:
[default] client_signing=no smb_neg=smb3_only