Performance
The MTP/IP networking technology that transports data for ExpeDat is designed to adapt to a wide variety of network circumstances. The default settings will allow you to use the full available resources of your network while still being fair to other traffic.
If you are not getting the performance you expect, read this section carefully or contact technical support.
In addition to the information contained in this manual, detailed technical information about optimizing network performance can be found in the DEI Tech Notes. An index of the Tech Notes most relevant to performance can be found at the bottom of this page.
The sections below summarize the network and data storage factors most likely to affect ExpeDat, and describe techniques for increasing or decreasing ExpeDat's speed.
Network Limitations
Most Wide Area Network paths involve hundreds of different components, any one of which may be limiting your available bandwidth. Here are a few of the more common types to consider:
- Slowest physical link in the path
- Storage I/O speed
- Network emulators (See Tech Note 0022.)
- Routers configured to meet Quality of Service goals
- Firewalls, especially those with Stateful Packet Inspection or DDoS protection
- Anti-virus and network monitoring software
- VPNs and network routing software
- CPU speed and cores
- Other applications using CPU or storage
- Bonded or multiplexed network links
- Faulty, non-standard, or obsolete routers and switches
- UDP buffers on Linux, FreeBSD, and Solaris (See Tech Note 0024.)
- MTU below 1500 bytes. (See Small MTU.)
ExpeDat cannot move data faster than the slowest of these capacities. Rate-limiting devices, especially firewalls, emulators, and routers, may not be treating all traffic equally! So if performance is not meeting your expectations, check your systems for the types of limitations listed above. DEI Tech Note 0003 offers detailed advice on testing and finding network limitations.
Storage Limitations
Filesystem bottlenecks can severely limit performance. This is particularly true for network attached storage. Below are some tips for optimizing storage performance:
- Use direct attached storage whenever possible.
- If you must use network attached storage (NAS), see Tech Note 0029 for performance tuning tips.
- RAIDs are not always faster. See Tech Note 0018.
- Linux systems may suffer from excessive caching. See Tech Note 0035 for adjusting cache sizes.
- Hard-disk drive performance drops exponentially with the number of users:
Two parallel high-speed transfers will add up to less than one! - For Windows, NTFS formatted filesystems may provide much better write performance than FAT.
- Hard-disk drive performance drops when the volume is nearly full, make sure you have plenty of free space.
- When using NFS filesystems, make sure you have an adequate number of biod and nfsd processes on the NFS clients and servers. To avoid potentially critical NFS bugs, use version 4 or later with up-to-date patches.
- For the most consistent performance with EC2 Elastic Block Storage (EBS), use an EBS-Optimized volume with the desired bitrate. See Tech Note 0025 for more about installing on EC2.
- Inline decompression can create a much higher load on the filesystem than it does on the network. See Tech Note 0014 for more about compression performance.
- Environmental factors, such as vibration in rack-mounted systems, can reduce hard-disk performance far below rated specifications. Be sure your mount points are well isolated and check all cables and connections.
- File transfers include the time it takes the OS to commit data to storage. If storage is slower than the network, there may be a long pause at the end of each transfer. See Write Confirmation for details.
Any time the total network throughput may be faster than about 100 megabits per second or you are using any form of network attached storage, you should investigate your filesystem's I/O capabilities. See Tech Note 0023 for further hardware recommendations.
CPU Limitations
Each real CPU core (or 2 vCPUs) can support about one gigabit per second of throughput with content encryption enabled. A 10 gigabit per second data transfer with encryption requires about 10 real CPU cores or 20 virtual CPU cores. This does not include the needs of other software which may be running on the same system.
If other CPU intensive applications are running on the same system, network performance may be reduced. This may happen even if there are idle CPU cores available. Applications which filter or monitor network traffic may substantially reduce network performance. For example, running Windows Performance Monitor may reduce network throughput by 25% or more. For the best performance, avoid running anti-virus, firewall, or monitoring software on the same hardware as ExpeDat.
Operating systems may limit CPU performance under some conditions, such when the device is operating on battery power or when the user is interacting with another application. Many Linux distributions configure CPUs to operate in power conservation mode by default. See Tech Note 0035 for instructions to check and adjust the Linux scaling governor.
Security Software
Some anti-virus software, including Windows Defender and all network monitoring software, can greatly reduce data transfer performance. Symptoms include poor network speeds accompanied by high CPU and filesystem usage, long delays near the end of a download, and slow scan rates when sending folders with large numbers of files. If you are experiencing slow performance, particularly on a Windows system, test whether anti-virus or other security software is the cause by temporarily disabling it to see if performance improves.
The best way to avoid problems with security software is to offload its functions to a purpose-built device such as a firewall or hardware VPN with sufficient capacity to handle the throughput you require. If security software must share the same hardware as ExpeDat, white-list the ExpeDat applications and file transfer directories. If applicable, configure anti-virus software to delay scanning new files until after they are finished writing, disable heuristic analysis, and disable or make an exception for scanning network packets.
Moving Data Even Faster
Normally MTP/IP makes an effort to allow other traffic on the network a fair share of available resources. If you want your ExpeDat data flows to achieve the absolute maximum speed, regardless of other users, you may do so by increasing the network "aggression" of MTP/IP.
Aggression affects relative performance compared to other data flows sharing the same network path, similar to prioritization. If there are no other data flows, or if performance is limited by hardware or policy, then increasing aggression may have little or no effect.
Aggression is represented by a number between -3 and +5 in user interfaces, or 125 to 133 in SDKs and logs, which determines how hard MTP/IP presses the limits of the underlying network hardware. The default value is +2 (130) which tries to strike a balance between speed and fairness. It is very rare to need to set aggression higher than +2.
Elevated aggression may cause reduced performance or loss of connectivity!
Aggression +4 and higher will always create packet loss. Never set an aggression level of 4 or higher without careful testing.
Always test at level 2 before trying higher levels.
Setting aggression to +3, +4, or +5 may improve ExpeDat throughput, but it may also be disruptive to other users of the network. Such high aggression levels can sometimes result in worse performance if firewalls or other bandwidth management devices are present.
Carefully read and understand the following paragraphs before using aggression +5 or MinDatagram.
MTP will automatically attempt to utilize Jumbo frames (MTU 9000) whenever a transfer exceeds about 1.1 gigabits per second. If every device along the path supports Jumbo frames, the MTP datagram payload size will switch to 8192 bytes for as long as that transaction speed remains high.
If you are certain that every device in your network supports Jumbo frames (MTU 9000) or Super Jumbo frames (MTU up to 65535), then you may be able to reduce I/O overhead and further improve speed by manually advising MTP/IP to use larger datagrams. This can be accomplished by setting MinDatagram to 8192 for Jumbo frame networks or up to 61440 for Super Jumbo. Setting aggression to level 5 will also attempt to use larger datagrams, but setting MinDatagram is preferred if you are sure the network is capable. If any component of the path does not fully support large datagrams, their use may cause severe performance degradation or loss of connectivity.
Refer to each applications's documentation for instructions on setting aggression and Tech Note 0005 for more information about MTP/IP bandwidth management features.
Moving Data Slower
For shared network environments, you may wish to limit ExpeDat's impact on other users even more than the default settings.
The most flexible way to give priority to other users is to reduce the aggression level described above. Values below 2 will limit the resources MTP/IP is willing to use, leaving more resources for other traffic flows. Note that on networks with inherently high-latency or packet loss (such as satellite or packet radio), lower aggression may reduce performance even when there is no other traffic.
Aggression affects relative performance compared to other data flows sharing the same network path. If there are no other data flows and the network path is "clean", decreasing aggression may not have an obvious effect on performance. But it will still cause performance to slow down in the presence of other data flows.
You can also set specific bandwidth limits for ExpeDat. The MaxRate settings can be used to establish speed limits for both individual transactions and the server as a whole. If your data path has a fixed known capacity, you can set MaxRate to a value less than that to apportion the bandwidth between ExpeDat and other users. You can even combine this with increased aggression to throttle ExpeDat to exactly that amount of bandwidth.
If you have applications on your network which are sensitive to latency issues, such as voice or video over IP, you can set a specific latency ceiling for ExpeDat. The MaxRTT setting will cause MTP/IP to slow down whenever it observes latency above the given value.
In order to avoid stalls and other inefficiencies, MTP will always maintain at least one datagram on the network at any time. Thus, transactions cannot be throttled below 256 bytes per Round-Trip-Time. For example, a path with an RTT of 1ms has a minimum rate limit of about 2 megabits per second, while a path with 10ms latency has a minimum of about 200 kilobits per second. This affects both MaxRate and MaxRTT, primarily on LANs.
Refer to the documentation for servedat, ExpeDat Desktop, and movedat for details on how to set these values. Also see DEI Tech Note 0005 for more information about MTP/IP bandwidth management features.
Client, Server, User, and Network Settings
Performance can be regulated at either the client or server. Server-side settings can be targeted to specific users or client networks. Most server-side performance settings have a default value and a limit value. The default is used when the client does not specify a value, but the client may not go past the limit value.
Many server performance settings can be set according by the source IP address of the client using the SiteOptions configuration variable. This allows administrators to regulate performance based on the client network.
Many server performance settings can be set according to the authenticated username by specifying the appropriate options in the AuthFile or AuthHandler.
The server MaxRate settings are applied to individual network transactions. The MaxRateTotal settings are enforced against all transactions in aggregate.
The ExpeDat Desktop client also allows administrators embed settings within a copy of the application. This allows different performance profiles to be distributed to specific persons or systems. Each embedded settings can be locked to prevent the user from modifying it.
See Also
The following Tech Notes provide further guidance on performance tuning:
0023 | Recommended minimum hardware for high speed networking. |
0005 | Features for regulating MTP and ExpeDat bandwidth usage. |
0029 | Optimizing Network Attached Storage (NAS). |
0003 | Important tips on conducting accurate and meaningful performance tests. |
0033 | Instructions for capturing network diagnostics. |
0021 | Guidance for accurately measuring network Loss, Latency, and Speed. |
0022 | See this note if you are considering use of a network emulator for testing. |
0009 | Details about specific devices and factors which may degrade network performance. |
0024 | Operating system tuning for certain unix variants. |
0018 | Configuration requirements for RAIDs. |
0014 | Pros and cons of using inline compression. |
0025 | Using ExpeDat with Amazon Web Services EC2 instances. |
0035 | Linux Performance Tuning. |
0036 | MTP Performance Statistics. |