Monitoring
Server status is recorded in the server log and can be remotely monitored using the mtping utility and the ExpeDat Server Dashboard.. This mtping utility can be found in the "Server Files" folder of each ExpeDat and SyncDat distribution package. The Server Dashboard can be found in the "Web" folder of each ExpeDat distribution package.
Using mtping
Pinging servedat will return version, license, and runtime information. The exact information available may vary depending on the version and configuration.
host-14# mtping california.mtpip.net ACT_IDSTR california.mtpip.net:8080 (107.170.213.228:8080) Response 0 of 240 bytes after 0 repeats in 59ms (Version 2/0): servedat - 1.18.0 October 2017 - DEI, DOC-2.3.0 MTP-linux64-4.2.6 2414 00000001: ExpeDat-1.18A Data Expedition Inc., california.mtpip.net Server Statistics (11800): Name: servedat Version: 1.18.0 Platform: linux Clock: Thu Nov 2 08:41:28 2017 Clock Skew: 2 seconds Load: Current: 3, MaxConfig: 3, MaxStart: 3, TotalStart: 140 Capacity: Limit: 10, Exceeded: 0 Up Time: 6.71 weeks Config Time: 2.54 weeks Idle Time: 0 seconds Errors: Errors: 0, Failures: 17, Warnings: 0, Denials: 3 Data: In: 2.77 gigabytes, Out: 1.11 gigabytes Bandwidth: In: 93.6 megabits/sec, Out: 22.4 megabits/sec
See the mtping section for more details on using mtping.
Object Handlers also have access to server configuration and load details through environment variables such as SV_SERVERLOAD.
Monitoring Server Health
The Load line tells you how busy your server has been.
| Current | The number of transactions currently active. |
| MaxConfig | The peak number of simultaneous transactions since the last configuration reload. |
| MaxStart | The peak number of simultaneous transactions since server startup. |
| TotalStart | The total number of transactions started or attempted since server startup. |
The Capacity line tells you the maximum possible Load and how often this has been Exceeded.
| Limit | The maximum possible Load. |
| Exceeded | The number of transactions which were denied because the server load was at maximum capacity. If this number is non-zero, consider upgrading your server's capacity. This count is included among the Deny count below. |
The Errors line counts how many times problems have been observed since the server was started.
| Errors | Serious problems which should be investigated and corrected. Check the server log for lines starting with 'E'. |
| Failures | Transactions which started successfully, then failed before completing. A large number of these compared to TotalStart could indicate network problems. |
| Warnings | Problems with the configuration or unusual run-time events. Check the server log for lines starting with 'W'. |
| Denials | Transactions which could not begin. Incorrect passwords, mistyped file names, and file permission issues are the most common causes. |
The Data line counts the total number of bytes transferred in and out since since the last configuration reload.
Bandwidth measures the average throughput, in and out, over the last 60 second measurement period. Because measurements are made over full 60 second periods, the rate shown may reflect activity up to 59 seconds in the past.
It is a good idea to check these values periodically, either with mtping, by looking for 'S' entries in the log file, or with the ExpeDat Server Dashboard. Increasing problem counts may indicate a persistent problem that requires attention. Pinging server health in this way consumes very few server resources and does not use a capacity slot.
Monitoring Transactions
Users with the AdminReport privilege may retrieve a list of all current transactions and their performance details by running "mtping -T" or by using an ExpeDat client requesting built-in action code "*rt". See the Server Dashboard for a graphical example of this.
To quickly enable transaction monitoring, create an AuthFile record similar to the one below by following the instructions in the Adding Users chapter.
rpadmin:rppass::::AdminOnly,AdminReport
You may then add those credentials into the status.php list of servers, as shown in the Server Dashboard Installation instructions.
Both "mtping -T" and movedat will produce the same output and both will consume a capacity slot while the report is being retrieved. The only difference is in execution options. mtping will always encrypt the report contents, will always send the results to stdout, an requires that the password be given on the command line. movedat may request unencrypted results, can save the output to a file, and may use an authentication cache. The following commands are equivalent:
# mtping -T rpadmin:rppassword@example.com # movedat -D rpadmin:rppassword@example.com=\*rt:
In most cases, it is preferable to use mtping with credentials that are restricted for use only for such reports by setting AdminOnly. Below is an example of output showing three transactions, including the report request itself.
servedat - 1.18.0 August 2017 - DEI, DOC-2.3.0 MTP-linux64-4.2.6 2373 00000001: ExpeDat-1.18A Data Expedition Inc., california.mtpip.net servedat 1.18.0 linux 59DCE59F UTC / rh,nl 3,3,3,56 10,0 17988B 179813 0 0,5,0,10 D871B1F5 D AFC1BAB8,C2BA,E dei A /var/ftp/test100mb.dat 0 59DD1E03,1BAFA58,3C,3489A90,2F76570 6400000,0,5702BA2B 0,0,0 A028,0,0,0,0,0,0 D871B1F7 D B2793C45,BA14,E web A test1000mb.dat AES 0 59DD1E10,39D2BC0,A1,2110290,3C6EFD70 3E800000,0,5702BA38 0,0,0 A028,0,0,0,0,0,0 D871B1F8 A ACC1BAB8,D36B,E - *rt 0 59DD1E14,0,0,0, ,0,0 0,0,0 0,0,0,0,0,0,0
The first three lines show the server identity and health information in the same format as mtping -R.
The remaining lines, if any, show details for each transaction which is active at that time. Lines are line-feed delimited, fields are tab separated, sub-fields are comma separated, and all text is formatted in UTF-8.
| Doc ID | An 8 character hexadecimal number (32-bits) which uniquely identifies each active transaction. This is the same transaction identifier used in the server log. |
| Type | One character indicating the transaction type: (U)pload, (D)ownload, (L)isting, (I)nbound object handler, (O)utbound object handler, (G)lobbing, (S)tat, (A)ction. |
| Client | Three comma separated fields consisting of: the client IP address in hexadecimal network byte order, the client port number in hexadecimal network byte order, and a single character client type code. Common client type codes are (E)xpedat, (S)yncdat, and the client SD(K). |
| Username | The username associated with the transaction, if any. Some incidental transactions, including the transaction report itself, may not show the username even if one was used. |
| Auth Type | One character indicating the type of authentication: (A)uthFile, (S)ysAuth, or (-) unavailable. Shadow authentication records are considered SysAuth. Some incidental transactions, including the transaction report itself, may indicate '-' even if authentication was used. |
| Path | The client supplied pathname associated with this transaction, if any. |
| Encryption | If content encryption is enabled, the content type is given as a string, such as "AES". Otherwise this field is empty. |
| Action | If an action or object handler code was specified, the string is shown here. |
| State | A single digit indicating the state of the transaction: (0) active, (1) network transfer complete, (2) upload waiting for post-processing. |
| Statistics | Five comma separated fields consisting of hexadecimal numbers describing characteristics of the transaction. |
| Start | Start time of the transaction in unix time. |
| Speed | The currently estimated net throughput in bits per second. May be reported as 0 if speed information is not available. |
| RTT | The estimated round trip time between client and server in milliseconds. May be reported as 0 if RTT information is not available. |
| Sequenced | The amount of contiguous data received in bytes, not counting data received out of order. May be reported as 0 if sequence information is not available. |
| Remaining | The number of bytes between the expected end of the transaction and Sequenced. This field will be empty if the end of the transaction is not known. |
| File Info | Three comma separated fields consisting of hexadecimal numbers describing the file associated with this transaction, if this is a file based transaction. |
| Size | The expected final file size, in bytes. This may include data transferred in a previous session if this session was resumed. This field will be empty if the size is not known. |
| Offset | The starting offset for resumed transfers. This will be zero for transfers which were not resumed. |
| Modified | The modification date stamp in unix time. May be reported as 0 if the modification time is not available. |
| Warning | Three comma separated hexadecimal numbers describing the last non-fatal warning reported for this transaction. See Tech Note 0013 for a list of MTP classes and codes, and their meanings. |
| Warned | 1 if this transaction has received a warning (non-fatal error), otherwise 0. |
| Class | The hexadecimal MTP error class, if Warned is true.. |
| Code | The hexadecimal MTP error code, if Warned is true.. |
| Performance | A comma separated list of the current performance parameters as hexadecimal numbers: expire, max rate, aggression, max RTT, max datagram, min datagram, and noise ratio. A zero value indicates that no value has specified and the default value is in effect. |
Additional tab or comma separated fields will be added in the future, so be tolerant of extra data and empty fields when parsing. Unless otherwise specified, assume that all hexadecimal values may be 64-bits.