Object Handlers
Clients can request that servedat launch external processes to perform actions or access external data storage. An Object Handler can be used to perform tasks such as processing data, sending notifications, or moving files. It can also be used to stream data to or from other applications. For example, CloudDat for S3 uses an Object Handler to access S3 object storage.
This section applies to ExpeDat. The SyncDat client does not currently support Object Handlers.
Windows: Due to I/O processing limitations, Windows is not recommended for hosting Object Handlers.
ExpeDat's inline compression and Streaming Folders features cannot be used with an Object Handler, but the handler itself may perform compression, packaging, or similar functions. For example, a movedat client might pipe data through tar and call on the server to do the same with an Object Handler.
Setup
To make a server-side program available to ExpeDat clients, its identifying action code and executable path must be declared using the –o option or ObjectHandler configuration variable:
ObjectHandler <action_code>,<exec_path>
The action code should be a short alphanumeric text string, at most 15 UTF-8 bytes. The executable path must be a pathname that is accessible to servedat. No command-line arguments are permitted and the executable will be run without shell interpretation.
For example, servedat.cf might declare:
ObjectHandler Notice,/usr/local/libexec/sendnotice.sh
The movedat client could be used to run this handler using the following syntax:
echo "This is a message for Joe" | movedat user@example.com=Notice:joe
The command above would cause servedat execute /usr/local/libexec/sendnotice.sh. That script would receive "This is a message for Joe" via stdin and the string "joe" via an environment variable. The script could then send an email to "joe" containing the message. A example of sendnotice.sh is shown below.
Access Restrictions
By default, all users may access an object handler by specifying the appropriate action code. Setting the global ObjectOnly option will prevent access to the filesystem by default. That is, only object handlers would be allowed.
Access rights for individual users can be specified using the AuthFile ObjectOnly option. This allows users to be given or denied access to any combination of the filesystem and available object handlers. In other words, only the object handlers listed (and the filesystem if the special "*files" code is used) would be allowed.
The state of some server access privileges and user options are passed to the object handler through its environment. It is the responsibility of the object handler to decide how, or if, such restrictions should apply to its actions. For example, the CloudDat S3 object handler uses RestrictHome to indicate that the user's home path should be prepended to the object key name. The server may also enforce some basic restrictions, such as disallowing an upload when GetOnly is in effect.
Structure
Object handlers must conform to the following requirements:
- All input parameters are received from the environment variables described below.
- Expect no other environment variables or command-line arguments.
- SEND transactions will deliver data to the handler via stdin.
- GET transactions will pull data from the handler via stdout.
- Exit with 0 on success.
- On failure, exit with a non-zero exit code from the MTP Application error class (see Tech Note 0013).
When servedat receives a client request declaring a known Action Code, it will attempt to launch the corresponding program. If the transaction was authenticated, the executable is launched with the user identity associated with the given username. The transaction will fail if that user does not have permission to run the executable.
Once running, the handler should examine SV_TYPE to determine whether this is a GET or SEND transaction, and SV_ACTION to determine what task it should perform. It can then perform the appropriate tasks and either read from stdin (for SEND) or write to stdout (for GET). Two-way communication with the client (both stdin and stdout) is not available.
Following is a template for the bash shell showing a handler structure supporting multiple ExpeDat functions:
#!/bin/bash # Identify the handler and sub-action codes IFS=',' read -r -a ACTS <<< "$SV_ACTION" # ${ACTS[0]} is the identifier used to trigger this script # ${ACTS[1]}, if present, is an action code such as: *dl, *mk, or *mv if [ "$SV_TYPE" == "GET" ]; then if [ "${ACTS[1]}" == "" ]; then # Plain Get - send the path contents to stdout /bin/cat "${SV_PATHRAW}" elif [ "${ACTS[1]}" == "*dl" ]; then # Delete - delete the indicated path /bin/rm "${SV_PATHRAW}" elif [ "${ACTS[1]}" == "*mv" ]; then # Move - rename the path /bin/mv "${SV_PATHRAW}" "${SV_ARG}" elif [ "${ACTS[1]}" == "*mk" ]; then # MkDir - Create a folder /bin/mkdir "${SV_PATHRAW}" else # MTP_APP_FEATURE echo "Unsupported action: ${ACTS[1]}" exit 28 fi else # Send - Process data arriving on stdin /bin/cat > "${SV_PATHRAW}" fi; if [ $? -ne 0 ]; then exit 29 # MTP_APP_SYSTEM fi exit 0
For a plain data transfer, the SV_ACTION environment variable contains just the action code associated with the Object Handler. When a standard ExpeDat client is called upon to perform a task such as deletion or folder creation with an Object Handler, it will invoke a GET transaction and append a secondary action code. For example, the following two movedat commands are equivalent:
movedat -x user@example.com:deleteme=MyHandler movedat 'user@example.com=MyHandler,*dl:deleteme'
The handler example above shows just one possible structure for handling file transfers. Any program meeting the requirements above, not just shell scripts, can be used. Any action codes codes might be supported. Any type of data or processing can be handled.
The example above performs only limited error checking. A more complete implementation might enforce access restrictions or attempt to translate error conditions into more precise MTP_APP constants. All input strings should be handled carefully to ensure that unexpected characters cannot produce security problems.
Exit Codes
The exit code of the handler must be 0 for success or an MTP_APP constant on failure. Guard against returning exit codes from utilities called by the handler as their values may result in confusing or incorrect error reports to the end-user.
The following codes are reserved: MTP_APP_ACTION, MTP_APP_OBJID, MTP_APP_FLAG, MTP_APP_ADDRESS, MTP_APP_EOD, MTP_APP_WAIT, MTP_APP_SKIP, MTP_APP_LOST, MTP_APP_ENCREQ, MTP_APP_ENCUNAVAIL, MTP_APP_ENCBADKEY, MTP_APP_DEFER. If returned by the handler, servedat will instead report a value of 1 (MTP_APP_UNKNOWN) to the client.
Environment
The handler program is launched directly, without shell interpretation. It should expect no command line options or shell environment variables. The only communication with servedat is from the servedat environment variables described below, the executable's exit code, and either stdin or stdout. The program will be launched with the system privileges associated with the given username, if any.
The Object Handler is responsible for interpreting and enforcing Access Privileges based on the information provided in the environment variables below.
Some command line utilities depend on the existence of HOME or PATH environment variables. The Object Handler must create these variables if they are needed. In some cases, such as an AuthFile user, suitable values may not exist.
Windows based object handlers must pay special attention to the formatting of text and paths. All environment variables are Unicode and all paths are forward-slash delimited. When using a command shell as an object handler, set its code page to UTF-8 with the command "chcp 65001".
If HandlerTimeout is set, then the executable will be terminated by servedat if it runs longer than the specified time. Otherwise, the transaction with the client will remain open until the executable exits, the client aborts, or a network error occurs. In the event of an abort or error, servedat will attempt to terminate the handler and its descendants.
The following variables are provided in the handler program's environment. Those marked "optional" may not be present if the corresponding value is not set.
| Name | Value |
| Authentication | How this transaction was authenticated. |
| SV_AUTHTYPE | NONE, PRIVATE, or SYSTEM |
| The authentication mechanism used to validate this transaction. NONE indicates that the server is operating without authentication. PRIVATE means the username was authenticated with AuthFile or AuthHandler. SYSTEM means the username matched a System User. | |
| SV_USER | <Username> |
| Optional | The username given by the client and authenticated by the server. A value which is blank or "ANONYMOUS" indicates that authentication is enabled and no username was given. If this variable is not set, then authentication is not enabled and this was an implicit anonymous request. |
| Transaction | Information about the requested action. |
| SV_ACTION | <Action>[,Secondary_Action] |
| The first string is the Action Code used to activate this executable. This may be followed by a comma and a secondary Action Code. Clients may use the secondary Action Code to request file maintenance actions such as deletion or folder creation. You may use movedat or the ExpeDat Client SDK to specify any string for SV_ACTION. See below for handling '*li' or '*lr' listing requests from ExpeDat Desktop. | |
| SV_ARG | <String> |
| Optional | An additional string provided by the client. For move/rename actions, it provides the destination path. For mkdir and touch actions it provides file meta-data. For all other actions, standard clients will leave SV_ARG empty and the ExpeDat Client SDK may specify a custom string. |
| SV_BULK | 0 or 1 |
| When present and non-zero, this indicates that the client has declared this to be a "bulk" transaction for the purposes of logging and session identification. | |
| SV_CLIENTIP | <IP Address> |
| The client IP address, as seen by the server. If the client is behind an NAT gateway, this will be a public address of that gateway. | |
| SV_CLIENTPORT | 1 to 65535 |
| The UDP port of the client, as seen by the server. If the client is behind an NAT gateway, this will be a public port of that gateway. | |
| SV_DOCID | <Hexadecimal> |
| An 8 character hexadecimal number which uniquely identifies this transaction. It corresponds to the "DocID" field in the log file. | |
| SV_ENCRYPT | AES |
| Optional | If present, this variable indicates that content encryption is being used with the named type. Currently, only AES is supported. |
| SV_FEATURES | <String> |
| Optional | If present, this variable contains a comma separated list of transaction feature codes requested by the client. Unknown or unsupported features should be ignored. |
| SV_HEADER | <Listing Header> |
| A structured listing header line, describing the current state of the server. | |
| SV_HOME | <Pathname> |
| Optional | If present, this string contains the Home Directory associated with this transaction. When SV_AUTHTYPE is SYSTEM, this is likely (but not guaranteed) to be equivalent to a shell $HOME path. |
| SV_LOGPREFIX | DocID SEND|GET Client User Encrypt "Path" "Action" |
| A pre-formatted log line using the same format as that of I and F log file records. | |
| SV_PATHRAW | <Target String> |
| The raw target string as given by the client. Although considered by clients to be the "remote path", this can be any UTF-8 string and is not interpreted or modified by the server. It is not guaranteed to be valid unicode. | |
| SV_SIZE | <Bytes> |
| Optional | For SEND transactions where the size of the source is known, the number of bytes expected. |
| SV_TYPE | GET or SEND |
| The type of transaction. GET indicates that data sent to stdout will be delivered to the client. SEND indicates that data from the client must be read from stdin. | |
| Restrictions | Access restrictions which the handler should apply to this transaction. |
| SV_GETONLY | 0 or 1 |
| When present and non-zero, this indicates that this transaction is subject to the GetOnly restriction. The Object Handler is responsible for interpreting and enforcing this restriction. | |
| SV_NOOVERWRITE | 0 or 1 |
| A value of 1 indicates that this transaction is subject to the NoOverwrite restriction. The Object Handler is responsible for interpreting and enforcing this restriction. | |
| SV_READONLY | 0 or 1 |
| A value of 1 indicates that this transaction is subject to the ReadOnly restriction. The Object Handler is responsible for interpreting and enforcing this restriction. | |
| SV_RESTRICTHOME | 0 or 1 |
| A value of 1 indicates that this transaction is subject to the RestrictHome restriction. The Object Handler is responsible for interpreting and enforcing this restriction. | |
| SV_WRITEONLY | 0 or 1 |
| A value of 1 indicates that this transaction is subject to the WriteOnly restriction. The Object Handler is responsible for interpreting and enforcing this restriction. | |
| SV_WRITELISTONLY | 0 or 1 |
| A value of 1 indicates that this transaction is subject to the WriteListOnly restriction. The Object Handler is responsible for interpreting and enforcing this restriction. | |
| Server | Information about the server environment. |
| SV_LICENSE | <String> |
| A description of this server's software license. See Tech Note 0001 for more information about interpreting identification strings. | |
| SV_MAXLOADSTART | <Integer> |
| The maximum server load since startup. | |
| SV_MAXLOADCONFIG | <Integer> |
| The maximum server load since the last configuration reload. | |
| SV_SERVEDAT | servedat - 1.20.1 February 2022 - DEI, DOC-2.3.7 MTP-osx-4.4.1 3969 |
| The server's identification string. See Tech Note 0001 for more information about interpreting identification strings. | |
| SV_SERVERLOAD | 0 to Capacity |
| The number of simultaneous transactions currently being served, including this one. | |
| SV_SERVERCAP | <Integer> |
| The maximum number of simultaneous transactions supportable by this server. | |
| SV_TIMEOUT | <Seconds> |
| Optional | If HandlerTimeout is set, then this variable will be present and contain the maximum number seconds the script will be allowed to run. |
Additional environment variables are likely to be introduced in future releases. Please let us know if you have suggestions.
Support for Listings
To fully support an ExpeDat Desktop client, an Object Handler must respond to requests for a listing of objects. The listing will be displayed in the Remote browser pane of the ExpeDat Desktop window.
ExpeDat Desktop and other clients request a listing by sending a GET transaction to the handler with the secondary action "*li". The handler must respond with the contents of SV_HEADER, followed by the list of objects (if any) in the ExpeDat Structured List format. All lines must end with a single Line Feed character and all pathnames must be UTF-8 encoded.
The "Free Space" field of SV_HEADER may not be accurate for your handler. Modify it to show a relevant value or erase it if a value is not known.
If the listing includes objects marked as directories, then the client may send a '*lr' request for a recursive listing of the directory in preparation for a download of the entire directory's contents. If your handler is to support such a function, the listing should be returned in the same format as '*li', in depth-first order.
Supporting listings is not essential for movedat.
Performance
Data streamed through stdio will be buffered as per the StreamSize setting. Because of this buffering, clients may not accurately report progress and cannot estimate a time of completion for GET transactions.
Because of the overhead of launching the Object Handler program, the response time for even simple actions may be significantly longer than an equivalent action handled entirely within the server. If the Object Handler stalls or blocks during its processing of data, overall server performance will not be impaired. Only the throughput of the individual transaction will be slowed by bottlenecks in the Object Handler. Data processing which is bursty or uneven may benefit from a larger StreamSize buffer. For example, if it takes significant time for an Object Handler to prepare for incoming data, use a buffer large enough to absorb arriving data for that time.
Example
Below is an example of the "sendnotice.sh" email notification Object Handler referenced above. Unlike the general purpose example, this script only accepts SEND transactions and treats the path as an email address rather than a filesystem path.
#!/bin/bash # servedat object handler for sending an email - 130715 # # The recipient address is the SV_PATHRAW. # The message body is read from stdin. # The path to "mail" varies by operating system. MAILER=/usr/bin/mail # Only support SEND transactions if [ "$SV_TYPE" == "GET" ]; then echo "This Object Handler only works with Send transactions" exit 28 # MTP_APP_FEATURE fi # Send the email $MAILER -s "ExpeDat Email Notice" ${SV_PATHRAW} STATUS=$? if [ $STATUS -ne 0 ]; then exit 29 # MTP_APP_SYSTEM fi exit 0
This handler could be declared in the servedat configuration as follows:
ObjectHandler Notice,/usr/local/libexec/sendnotice.sh
It would then be run by movedat like this:
echo "This is a message for Joe" | movedat user@example.com=Notice:joe
A more sophisticated implementation might try to translate the exit codes of the "mail" command, perform user or input validation, or lookup recipient aliases.