Standard Setup

A standard install of the CloudDat on AWS uses the following ExpeDat configuration options.

• The following line in /etc/servedat.cf ties the “S3” action code supplied by clients to the s3handler.sh shell script on the CloudDat EC2 instance.  See the Object Handlers documentation.

  ObjectHandler S3,/usr/local/expedat/s3handler.sh

• AuthFile /etc/svpasswd
  ExpeDat users may be defined in /etc/svpasswd, which resembles a unix password file.  The s3user created during initial setup is given the ObjectOnly restriction, which means it cannot access the instance filesystem.  See the ExpeDat AuthFile documentation.  If you customize this file, be sure to answer ‘n’ if you re-run an installer and it asks about replacing existing credentials.
• SysAuth 1
  This option in /etc/servedat.cf allows system accounts on the CloudDat instance, such as ec2-user, to access both S3 and the instance filesystem.  You must assign a password to a system account for it to be useable with ExpeDat.  For example, “sudo passwd ec2-user".  See the ExpeDat SysAuth documentation.
• /var/www/html/index.php
  This is the CloudDat instance web page, available with the AWS Marketplace version of CloudDat, for downloading clients and monitoring server status.  The settings at the top of this file allow you to easily customize its look.  You can also rename it to allow other web content on the instance.

Following are some of the ways you can modify these settings to customize CloudDat.

Updating to a New Version

New versions of CloudDat for perpetually licensed software can be downloaded directly from your DEI Customer Site.  New versions of CloudDat from AWS Marketplace are obtained by launching a new CloudDat instance from AWS Marketplace and copying over configuration and other customer data from the old instance to the new one.

Follow the standard setup procedure when deploying the new server instance.  Customizations, such as those described on this page, can be transferred from the old server to the new server by replicating these files:

• /etc/servedat.cf - Your old ExpeDat server configuration file may be preserved without modification, however you may wish to examine the new sample file for new features.  For example, the preferred StreamSize value may have increased.
• /etc/svpasswd - Your old ExpeDat password file may be preserved without modification, or you may copy over individual user lines.  You may also wish to examine the new sample file for new features.
• /usr/local/expedat/s3handler.sh - A new handler script is created and configured during instance setup.  If you made customizations to the handler script, or created new handler scripts, it is best to replicate those customizations using the new script as a template.  If you are copying an older handler script, pay particular attention to the S4CMD line as python has been updated to python3.  These scripts must be marked as executable.
• /var/www/html/ - If you made changes to the instance landing page (index.php) or any of the web integration templates, you may wish to copy those to the new instance or apply similar changes to the new versions of these files.  See the ExpeDat Web Integration documentation for more details.

General advice for updating DEI software can be found in Tech Note 0012.  A list of changes for each CloudDat release can be found in the CloudDat Change Log.

Opt-in Regions

AWS disables access to some regions by default, requiring you to opt-in before you can access resources such as S3 storage:

• af-south-1 (Cape Town, Africa)
• ap-east-1  (Hong Kong, Asia Pacific)
• eu-south-1 (Milan, Europe)
• me-south-1 (Bahrain, Middle East)

If you plan to use CloudDat in one of these regions, or any region created since 2019, follow these AWS instructions:

Managing AWS Regions

Alternate Endpoints

The default S3 endpoint used by CloudDat for AWS takes the form:

http://s3.region.amazonaws.com

Some regions or environments may require a different S3 endpoint.  To set a custom endpoint, edit /usr/local/expedat/s3handler.sh and modify the "S3_ENDPOINT_URL=" and "S3_ENDPOINT_REGION=" lines as needed.  For example:

S3_ENDPOINT_URL="http://s3.cn-north-1.amazonaws.com.cn" S3_ENDPOINT_REGION="cn-north-1"

Multiple Users

The initial installation creates a single S3 user.  To create additional credentials, edit /etc/svpasswd, using that first entry as a template.  For example, after adding a second user, svpasswd might look like:

s3user:frYypg…0HCxHA:99:99::ObjectOnly usertwo:SddKMR…jSHh7g:99:99::ObjectOnly

The second field of svpasswd can contain a plain text password, but it is safer to generate an SHA hash using the /usr/local/expedat/mkpasswd utility.  After modifying svpasswd, the server will automatically load the new credentials after about a minute.

With multiple user credentials, you can apply different restrictions and configurations to different accounts, as discussed below.

Access Restrictions

The final field of an AuthFile (svpasswd) record contains a list of access restrictions.  Commonly used restrictions are:

• ObjectOnly — prevents access to the filesystem or specifies which storage is allowed.
• ReadOnly — no uploads or changes allowed.
• WriteListOnly — downloading prohibited.
• WriteOnly — downloading and listing prohibited.
• GetOnly — download only, listing other actions prohibited.
• NoOverwrite — this option is not supported.
• RestrictHome — the user home directory is prepended to all paths.  See Segmented Buckets below.  The user home directory is otherwise ignored by the default S3 handler.

See the ExpeDat AuthFile documentation for more options.

Segmented Buckets

The default CloudDat user account, shown above, has access to the entire S3 bucket.  You can restrict gateway users to just a portion of the bucket by assigning a RestrictHome prefix.  For example,

s3user:frYypg…0HCxHA:99:99:webfolder:ObjectOnly,RestrictHome

This will insert the prefix “webfolder/” in front of all paths accessed by ExpeDat clients using the “s3user” name and will not allow them to access any other objects.  It is like restricting the user to a single folder.  You can create multiple users, each restricted to a different prefix.  Changes to svpasswd will be automatically loaded about a minute after your last modification.

Multiple Buckets

Step 1 of the Install process asks you to choose an S3 bucket to access.  Setting a fixed name allows clients to access S3 storage without knowing the bucket name.  If you leave this blank, clients must specify a bucket name as the first part of the remote path.  This allows you to access multiple buckets through a single CloudDat instance.

If you’ve already setup the gateway, the easiest way to modify the bucket name is by editing /usr/local/expedat/s3handler.sh to change the “S3BUCKET=” setting at the top.

When S3BUCKET is blank, the bucket must be specified either by the client or as part of the svpasswd home prefix.  If you follow the Multiple Users and Segmented Buckets instructions above, you can have different users accessing different buckets all from the same CloudDat instance.

joanne:frYypg…0HCxHA:99:99:bucketone:ObjectOnly,RestrictHome robert:SddKMR…jSHh7g:99:99:buckettwo:ObjectOnly,RestrictHome samuel:Yd3751…PWFgIQ:99:99:bucketthree/sam:ObjectOnly,RestrictHome nancy:nCcR61…TXNybU:99:99:bucketthree/nancy:ObjectOnly,RestrictHome

In the example above, joanne and robert each have access to an entire bucket.  Users samuel and nancy each have access to a psuedo-folder within a third bucket.

To have clients choose the bucket, leave off the RestrictHome option.  When RestrictHome is not set, the home directory field is ignored and clients must specify the bucket name as part of their remote path.  With movedat, this just means typing the bucket name in front of the object name.

movedat local.dat s3user@aws.example.com=S3:bucketone/remote.dat

In ExpeDat Desktop, the bucketname will need to be entered into the Remote Prefix field.  Users can type this manually, or you can make it part of an expedat:// URL.  For example, you could create a custom landing page as described below and modify it to include a bucket prefix.

Changes to svpasswd will be automatically loaded about a minute after your last modification.

Server Side Encryption

Server Side Encryption (SSE) is enabled by editing /usr/local/expedat/s3handler.sh on the gateway instance.  Modify the "KMS_ID=" line by adding the KMS ID of the key you wish to use.  For example:

KMS_ID=99999999-9999-9999-9999-999999999999

Sending Links

You can direct users to a specific location in your S3 bucket by creating a gateway link.  The easiest way to do this is to navigate to the location using the ExpeDat Desktop client.  If you want to make a link for downloading a specific file, select that file, then click the Bookmark button.

You will now have an expedat:// link copied into your desktop clipboard.  To create a gateway link, just paste the link onto the end of your gateway URL with a question mark in between.  For example:

http://aws.example.com/?expedat://aws.example.com:8080/Downloadme.dat?u=&h=S3

When a user clicks that link, they will be taken to your gateway page where they receive instructions, an opportunity to download the client software, and a final link to direct the client to download the targeted object.

You can modify the link by pre-filling the username ( u=s3user ) or by prompting the user to upload a file instead of downloading ( &a=s ).  For example:

http://aws.example.com/?expedat://aws.example.com:8080/folder/?u=s3user&h=S3&a=s

See the expedat:// URLs chapter of the ExpeDat Documentation for more details about creating links.

Custom Landing Pages

The CloudDat instance web page from which you download ExpeDat clients is a single PHP file located at /var/www/html/index.php.  The settings at the top of this file allow you to easily customize its look.  You can also rename it to allow other web content on the instance, or make multiple copies with different configurations.

Customization of the instance web page is very similar to that of ExpeDat's User Links page.  See the User Links documentation for details.

Custom Clients

The clients available for download on the CloudDat instance from AWS Marketplace are pre-configured to show an “S3” object handler as an option in the Handler pop-up menu.  Standard ExpeDat clients are not pre-configured for S3 and must be setup either by the end-user or by embedding a list of handlers.

If you have a standard ExpeDat package, you can embed custom settings into ExpeDat Desktop executables, including a list of Object Handlers.  This list might include the standard “S3” handler as well as any custom handlers you create.  Once you have embedded this list, the Handler pop-up menu will automatically appear when that ExpeDat Desktop executable is run, without the end-user needing to perform any setup.

Filesystem Access

Instance system users, such as ec2-user can access the full filesystem of the CloudDat instance, including instance, EBS, and EFS volumes.  In ExpeDat Desktop, select “None” from the Handler pop-up menu.  In movedat, remove the “=S3” or similar suffix.  System user accounts must have a password assigned to be used with ExpeDat.  For example, “sudo passwd ec2-user".  See the ExpeDat SysAuth documentation for additional information about how system accounts interact with ExpeDat.

Users declared in the AuthFile (/etc/svpasswd) are blocked from accessing the instance filesystem by the ObjectOnly option.  If you remove that option from a user's record, or add *files to the list of allowed objects, then that user will also be able to access the full filesystem with the privileges of the UID and GID specified.  This is not recommended without additional restrictions such as RestrictHome and AllowPath to control which paths are visible to the user.

If you wish to prevent any users, including system users, from accessing the instance filesystem, add "ObjectOnly" to /etc/servedat.cf.  After modifying servedat.cf, you must signal the server to restart with "sudo killall -HUP servedat".

Glacier

Objects in Amazon Glacier cannot be accessed in real-time, but they can be moved to and from S3.  To upload an object to Glacier, use CloudDat to place the object into S3, then use the AWS APIs or AWS Console to move it to Glacier.  For automated operations, you can create S3 rules which automatically move new objects to Glacier.  Likewise, use the AWS APIs, web interface, or automation rules to pull objects from Glacier to S3.  Remember that AWS can take several hours to access Glacier objects.

Web Passwords

This topic requires proficiency with Apache webserver configuration.

You can restrict access to web pages using by sharing the same credentials used by the ExpeDat clients.  The format of the /etc/svpasswd file is compatible with that of an Apache htaccess file.  To restrict the contents of a folder in /var/www/htmlusing svpasswd credentials, create a file in that folder named “.htaccess” with the following contents:

AuthType Basic AuthUserFile /etc/svpasswd AuthName “My S3 Gateway”

For password compatibility, you must use unix-crypt hashes in /etc/svpasswd instead of the default SHA hashes.  To create a unix-crypt hash, run “/usr/local/expedat/mkpasswd -L“

Security Warning: If you password-protect your web pages, it is strongly recommended that you install an SSL certificate and configure the webserver to only access authenticated pages using SSL.  Otherwise web browsers will send usernames and passwords in an insecure manner.  ExpeDat clients always encrypt usernames and passwords, but web browsers will only do so if you use SSL.

Web Integration

Templates for creating custom web landing and monitoring pages are included in the /var/www/html folder.  See the ExpeDat Web Integration documentation for usage details.

S3 Upload Tuning

This topic requires proficiency with Bash shell scripts.

CloudDat uses multipart upload with the concurrency and chunk size chosen to allow speeds up to 5 gigabits per second.  These values are specified by the partsize and multipart variables near the bottom of the S3 object handler, /usr/local/expedat/s3handler.sh.  The ExpeDat server, servedat, uses an additional 384 megabytes per S3 transfer for buffering, as specified by the StreamSize variable in /etc/servedat.cf.  The total memory requirement is approximately 1 gigabyte per S3 upload.

AWS Marketplace instances of CloudDat will record the instance size and server transaction capacity in /usr/local/expedat/host.txt which will be used to reduce the chunk size and/or concurrency when running on an instance with too little memory to accommodate the full number of uploads.  Using too small an instance size for your server capacity may result in reduced speeds.

If your workflow does not require 5 gigabits per second upload speeds and you wish to use smaller instance sizes, you may reduce the CloudDat memory requirements by lowering partsize and multipart in s3handler.sh or any custom handlers.

AWS S3 limits the number of parts in a multipart upload to 10,000.  With a default part size of 32 megabytes, the largest object that could be uploaded would be about 312 gigabytes.  For file uploads, where the size of the file is known from the start, s3handler.sh will automatically increase the chunk size to accommodate larger objects, provided the instance has sufficient memory.

Custom Handling

This topic requires proficiency with Bash shell scripts.

You can create multiple custom versions of s3handler.sh and access them using different handler names.  Simply make a copy of the /usr/local/expedat/s3handler.sh shell script and modify it as needed.  For example, you might have different S3BUCKET targets or target different SSE options.

After creating a new handler, you must declare it in /etc/servedat.cf.  The following example shows two handlers:

ObjectHandler S3,/usr/local/expedat/s3handler.sh ObjectHandler S3custom,/usr/local/expedat/s3custom.sh

To access the custom handler with movedat, specify its action code on the command line:

movedat local.dat s3user@aws.example.com=S3custom:bucketone/remote.dat

To access the custom handler with ExpeDat Desktop, add its action code in the Opt Setup dialog, then select it in the Handler pop-up menu.  Alternatively, you can follow the instructions above to preconfigure ExpeDat Desktop clients with your custom handlers.

The maximum length of an action code is 15 bytes.  Treat them like file extensions: shorter is better.

After modifying servedat.cf, you must signal the server to restart with “sudo killall -HUP servedat".

You can specify which object handlers a user may access by adding a list to the ObjectOnly option.

More

Each CloudDat EC2 instance includes a fully functional ExpeDat server.  See the ExpeDat Documentation for more information about customizing it.