Datacenter Flow Guide

README – SPILLBOX DATACENTER TO DATACENTER (SPL-DC)

Spillbox is a hybrid Cloud product which can also be used to burst your on-premise compute job on a remote Datacenter to use its extra available resources. This is done in a few simple steps:

1. Download the executable and untar it.

curl -o splfs.tar.gz https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/latest/splfs.tar.gz
tar xvfz splfs.tar.gz

The above steps should create an executable for splrun, along with a few more executables.

"latest" can be replaced with a specific version tag (e.g. v1.5) if you need to install something other than the newest release. To see what versions are available, run:

./upgrade.sh -list-versions

To upgrade an existing install to a specific version later, run:

./upgrade.sh -version <version>

2. Default value in spillbox may be sufficient but if you want to customize or override a few defaults, please create a config file.

Following config file creates docker network with an overlay driver, default is host driver, needed for jobs where dockers need to communicate among itself.

cat > ./myconfig_file <<eoj
"docker_driver" = "overlay"
eoj

More details about the config file will be described in later as part of Q&A.

3. Apply the config file.

This should be done only once per project and on a host or VM having at least 4 cores and 32Gb RAM. The concept of the project will be described in later steps. The lines below create a project, myproject, using a config file, myconfig_file, created in step 2.

export SPILLBOX_VPN=1   # Not a Cloud flow but a Datacenter flow
export SPILLBOX_SERVER=10.10.10.10 # IP address of spillbox server at remote Datacenter
./splrun project apply  myproject ./myconfig_file 

4. You are now ready to run your existing executable/script at a top/leaf level in the remote Datacenter.

Above apply step will print the Docker command.

Syntax: ./splrun run <existing exe/script> <arg>

Below runs the script /depot/openssh-7.9p1/build in the other Datacenter.

export SPILLBOX_PROJECT=myproject
./splrun run /depot/openssh-7.9p1/build

Since you may have multiple projects, you need to set SPILLBOX_PROJECT to a project. Here SPILLBOX_PROJECT is set to myproject so that script /depot/openssh-7.9p1/build is run in project myproject.

./splrun brun /depot/openssh-7.9p1/build

runs in batch mode with return status of the job. run option may not return job exit status in all cases.

COMMON QUESTIONS AND THEIR ANSWERS

Q. How can I get help?

Ans. At any level, you can type help to get a help message. For example:

splrun help
splrun run help

Q. What is the format and possible entries in the config file?

Ans. The format of the config file is "key" = "value", both in quotes. If the value is in the form of a list, then the list syntax is ["value1", "value2"] . Possible keys and their default values are:

"mount_paths" = // default is derived automatically by Spillbox

This gives the list of NFS export paths on local Datacenter to be made available in the remote Datacenter. Spillbox automatically adds by default the list of top level directories that are not part of Linux OS. User home directories are not added if they are inside a Linux OS distribution path like /home. If the Spillbox automatic NFS mount point finder is not suitable to you, please override it by providing the list. You can add a full path of subdirectories also. For example ["/depot", "/home/user"].

"exclude_mount_paths" = [] // no default exclude 

Provide the list of mount_paths to be excluded.

"tmpfs_size"= // size of tmpfs mount in GB (integer value)

Specifies the size of tmpfs to be mounted on the cache directory. Example: "tmpfs_size" = 4 // mounts tmpfs of size 4G

Note:Must not exceed "spillbox_tmpfs_max" defined in spillbox_config.

"global_cache"= // Set to true to enable global cache project. Default is false.
"cloud_user" = // default is on-prem unix username
"cloud_group" = // default is on-prem unix  group
"cloud_uid" = // default is on-prem unix uid
"cloud_gid" = // default is on-prem unix gid
"cloud_shell" = // default is /bin/bash

Specifies the shell to use inside the container when running jobs. If not set in the project config, Spillbox will fall back to the cloud_shell setting in spillbox_config on the server. If neither is set, /bin/bash is used. For csh/tcsh users, set this to the full path of tcsh (e.g. /usr/bin/tcsh) and Spillbox will automatically adjust direnv hooks and shell invocations accordingly. Example: "cloud_shell" = "/usr/bin/tcsh"

"exclude_filter_path" = "" // default no file/directories are excluded

Excludes file/directory from getting uploaded to the remote Datacenter.

"docker_base_image" =   // default is centos:7

You can put your own docker image on the docker repo server ( spillbox server is docker repo server ) and select the image for the project using this variable.

"docker_driver" =  // default is host

You can choose overlay/macvlan as long as spillbox server supports overlay/macvlan networking and sysadmin has configured the spillbox server.

"disable_docker_image" = // default is false

User can bring their own docker image and disable per project docker image. Env SPILLBOX_DISABLE_DOCKER_IMAGE set to any value will also disable spillbox image.

"wan_timeout" = 1800 // Change WAN timeout, must be more than 900 (default 900)
"nfs_model"= // default is empty, falls back to the "default" model in the config file

Selects a model defined in terraform/etc/spillbox_nfsModel_config. Each model can set, independently:

model_nfs_submitopt_medium="-n 4" will allocate 4 Slot(s) on Host(s) of the job

getting submitted by the lsf job manager. Model names must be made of letters, digits and underscores only.

"hybrid_mount_paths" = "" // list of mount paths separated by spaces

Enable fully synchronized data sync between on-premise and Cloud for set of mount paths hierarchically. This is equivalent to enabling the snapshot options in SPILLBOX_HYBRID_OPTION (empty by default) for these directories. Environment variable SPILLBOX_HYBRID_MOUNT_PATHS also does the same. Special keyword "ALL" enables globally on all mount paths and "NONE" disables it if enabled globally by sysadmin.

"hybrid_mount_paths_exclude" = "" // list of mount paths separated by spaces

Disable fully synchronized data sync between on-premise and Cloud for set of mount paths hierarchically. This will be applied after hybrid_mount_paths Environment variable SPILLBOX_HYBRID_MOUNT_PATHS_EXCLUDE also does the same. Special keyword "PATH" excludes all paths in $PATH.

Spillbox provides wrapper script for LSF/UGE/RTDA/… in lsf/uge/rtda/… sub-directory so that users do not need to change their job submission command line embedded in their script/flow. Just picking up Spillbox job submission command by modifying $PATH will do the trick. There are few environment variables used in these wrappers:

Q. What are other important commands to know?

Ans. Other important commands are:

Syntax: SPILLBOX_PROJECT=<projectname> splrun log

Prints the server-side log for the given project. Useful for debugging when a project is in STAGE_APPLIED state and you want to see what the server is doing.

Syntax: splrun project destroy|destroy_all|snapshot|backup|restore <projectname>

destroy — To destroy compute machine and file server (keeps file cache) destroy_all — To destroy compute machine and file cache snapshot — To create snapshot at current time backup — To backup Cloud data to objectstore for later reuse of project restore — To restore backed up project

Once you are done running all your jobs, please run destroy_all to delete all resources in the other Datacenter. If your data upload is large and you want to come back again to use the remote Datacenter in the near future, please run destroy only. This keeps the file cache intact. This does, however, cost resources.

If you make changes on premise after creating a project and accessing data on the remote Datacenter, these changes are not updated in the remote Datacenter. To make your changes visible in the remote Datacenter, you have to run snapshot. snapshot creates a file instance in the remote data center based on time of the snapshot command. Each time you create changes on local data center and need the changes to be available at the remote data center, you need to update the file system instance time by running the snapshot command.

Note: Full paths are no longer required for the config file in project apply or the directory argument in snapshot -download-hier. Relative paths now work:

splrun project apply default myconfig_file
splrun project snapshot default -download-hier mydir

Syntax: splrun show <projectname> [<variable>]

In the case that you need to login to a remote Datacenter compute machine, you will need to know information such as the ssh key and the IP address of the machine. These and other important variables can be printed with their values using the show command.

Syntax: splrun precache <dir1> <dir2>…<dirN>

splrun download <src_file> <dst_dir>
splrun download_all [<dst_dir>]

The requirements of your job may be such that it will sync TBs of data to the remote Datacenter before the job starts running. Although Spillbox will do this automatically for you, you may not want to waste compute resources while data is being uploaded automatically by Spillbox. You may want to help the system by uploading some of the larger data yourself before running the job. splrun precache is a very useful tool for this. You can sync multiple directories at a time. Spillbox does not sync data from the remote Datacenter to local automatically. You will need to use the above download command to sync the selected result back to the local Datacenter. Upload/download can exclude/include selected files/directories and detailed usage can be available with command:

splrun precache/download help

There are few important features for data synchronization between on-premise and remote datacenter which can be activated through snapshot features:

splrun  project snapshot  help|<project_name> [-missing|-missing-hier|-missing-remove|-missing-hier-remove] [-missing-pattern-include <pattern>] [-missing-pattern-exclude <pattern>] [-sync|-sync-hier|-sync-remove|-sync-hier-remove] [-download|-download-hier|-download-remove|-download-hier-remove] [-download-pattern-include <pattern>] [-download-pattern-exclude <pattern>] [-reset-log <tag>] [-reset-appendlog <tag>] [<file>/<dir>]
    <file>/<dir>              -- Optionally snapshot a file/dir only
    -missing                  -- Search for missing files/dirs at current level in on-prem
    -missing-hier             -- Search for missing files/dirs at current and below level in on-prem
    -missing-remove           -- Remove search for missing files/dirs at current level in on-prem
    -missing-hier-remove      -- Remove search for missing files/dirs at current and below level in on-prem
    -missing-pattern-include  -- Pattern to include in handling missing files/dirs
    -missing-pattern-exclude  -- Pattern to include in handling missing files/dirs
    -sync                     -- Sync files/dirs at current level with on-prem all the time
    -sync-hier                -- Sync files/dirs at current level and below with on-prem all the time 
    -sync-remove              -- Remove -sync
    -sync-hier-remove         -- Remove -sync-hier
    -download                 -- Download new files/dirs at current level automatically at close
    -download-hier            -- Download new files/dirs at current and below level automatically at close
    -download-remove          -- Remove download new files/dirs at current level automatically at close
    -download-hier-remove     -- Remove download new files/dirs at current and below level automatically at close
    -download-pattern-include -- Pattern to include in handling download of files/dirs
    -download-pattern-exclude -- Pattern to exclude in handling download of files/dirs
    -reset-log                -- Create full snapshot and recreate log file for tracking data transfer
    -reset-appendlog          -- Create full snapshot and recreate log file in append mode for tracking data transfer 

With the activation of -missing*, -sync* and -download* features in the beginning of the project, Spillbox's cloud bursting can be made to look like two way fully synchronized hybrid cloud.

Q. How can I ssh out to spillbox worker machines when the ssh port is blocked at corporate level?

Ans. You can use spillbox data pipe to ssh out.

After project is created, you need to run the following command to ssh:

splrun ssh <worker_hostname> 

where <worker_hostname> is the spillbox machines you want to ssh to. No need to provide ssh private key. It is automatically taken care by splrun.

Implementation internally uses a local port, 2222 and above. You can control the port with an environment variable SPILLBOX_SSH_PORT by giving the starting port number.

Q. How can I pass environment variable to "splrun run"

Ans. Please see below usage:

splrun run [-env <var>=<value>] help|<command>  <argument>

Q. splrun run is not sourcing my .cshrc ( .bashrc ) in the container.

Ans: Please do "splrun run /bin/csh -c \"<command> <args> …\" (or similar for bash) in places where you want to source .cshrc in the container. It will be usually needed at the top level script, and not when calling splrun from the middle of the script. Alternatively, set "cloud_shell" in your project config or in spillbox_config on the server so that all jobs in the project automatically use the correct shell.

Q. How do I configure the shell used inside the container for my jobs?

Ans: By default, Spillbox uses /bin/bash inside the container. You can override this at two levels:

1. Per-project: set "cloud_shell" in your project config file:

"cloud_shell" = "/usr/bin/tcsh"

Then apply the project:

splrun project apply <projectname> <configfile>

2. Server-wide: set cloud_shell in terraform/etc/spillbox_config on the server:

cloud_shell="/usr/bin/tcsh"

This applies to all projects on the server unless overridden per-project.

When cloud_shell is set to a csh/tcsh variant, Spillbox automatically:

• Uses tcsh for all job executions inside the container
• Adjusts direnv hooks to use "direnv hook tcsh" instead of bash
• Passes SPILLBOX_CLOUD_SHELL through to the container environment

The per-project setting takes priority over the server-wide setting.

Q. How can I use Spillbox for test case archival and replication at a later time or at a third party?

Ans. If user runs into complex bug and need to send the test case to the third party for debugging by replicating the test case at third party then please follow the below steps:

splrun run -archive <dir1> <cmd> <args> // archive in dir1 at the end of running

Inspect dir1 for sensitive data as suggested by splrun at runtime and ship it to the third party.

Third party can restore  using Spillbox during project apply phase:

splrun project apply <projectname> -load <dir2>   // dir2 is directory where shipped
                                                  // data were untar'ed by the vendor.

Then run as

splrun run -rerun   // if no arguments are given, it runs the command used during archiving

Q. How do I use multiple container images?

Ans: Users can select container image using short name ( or alias) from a set of containers available on the server:

splrun run -image <short_name> <cmd> <args>

Q. How can I get an interactive terminal

Ans: Please run "splrun run $SHELL -i" to get an interactive terminal. Example : splrun run /bin/bash -i

Please type control-D to end the session. Typing "exit" in the shell will also end the session.

Q. How can I change the location of Spillbox config file $HOME/.config/spillbox ?

Ans: Please set environment variable SPILLBOX_CONFIGDIR to a location other than $HOME.

Q. How can I run multiple splrun in single command line?

Ans: Use multi_splrun splrun run <cmd1> ++ splrun run <cmd2> . Using "++" user can concatenate any number of splrun commands to the need. To run multiple splrun commands in background use only brun

multi_splrun splrun brun <cmd1> ++ splrun brun <cmd2> &

Example : multi_splrun splrun run hostname ++ splrun run whoami

multi_splrun  splrun brun hostname ++ splrun brun whoami &

Q. How do I enable support for Spot Instances?

Ans: Support for Spot Instances are done on the server by your sysadmin following the Spillbox Installation & Configuration Reference (https://spillbox.io/installation-configuration-reference/). In case support for the spot instances are enabled, you can choose the job to be automatically restarted on spot preemption ( default ) suitable for short jobs, or enable checkpoint at regular intervals. On preemption, if checkpoint exits, job will start from the last checkpoint. Following environment variables are used to control Checkpoint:

Q. How do I enable spillbox boostmode?

Ans: export SPILLBOX_BOOSTMODE=1

export SPILLBOX_BOOST_MOUNT_PATHS="DIR1 DIR2"     # Directory to be in boost mode

Q. How can I login to the container where my job (job_id) is running??

Ans: – For LSF jobs: Use the Spillbox bsub wrapper

bsub -jobattach <job id>

• For UGE jobs: Use the Spillbox qsub wrapper qsub -jobattach <jobid>

• For slurm jobs: Use the Spillbox sbatch wrapper sbatch -jobattach <jobid>

• For nc jobs: Use the Spillbox nc wrapper nc -jobattach <job id> Note: This will be useful for debugging purposes.

Q. How can I map a remote queue name to an on-premises queue name?

• To map your on-premises queue name with a remote queue name, create a setup.sh script in the slurm directory on your client side.
• Provide your on-premises queue name in the setup.sh file in the following format:

queue_map={"queue_name1"="compute-queue","queue_name2"="nfs-queue","queue_name3"="spot-queue"}

• "queue_name1" , "queue_name2" and "queue_name3" are your on-prem queue names (Replace them with your actual queue names).
• "compute-queue" , "nfs-queue" and "spot-queue" are remote queue names. NOTE: The format should be the same as provided, without any extra spaces or characters.

Q. How to Automatically Destroy an Idle Project ?

Ans: Set SPILLBOX_IDLE_TIMEOUT to the desired number of hours before creating a project. It specifies how long a project can remain idle before being automatically destroyed.

Q. How can I get help from Spillbox team?

Ans: Please email support@spillbox.io or call 1-833-SPILLBX(774-5529).