Installation & Configuration Reference
INSTALLATION
This document describes the installation of Spillbox software to utilize the free compute resources of remote data center. User level software and documentation is a separate bundle.
Requirements at remote data center
- Need a machine ( 32-64 cores, 128-256Gb, 2TB and above local NVME storage) to run Spillbox server. IT needs to create a Unix service account which will be used to run Spillbox server. Service accounts should have SUDO (super user do ) access without needing a password on the spillbox server machine. Service account should have Docker/Singularity access. This machine should have good amount of storage in /var ( or somewhere local, about 20gb ) to accommodate docker images.
- IT needs to have a list of OS patches to create Docker/Singularity images.
- IT needs to allocate an adequate set of machines with enough local storage (32-64 cores, 256Gb, 2TB and above local NVME storage ) for data caching to be used by Spillbox NFS servers. NVME storage should be mount locally with world writable permission.
- If using Docker, then IT needs to setup and maintain Docker image orchestration by installing docker daemon on all machines. Spillbox can create a container registry server on the spillbox server. Need docker-ce and docker-ce-cli containerd.io packages on all machines.
- Common storage ( NFS mounted, visible to all hosts ) to install Spillbox software
- Spillbox installation area creates two directories parallel to its installation ( ../sites and ../customers ) which may grow over time and it keeps per project/user data. This will be a few gb only.
- NIS/LDAP support required on host and in container. All Spillbox users must have an account with home directory on cloud hosts machines.
NFS server cannot be run in a container on azure Cloud due to a bug/limitation in Azure Cloud. For NFS server on run on host without container, followinngs are needed on all NFS servers:
- patchelf binary/pkg
- sudo without password by the service account on the nfs server
- max number of open files for all users should be 1024000. Entry in /etc/security/limits.conf on spillbox server and nfs-server
* soft nproc 65536 * hard nproc 65536 * soft nofile 1024000 * hard nofile 1024000
- Below kernel configurations on spillbox server and nfs-server ( /etc/sysctl.conf):
sunrpc.tcp_max_slot_table_entries=65536
sunrpc.tcp_slot_table_entries = 128
fs.file-max=10240000
vm.min_free_kbytes=4124416
net.core.somaxconn=8192
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.tun0.disable_ipv6=1
net.core.rmem_max=16777216
net.core.wmem_max=16777216
# please do "sysctl -w ..." for each above entry for already running kernel.There are two steps
In Phase 1: One time installation by an account having docker access. This account must have write permission in the area where Spillbox software is installed. Better way would be for this account to download and install the software at common NFS location.
This phase will create common docker image and docker registry , if needed. This account also configures the config file used by all.
In Phase 2: This step starts a spillbox server. Any user can start a spillbox server and need not have docker access if docker container can be spawned through LSF app.
If docker container cannot be spawned through LSF app then server should be run by a common service account which has docker access ( in docker group). All users can use this server for their clients on premise. In this case, $HOME of the servrice account should be on the local disk on the server machine, not on a common NFS storage.
Phase 1 Steps
- Login to designated spillbox server machine with account created for Spillbox
- Install docker and run docker server ( sudo service docker start )
- On NFS partition ( no. 6 above ), create a directory spillbox ( mkdir -p spillbox ) for Spillbox software download and installation.
- Download software from https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/latest/spill-dc.tar.gz ("latest" can be replaced with a specific version tag, e.g. v1.5; run "./upgrade.sh -list-versions" to see what versions are available)
- unzip and untar it in spillbox
- copy terraform/etc/spillbox_config.example to terraform/etc/spillbox_config
- Edit terraform/etc/spillbox_config configuration file as per your site ( described below )
- Run "terraform/server_image.sh build" This will create docker image and docker registry server
- To destroy, please run "terraform/server_image.sh destroy"
- This account should also configure system-wide terraform/etc/spillbox_config.
- If you are using Spillbox docker registry server then please make Spillbox registry server as insecure private registry server in /etc/docker/daemon.json on all of your machines.
Configuration Reference
Please note that all entries are not must. Tool will inform about missing must field at runtime.
| Parameter | Description |
|---|---|
disable_server_image=false | If you have your own docker image, Spillbox can use that. |
disable_registry_server=false | If you have your own docker orchestration system through LSF etc, Spillbox will not create docker registry server used as docker repo for docker orchestration. |
registry_server="" | Please provide host:port here if using own docker registry server. |
cache_dir=/cache_dir | This is a snapshot area where local data center's data will be cached. This should be a large local or NFS partition. This is required; an empty value is an error. |
cache_server=10.1.10.103 | if cache_dir is NAS then the IP address of the NAS server. |
hub_port_min=8090 | For all projects and users, currently hub will run on the spillbox server itself. So multiple hub will be on one machines and each hub will use a unique port to communicate with the nfs server of that project. Hub will hunt for free unix port starting from this port number. |
docker_base_image="centos:7" | If you have your own base image to be used company wide, please add here. Users can override this. |
local_mount_paths="/tools" | If remote data center has its own tool area and need not cache that from local data center, please add here. |
exclude_mount_paths="" // list seperated by spaces | Force exclusion of these nfs/local mount paths from being exported from on-premise. |
disable_docker_image=true | This disables docker image per user/project. |
disable_docker_privileged=false | Disable –privileged option in docker being passed. |
cloud_shell="/bin/csh" | Change default shell in Docker. |
nfs_portmin=10000 | Starting port number for NFS server. |
docker_wrapper="" // full paths separated by spaces | User's runtime commands are executed in a docker wrapper such that command runs as user. Spillbox has a default wrapper. Customer can bring their own wrapper. Multiple approved wrappers can be provided. Only approved wrappers are allowed. |
client_hosts="" | In case if you want to enforce all clients (onprem) to be run from a particular machine. |
container_with_automount=false | In case you are using your own container and container already mounts relevant local file systems and does not need help from Spillbox for local_mount_paths. This is not common. |
job_manager="" // slurm, uge, lsf, nc | Job manager used for running file server. |
job_submit_command="" // sbatch, bsub, qsub ... | job manager submit command |
job_kill_command="" // scancel, bkill ... | job manager' command to cancel a job |
user_setup_file="" // path to a setup file | If setup file exists, then it will be sourced before spawning the container(running the job/batch job). |
preexec_file="" // on-premise filepath to be executed as first thing in the container | If preexec_file exists in the container, then it will be sourced quietly before executing the user command. Please note the file path and file exists on premise. |
container="" // default docker | "docker" and "singularity" containers are supported ("host" runs without a container). Applies for NFS servers as well as for compute servers. |
docker_path= | In case "docker" in your network is at some other location. |
singularity_path= | In case "singularity" in your network is at some other location. |
singularity_base_image="" | Location of singularity .sif image. Please provide full path. |
runtime_container= // default is whatever is during NFS server | At compute time, one could use singularity container even if nfs servers are docker container. |
workdir= // default log/work directory is .spillbox directory in $HOME | of the service account which runs the server. User can configure this to any NFS directory, instead of default $HOME. |
batch_command_allowed= // default is ALL | This should be either left "" ( allow all) or filled with set of commands, separated by spaces. These are usually job manager's batch commands like sinfo, qsub, bsub, nc. Please provide full path also. |
docker_option = // default is "" | Instead of users providing options through "docker run -docker_option" for each command line, this adds globally for all commands. |
uge_dockerized_cluster = // default is false | UGE is configured to spin Spillbox docker. |
lsf_dockerized_cluster = // default is false | LSF is configured to spin Spillbox docker. Please see spillbox_lsf_docker_app_profile_configuration.docx. |
docker_registry_port = // default is 5000 | Port used by Spillbox docker registry server. |
server_base_image = // default is docker_base_image | If NFS servers are configured to use container then container image to be used by it. |
remote_command_allowed = // default is "", list seperated by spaces | <remote_cmd> in "splrun run -remote <remote_cmd>" will run by server with server userid, not non-prem user's userid. |
rootless_docker == // default is false | Enable rootless docker. |
nameserver = // default is nameservers on spillbox server host | List of nameservers seperated by spaces, to be used in container. |
uge_docker_option = // default is "" | In case of UGE spawning docker container, extra docker option to be supplied. |
setup_bapp = // default is false | Set this to true if the spillbox specific lsf bapp setup guide needs to be enabled. |
bapp_name = // default is the image tag | Set this to the name of lsf application profile. This in effect if setup_bapp is true. |
bapp_image_name= // default is none | Set this to the docker image that has to be used. This in effect if setup_bapp is true. |
lsf_cluster_name= // required, no default | Set this to the lsf clustername that is needed. This in effect if setup_bapp is true. |
nc_setup_bapp = // default is false | Set this to true if the spillbox specific nc bapp setup guide needs to be enabled. |
nc_bapp_name = // default is none | Set this to the name of nc resource name. This in effect if nc_setup_bapp is true. |
nc_bapp_image_name = // default is none | Set this to the docker image that has to be used. This in effect if nc_setup_bapp is true. |
nc_vov_swd= // default is none | Set this to the main vov server's swd path. This in effect if nc_setup_bapp is true. |
nc_vov_project= // default is none | Set this to the vovproject for which the setup to be done. This in effect if nc_setup_bapp is true. |
checkpoint_objstore= // true/false | Provide a object store file system (juicefs with S3) to keep the checkpoint data To enable object store (To enable object store, set these variable in spillbox_config: checkpoint_objstore, checkpoint_image_dir, checkpoint_bucket, checkpoint_objport ) |
checkpoint_image_dir= // no default | Provide a world writable NFS directory with enough space for checkpoint to keep data in this directory. |
checkpoint_bucket= // (only alphabet, number and - are allowed, and the length should be 3 to 63 characters) | Bucket used for storing juicefs data, only empty buckets is allowed to be configured for juicefs storage If bucket doesn't exist, new bucket will be created. |
checkpoint_objport= // port number | Currently used fo redis database, necessary for juicefs to track checkpoint metadata |
backup_bucket= // object-store URL for project archive/restore | (Different from checkpoint_bucket above which is used by JuiceFS for metadata storage) Cloud storage location where project archives are saved and restored. Format depends on the cloud provider: AWS: backup_bucket=s3://<bucket-name> e.g. backup_bucket=s3://spillboxbackup Auth: instance IAM role with s3:GetObject / s3:PutObject / s3:ListBucket / s3:DeleteObject on the bucket. Azure: backup_bucket=https://<storage_account>.blob.core.windows.net/<container> e.g. backup_bucket=https://myaccount.blob.core.windows.net/spillboxbackup Auth: managed identity — do NOT include a SAS token in the URL. The VM managed identity must have the "Storage Blob Data Contributor" role assigned on the target container. GCP: backup_bucket=gs://<bucket-name> e.g. backup_bucket=gs://spillboxbackup Auth: service account attached to the instance with storage.objects.create / storage.objects.get / storage.objects.list / storage.objects.delete on the bucket. |
hybrid_mount_paths = "" // list of mount paths separated by spaces | Enable fully synchronized data sync between on-premise and Cloud for set of mount paths hierarchically. This is equivalent to enabling -download-hier, missing-hier and sync-hier in the snapshot option for these directories. Special keyword "ALL" enables globally on all mount paths. |
"hybrid_mount_paths_exclude" = "" // list of mount paths separated by spaces | Diasble fully synchronized data sync between on-premise and Cloud for set of mount paths hierarchically. This will be applied after hybrid_mount_paths Environment variable SPILLBOX_HYBRID_MOUNT_PATHS_EXCLUDE also does the same. |
boostmode= // default is false | Set this to true to enable boostmode |
boost_server_cache_size="" // default will be 500M, possible values K|M|G | Configures the cache size on the server. Setting this variable will allocate the specified amount of host RAM for the spillbox project |
boost_client_cache_dir="" // default will be /var/cache/fscache | Configures the cache directory on the client. |
"wan_timeout" = 900 // set WAN timeout in seconds ( 0 for no timeout ) | |
max_file_size_gb= // default is unset, no limit | Caps the size of any single file a job writes. Whole GB values only. A file that hits the cap fails immediately and is left truncated near that size — this limits one file at a time, not a job's total disk usage. Set it above your largest known legitimate file, or that file will fail too. |
job_monitor= // default is false | Set this to true to launch job_monitor.sh alongside the job. It watches for the job's NFS server/mount going unreachable and force-kills the job's container so it doesn't hang around as an unkillable zombie. |
max_project=0 // default is 0, unlimited | Caps the number of concurrently alive project hubs. Once that many are alive, a new "start" hub request is rejected (HTTP 503) with "Project limit reached (max N). Destroy an existing project before applying a new one." until an existing one is destroyed. |
admin_users="" // default is unset | Space-separated list of usernames allowed to log in to the web GUI as admin. If set, only these usernames may do so (each still authenticated against their own Linux/sudo password). If unset, only the account that owns the spld process itself can log in as admin. |
gui_port=8888 // default is 8888 | Port for the web GUI (spld –gui-only), separate from the main server port above. The GUI is started by default alongside the main server; this only needs to be set to use a non-default port. |
cron_lockfile="" // default is unset (no scheduler) | Absolute path of the lock file of the crontab scheduler, in a directory the user running run_server.sh can write to. If set, the web GUI spld (spld –gui-only) runs the jobs in <cron_lockfile>.crontab on a schedule, for example the hold queue to send queue move (see doc/BMOD_WORKFLOW.md). The first time, the shipped etc/spillbox_cron.crontab.example is copied there with every entry commented out, so nothing runs until a # is taken off a line. Only one scheduler runs at a time per lock file. |
spld_systemctl="" // default is unset (plain background process) | Set to "yes" to run spld as a systemd service instead of a plain nohup'd background process. |
mcp_server=true // default is true (when the MCP server code is installed) | Set to true (also accepts yes/on/1) to start an MCP (Model Context Protocol) server alongside spld, so an AI assistant (e.g. Claude Code) can drive Spillbox project actions directly. |
mcp_server_dir="" // default: <install_dir>/mcp, or ~/spillbox-mcp-vpc | Directory containing the MCP server's app.py (or vpc/app.py). Only needed if the MCP server code isn't in one of those default locations. |
mcp_server_port=8000 // default is 8000 (8001 for run_cloud_server.sh) | Port the MCP server listens on. |
mcp_server_bind=127.0.0.1 // default is 127.0.0.1 (loopback only) | Address the MCP server binds to. At the loopback default it's reachable only by something that already has a shell on the host, and needs no credential of its own. Widening this means anything that can reach the port can call every MCP tool, including destroying a project — every call is still recorded by address in the MCP server's own audit.log regardless. Only read by run_cloud_server.sh; with run_server.sh the bind comes from SPILLBOX_MCP_BIND (default 0.0.0.0). |
Phase 2 Steps
Can be run as any user if docker orchestration does not need user to be in docker group
- Run the spillbox server on any machine through LSF/ssh:
bash
terraform/run_server.sh [-kill] [-p <portnumber>] <tag>
This starts spillbox server and returns SPILLBOX_SERVER value as host:port. Clients onprem should set this value and SPILLBOX_VPN=1.