LSF Job Modification Workflow (bmod)

Spillbox LSF Job Modification Workflow (bmod)

Overview

This workflow lets you take an existing LSF job that was submitted on the local (client) cluster and redirect it to run on the remote Spillbox cluster — without the user having to resubmit. The job is modified in-place using bmod, transparently wiring in Spillbox’s pre/post exec hooks, snapshot, and download steps.

Two scripts drive this:

Prerequisites

• LSF must be sourced (LSF_ENVDIR set)
• bsub, bmod, bresume must be in PATH
• A Spillbox project config file must exist on both the client side and the server side — see Client vs. server config files below for the two paths and which values each one needs
• The operator must have sudo access to run commands as other users (or run as the same user)

Client vs. server config files

Two separate copies of the same-named config file (spillbox_singularity_config.<project>) are needed — one in each data center — and each only needs the values that data center’s scripts actually use.

Client side (where you run modify_lsf_job.sh)

Path: terraform/etc/spillbox_singularity_config.<project>

export SPILLBOX_SERVER=S10.10.10.10:8080:C192.168.1.1:8080
export SPILLBOX_SITE=mysite
export SPILLBOX_REMOTE_WORKDIR=/workdir

Used by: modify_lsf_job.sh only. The client side never actually runs a job’s container — it just submits jobs and tells them where to go, so it only needs to know which server to burst to (SPILLBOX_SERVER), which site this is (SPILLBOX_SITE), and the remote working directory (SPILLBOX_REMOTE_WORKDIR).

Server side (where the job actually runs)

Path: $SPILLBOX_REMOTE_WORKDIR/.spillbox/sites/local/<SITE>/terraform/etc/spillbox_singularity_config.<project>

export SPILLBOX_SERVER=S10.10.10.10:8080:C192.168.1.1:8080
export SPILLBOX_SITE=mysite
export SPILLBOX_SINGULARITY_EXECUTABLE=/workdir/singularity_wrapper
export SPILLBOX_HYBRID_MOUNT_PATHS=ALL
export SPILLBOX_HYBRID_OPTION="-download"
export SPILLBOX_REMOTE_WORKDIR=/workdir
# Other optional variables (see Environment Variables Reference below for the full list)
#export SPILLBOX_DOWNLOAD_INTERVAL=10
#export SPILLBOX_HOLD_EXCLUDE_SIZE=1024

Used by: singularity_preexec.sh, singularity_jobstarter.sh, singularity_postexec.sh — the three scripts that actually run when a job executes in its container, so this copy needs the extra runtime/mount settings the client-side copy doesn’t.

Note: “client” and “server” are roles for a given burst direction, not fixed machine identities — if jobs ever burst the other way, the roles and the S/C prefixes in SPILLBOX_SERVER flip accordingly.

Quick example

# 1. User submits job on hold
bsub -H -q normal -o /results/myjob.out myjob.sh
# Job <54321> is submitted to queue <normal>.

# 2. Operator modifies it to run via Spillbox
terraform/bin/modify_lsf_job.sh \
  -project default \
  -localq normal \
  -sendq sendq \
  54321

# Output:
# Processing Input: 54321
# SUCCESS: Job 54321 updated.
# USER: jsmith
# Job <54322> is submitted to queue <normal>.   <- download job

# 3. If something goes wrong and you want to revert
terraform/bin/reverse_modify_lsf.sh -localq normal 54321
# SUCCESS: 54321 reversed

That’s the whole workflow in miniature. The sections below walk through each step in detail, and the Reference section at the bottom covers every flag, environment variable, and internal detail for when you need to look something up.

Workflow

Step 1 — User submits job in HOLD state

The user (or an automated system) submits the job to the local queue with -H to hold it before it runs:

bsub -H -q normal myjob.sh

The job sits in PSUSP/PEND state — it will not run until resumed.

Step 2 — Operator runs modify_lsf_job.sh

Script location: terraform/bin/modify_lsf_job.sh

terraform/bin/modify_lsf_job.sh \
  -project <projectname> \
  -localq <local_queue> \
  -sendq <send_queue> \
  [-queue <queue_names>] \
  [<job_ids_or_usernames>]

This is the minimum you need: a project name, a local queue, a remote send queue, and the job IDs (or usernames) to modify. There are more optional flags for less common cases — interactive jobs, archiving for later resubmission, delaying the start, and so on — covered in the full flag reference below.

Job targets (comma-separated, mixed is OK):

# By job ID
terraform/bin/modify_lsf_job.sh -project default -localq normal -sendq sendq 12345

# By job array
terraform/bin/modify_lsf_job.sh -project default -localq normal -sendq sendq "12345[1-10]"

# By username (modifies all jobs for that user in the local queue)
terraform/bin/modify_lsf_job.sh -project default -localq normal -sendq sendq jsmith

# By queue name (modifies EVERY job currently in that queue, across ALL
# users — not just the one running the script). A queue must be given
# with -queue; a bare name is always taken as a username.
terraform/bin/modify_lsf_job.sh -project default -localq normal -sendq sendq -queue batchq

# Multiple, mixed (comma-separated, no spaces)
terraform/bin/modify_lsf_job.sh -project default -localq normal -sendq sendq 12345,67890,jsmith

# Job IDs or usernames together with a queue
terraform/bin/modify_lsf_job.sh -project default -localq normal -sendq sendq -queue batchq 12345,jsmith

A queue given with -queue modifies every job currently in that queue.

Step 3 — Job runs on remote cluster

Spillbox’s singularity_preexec.sh and singularity_postexec.sh handle:

• Staging input data to the remote site
• Running the job inside a container (Singularity/Docker)
• Writing output/error back to the original paths

The download job then pulls results back automatically.

Automatic mode: the hold queue and the crontab

Instead of running modify_lsf_job.sh by hand, the customer can have Spillbox run it on a schedule. Users submit jobs to a hold queue, and a crontab entry moves every job in that queue to the send queue. Users only change the queue name; nothing else.

Turning it on. It is off by default.

1. In etc/spillbox_config set cron_lockfile to an absolute path the user running run_server.sh can write to, for example cron_lockfile="/var/lib/spillbox/cron.lock".
2. Start (or restart) the server with run_server.sh. It copies etc/spillbox_cron.crontab.example to <cron_lockfile>.crontab the first time, with every entry commented out.
3. Edit <cron_lockfile>.crontab: take the # off the modify_lsf_job.sh line and set the project name, queues and schedule. The scheduler runs in the web GUI spld (gui.log), not the main one.

The crontab is an ordinary crontab: one job per line as <schedule> <command>, # starts a comment, NAME=value sets an environment variable for every job. The schedule is five fields (minute hour day-of-month month day-of-week) or @hourly, @daily; there is no seconds field. The customer decides how often by editing the schedule. A # in front of a line turns it off.

# every 5 minutes
*/5 * * * * sudo -n -u lsfadmin /opt/spillbox/terraform/bin/modify_lsf_job.sh -project amd -localq normal -sendq send_queue -queue hold

Changing it while running. The file is re-read every second; no restart is needed. A job that is still running when the file changes is not started a second time, and a run is skipped while the last one is still going. If an edit leaves a line that is not valid, the scheduler keeps the previous schedule and logs the line.

Times are UTC; set CRONWRAPPER_TZ=America/Los_Angeles (or another zone) in the environment of spld to change that. The user running spld needs passwordless sudo to the LSF account if the entry uses sudo -n -u. Every run (start, finish time, exit code, output) is logged in gui.log. More detail: pkg/cronwrapper/README.md in splpipe.

Reversing a modification

If a modified job gets stuck in pending state on the remote cluster and you want to undo the modification and bring it back to its original local state:

Script location: terraform/bin/reverse_modify_lsf.sh

terraform/bin/reverse_modify_lsf.sh -localq <local_queue> <job_ids>

Example:

terraform/bin/reverse_modify_lsf.sh -localq normal 12345,67890

reverse_modify_lsf.sh reads the encoded original values from the job’s current pre-exec command (where modify_lsf_job.sh stored them) and uses bmod to restore:

• Original job name
• Queue set to the -localq value
• Original working directory
• Original output/error files
• Original pre/post exec commands
• Original resource requirements
• Original dependencies
• Original app profile

It skips internal Spillbox jobs (spillbox_snapshot, spillbox_download_*) automatically.

Reference

modify_lsf_job.sh flag reference

Skipping users with recent jobs — SPLAPP_SUBMIT_TIME. Set this in the environment of the run (or in the project config file) to a job age, not a clock time: seconds, or a number with an m, h or d suffix (3600, 30m, 2h, 1d). If any of a user’s jobs selected in this run was submitted more recently than that, all of that user’s jobs are skipped — including their older ones — and no snapshot, sleep or download job is submitted for them. Other users are modified as usual, so only users whose jobs are all older than the age are processed. Unset or empty = no limit (default); any other value stops the script with an error. modify_slurm_job.sh reads it the same way.

SPLAPP_SUBMIT_TIME=2h terraform/bin/modify_lsf_job.sh -project default -localq normal -sendq sendq -queue holdq
# USER=jsmith has a job submitted less than SPLAPP_SUBMIT_TIME=2h ago; skipping all their jobs
# Skipping 12345 : user jsmith has a job submitted less than SPLAPP_SUBMIT_TIME=2h ago

Separate mode — resubmitting an archived job (not part of the normal modify flow above; mutually exclusive with <job_ids_or_usernames> and -queue):

terraform/bin/modify_lsf_job.sh -project <projectname> -resubmit <job_id> [-hold]

Options updated by modify_lsf_job.sh on the job

When modify_lsf_job.sh runs bmod, it updates the following on each job:

All original values (name, pre/post exec, output, cwd, dependencies, resource requirements, app) are encoded into the pre-exec command so they can be restored by reverse_modify_lsf.sh.

What modify_lsf_job.sh does internally

For each job it processes:

1. Reads existing job state via bjobs:
2. Job name, pre/post exec commands, output/error files, working directory
3. Existing dependencies (bjdepinfo)
4. App profile, resource requirements

5. Whether it’s interactive

6. Submits a snapshot job (spillbox_snapshot) — this runs first on the remote cluster and snapshots the project filesystem so the job has the right data when it starts. Only one snapshot job is submitted per user per run (shared across all that user’s jobs).

7. Encodes original job state — all original values (name, pre/post exec, output, cwd, dependencies, resource requirements, app) are encoded as environment variables inside the new pre-exec command string, so they can be recovered later by reverse_modify_lsf.sh.

8. Calls bmod to modify the held job:

9. Changes queue to sendq (remote cluster)
10. Replaces pre-exec with Spillbox’s singularity_preexec.sh (carries all encoded original values)
11. Replaces post-exec with Spillbox’s singularity_postexec.sh
12. Redirects output/error to remote log directory: $SPILLBOX_REMOTE_WORKDIR/.spillbox/customers/<user>/<project>-<site>/stacks/modules/docker-nfs/install/spillbox_splfs/LOG/LSFLOG/
13. Sets working directory to /tmp (actual workdir restored at runtime via SPLAPP_EXEC_CWD)

14. Adds dependency on the snapshot job

15. Calls bresume — releases the job so it can now run via Spillbox.

16. Submits a download job (spillbox_download_<project>) — runs after all modified jobs complete, downloads results back to the local cluster.

Environment Variables Reference

Variables set in the project config file (terraform/etc/spillbox_singularity_config.<projectname>):

Example (terraform/etc/spillbox_singularity_config.<projectname>):

# --- Required ---

# Single server
export SPILLBOX_SERVER=10.10.10.10:8080

# Two-server setup: S<remote_ip>:<port>:C<client_ip>:<port>
# S = remote/server-side IP, C = local/client-side IP
# Use when the Spillbox server has separate IPs for each network
# export SPILLBOX_SERVER=S10.10.10.10:8080:C192.168.1.1:8080

export SPILLBOX_SITE=mysite                   # unique on-prem site ID
export SPILLBOX_REMOTE_WORKDIR=/workdir       # cloud-side job working directory, required

# --- Container executable: set the one matching runtime_container ---

export SPILLBOX_SINGULARITY_EXECUTABLE=/workdir/singularity_wrapper   # compiled setuid wrapper, used when runtime_container=singularity
export SPILLBOX_DOCKER_EXECUTABLE=/workdir/docker_wrapper              # compiled setuid wrapper, used when runtime_container=docker

# --- Hybrid mount behavior ---

export SPILLBOX_HYBRID_MOUNT_PATHS=ALL        # which on-prem paths get hybrid-mounted ("ALL" = everything)
export SPILLBOX_HYBRID_OPTION="-download"     # fetch data in, not just reference it

# --- Optional: uncomment and adjust only if the defaults don't fit ---

#export SPILLBOX_DOWNLOAD_INTERVAL=10                         # "hold" = hold until released, a number = hold that many minutes, unset/0 = no hold (default)
#export SPILLBOX_LARGEDIR_OPT=100000                          # entry-count threshold for "large directory" handling
#export SPILLBOX_LARGEDIR_ENTRY_SIZE=88                       # expected avg entry size hint for large-directory handling
#export SPILLBOX_LARGEDIR_PAT='/cache/sync[0-9]+/[^/]+'       # regex matching dirs that get large-directory handling
#export SPILLBOX_HOLD_EXCLUDE_PAT='/runpointsfile|/foo_\d+$'  # regex of files excluded from project snapshot/hold
#export SPILLBOX_HOLD_EXCLUDE_SIZE=1024                       # size (bytes) above which files are excluded from snapshot/hold
#export SPLAPP_SUBMIT_TIME=300                                # skip all jobs of a user with any job newer than this (seconds, or 30m/2h/1d)

Notes

• Jobs are processed in parallel — large batches of job IDs run concurrently.
• A queue name as input modifies every job currently in that queue, including other users’ jobs, not just the one running the script — the operator needs sudo rights to act as those other users. Each affected user gets their own separate snapshot/download job (never combined), and if a user doesn’t already have the given -project set up, it’s created for them on first use.
• If a user already has a spillbox_download_<project> job running, that job is skipped (incremental mode, currently disabled).
• Interactive jobs (bsub -I) require -spillboxlocalq and -spillboxsendq to be explicitly set.
• Spaces in resource requirements, pre/post exec, and dependencies are encoded as -s_b-o_x- internally to survive shell quoting through bmod.
• The SNAPSHOT_TIME is stamped at the start of the modify_lsf_job.sh run and shared across all jobs in that batch so they all refer to the same snapshot.
• Hold time is a minimum, not exact — files release within N to N+1 minutes (checked once a minute).