LSF Docker Application Profile

Configuring LSF Docker Application profile for Spillbox flow

Use model

Set this environmental variable

SPLAPP_BATCH_RUN_OPTION="-app <application name (splapp)>"

splapp is the name of the app configured in lsb.applications file, an example can be found at the end of this doc

Invoke bsub using spillbox bsub wrapper : <spillbox_installation>/lsf/bsub -o <path-to-outputfile> -e <path-to-errorfile> <Your Command>

Assuming normal LSF installation is already done. Source the lsf setup like below example

do ". /usr/share/lsf/conf/profile.lsf"

Main URL : https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbapplications

LSF top dir is /usr/share/lsf

The path may vary depending on where LSF is installed.

Configuration steps for lsf application (The files which are to be edited, will have steps to follow at the beginning of the file) The Docker Engine, Version 1.12, or later required and should be accessible in the lsf master host

1. Edit the <lsf top dir>/conf/lsf.conf file and configure the following parameters:

LSF_PROCESS_TRACKING=Y

LSF_LINUX_CGROUP_ACCT=Y

LSB_RESOURCE_ENFORCE="cpu memory"

2. Edit the <lsf top dir>/conf/lsf.shared file and configure a new Boolean resource named docker.

…

Begin Resource

RESOURCENAME  TYPE    INTERVAL INCREASING CONSUMABLE DESCRIPTION  # Keywords

...

   docker     Boolean ()       ()         ()         (Docker container)

...

End Resource

3. Edit the <lsf top dir>/conf/lsf.cluster.<clustername> file and attach the docker Boolean resource to the LSF server host that is running Docker Engine. The format of this file and lsf.shared file may vary based on the version of LSF.

…

Begin Host

    HOSTNAME  model  type  server  r1m  mem  swp  RESOURCES

    ...

    host1     !      !     1       3.5  ()   ()   (docker mg)

    ...

End Host

After making any changes to lsf.shared & lsf.cluster.<clustername>, run the following commands: "badmin reconfig", "lsadmin reconfig" to reconfigure LIM and "badmin mbdrestart" to restart mbatchd

4. Check if /usr/bin/python is linked to the proper Python interpreter.

In some versions of Linux (such as RHEL 8.x), there is no /usr/bin/python executable file on the host. Since the execution driver uses /usr/bin/python to execute the Python script, you must create a link from /usr/bin/python linking to the available Python executable file (both python2.x and python3.x are available).

5. Edit the <lsf top dir>/conf/lsbatch/<cluster name>/configdir/lsb.applications to define the CONTAINER parameter for the application profile or queue to run Docker jobs.

CONTAINER=docker[image(image_name) options(docker_run_options)]

Example:

For sequential jobs:

CONTAINER=docker[image(ubuntu) options(--rm)]

For parallel jobs:

CONTAINER = docker[image(ubuntu)  options(--rm --net=host --ipc=host)]

6. Edit the <lsf top dir>/conf/lsbatch/<clustername>/lsb.applications and define the EXEC_DRIVER parameter for the application profile or queue to run Docker jobs. The docker-*.py should be owned by the user running the container/docker and should have specific permission 700 or 500 . (Original scripts can be found under, similar to this /<lsf top dir>/10.1/linux2.6-glibc2.3-x86_64/etc/docker-starter.py)

EXEC_DRIVER=context[user(lsfadmin)]

starter[/path/to/spillbox installation/etc/docker-starter.py]

controller[/path/to/spillbox installation/etc/docker-control.py]

monitor[/path/to/spillbox installation/etc/docker-monitor.py]

“lsfadmin” should be replaced with the userid or username with docker permission.

An example entry for lsb.applications

Assuming the Spillbox installation is in /splbox/ and lsf top dir is /usr/share/lsf

Begin Application

NAME = splapp

CONTAINER = docker[image(10.0.0.74:5000/spillbox) options(--rm --net=host --privileged -e SPILLBOX_SITE=ip-10-0-0-74.us-west-1.compute.internal -v /splbox/terraform/bin:/spillbox -v /cache_dir:/spillbox_workdir -v /fsx/tools/sbx_lsf:/fsx/tools/sbx_lsf  --entrypoint=/spillbox/bapp_entrypoint.sh)]

DESCRIPTION = Docker User Service

EXEC_DRIVER=context[user(root)] starter[/splbox/etc/docker-starter.py] controller[/splbox/etc/docker-control.py] monitor[/splbox/etc/docker-monitor.py]

End Application

7. Edit the <lsf top dir>/conf/lsbatch/<clustername>/lsb.params to define the spool directory, which should be kept to /tmp

JOB_SPOOL_DIR = /tmp

Run the following commands: "badmin reconfig", "lsadmin reconfig" to reconfigure LIM and "badmin mbdrestart" to restart mbatchd , to make your changes in effect.

To validate the bapp application can access docker resources , run bapp -l <application name> check the output has the associated docker CONTAINER details: :$bapp -l splapp

APPLICATION NAME: splapp — Docker User Service

PARAMETERS:

CONTAINER: docker[image(10.0.0.74:5000/spillbox) options(–rm –net=host –privileged -e SPILLBOX_SITE=ip-10-0-0-74.us-west-1.compute.internal -v /splbox/terraform/bin:/spillbox -v /cache_dir:/spillbox_workdir -v /fsx/tools/sbx_lsf:/fsx/tools/sbx_lsf –entrypoint=/spillbox/bapp_entrypoint.sh)]

If your output does not have the CONTAINER section. Ensure the lsf.shared and lsf.cluster.clustername has the appropriate docker resource entry and access are provided.

Spillbox support for LSF multicluster

LSF has to be configured with docker and application profile, refer to

spillbox_lsf_docker_app_profile_configuration section in this file has steps to create a docker application profile. Assuming One LSF cluster on the cloud and another one on the on-prem has been created and connected using LSFconnect.

If user wants to run their jobs in both on-prem and the cloud, following two scenarios

describe the setup required for Spillbox to work with LSF multicluster.

Command line is the same for both scenarios

Usage: Assuming the Spillbox client distribution is setup in /splbox

Export the LSF_ENVDIR path(which will be part of LSF setup sourcing file) and

        SPLAPP_BATCH_RUN_OPTION = “-app <application name>”

/splbox/lsf/bsub -multi_cluster <lsf_option> <usrcmd>

You can use this application profile name when you are using Spillbox bsub wrapper (found

in lsf directory in the Spillbox client distribution).

Scenario 1

Run all the jobs on a docker container in the cloud and run as normal jobs on the on-prem (job runs without docker)

You need to configure your LSF with a docker application profile in the lsb.application on the cloud and on the on-prem you need to create normal application profile with the following contents.

The file exewrapper.sh used in the example setup can be found in the Spillbox client distribution/setup.

On-prem lsb.application should have

Begin Application

NAME = splapp

JOB_STARTER =/<full_path_to_/exewrapper.sh> %USRCMD

End Application

Cloud lsb.application should have

Begin Application

NAME = splapp

CONTAINER = docker[image(10.0.0.74:5000/spillbox) options(--rm --net=host --privileged -e SPILLBOX_SITE=ip-10-0-0-74.us-west-1.compute.internal -v /splbox/terraform/bin:/spillbox -v /cache_dir:/spillbox_workdir -v /fsx/tools/sbx_lsf:/fsx/tools/sbx_lsf  --entrypoint=/spillbox/bapp_entrypoint.sh)]

DESCRIPTION = Docker User Service

EXEC_DRIVER=context[user(root)] starter[/splbox/etc/docker-starter.py] controller[/splbox/etc/docker-control.py] monitor[/splbox/etc/docker-monitor.py]

End Application

Scenario 2

Run all the jobs in docker both on-prem and the cloud.

You need to configure the LSF with docker application profile in the lsb.application on both the cloud and on-prem.

Note: The Application name should be same both in the cloud and on-prem LSF cluster configuration. Spillbox shipped exewrapper.sh should be accessible on all job executing machines.

Cloud and on-prem lsb.application should have similar or same (path sometimes might change in the two clusters, if they are same then the lsb.application entries will be the same)

Begin Application

NAME = splapp

CONTAINER = docker[image(10.0.0.74:5000/spillbox) options(--rm --net=host --privileged -e SPILLBOX_SITE=ip-10-0-0-74.us-west-1.compute.internal -v /splbox/terraform/bin:/spillbox -v /cache_dir:/spillbox_workdir -v /fsx/tools/sbx_lsf:/fsx/tools/sbx_lsf  --entrypoint=/spillbox/bapp_entrypoint.sh)]

DESCRIPTION = Docker User Service

EXEC_DRIVER=context[user(root)] starter[/splbox/etc/docker-starter.py] controller[/splbox/etc/docker-control.py] monitor[/splbox/etc/docker-monitor.py]

End Application