Grid Orchestration — LSF

Docs / Administrator Guide / LSF / Grid Orchestration — LSF
Administrator Guide

Grid Orchestration — LSF

SPILLBOX GRID ORCHESTRATION support for IBM LSF standard Edition 10.1

Purpose

This is a user reference manual that allows customers to manage infrastructure deployment and set up the IBM Licensed LSF 10.1 workload manager with elasticity provisioning. SPILLBOX GRID ORCHESTRATION supports both AWS and Azure cloud providers.

Prerequisites

  • Cloud Credentials:
    • For Aws : AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
    • For Azure : ARM_TENANT_ID, ARM_SUBSCRIPTION_ID, ARM_CLIENT_ID, ARM_CLIENT_SECRET
    • For Google : GOOGLE_APPLICATION_CREDENTIALS, GOOGLE_PROJECT
  • Understand the supported deployment model before triggering orchestration.
  • For troubleshooting, refer to the questionnaire at the end of this document or contact Spillbox Support at support@spillbox.io.
  • Required ports to be opened inside your subnet for LSF autoscaling:
    • "22" "111" "443" "988" "2049" "5000" "6000" "6878" "6881" "6882" "6891" "7070-8090"
  • Azure CycleCloud specific prerequisites:

NOTE: Azure CycleCloud server and the cyclecloud CLI are automatically installed on the machine running this script during the node create step. They do not need to be pre-installed.

NOTE: A user-assigned managed identity is automatically created during cluster create and attached to the CycleCloud webserver, NFS, and compute nodes. This allows those VMs to authenticate to Azure (for objstore.sh and other operations) without needing explicit ARM_* credentials — equivalent to AWS instance profiles. The user running the script needs Microsoft.Authorization/roleAssignments/write (e.g. User Access Administrator or Owner on the resource group) to assign Contributor to this identity.

  • Run the script from the machine that will act as the CycleCloud hub. This machine is where CycleCloud server will be installed; the LSF master runs on the CycleCloud webserver node.
  • Log in to Azure CLI before running the script:
bash
$ az login

Verify with: az account show

  • Export service principal credentials as environment variables:
bash
export ARM_CLIENT_ID="<appId>"
export ARM_CLIENT_SECRET="<password>"
export ARM_SUBSCRIPTION_ID="<subscriptionId>"
export ARM_TENANT_ID="<tenantId>"

To create a service principal:

bash
$ az ad sp create-for-rbac --name "spillbox-sp" --role Contributor \
    --scopes /subscriptions/<ARM_SUBSCRIPTION_ID>
  • Pre-create the following Azure networking resources (the script does not create these):
    • Virtual Network (VNet) with a subnet SUBNET_ID format: /subscriptions/<sub>/resourceGroups/<rg>/providers/ Microsoft.Network/virtualNetworks/<vnet>/subnets/<subnet>
    • Network Security Group (NSG) attached to the subnet with required ports open SECURITY_GROUP_ID format: /subscriptions/<sub>/resourceGroups/<rg>/providers/ Microsoft.Network/networkSecurityGroups/<nsg>
  • Outbound internet access is required on the hub machine during cluster create to download:
    • cyclecloud-lsf project : github.com/Azure/cyclecloud-lsf/releases/3.3.1
    • LSF installer tarballs : s3.us-west-1.amazonaws.com/spillbox.io-internal
  • Additional config entries required in setup_spillbox_newclus.config for Azure CycleCloud:
SHARED_FILESYSTEM_SIZE=""   # (OPTIONAL, default 50) Size in GB for the CycleCloud built-in NFS share
NFS_CPU_LIMIT="2"           # CPU slots reserved on NFS node for LSF scheduling
MIN_NFS_COUNT=0             # Minimum number of NFS nodes (CycleCloud autoscale)
MAX_NFS_COUNT=3             # Maximum number of NFS nodes (not used on Azure; scale comes from the CycleCloud template)
MIN_COMPUTE_COUNT=0         # Minimum number of compute nodes
MAX_COMPUTE_COUNT=3         # Maximum number of compute nodes (not used on Azure; scale comes from the CycleCloud template)
  • Additional config files required for cluster create: config/spillbox_config # Spillbox config copied into CycleCloud cluster-init config/lsb.appilcations # LSF application profile copied into install scripts
  • Running objstore.sh on the CycleCloud webserver: ARM_RESOURCE_GROUP and ARM_SUBSCRIPTION_ID are auto-detected from Azure IMDS. Authentication uses the managed identity — no ARM_* credentials needed.
bash
		    $ /fsx_shared/spillbox_server/terraform/objstore.sh azure create

#********************************************************************

Steps to Build Spillbox Grid Orchestration

  1. Download and Extract Software
    • Download from: https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/latest/spill-dc.tar.gz
  1. Copy Configuration File and Edit
bash
$ cp terraform/grid_orchestration/rc-lsf/config/setup_spillbox_newclus.config.example terraform/grid_orchestration/rc-lsf/config/setup_spillbox_newclus.config
  • Edit the configuration file as per your site requirements.
  1. Required Configurations
    • Aws : Provide BASE_AMI_NAME, DOCKER_BASEAMI_ID, VPC ID, SUBNET ID, SECURITY_GROUP ID, REGION, ADMIN_USER, DOCKER_ADMIN_USER.
    • Azure : Provide BASE_AMI_NAME, DOCKER_BASEAMI_ID, VPC_ID, SUBNET_ID, SECURITY_GROUP_ID, RESOURCE_GROUP, REGION (display name e.g. "East US"), ADMIN_USER, DOCKER_BASEAMI_ID, SERVICE_USERID, NFS_INSTANCE_TYPE (must be NVMe-supported), SHARED_FILESYSTEM_SIZE (optional, default 50). NOTE: WEBSERVER_CREATE has no effect on Azure — CycleCloud always manages the scheduler. NOTE: NFS_TAG and COMPUTE_TAG are not supported on Azure.
    • Google : Provide AMI ID, DOCKER_AMI ID, VPC ID, SUBNET ID, REGION, ZONE, ADMIN_USER, DOCKER_ADMIN_USER.
  1. Provide the LSF License in the terraform/grid_orchestration/rc-lsf/config/lsf_std_entitlement.dat file
  1. Set Cloud Credentials
    • A list of required permissions is available at the end of this document.
  1. Run the Script
bash
$ ./terraform/grid_orchestration/rc-lsf/spillbox_gridorchestration.sh <aws|azure|google> -node <create> -cluster <create> -tag <tagname>
  • Populate webserver_create in setup_spillbox_newclus.config:
    • true: Creates two nodes (one Spillbox Server, one Image node).
    • false: Creates one Image node; current node acts as Spillbox Webserver.
  • The script creates required nodes and displays credentials.
  1. Access Webserver Machine
    • A directory named /fsx_shared/spillbox_server will be available.

To Delete the Lsf autoscaling

  1. To delete the cluster and node
bash
$ ./terraform/grid_orchestration/rc-lsf/spillbox_gridorchestration.sh <aws|azure|google> -node <delete> -cluster <delete> -tag <tagname>
  1. To delete the cluster alone
bash
        $ ./terraform/grid_orchestration/rc-lsf/spillbox_gridorchestration.sh <aws|azure|google> -cluster <delete> -tag <tagname> 

#********************************************************************

Configuration File Entries (setup_spillbox_newclus.config)

#********************************************************************

-----------------------Region Configuration------------------------
REGION=""
ZONE=""                                    # (For Google only)

-----------------------Network Configuration-----------------------
VPC_ID=""
SUBNET_ID=""
PUBLIC_SUBNET_CIDR=""                   # (OPTIONAL)
SECURITY_GROUP_ID=""                    # (Not needed for Google)
RESOURCE_GROUP=""                       # (For Azure only)

-----------------------Machine Configuration-----------------------
BASE_AMI_NAME=""
ADMIN_USER=""
DOCKER_ADMIN_USER=""
DOCKER_BASEAMI_ID=""
KMS_KEY_ID=""                           # (For encrypted AMI/Image in AWS & Google)
SERVICE_USERID=""
    # Format: uid:gid:username  e.g. SERVICE_USERID="1001:1001:srv-splfs"
    # Azure: default admin user (azureuser) takes UID/GID 1000.
    #        Use a UID/GID that does not conflict with existing users on the base image.
MODEL="small|medium|large"

------------------------Webserver Configuration--------------------
WEBSERVER_CREATE="true|false"
WEBSERVER_INSTANCE_TYPE=""
WEBSERVER_VOLUME_SIZE=""

-------------------------NFS Configuration-------------------------
NFS_INSTANCE_TYPE=""                    # Specify only nvme instance type.
MIN_NFS_COUNT=0
MAX_NFS_COUNT=3
NFS_TAG="nfs_node=1,nfs_queue=1,queue_name=sbox_nfs_queue"
NFS_VOLUME_SIZE=""

-------------------------Compute Configuration----------------------
COMPUTE_INSTANCE_TYPE=""
COMPUTE_VOLUME_SIZE=""
MIN_COMPUTE_COUNT=0
MAX_COMPUTE_COUNT=3
COMPUTE_TAG="compute_node=1,compute_queue=1"

--------------------Spot Configuration (Optional)-------------------
SPOT_INSTANCE_TYPE=""
MAX_SPOT_COUNT=50
MAX_SPOT_PRICE=0.5
SPOT_ALLOCATION_STRATEGY="LowestPrice"
SPOT_TAG="spot_node=1,spot_queue=1"

----------------------Container Configuration-----------------------
SPILLBOX_CONTAINER="docker"             # Supports "docker" (default) and "singularity"
SINGULARITY_SIFPATH=""                  # (Only if using singularity)

----------------------Boot-Time Script Execution---------------------
SETUP_SCRIPT=""
#********************************************************************

FAQ's

1. What roles need to be assigned for the cloud credentials?

  • The cloud credentials should allow the creation/destruction of VPC, subnet, security group, FSx (AWS), EC2, S3, and resource group (Azure).

2. Can users bring their own AMI?

  • Yes, users can bring their own AMI to create VMs, VPC, subnet, security group, and even configure the root volume size of the nodes being created.

3. Can we sync up a Docker image at runtime (after creating the LSF cluster)?

  • Add your Docker image and registry information to /fsx_shared/lsf_shared_directory/tools/userdata.json. Dynamic nodes will pull the images at creation time. Note: This applies only to dynamic nodes, not static nodes. The registry must allow image pulls without authentication.

Example:

{
  "registries": ["10.0.0.123:5000", "ghcr.io"],
  "images": ["10.0.0.123:5000/image1", "ghcr.io/image2"]
}
  1. How do I enable checkpoint/restore on the client side?
    • Set the following environment variables:
SPLAPP_CR_INTERVAL=1
SPLAPP_CR_ENABLE=1

5. What should I do if I accidentally close or lose the terminal that displayed all the IP information?

  • No problem. All the essential setup details, including IP addresses and configuration information, are saved in a setup_info file located inside the directory with your tagname.

Required IAM Roles for Cloud Providers

1. AWS (Amazon Web Services)

  • IAM roles required on AWS > AmazonS3FullAccess > Specific EC2 access > Specific FSx access > AmazonSSMManagedInstanceCore (Required for SSM connectivity) > EC2InstanceProfileForImageBuilder (Required for EC2 Image Builder)
  • Specific IAM roles (JSON format):
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "ec2:AttachVolume",
                "ec2:CreateImage",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:DeleteKeyPair",
                "ec2:DeleteVolume",
                "ec2:DeregisterImage",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInstanceAttribute",
                "ec2:DescribeInstanceCreditSpecifications",
                "ec2:DescribeInstanceTypeOfferings",
                "ec2:DescribeSubnets",
                "ec2:DescribeImages",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeTags",
                "ec2:DescribeVolumes",
                "ec2:DetachVolume",
                "ec2:ImportKeyPair",
                "ec2:ModifyInstanceAttribute",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:DeleteSnapshot",
                "ec2:TerminateInstances",
                "ec2:RequestSpotFleet",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:DescribeSpotFleetInstances",
                "ec2:DescribeSpotFleetRequests",
                "ec2:CancelSpotFleetRequests",
                "ec2:ModifySpotFleetRequest",
                "imagebuilder:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Statement2",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Statement3",
            "Effect": "Allow",
            "Action": [
                "fsx:CreateFileSystem",
                "fsx:TagResource",
                "fsx:DescribeFileSystems",
                "fsx:DeleteFileSystem",
                "fsx:DeleteBackup",
                "fsx:DeleteSnapshot",
                "fsx:DeleteStorageVirtualMachine",
                "fsx:DeleteVolume",
                "fsx:DeleteFileCache",
                "fsx:DeleteDataRepositoryAssociation",
                "fsx:DeleteResourcePolicy"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Statement4",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:UpdateRole",
                "iam:ListRoles",
                "iam:GetRole",
                "iam:AttachRolePolicy",
                "iam:DetachRolePolicy",
                "iam:CreatePolicy",
                "iam:DeletePolicy",
                "iam:ListPolicies",
                "iam:GetPolicy",
                "iam:PutRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:PassRole",
                "iam:ListAttachedRolePolicies",
                "iam:ListInstanceProfilesForRole",
                "iam:ListPolicyVersions",
                "iam:ListRolePolicies",
                "iam:GetPolicyVersion",
                "iam:DeletePolicyVersion",
                "iam:CreateInstanceProfile",
                "iam:GetInstanceProfile",
                "iam:DeleteInstanceProfile",
                "iam:AddRoleToInstanceProfile",
                "iam:RemoveRoleFromInstanceProfile"
            ],
            "Resource": [
                "arn:aws:iam::************:role/spillbox-*",
                "arn:aws:iam::************:policy/spillbox-*",
                "arn:aws:iam::************:instance-profile/*"
            ]
        }
    ]
}

2. AZURE (Microsoft Azure)

  • Roles needed for Application account on Azure > Microsoft.Storage: FullAccess > Microsoft.Network: FullAccess > Microsoft.Compute: FullAccess > Microsoft.Resources/subscriptions/resourceGroups: FullAccess > Microsoft.Resources/deployments: FullAccess > Microsoft.Insights/alertRules: FullAccess > Microsoft.Insights/diagnosticSettings: FullAccess > Microsoft.Support: FullAccess > Microsoft.ResourceHealth/availabilityStatuses: ReadAccess > Microsoft.Authorization/policyExemptions: Write + Delete (OPTIONAL)

NOTE: Microsoft.Authorization/policyExemptions write and delete are optional. Only required if your subscription has a subscription-level auto-shutdown policy that would shut down cluster VMs at a scheduled time. The script automatically attempts to create a policy exemption for the cluster resource group — if the permission is not present it skips silently and has no impact. Most subscriptions do not have such a policy and this permission is not needed.

You can also refer lsf resource connector for azure document: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=providers-configuring-microsoft-azure-lsf-resource-connector

  • Specific IAM roles (JSON format):
"permissions": [
        {
            "actions": [
                "Microsoft.Storage/*",
                "Microsoft.Network/*",
                "Microsoft.Compute/*",
                "Microsoft.Authorization/*/read",
                "Microsoft.Authorization/policyExemptions/write",
                "Microsoft.Authorization/policyExemptions/delete",
                "Microsoft.Resources/subscriptions/resourceGroups/*",
                "Microsoft.Resources/deployments/*",
                "Microsoft.Insights/alertRules/*",
                "Microsoft.Insights/diagnosticSettings/*",
                "Microsoft.Support/*",
                "Microsoft.ResourceHealth/availabilityStatuses/read"
            ],
            "notActions": [],
            "dataActions": [],
            "notDataActions": []
        }
    ]

The ready-made custom role definition including all required permissions is provided in terraform/grid_orchestration/rc-lsf/custom_role.json.

To create and assign (first time):
Set AssignableScopes in custom_role.json to /subscriptions/<your SUBSCRIPTION_ID> first.
bash
$ az role definition create \
    --role-definition terraform/grid_orchestration/rc-lsf/custom_role.json
$ az role assignment create \
    --assignee <ARM_CLIENT_ID> \
    --role "LSF Resource Connector" \
    --scope /subscriptions/<SUBSCRIPTION_ID>

To update after changes to custom_role.json:

bash
$ az role definition update \
    --role-definition terraform/grid_orchestration/rc-lsf/custom_role.json

3. GCP (Google Cloud Platform)

  • IAM roles and permissions required for service account on GCP >Read-Write on Compute Engine resources. >Full control on buckets and objects.
  • Specific roles are:
compute.disks.create
compute.disks.useReadOnly
compute.firewalls.create
compute.firewalls.delete
compute.globalOperations.get
compute.images.create
compute.images.createTagBinding
compute.images.delete
compute.images.deleteTagBinding
compute.images.get
compute.images.list
compute.images.useReadOnly
compute.instanceTemplates.create
compute.instanceTemplates.delete
compute.instanceTemplates.get
compute.instanceTemplates.useReadOnly
compute.instances.create
compute.instances.delete
compute.instances.get
compute.instances.setMetadata
compute.instances.useReadOnly
compute.machineImages.useReadOnly
compute.projects.get
compute.projects.setCommonInstanceMetadata
compute.subnetworks.get
compute.subnetworks.use
compute.subnetworks.useExternalIp
compute.zones.get
file.instances.create
file.instances.delete
file.instances.get
file.operations.get
iam.serviceAccounts.actAs

GCLOUD_CREDENTIALS_FILE: Google Cloud service account key. The service account must have read-write permission for Google Compute Engine and full control of Google Cloud Storage. This parameter is optional if the LSF management host is on Google Cloud, and the service account that is attached to the LSF management host instance has the required permission.

On this page
💬 Ask a Question