Grid Orchestration — Slurm

AWS Parallel Cluster with Slurm Scheduler to support autoscaling:

PREREQUISITES

Cloud Credentials

• "AWS_ACCESS_KEY_ID" "AWS_SECRET_ACCESS_KEY" "AWS_DEFAULT_REGION" (should have all the IAM roles listed below)
• Download software from https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/latest/spill-dc.tar.gz
• Unzip and untar using tar xvf spill-dc.tar.gz
• Understand the supported deployment model below before triggering the orchestration for effective results.
• For any support, reach the spillbox support team at support@spillbox.io.

IAM user roles

The IAM user should have all the below IAM roles associated. To make it easy, you can find two policy json file in our installation tar file (in the directory ./terraform/grid_orchestration/pc-slurm/ – policy1.json and policy2.json). Create two inline policies using the two given json file and attach them to the IAM USER. Below are the list of roles that will be associated using the json file.

• AmazonDynamoDB
• AmazonEC2
• AmazonEC2SpotFleetAutoscaleRole
• AmazonSNS
• AmazonFSx
• AmazonRoute53
• AWSCloudFormation
• AWSImageBuilder
• AWSLambda
• CloudWatchLogsFull
• AWSKMS (Job Accounting)
• SecretsManagerReadWrite (Job Accounting)
• AmazonSSMManagedInstanceCore (Required for SSM connectivity)
• EC2InstanceProfileForImageBuilder (Required for EC2 Image Builder)

SETUP Procedure

I. Setting Spillbox With Existing Cluster

Prerequisites:

• Customer should need to allocate one of their Login Node as Webserver
• Should create an USER (spillbox service-account user) with passwordless SUDO power
• Shared Mount Area between your cluster.

Steps:

1. Login into one of your Login Node and switch to the service account user
2. Download software from https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/latest/spill-dc.tar.gz and untar it
3. copy the file terraform/grid_orchestration/pc-slurm/setup_spillbox_existclus.config.example to terraform/grid_orchestration/pc-slurm/setup_spillbox_existclus.config and Edit the configuration file as per your site..

* SUBNET_ID
* SECURITYGROUP_ID
* ADMIN_USER
* BASEAMI_ID
* KMS_KEY_ID
* SERVICE_USERID
* SHARED_MOUNTDIR

NOTE: Details of this entries is described below

4. Run the scipt

$ terraform/grid_orchestration/pc-slurm/setup_spillbox_pc.sh  -node create -ami create -tag <tagname>

Once the script is done, You will get some Entries.

5. Edit/Append the entries in your SLURM cluster configuration and update your cluster NOTE : The entries consist of 3 New queues which will be used by the Spillbox to run your jobs.

II. Setting Spillbox In A New Cluster

Prerequisites:

• Ensure jq and awscli packages are pre-installed in the current node
• Required Ports needed to be opened inside your subnet for Slurm autoscaling: "22" "111" "443" "988" "2049" "5000" "6000" "6817" "6818" "6819" "7070-8090"

TO SETUP the SLURM CLUSTER

Steps:

1. Download software from https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/latest/spill-dc.tar.gz and untar it
2. copy the file terraform/grid_orchestration/pc-slurm/setup_spillbox_newclus.config.example to terraform/grid_orchestration/pc-slurm/setup_spillbox_newclus.config and Edit the configuration file as per your site..
3. Run the script:

$ terraform/grid_orchestration/pc-slurm/setup_spillbox_pc.sh  -node create -tag <tagname>    

Once the script is done, You see an Webserver Node and Image node

• Login using the credentials, and configure the node as required (Docker images)

4. Once the changes in the Image node is done, run the following script

$ ./terraform/grid_orchestration/pc-slurm/setup_spillbox_pc.sh -ami create -tag <tagname>  

• It will create an AMI(Image node) which will have all your changes and spillbox setup

5.Login into the Webserver Machine, You will find a directory named spillbox_aws_pc in your home directory. Run the script $HOME/spillbox_aws_pc/setup_new_cluster.sh

$ $HOME/spillbox_aws_pc/setup_new_cluster.sh  <cluster-name> create <tagname> 

• In order to override any slurm configuration ( which is given in the ./terraform/grid_orchestration/pc-slurm/setup_spillbox_newclus.config), you can edit the config file ($HOME/spillbox_aws_pc/<tagname>.conf in Webserver node)
• Tagname must be the name which you have given for node creation
• This will take around 1 hour. Monitor the AWS cloudFormation for errors and EC2 image builder console to see the progress of image building stages(Parallel Cluster) KEY PATH :
  • You can find WEBSERVER/CLUSTER node's ssh key at ./terraform/grid_orchestration/pc-slurm/.ssh/

NOTE : To CREATE CLUSTER with more static nodes

• Configure the NFS_MIN_NODE_COUNT / COMPUTE_MIN_NODE_COUNT / SPOT_MIN_NODE_COUNT variables in ./terraform/grid_orchestration/pc-slurm/setup_spillbox_newclus.config under SLURM CLUSTER configuration section

To DESTROY the SLURM Cluster

1. Remove the cluster nodes : Login into the Webserver node and run :

$ $HOME/spillbox_aws_pc/setup_new_cluster.sh  <cluster-name> delete <tagname>

2. Remove the Parallel cluster AMI

$ ./terraform/grid_orchestration/pc-slurm/setup_spillbox_pc.sh -ami delete -tag <tagname>

3. Remove the Webserver/ssh-keys/s3-bucket

$ ./terraform/grid_orchestration/pc-slurm/setup_spillbox_pc.sh -node delete -tag <tagname>

To Enable Job Accounting

SLURM_ACCOUNTING="true"/"false"     # If set to "true", an SQL server will be installed on the current node, which will act as the database.
                                    # By Setting it to false, you can bring your own database and you need to add the following entries: MYSQL_SERVER, MYSQL_USERNAME, MYSQL_SECRET in the config file

Note: Slurm accounting is enabled by default (SLURM_ACCOUNTING="true").

Store your password in the AWS Secret Manager:

* Store type: Other type of secret
* Save as Plain Text
* The AWS user executing this build must have access to the KMS key for encryption/decryption
* Specify the Secret ARN ID of the Secret in the config file as MYSQL_SECRET="arn:aws:secretsmanager:***"

SLurm Info

• SLurm will be installed in the /opt/slurm
• Fsx will be mounted on /fsx_shared/parallelcluster (A shared storage for spillbox work)
• If changing any slurm configuration, Login into the Head Node and do the following

           $ sudo systemctl restart slurmctld 
           $ sudo -E -u slurm /opt/slurm/bin/scontrol reconfigure 

---------------------------------------------------------------------------------------

Example entry for setup_spillbox_existclus/setup_spillbox_newclus.config and explanation

1. SUBNET_ID="subnet-***************"

SECURITYGROUP_ID="sg-************" #These IDs specify the network and security settings for your cluster.

2. ADMIN_USER="centos" #The adminstrative user for the AMI

BASEAMI_ID="ami-***************"  # AMI used to create ParallelCluster
OS_TYPE="centos7"/"alinux2"/"ubuntu2204"/"ubuntu2004"/"rhel8"/"rocky8"    #The ID and type of the base Amazon Machine Image (AMI) that will be used for the cluster. The image should not have SLURM pre-installed.

3. DOCKER_BASEAMI_ID="ami-**************" #AMI used to build docker image

DOCKER_ADMIN_USER="ec2-user"           #The adminstrative user for the AMI

4. Webserver Configuration:

WEBSERVER_CREATE="true"/"false"     #If set to "true," a new web server node will be created as part of the cluster. If set to "false," the current node will act as the web server.
WEBSERVER_INSTANCE_TYPE="i3.large"  #Instance type that needs to support NVMe-based SSDs
WEBSERVER_ROOT_VOLUME_SIZE="80"     #Storage size for web server node.This is required if WEBSERVER_SERVER="true".

5. ImageNode configuration:

IMAGE_ROOT_VOLUME_SIZE="50"        #Storage size for image node.
IMAGE_INSTANCE_TYPE="t2.medium"    #Instance type for the image node.

6. SLURM CLUSTER configuration:

SETUP_SCRIPT="/home/dummy/sample_script"   #The file path for a setup script that runs on all cluster nodes at boot time (optional).
SHARED_STORAGE="lustre|efs"        # Specify the shared storage 
                                   # You cannot specify lustre, when using multiple subnets

7. HEADNODE Configuration:

HEAD_INSTANCE_TYPE="t2.medium"     #Instance type for the head node.
HEAD_ROOT_VOLUME=80                #Storage size for head node.

8. NFS Node Configuration:

NFS_QUEUE_NAME="nfs-queue"                     #Name of the NFS queue.
NFS_INSTANCE_TYPE="t2.medium"                  #Instance type for the NFS node.
NFS_MIN_NODE_COUNT=1 and NFS_MAX_NODE_COUNT=3  #The minimum and maximum number of NFS nodes.
NFS_ROOT_VOLUME=80                             #Storage size for the NFS node (must be at least IMAGE_ROOT_VOLUME_SIZE + 27).

9. Compute Node Configuration:

COMPUTE_QUEUE_NAME="compute-queue"                     #Name of the compute queue.
COMPUTE_INSTANCE_TYPE="t2.medium"                      #Instance type for the compute node.
COMPUTE_MIN_NODE_COUNT=1 and COMPUTE_MAX_NODE_COUNT=3  #The minimum and maximum number of compute nodes.
COMPUTE_ROOT_VOLUME="80"                               #Storage size for the compute node (must be at least IMAGE_ROOT_VOLUME_SIZE + 27).

10. SPOT Queue Configuration:

SPOT_QUEUE_NAME="spot-queue"                       #Name of the spot queue.
SPOT_INSTANCE_TYPE="t2.medium"                     #Instance type for the spot node.
SPOT_MIN_NODE_COUNT=0 and SPOT_MAX_NODE_COUNT=3    #The minimum and maximum number of spot nodes.
SPOT_ROOT_VOLUME=80                                #Storage size for the spot node (must be at least IMAGE_ROOT_VOLUME_SIZE + 27).
SPOT_ALLOCATION_STRATEGY="lowest-price"            #The strategy for allocating spot instances (lowest-price or capacity-optimized).
SPOT_PRICE=1.1                                     #The maximum price for the spot instance.

11. JOB Accounting enable and Database Configuration

SLURM_ACCOUNTING="true"/"false"         #If set to "true", an SQL server will be installed on the current node, which will act as the database.

MYSQL_SECRET #Password used to login to the database. [arn:aws:secretsmanager:*****], if not give then spillbox will create and store the password in aws secretsmanager. Refer Job Accounting Enabled section for setting up the password in AWS Secret Manager

FAQ's

1. How do I submit jobs on spot instances?

• To submit jobs on spot instances, use SPOT_QUEUE_NAME Example : sbatch -p < SPOT_QUEUE_NAME > <./testscript.sh >. This will trigger the spot instance dynamically and submit the testscript.sh job execution on the spot instance.

2. How to update the MAX/MIN NODE_COUNT after the cluster is created?

• Login into the Webserver Node and set your Cloud Credientials
  1. Edit the yaml file $HOME/spillbox_aws_pc/awspc-cluster_config.yaml
  2. Run the command: pcluster update-cluster –cluster-configuration $HOME/spillbox_aws_pc/awspc-cluster_config.yaml –cluster-name <clustername>

3. How can I map a remote queue name to an on-premises queue name?

• To map your on-premises queue name with a remote queue name, create a setup.sh script in the slurm directory on your client side.
• Provide your on-premises queue name in the setup.sh file in the following format:

queue_map={"queue_name1"="compute-queue","queue_name2"="nfs-queue","queue_name3"="spot-queue"}

• "queue_name1" , "queue_name2" and "queue_name3" are your on-prem queue names (Replace them with your actual queue names).
• "compute-queue" , "nfs-queue" and "spot-queue" are remote queue names.

4. What should I do if I accidentally close or lose the terminal that displayed all the IP information?

• No problem. All the essential setup details, including IP addresses and configuration information, are saved in a setup_info file located inside the directory with your tagname.

5. What if I lose the cluster information on the webserver machine?

• Log in to the webserver machine.
• In your home directory, there will be a folder named spillbox_aws_pc.
• Inside that folder, you'll find a file named <cluster-name>-cluster.log which contains the cluster information.