Using an X11 Virtual Frame Buffer to run GUI jobs in batch mode on the HPC.
Thu 20 December 2018 by Dr. Dirk ColbryI have an example program that came with BCCD called Pandemic which I wanted to run on our local HPCC. Unfortunately Pandemic requires X11 to run and I would get a segmentation fault every time I ran it in the batch system.
This blog post shows how I used the X11 Virtual Frame Buffer (Xvfb) to enable X11 in batch mode. This example uses SLURM running in CentOS7. This trick can come in really handing when you are using MATLAB to because last I checked it needed X11 to run in order to generate and save figures. This will let you do that even in batch mode.
First, here is the batch script for the OpenMP version (Should work for serial jobs as well).
#!/bin/bash
#SBATCH --time=00:10:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=10G
#SBATCH --job-name Pandemic-OpenMP
cd ${SLURM_SUBMIT_DIR}
# Have X11 find an open display number and communicate the number
# though a temporary file to set the environment variable.
display_file=.tmp_display.txt
Xvfb -displayfd 1 -auth /dev/null 1>$display_file 2> /dev/null &
sleep 5
export DISPLAY=:`cat $display_file`
echo "DISPLAY set to $DISPLAY"
rm $display_file
# Benchmark program with different numbers of processes
export OMP_NUM_THREADS=16
time ./Pandemic.c-openmp
export OMP_NUM_THREADS=8
time ./Pandemic.c-openmp
export OMP_NUM_THREADS=4
time ./Pandemic.c-openmp
export OMP_NUM_THREADS=2
time ./Pandemic.c-openmp
export OMP_NUM_THREADS=1
time ./Pandemic.c-openmp
echo done
# Report job statistics
scontrol show job $SLURM_JOB_ID
And here is the job I got working with MPI. Notice it is basically the same but I need to pass the DISPLAY variable though MPI for it to work.
#!/bin/bash
#SBATCH --time=00:10:00
#SBATCH --ntasks=16
#SBATCH --cpus-per-task=1
#SBATCH --mem=10G
#SBATCH --job-name Pandemic-MPI
cd ${SLURM_SUBMIT_DIR}
module swap GNU Intel
module load OpenMPI
# Have X11 find an open display number and communicate the number
# though a temporary file to set the environment variable.
display_file=.tmp_display.txt
Xvfb -displayfd 1 -auth /dev/null 1>$display_file 2> /dev/null &
sleep 5
export DISPLAY=:`cat $display_file`
echo "DISPLAY set to $DISPLAY"
rm $display_file
# Benchmark program with different numbers of processes
time mpirun -n 16 -x DISPLAY=$HOSTNAME:$DISPLAY ./Pandemic.c-mpi
time mpirun -n 8 -x DISPLAY=$HOSTNAME$DISPLAY ./Pandemic.c-mpi
time mpirun -n 4 -x DISPLAY=$HOSTNAME$DISPLAY ./Pandemic.c-mpi
time mpirun -n 2 -x DISPLAY=$HOSTNAME$DISPLAY ./Pandemic.c-mpi
time mpirun -n 1 -x DISPLAY=$HOSTNAME$DISPLAY ./Pandemic.c-mpi
# Report job statistics
scontrol show job $SLURM_JOB_ID
Hope you find this useful.
Emerging Technologies (FPGAs) @ 2018 ACI-REF VR
Today is the last day of the 2018 ACI-REF Virtual residency workshop here in Norman Ok. It was a great week with a lot of great people. I highly recommend the workshop for anyone in the area of Advanced Computing Instruction especially those do some Research and Education Facilitation in …
read moreDeciding Which Technologies to Adopt, and When @ 2018 ACI-REF VR
I had fun leading a discussion on Deciding Which Technologies to Adopt, and When at the 2018 ACI-REF Virtual residency event in Norman Oklahoma. Since many of the participants were virtual I experimented with doing a Brainstorming exercise over zoom. We had local volunteers on typing in comments from our …
read moreTeams of CI Professionals: Recruitment & Retention, Management, Team-building, and Motivation Panel @ 2018 ACI-REF VR
Thank you Aaron Bergstrom (U North Dakota) for facilitating the panel discussion on Teams of CI Professionals: Recruitment & Retention, Management, Team-building, and Motivation at the 2018 ACI-REF Virtual residency. My fellow panel members included:
- Jerry Perez, U Texas Dallas
- Derek Leydig, Pennsylvania State U
- Claire Mizumoto, U California San Diego …
Installing RStudio on the MSU HPCC
These instructions are for installing RStudio on the HPCC. RStudio provides a variety of installers on their website https://www.rstudio.com/products/rstudio/download2/. However, the HPCC uses an older version of Linux and the precompiled binaries are not compatible. Trying to install RStudio from the source code is …
read moreManaging Inconsistent Runtimes on the HPCC
Inverse Problems Seminar
I was invited to give a talk at the inverse seminars symposium:
https://inverseproblems2018.org/
At first I was a little nervous since I don't consider myself an expert in inverse problems so I did not fully understand what they wanted me to present. However after talking to the organizers …
read more