Dirk's Home Pagehttps://colbrydi.github.io/2023-10-18T00:00:00-04:00ASEE Hall of Fame2023-10-18T00:00:00-04:002023-10-18T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2023-10-18:/asee-hall-of-fame.html<p><img alt="ASEE Gala" src="https://www.asee.org/getattachment/0ceb21c6-cbdf-4d52-8140-1ba0720d8e85/Gala_Square.jpg"></p>
<p>I am horned to be a part of the CyberAmbassador Team's induction into the Amarican Society for Engineering Education's (ASEE) Hall of fame.</p>
<p><a href="https://www.asee.org/membership-and-communities/AWARDS-HONORS/Hall-of-Fame">Link to Hall of Fame Annocement</a></p>Python Mailmerge Library2023-06-02T00:00:00-04:002023-06-02T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2023-06-02:/python-mailmerge-library.html<p><img alt="Picture of a black hole to represent a singularity" src="https://www.nasa.gov/sites/default/files/thumbnails/image/simulated_bh.jpg"></p>
<p>I just posted a library in my gitlab repository which runs a mailmerge program on our jupyterhub server here at MSU. I use this every week to improve a lot of my email workflows from sending feedback to students to emailing different groups weekly reports. I like that it just …</p><p><img alt="Picture of a black hole to represent a singularity" src="https://www.nasa.gov/sites/default/files/thumbnails/image/simulated_bh.jpg"></p>
<p>I just posted a library in my gitlab repository which runs a mailmerge program on our jupyterhub server here at MSU. I use this every week to improve a lot of my email workflows from sending feedback to students to emailing different groups weekly reports. I like that it just uses a pandas dataframe. Here are the basic steps to use the program.</p>
<ol>
<li>Create a pandas dataframe with each row representing an email you want to send and each column representing the things you would like to be different in each email. Typically there is an Email Address column and maybe a Name column but you can add anything you want.</li>
<li>Create an eamil "template" using the library. THe template consists of some common email feilds. You want to fill out each feild as a string. You include \<\<TAG>> syntax where the TAG is the name of a column in your pandas dataframe. This is what will get replaced when you send each email.</li>
<li>Run the sendmail function using the template and the dataframe. It is easy as that.</li>
</ol>
<p>Please check it out and let me know what you think.</p>
<p><a href="https://gitlab.msu.edu/colbrydi/mailmerge">https://gitlab.msu.edu/colbrydi/mailmerge</a></p>
<ul>
<li>Dirk</li>
</ul>Director of User Support Video2022-09-02T00:00:00-04:002022-09-02T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2022-09-02:/director-of-user-support-video.html<p><img alt="Image from ICER User support video" src="//colbrydi.github.io/images/UserSupportVideo.jpg"></p>
<p>I am excited to say that I got to rejoin ICER and am now helping lead the User Support Team. </p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/QSyZnuZfKs0?si=X7EQ0v0KdBvoifJd" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>Singularity Overview2022-07-15T00:00:00-04:002022-07-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2022-07-15:/singularity-overview.html<p><img alt="Picture of a black hole to represent a singularity" src="https://www.nasa.gov/sites/default/files/thumbnails/image/simulated_bh.jpg"></p>
<p>Singularity is a versitle tool to give researchers more flexibility installing software and running their workflows on the HPC. I am figuring out singularity for my own research, to better use the Open Science Grid and hopefully to help other people on the HPCC.</p>
<p>Most workflows don't need singilarity but …</p><p><img alt="Picture of a black hole to represent a singularity" src="https://www.nasa.gov/sites/default/files/thumbnails/image/simulated_bh.jpg"></p>
<p>Singularity is a versitle tool to give researchers more flexibility installing software and running their workflows on the HPC. I am figuring out singularity for my own research, to better use the Open Science Grid and hopefully to help other people on the HPCC.</p>
<p>Most workflows don't need singilarity but it can be extreamly helpful to solve certain weridly difficult problems. Some common examples for researchers using singularity on the HPC include:</p>
<ul>
<li>Installing software that needs a special/different base operating system.</li>
<li>Installing software that requires adminstrator privliges (aka root, su and/or sudo). </li>
<li>Installing complex dependancy trees (like python and R)</li>
<li>Using existing software inside a pre-built vitural machine.</li>
<li>Working with lots of tiny files on the HPC filesystems which are better designed for smaller numbers of big files. </li>
<li>Building workflows that can easily move between different resources.</li>
</ul>
<p><strong><em>NOTE</em></strong> This overview is specific to the High Performace Computing Center (HPCC) at Michigan State University (MSU). For a complete tutorial see the <a href="https://docs.sylabs.io/guides/3.2/user-guide/index.html">Singularity documentation</a>. This overview assumes that you have an HPCC account and know how to navigate to and use a development node. </p>
<h2>Step 1: Get a singularity image</h2>
<p>As a starting point we need a singularity image, also known as a container or virtual machine. You can think of a singularity image as a "software hard drive" that contains an entire opperating system in a file. There are three main ways to get these images:</p>
<ol>
<li>Use one of the Singularity images already on the HPCC. </li>
<li>Download an image form one of the many online libraries.</li>
<li>Build your own image.</li>
</ol>
<p>If you don't know which one of the above to use, I recommend that you pick number 1 and just use the singularity image we already have on the system.</p>
<h3>1. Use one of the Singularity images already on the HPCC.</h3>
<p>For this introduction, we can keep things simple and just use one of the singilarity images already on the HPCC. This image runs CentOS 7 linux and is a good starting point. Use the following command to start singularity in a "shell" using the provided image:</p>
<p><code>singularity shell --env TERM=vt100 /opt/software/CentOS.container/7.4/bin/centos</code></p>
<p>Once you run this command you should see the "Singularity" prompt which will look something like the following:</p>
<p><code>Singularity></code></p>
<p>You did it! You are running a different operating system (OS) than the base opporating system. All of the main HPCC folders are still accessable from this "container" (ex. /mnt/home, /mnt/research, /mnt/scratch/, etc) so it shouldn't look much different than before (except for the different prompt and you no longer have access to some of the base HPCC software). </p>
<p>At this point, if you know what you need, you should be able to install your software and it will compile/run using the singularity OS instead of the base OS. </p>
<p><strong><em>NOTE</em></strong>: You can just install software in your <code>/mnt/home/$USER</code> and/or <code>/mnt/research</code> folders. The software you install will probably only work from "inside" this singularity image. However, you will also be able to see and minipulate the files from within your standard HPC account. This is fine for many researchers but I recommend you jump down to "Step 2: Overlays" to make Singularity even more flexible. </p>
<h3>2. Download an image form one of the many online libraries</h3>
<p>Many people publish singularity images and post them on public "libraries" for easy install. Here is a list of online libraries you can brouse:</p>
<h4>Sylabs Library</h4>
<p><a href="https://cloud.sylabs.io/library">Link to Browse Sylabs</a><br>
example:</p>
<div class="highlight"><pre><span></span><code><span class="nx">singularity</span><span class="w"> </span><span class="nx">pull</span><span class="w"> </span><span class="nx">alpine</span><span class="p">.</span><span class="nx">sif</span><span class="w"> </span><span class="kn">library</span><span class="p">:</span><span class="c1">//alpine:latest</span>
<span class="nx">singularity</span><span class="w"> </span><span class="nx">shell</span><span class="w"> </span><span class="nx">alpine</span><span class="p">.</span><span class="nx">sif</span>
</code></pre></div>
<h4>Dockerhub</h4>
<p><a href="https://hub.docker.com/search?q=&type=image">Link to Browse Dockerhub</a><br>
example:</p>
<div class="highlight"><pre><span></span><code>singularity pull tensorflow.sif docker://tensorflow/tensorflow:latest
singularity shell tensorflow.sif
</code></pre></div>
<h4>Singularity Hub (aka shub)</h4>
<p><a href="https://singularityhub.github.io/singularity-catalog/">Link to Browse Singularity Hub</a><br>
example:</p>
<div class="highlight"><pre><span></span><code>singularity pull shub_image.sif shub://vsoch/singularity-images
singularity shell shub_image.sif
</code></pre></div>
<h3>3. build your own image</h3>
<p>This one is more complex and outside the scope of the overview. However, if you are intersted I recommend you try using the build command with a Docker image since it is fairly easy to install on your personal computer. Here is a link to how to use docker to make a singularity image:</p>
<ul>
<li><a href="https://docs.sylabs.io/guides/3.2/user-guide/build_a_container.html?highlight=docker">Link to singularity build command</a></li>
</ul>
<h2>Step 2: Overlays</h2>
<p>One problem we often encounter on the HPCC is "lots-of-small-files" (hundreds of files where each one is < 50MB). The filesystem is optmized for large files. Lots of small files end up "clogging" things up which can slow things down for everyone. One useful trick of singularity is you can make a single large file called an "Overlay" which can be attached to a singularity session. You can use an Overlay as a "filesystem inside a single file" where you can store lots of the small files inside the single overlay file. From the user point of view you can have as many small files as you want accessable from the singularity image (within reasonable limits). However, these small files act as a single file from the HPCC point of view and dosn't clog things up.</p>
<p>This technique is really helpful if you are using complex software installs such as lots of Python, R or Conda installs. It can also be helpful if your research data is lots of small files. </p>
<h3>Make your overlay file</h3>
<p>Making an overlay is not hard but takes multiple steps. For details on how to make an overlay we recommend viewing the <a href="https://docs.sylabs.io/guides/3.2/user-guide/persistent_overlays.html?highlight=overlay">singularity overlay documentation</a>.</p>
<p>Fortunatly the HPCC has a "powertool" that can make a basic overlay for you. All you need to do is run the following command:</p>
<div class="highlight"><pre><span></span><code>mk_overlay
</code></pre></div>
<p>To inspect and learn about what this command is doing type the following (note the tick charicter is not the single quote and is typically found about the tab key on a standard keyboard):</p>
<div class="highlight"><pre><span></span><code>cat `which mk_overlay`
</code></pre></div>
<p>This overlay can be applied to an image using the <code>--overlay</code> option as follows:</p>
<div class="highlight"><pre><span></span><code>singularity shell --overlay overlay.img --env TERM=vt100 /opt/software
/CentOS.container/7.4/bin/centos
</code></pre></div>
<p>Once you are in the singularity shell you can now write to the overlay as if you were adding files to the "root" directory. For example, run the following commands from the singularity prompt (with the overlay enabled). This will download a copy of miniconda and install it in the /miniconda3/ directory:</p>
<div class="highlight"><pre><span></span><code>wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh
./Miniconda3-py39_4.12.0-Linux-x86_64.sh -b -p /miniconda3/
</code></pre></div>
<p>To use this miniconda you will need to add the folder <code>/miniconda3/</code> to the path with the following command:</p>
<div class="highlight"><pre><span></span><code><span class="k">export</span><span class="w"> </span><span class="n">PATH</span><span class="o">=/</span><span class="n">miniconda3</span><span class="o">/</span><span class="n">bin</span><span class="p">:</span><span class="o">$</span><span class="n">PATH</span>
<span class="n">conda</span><span class="w"> </span><span class="o">--</span><span class="n">init</span>
</code></pre></div>
<p>At this point you can use <code>pip</code> and <code>conda</code> installs as much as you like. These generate hundreds of small files but it dosn't matter because everything will be stored in the overlay.img file as one big file. </p>
<p>I recommend creating a file called <code>start-conda.sh</code> which contains the following. This will enable easy startup of your singularity enviornment with the overlay and conda path already set up:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
singularity<span class="w"> </span>shell<span class="w"> </span>--env<span class="w"> </span><span class="nv">TERM</span><span class="o">=</span>vt100<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--env<span class="w"> </span><span class="nv">PATH</span><span class="o">=</span>/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sysbin/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--overlay<span class="w"> </span>overlay.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>/opt/software/CentOS.container/7.4/bin/centos<span class="w"> </span>
</code></pre></div>
<h2>Step 3: Submitting Jobs</h2>
<p>Once we have our image and our overlay working in a development node we can execute a script inside of the singularity image using the <code>exec</code>. For example, this script uses our miniconda installed overlay and runs the python3 script called "mypython.py" which is stored in my home directory on the HPCC.</p>
<div class="highlight"><pre><span></span><code><span class="nv">singularity</span><span class="w"> </span><span class="k">exec</span><span class="w"> </span><span class="o">--</span><span class="nv">overlay</span><span class="w"> </span><span class="nv">overlay</span>.<span class="nv">img</span><span class="w"> </span><span class="o">/</span><span class="nv">opt</span><span class="o">/</span><span class="nv">software</span><span class="o">/</span><span class="nv">CentOS</span>.<span class="nv">container</span><span class="o">/</span><span class="mi">7</span>.<span class="mi">4</span><span class="o">/</span><span class="nv">bin</span><span class="o">/</span><span class="nv">centos</span><span class="w"> </span><span class="nv">python3</span><span class="w"> </span><span class="o">/</span><span class="nv">mnt</span><span class="o">/</span><span class="nv">home</span><span class="o">/</span>$<span class="nv">USER</span><span class="o">/</span><span class="nv">mypython</span>.<span class="nv">py</span>
</code></pre></div>
<p>Once the above is running on a development node we can just submit this as a job to the HPCC using the following submissions script:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#SBATCH --walltime=04:00:00</span>
<span class="c1">#SBATCH --mem=5gb</span>
<span class="c1">#SBATCH -c 1</span>
<span class="c1">#SBATCH -N 1</span>
singularity<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>--overlay<span class="w"> </span>overlay.img<span class="w"> </span>/opt/software/CentOS.container/7.4/bin/centos<span class="w"> </span>python3<span class="w"> </span>/mnt/home/<span class="nv">$USER</span>/mypython.py
</code></pre></div>
<p>If you need to have multiple jobs running the same software (such as for a job array). You can't have them all writting to the same overlay file. The following script still allows the overlay to work but all of the changes will be discared after the run so make sure you copy your results back to your home drive:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#SBATCH --walltime=04:00:00</span>
<span class="c1">#SBATCH --mem=5gb</span>
<span class="c1">#SBATCh --array 1-10</span>
<span class="c1">#SBATCH -c 1</span>
<span class="c1">#SBATCH -N 1</span>
singularity<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>--write-tmp<span class="w"> </span>--overlay<span class="w"> </span>overlay.img<span class="w"> </span>/opt/software/CentOS.container/7.4/bin/centos<span class="w"> </span>python3<span class="w"> </span>/mnt/home/<span class="nv">$USER</span>/mypython.py<span class="w"> </span><span class="nv">$SLURM_ARRAY_ID</span>
</code></pre></div>
<p>This overview of singularity was inicially written by Dirk Colbry. Please contact the ICER User Support Team if you need any help getting your workflow up and running.</p>
<p><a href="https://contact.icer.msu.edu/contact">link to ICER User Support Team online contact form</a></p>Matrix Algebra with Computaional Applications2021-09-30T00:00:00-04:002021-09-30T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-09-30:/matrix-algebra-with-computaional-applications.html<p><img alt="Printout of my textbook on a table of MSU publications" src="//colbrydi.github.io/images/MTH314.jpg"> </p>
<p>Matrix Algebra with Computational Applications is a collection of Open Educational Resource (OER) materials designed to introduce students to the use of Linear Algebra to solve real-world problems. These materials were developed specifically for students and instructors working in a "flipped classroom" model that emphasizes hands-on problem-solving activities during class …</p><p><img alt="Printout of my textbook on a table of MSU publications" src="//colbrydi.github.io/images/MTH314.jpg"> </p>
<p>Matrix Algebra with Computational Applications is a collection of Open Educational Resource (OER) materials designed to introduce students to the use of Linear Algebra to solve real-world problems. These materials were developed specifically for students and instructors working in a "flipped classroom" model that emphasizes hands-on problem-solving activities during class meetings, with students watching lectures and completing readings and assignments outside of the classroom. </p>
<p><a href="https://openbooks.lib.msu.edu/cmse314/"><img alt="MTH314 Cover" src="https://openbooks.lib.msu.edu/app/uploads/sites/18/2021/03/9781626101074_Cover-350x525.jpg"></a> </p>IEEE HPEC 2021 Presentation2021-09-21T00:00:00-04:002021-09-21T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-09-21:/ieee-hpec-2021-presentation.html<p><img alt="IEEE HPEC ICON" src="https://ieee-hpec.org/index_htm_files/15909.jpg"> </p>
<p>Nicholas Grabill, Kai Pinckard and I presented our most recent work on the <a href="https://see-insight.github.io/see-segment/">SEE-Segment</a> Project at the <a href="https://ieee-hpec.org/">IEEE HPEC Converence</a>. This is work that both students conducted during the summer of 2020 as part of the <a href="http://computationalscience.org/xsede-empower">XSEDE Empower</a> program. </p>
<h2>Scaling of Evolutionary Search of Algorithm Space to Speed-Up Scientific ImageUnderstanding …</h2><p><img alt="IEEE HPEC ICON" src="https://ieee-hpec.org/index_htm_files/15909.jpg"> </p>
<p>Nicholas Grabill, Kai Pinckard and I presented our most recent work on the <a href="https://see-insight.github.io/see-segment/">SEE-Segment</a> Project at the <a href="https://ieee-hpec.org/">IEEE HPEC Converence</a>. This is work that both students conducted during the summer of 2020 as part of the <a href="http://computationalscience.org/xsede-empower">XSEDE Empower</a> program. </p>
<h2>Scaling of Evolutionary Search of Algorithm Space to Speed-Up Scientific ImageUnderstanding Workflows</h2>
<blockquote>
<p>Scientific image understanding is an integral part of many research workflows. Studies in everything from self-driving cars to the makeup of cells rely on image understanding through computer vision techniques. Unfortunately, almost every new scientific question requires a new algorithm to be developed by researchers. Exploring the possible algorithms (and their parameters) to find one that fits a particular scientific problem requires time and a broad understanding of the many options available. For this reason, many scientists resort to making measurements "by hand" instead of making the considerable investment required to develop a tool that can automate their specialized workflow. This paper explores the scaling of a tool (SEE-Segment) that searches the "algorithm space" of possible image segmentation algorithms and their parameters for solutions to unique scientific problems. The goal is to have the researchers do this manual annotation of their images ("measure by hand") using a Graphical User Interface front end while the SEE-Segment program searches through the algorithms and their parameters on the back end. In the best case, the SEE-Segment tool finds a good candidate algorithm that can be used on the remaining images in the researcher's data-set. In the worst case, the researchers annotate their images as they would without the tool. Unfortunately, the search space for different segmentation algorithms and their parameters is quite large and non-linear. This research leverages the pleasantly parallel nature of Genetic Algorithms and explores the use of large-scale computing to speed up the search both on a local HPC and on the cloud using Kubernetes. </p>
</blockquote>
<ul>
<li><a href="//colbrydi.github.io/images/20210021_IEEE_HPEC.pdf">IEEE HPEC Slides</a></li>
</ul>OBS Default Backdrop on Mac2021-08-17T00:00:00-04:002021-08-17T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-08-17:/obs-default-backdrop-on-mac.html<p><img alt="My new default background picture" src="//colbrydi.github.io/images/placeholder.png"> </p>
<p>As we move online I have been experimenting with <a href="https://obsproject.com/">Open Broadcast Software (OBS Studio)</a> as a way to improve my video input while I am in meetings and conducting lectures. </p>
<p>Although it had a small learning curve and often produces a lag between the audio and video; I really like …</p><p><img alt="My new default background picture" src="//colbrydi.github.io/images/placeholder.png"> </p>
<p>As we move online I have been experimenting with <a href="https://obsproject.com/">Open Broadcast Software (OBS Studio)</a> as a way to improve my video input while I am in meetings and conducting lectures. </p>
<p>Although it had a small learning curve and often produces a lag between the audio and video; I really like the flexibility that OBS brings to my remote meetings. Some tools I particularly like include:</p>
<ul>
<li>Custom away messages</li>
<li>On screen logos</li>
<li>Changing background images</li>
<li>Over the shoulder coding</li>
</ul>
<p>Today I figured out how to change the default OBS image that appears when the software is not running. Although I didn't mind the built in default I wanted to have something that was more personal and less distracting than an advertisement for OBS. It took me a little searching so I thought I would share the steps that worked for me: </p>
<ol>
<li>Create a 1920x1080 image (Not sure if this exact size is required but I matched the size I found). Name this file <code>placeholder.png</code></li>
<li>
<p>Copy and replace the image in the corresponding<code>/Application/OBS.app</code> folder and the <code>obs-mac-virtualcam.plugin</code> folder. I assume that only the second one is required but I went ahead and copied both. here are the exact commands (note the second one requires root privliges)</p>
<p><code>cp placeholder.png /Applications/OBS.app/Contents/Resources//data/obs-plugins/mac-virtualcam/obs-mac-virtualcam.plugin/Contents/Resources/placeholder.png</code></p>
<p><code>sudo cp placeholder.png /Library/CoreMediaIO/Plug-Ins/DAL/obs-mac-virtualcam.plugin/Contents/Resources</code></p>
</li>
</ol>
<p>That should be it! The new image started working for me right away. I haven't done much else checking to see if I could use a different size image and/or if I needed to coy both files. </p>
<h3>Edited Oct 6, 2021</h3>
<p>Recent update to OBS changed the wrokign path and overwrite my placeholder file. I updated the path in the first command from <code>/Applications/OBS.app/Contents/Resources/data/obs-mac-virtualcam.plugin/Contents/Resources/placeholder.png</code> to
<code>/Applications/OBS.app/Contents/Resources//data/obs-plugins/mac-virtualcam/obs-mac-virtualcam.plugin/Contents/Resources/placeholder.png</code>. Again, I am not sure if this copy is required. </p>Davin Lin's Final Project Presentation (SEE-Classify)2021-07-30T00:00:00-04:002021-07-30T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-07-30:/davin-lins-final-project-presentation-see-classify.html<p><a href="//colbrydi.github.io/images/Davin_midSURE-2021-final.pdf"><img alt="Davin's Poster" src="//colbrydi.github.io/images/Davin_midSURE-2021-final.png"></a> </p>
<p>In the summer of 2021 Davin Lin worked with me as part of the ICER/ACERS REu program on his project titled "Simple Evolutionary Exploration in Classification Algorithms for Supervised Learning"</p>
<blockquote>
<p>Genetic algorithms (GAs) find good solutions to search problems through a process inspired by evolution. Solutions are randomly selected …</p></blockquote><p><a href="//colbrydi.github.io/images/Davin_midSURE-2021-final.pdf"><img alt="Davin's Poster" src="//colbrydi.github.io/images/Davin_midSURE-2021-final.png"></a> </p>
<p>In the summer of 2021 Davin Lin worked with me as part of the ICER/ACERS REu program on his project titled "Simple Evolutionary Exploration in Classification Algorithms for Supervised Learning"</p>
<blockquote>
<p>Genetic algorithms (GAs) find good solutions to search problems through a process inspired by evolution. Solutions are randomly selected and tested using a fitness function. The best solutions undergo changes (mutations) over multiple iterations (generations) to try and find better solutions. There have been several studies that use GAs to search over hyper-parameters of machine learning algorithms to learn values that work well for specific problems. For example, one popular study performs genetic search over breast cancer data to find the best supervised learning classification algorithm [ref]. One of the main benefits of using genetic search is that it can be domain independent. Any scientist can use this method to find a well-performing classification algorithm for their dataset. This summer, we will leverage this property by extending an existing GA framework to adopt classifiers. This will allow scientists from any field to search a classifier “algorithm space” without having to recreate the machinery to do so. We will test our software and perform a proof-of-concept by attempting to partially reproduce the experimental results of previous works.</p>
</blockquote>
<p>Here is a video from his final presentation:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/J3khxAWeYCs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Nate Britton's Mid-Sure Project Presentation (Transfer learning)2021-07-29T00:00:00-04:002021-07-29T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-07-29:/nate-brittons-mid-sure-project-presentation-transfer-learning.html<p>In the summer of 2021 Nate Britton worked with me as project titled "Reexamining Transfer Learning Image Segmentation Hypothesis by Scaling Up"</p>
<blockquote>
<p>Image segmentation is a process in which an image is separated into foreground (areas of interest) and background regions. Image segmentation is used as a first step in …</p></blockquote><p>In the summer of 2021 Nate Britton worked with me as project titled "Reexamining Transfer Learning Image Segmentation Hypothesis by Scaling Up"</p>
<blockquote>
<p>Image segmentation is a process in which an image is separated into foreground (areas of interest) and background regions. Image segmentation is used as a first step in many research fields, which is why the SEE-Insight Team has decided to focus on it as one of its primary scientific image analysis workflows. Building upon years of software developed by the SEE-Insight Team as well as previous work, this research will further explore how transfer learning (using previous results to inform better future ones) can help facilitate the search for segmentation algorithms. This work builds off prior research with transfer learning that used a low number of iterations. This prior work found that there was little to no difference in results between transfer learning and randomized algorithm/parameter selection in image segmentation accuracy. As there is still good reason to expect that transfer learning should yield better-than-random results, the goal of this project is to continue the research by experimenting with finding better baseline algorithms by exploring more iterations and greater population sizes. Research on this project has been and will continue to be implemented using Python and shared as an open-source project on GitHub. In the presentation of this research, I will explore the additional progress that has been made thus far in researching and implementing transfer learning.</p>
</blockquote>
<p>Here is a video from his final presentation:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/DWCA7Bm-dAA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Making an Effective Video Presentation2021-06-30T00:00:00-04:002021-06-30T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-06-30:/making-an-effective-video-presentation.html<p><img alt="Texas A and M logo" src="//colbrydi.github.io/images/Texas_AM.png"> </p>
<p>I was asked to give an second of two "effective presentation" tutorials to a group of REU students at Texas A&M. This second part covers how to make an effective presentation with a focus on scientific introductions. I pulled heavily from the CCQH method described here:</p>
<p><a href="https://www.northwestern.edu/climb/resources/oral-communication-skills/creating-an-intro.html">CCQH refernece</a></p>
<p>Here …</p><p><img alt="Texas A and M logo" src="//colbrydi.github.io/images/Texas_AM.png"> </p>
<p>I was asked to give an second of two "effective presentation" tutorials to a group of REU students at Texas A&M. This second part covers how to make an effective presentation with a focus on scientific introductions. I pulled heavily from the CCQH method described here:</p>
<p><a href="https://www.northwestern.edu/climb/resources/oral-communication-skills/creating-an-intro.html">CCQH refernece</a></p>
<p>Here is a video:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/6og_GQ60J28" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Camtasia Studio tutorial2021-06-09T00:00:00-04:002021-06-09T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-06-09:/camtasia-studio-tutorial.html<p><img alt="Camtasia Logo" src="//techdows.com/wp-content/uploads/2014/05/TechSmith-Camtasia-Studio-8.4.png"></p>
<p>I was asked to give an "effective presentation" tutorial to a group of REU students at Texas A&M. As I talked with the organizers we thought it would be helpful to do a two part tutorial. The first is more of a technical, hands-on workshop on how to use …</p><p><img alt="Camtasia Logo" src="//techdows.com/wp-content/uploads/2014/05/TechSmith-Camtasia-Studio-8.4.png"></p>
<p>I was asked to give an "effective presentation" tutorial to a group of REU students at Texas A&M. As I talked with the organizers we thought it would be helpful to do a two part tutorial. The first is more of a technical, hands-on workshop on how to use Camtasia Studio and the second will be more about how to make an effective presentation. I had fun putting the first workshop together and thought it went quite well. Here is a video:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/fg-bEbcjlyw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>SEE-Insight Research Group2021-05-25T00:00:00-04:002021-05-25T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-05-25:/see-insight-research-group.html<p><img alt="Title Slide for the SEE-Insight Overview Video" src="//colbrydi.github.io/images/SEE-Insight.png"></p>
<p>This week we had a bunch of new students join the SEE-Insight research group. I put together a quick introductory presentation for them. I thought it came out fairly well so I decided to post in here in case anyone is interested in learning more about what our group is …</p><p><img alt="Title Slide for the SEE-Insight Overview Video" src="//colbrydi.github.io/images/SEE-Insight.png"></p>
<p>This week we had a bunch of new students join the SEE-Insight research group. I put together a quick introductory presentation for them. I thought it came out fairly well so I decided to post in here in case anyone is interested in learning more about what our group is doing:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/5inNPZ45Lsc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Refrigerator Shelf Repair2021-03-06T00:00:00-05:002021-03-06T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-03-06:/refrigerator-shelf-repair.html<p><img alt="Broken Connector" src="../images/Broken_Connector_Tiny.jpg"></p>
<p>Our GE refrigerator shelf broke a few weeks ago. The connector on the door side was really weak and poorly designed. </p>
<p>Here is the other side to show you what it looks like when it is not broken. </p>
<p><img alt="Unbroken Fixture" src="../images/Unbroken_Fixture.jpg"></p>
<p>I decided to 3D print a replacement. The design is fairly simple …</p><p><img alt="Broken Connector" src="../images/Broken_Connector_Tiny.jpg"></p>
<p>Our GE refrigerator shelf broke a few weeks ago. The connector on the door side was really weak and poorly designed. </p>
<p>Here is the other side to show you what it looks like when it is not broken. </p>
<p><img alt="Unbroken Fixture" src="../images/Unbroken_Fixture.jpg"></p>
<p>I decided to 3D print a replacement. The design is fairly simple so I used <a href="https://www.openscad.org/">OpenSCAD</a>. </p>
<div class="highlight"><pre><span></span><code><span class="n">difference</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">union</span><span class="p">(){</span>
<span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">-15</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="mi">-5</span><span class="p">])</span><span class="w"> </span><span class="n">cube</span><span class="p">([</span><span class="mi">30</span><span class="p">,</span><span class="mf">0.6</span><span class="p">,</span><span class="mi">65</span><span class="p">]);</span>
<span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">-5</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">cube</span><span class="p">([</span><span class="mi">10</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">35</span><span class="p">]);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">-4</span><span class="p">,</span><span class="mi">-1</span><span class="p">,</span><span class="mi">1</span><span class="p">])</span><span class="w"> </span><span class="n">cube</span><span class="p">([</span><span class="mi">8</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">33</span><span class="p">]);</span>
<span class="p">}</span>
</code></pre></div>
<p><img alt="OpenSCAD_Render" src="../images/OpenScad_Refrigerator_Design_1.png">
<img alt="OpenSCAD_Render" src="../images/OpenScad_Refrigerator_Design_2.png"></p>
<p>The final design had a large void which I printed with supports. By leaving in the supports I was able to fill the void with epoxy to add strength to the design. I also included a large 0.6mm brim to increase the glue surface area. </p>
<p><img alt="Glued Repair" src="../images/Glued_repair.jpg"></p>
<p>When gluing this in, I used the shelf to hold it in place and a layer of cellophane wrap to protect shelf from getting glue on it while it dried. The entire repair turned out great. Just needs a little cleaning of the leftover glue residue. </p>
<p><img alt="Fixed Shelf" src="../images/Fixed_Shelf.jpg">
<img alt="Fixed Shelf" src="../images/Fixed_Shelf2.jpg"></p>MakerCrate Projects2021-03-05T00:00:00-05:002021-03-05T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-03-05:/makercrate-projects.html<p><img alt="Safe Project" src="../images/Kiwi_Safe.jpg"></p>
<p>For Christmas my family got me a box full of <a href="https://www.kiwico.com/maker">MakerCrate projects from KiwiCo</a>. There were five kits in the gift; one for me to work with each of my kids and one for myself. We had a lot of fun putting things together. The quality of the kits are …</p><p><img alt="Safe Project" src="../images/Kiwi_Safe.jpg"></p>
<p>For Christmas my family got me a box full of <a href="https://www.kiwico.com/maker">MakerCrate projects from KiwiCo</a>. There were five kits in the gift; one for me to work with each of my kids and one for myself. We had a lot of fun putting things together. The quality of the kits are outstanding, the directions are clear. </p>
<p><img alt="Safe Project" src="../images/Kiwi_1.jpg">
<img alt="Safe Project" src="../images/Kiwi_2.jpg">
<img alt="Safe Project" src="../images/Kiwi_3.jpg">
<img alt="Safe Project" src="../images/Kiwi_4.jpg"></p>Design of MTH314 - Matrix Algebra with Applications2021-02-09T00:00:00-05:002021-02-09T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2021-02-09:/design-of-mth314-matrix-algebra-with-applications.html<p><img alt="Example" src="https://lh5.googleusercontent.com/T7SNdiXctkIi2iWHjjdAZCRBASzRjysFkmivxtD8cZ80QlKFPjOeYa--uiDoHEUdb7daadZQ-fW-l0SVBQJDj90vPFaCKke9JSf6jf8Rk71zpy62TP_hmBpdFONBoA=w740">
I was honored to be invited to give a presentation to the Math department about the redesign of MTH314 to focus more on applications in python. Here is my presentation:</p>
<iframe id="kaltura_player" src="https://cdnapisec.kaltura.com/p/811482/sp/81148200/embedIframeJs/uiconf_id/27551951/partner_id/811482?iframeembed=true&playerId=kaltura_player&entry_id=1_i92323bd&flashvars[streamerType]=auto&flashvars[localizationCode]=en&flashvars[leadWithHTML5]=true&flashvars[sideBarContainer.plugin]=true&flashvars[sideBarContainer.position]=left&flashvars[sideBarContainer.clickToClose]=true&flashvars[chapters.plugin]=true&flashvars[chapters.layout]=vertical&flashvars[chapters.thumbnailRotator]=false&flashvars[streamSelector.plugin]=true&flashvars[EmbedPlayer.SpinnerTarget]=videoHolder&flashvars[dualScreen.plugin]=true&flashvars[hotspots.plugin]=1&flashvars[Kaltura.addCrossoriginToIframe]=true&&wid=1_xoqag17c" width="640" height="396" allowfullscreen webkitallowfullscreen mozAllowFullScreen allow="autoplay *; fullscreen *; encrypted-media *" sandbox="allow-forms allow-same-origin allow-scripts allow-top-navigation allow-pointer-lock allow-popups allow-modals allow-orientation-lock allow-popups-to-escape-sandbox allow-presentation allow-top-navigation-by-user-activation" frameborder="0" title="Kaltura Player"></iframe>STL files to CNC2020-12-13T00:00:00-05:002020-12-13T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-12-13:/stl-files-to-cnc.html<p>I purchased a discount Workbench CNC machine. I've been a 3D printing person for a while and was suprised how hard it has been to get gcode for a CNC machine. I assumed that I could just upload something like an STL file but there are not a lot of …</p><p>I purchased a discount Workbench CNC machine. I've been a 3D printing person for a while and was suprised how hard it has been to get gcode for a CNC machine. I assumed that I could just upload something like an STL file but there are not a lot of software tools. Or better yet, there are too many software tools and I can't figure out which ones work the best. </p>
<p>Part of the problem is that CNC milling is not true 3D. Instead it is more like 2.5D, where the tool only comes in from one direction. </p>
<p>There are lot of STl designs that should work fine in a 2.5D setting. For example, I have been fascinated by these project cards you can find on Thingiverse:</p>
<ul>
<li><a href="https://www.thingiverse.com/thing:4601489">Enterprise-D</a></li>
</ul>
<p>In this project I am exploring ways to convert these types of one-directional stl files into something I can cut using my CNC machine. So far I am trying to use the following steps:</p>
<ol>
<li>Generate a depthmap from STL file using <a href="https://www.meshlab.net/">Meshlab</a> and following directions <a href="https://community.glowforge.com/t/tutorial-creating-a-depth-map-from-a-3d-model-for-3d-engraving">posted here</a>.</li>
<li>Adjust the contrast of the depthmap to use full grayscale range. (I used preview on mac but any image processing tool should work)</li>
<li>Generate gcode from depthmap using <a href="http://scorchworks.com/Dmap2gcode/dmap2gcode.htm">Dmap2gcode</a> python tool</li>
<li>Review the gcode using <a href="https://nraynaud.github.io/webgcode/">g-code simulator</a></li>
<li><a href="https://www.youtube.com/watch?v=gTUAAm0DFOQ">Hold down method for thin stock using glue and Painters tape</a></li>
</ol>
<h2>Step 1: STL to Depthmap</h2>
<p>Using this <a href="https://community.glowforge.com/t/tutorial-creating-a-depth-map-from-a-3d-model-for-3d-engraving">tutorial</a> as a guild, I downloaded <a href="https://www.meshlab.net/">Meshlab</a> and imported the <a href="https://www.thingiverse.com/thing:4601489">Enterprise-D</a> stl file. </p>
<p>I do not use Meshlab that much so I was not sure how to set the camera view. Fortunately the stl file loaded in the way I wanted so as long as I did not rotate it I could follow the tutorial directions and generate a depthmap. It was a little tricky to figure out where to set the zmax and zmin values. I tried really hard to pick settings that would show the most color contrast in the image. </p>
<p>I found that values near the right side of the slider seemed to work well for this stl model.</p>
<p>When taking the image snapshot I set the background to black because I wanted to cut away the background.</p>
<p><strong><em>Alternative to Step1:</em></strong> I found this website which seems to be much quicker and easier than using meshgrid. I can't adjust anything but the results are clean and could allow someone to skip step 2:</p>
<ul>
<li><a href="https://fenrus75.github.io/FenrusCNCtools/javascript/stl2png.html">STL2PNG</a></li>
</ul>
<h2>Step 2: Adjust contrast</h2>
<p>I found it difficult to get the colors right in Meshlab so I loaded the image into preview (on mac), cropped the image and fiddled with the contrast. Try using histogram equalization if it is available in your image processing program. The goal is to use the entire color range for the file with white being the highest peak in your cut and black being everything is removed. </p>
<p><img alt="Enterpise-D01" src="Enterpise-D01.png"></p>
<h2>Step 3: dmap2gcode</h2>
<p>I found this interesting python project to convert depthmap files to gcode. </p>
<ul>
<li><a href="http://scorchworks.com/Dmap2gcode/dmap2gcode.html">dmap2gcode</a></li>
</ul>
<p>I have no idea if this will work. Also the developer has executables for windows but not mac. Turns out not to be a big deal. I am a python programmer and I was easily able to download and run the python code on my laptop. <strong><em>NOTE</em></strong> python 3.9 did not work but python 3.7 was fine. I ended up making a conda environment, installing the necessary packages and running from there. Here is a link to my environment.yml file if you want to try it out.</p>
<ul>
<li><a href="">environment.yml</a></li>
</ul>
<p>Importing the image file seemed to work fine. The gcode takes hours to generate but once I downgraded to python 3.7 everything seemed to work fine. </p>
<h2>Step 4: Preview G-code</h2>
<p>I have no idea what settings I should be using for the dmap2gcode interface. I found this online simulator which allows me to copy/paste my code and view the gcode (neat). I used the following mac trick to copy the code:</p>
<p><code>cat Enterprise-D.ngc | pbcopy</code></p>
<p>The first few times generating the gcode did not seem like it worked well. I fiddled for a bit and came up with this answer.</p>
<h2>Step 5: Milling a thin sheet</h2>
<p>Next, I needed to figure out how to hold my sample into place. I want to use a thin piece of wood and cut though the material. This means I need a way to hold the entire sample down. I found <a href="https://www.youtube.com/watch?v=gTUAAm0DFOQ">this video tutorial</a> for using a special kind of super glue and painters tape. Seems like a cool trick to do what I want. </p>
<h2>Final Product</h2>
<p>I haven't actually got this working yet. I am keeping this blog post for notes and plan to update it as I get closer.</p>
<p>Like any project, there was a lot of learning for each of the above steps. What I have shown here are some notes to remind my future self what I did to get this working. I am posting these notes on-line just in case anyone else wants to give it a try. </p>
<p>Please feel free to email me questions and/or let me know if you have found better ways to do any of these steps.</p>
<p>Take care,</p>
<ul>
<li>Dirk</li>
</ul>BlocksCAD2020-12-11T00:00:00-05:002020-12-11T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-12-11:/blockscad.html<p><img alt="Picture with Part number" src="../images/Jack_AmongUs.jpg">
<img alt="Picture with Part number" src="../images/Ken_AmongUs.jpg"></p>
<p>I am a big fan of <a href="http://www.openscad.org/">OpenSCAD</a>, it is my goto tool for making 3D designs. My kids have been getting into block coding like scratch and I thought, "wouldn't it be cool if there was a block code program that used OpenSCAD?" A quick google search showed me exactly …</p><p><img alt="Picture with Part number" src="../images/Jack_AmongUs.jpg">
<img alt="Picture with Part number" src="../images/Ken_AmongUs.jpg"></p>
<p>I am a big fan of <a href="http://www.openscad.org/">OpenSCAD</a>, it is my goto tool for making 3D designs. My kids have been getting into block coding like scratch and I thought, "wouldn't it be cool if there was a block code program that used OpenSCAD?" A quick google search showed me exactly that:</p>
<p><a href="https://www.blockscad3d.com/"><img alt="BlocksCAD" src="https://www.blockscad3d.com/themes/custom/blockscad/logo.svg"></a></p>
<p>It is a really great interface. I showed my two youngest and then challenged them to make an AmongUs figure (They are really into the game and the shapes are really interesting). In less than a few hours (with almost no help) they cam eup with some really amazing designs. You can output the designs as stl files or scad files. Very cool!</p>JupyterInstruct Python library for Instructors maintaining curriculum in Jupyter2020-12-11T00:00:00-05:002020-12-11T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-12-11:/jupyterinstruct-python-library-for-instructors-maintaining-curriculum-in-jupyter.html<p><a href="https://github.com/colbrydi/jupyterinstruct"><img align="right," alt="JupyterInstruct Logo with a cartoon Jupiter at a chalkboard" hspace="10" src="https://raw.githubusercontent.com/colbrydi/jupyterinstruct/master/docs/images/JupyterInstruct_icon.png" width="200,"></a></p>
<p>I just uploaded my first developer release of my Python <a href="https://github.com/colbrydi/jupyterinstruct">jupyterinstruct library on github</a>. This library has been in development over the last couple of years but has really materialized in the last few months as I have been planning to pass on my curriculum materials to other faculty. The …</p><p><a href="https://github.com/colbrydi/jupyterinstruct"><img align="right," alt="JupyterInstruct Logo with a cartoon Jupiter at a chalkboard" hspace="10" src="https://raw.githubusercontent.com/colbrydi/jupyterinstruct/master/docs/images/JupyterInstruct_icon.png" width="200,"></a></p>
<p>I just uploaded my first developer release of my Python <a href="https://github.com/colbrydi/jupyterinstruct">jupyterinstruct library on github</a>. This library has been in development over the last couple of years but has really materialized in the last few months as I have been planning to pass on my curriculum materials to other faculty. The library has a number of main goals including:</p>
<ul>
<li>Tools to help instructors maintain course materials all in one place including instructor notes and Answers. </li>
<li>Tools to help migrate curriculum form one semester to the next.</li>
<li>Tools to automatically generate websites and ebooks from notebooks. </li>
<li>Notebook validation tools to identify common problems with links, images and accessibility.</li>
<li>Tools to interface nbgrader with the MSU jupyterhub servers and MSU Desire2Learn course management systems. </li>
</ul>
<p>Obviously there is a lot of work that still needs done but I am excited to continue develop these tools and hopefully make it easer to curate curriculum materials between semesters with the long term goals of saving instructor time while improving the learning experience for students.</p>Using the Michigan Sate University Virtual Desktop System2020-12-04T00:00:00-05:002020-12-04T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-12-04:/using-the-michigan-sate-university-virtual-desktop-system.html<p><img align="right," alt="MSU VID System" hspace="10" src="//colbrydi.github.io/images/VDI1.png" width="200,"></p>
<p>Michigan State University (MSU) has a <a href="https://tech.msu.edu/technology/virtual-desktop/">Virtual Desktop System</a> that is accessible to students and staff. If you have an account you can access it from here:</p>
<ul>
<li><a href="http://vdi.msu.edu">vdi.msu.edu</a></li>
</ul>
<p>I find it is a great for when I need to do something in windows or I need a backup …</p><p><img align="right," alt="MSU VID System" hspace="10" src="//colbrydi.github.io/images/VDI1.png" width="200,"></p>
<p>Michigan State University (MSU) has a <a href="https://tech.msu.edu/technology/virtual-desktop/">Virtual Desktop System</a> that is accessible to students and staff. If you have an account you can access it from here:</p>
<ul>
<li><a href="http://vdi.msu.edu">vdi.msu.edu</a></li>
</ul>
<p>I find it is a great for when I need to do something in windows or I need a backup for jupyterhub. Here is a video I made explaining how to use the system for my students.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/8sZpyFhzwfo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Air Conditioner CAM Replacement2020-08-25T00:00:00-04:002020-08-25T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-08-25:/air-conditioner-cam-replacement.html<p><img alt="Picture with Part number" src="../images/ACCAM-PartNum.jpg"></p>
<p>Our house has a 2-zone heating and air conditioning system. Today is the third time in the same number of years that one of the automated damper has broken. The problem seems to be the CAM that triggers the limit switches inside the motor is made of a flimsy plastic …</p><p><img alt="Picture with Part number" src="../images/ACCAM-PartNum.jpg"></p>
<p>Our house has a 2-zone heating and air conditioning system. Today is the third time in the same number of years that one of the automated damper has broken. The problem seems to be the CAM that triggers the limit switches inside the motor is made of a flimsy plastic and easily breaks (see following pictures).</p>
<p><img alt="Original CAMs with a broken piece in the corner" src="../images/ACCAM-Open.jpg"></p>
<p><img alt="Picture of the broken CAMs" src="../images/ACCAM-Broken.jpg"></p>
<p>I decided to build 3D print a stronger CAM. Instead of trying to replace with two adjustable CAMS my design includes one, thicker CAM which I modeled in OPENSCAD using the following code:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nt">difference</span><span class="o">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">union()</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="err">translate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mf">12.5</span><span class="p">,</span><span class="mi">0</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cube(</span><span class="cp">[</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">4</span><span class="cp">]</span><span class="err">,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="err">translate(</span><span class="cp">[</span><span class="mi">2</span><span class="p">,</span><span class="mf">9.5</span><span class="p">,</span><span class="mi">0</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">rotate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">45</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cube(</span><span class="cp">[</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"> </span><span class="mi">4</span><span class="cp">]</span><span class="err">,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="err">translate(</span><span class="cp">[</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span><span class="mf">9.5</span><span class="p">,</span><span class="mi">0</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">rotate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="o">-</span><span class="mi">45</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cube(</span><span class="cp">[</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"> </span><span class="mi">4</span><span class="cp">]</span><span class="err">,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="err">rotate</span><span class="w"> </span><span class="err">(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">90</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">union()</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="err">translate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mf">12.5</span><span class="p">,</span><span class="mi">0</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cube(</span><span class="cp">[</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">4</span><span class="cp">]</span><span class="err">,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="err">translate(</span><span class="cp">[</span><span class="mi">2</span><span class="p">,</span><span class="mf">9.5</span><span class="p">,</span><span class="mi">0</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">rotate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">45</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cube(</span><span class="cp">[</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"> </span><span class="mi">4</span><span class="cp">]</span><span class="err">,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="err">translate(</span><span class="cp">[</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span><span class="mf">9.5</span><span class="p">,</span><span class="mi">0</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">rotate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="o">-</span><span class="mi">45</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cube(</span><span class="cp">[</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"> </span><span class="mi">4</span><span class="cp">]</span><span class="err">,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nt">difference</span><span class="o">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">scale(</span><span class="cp">[</span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cylinder(h</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">4,</span><span class="w"> </span><span class="err">r</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">100,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="err">scale(</span><span class="cp">[</span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cylinder(h</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">5,</span><span class="w"> </span><span class="err">r</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">35,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="o">;</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="o">(</span><span class="nt">a</span><span class="w"> </span><span class="o">=</span><span class="cp">[</span><span class="mi">15</span><span class="p">:</span><span class="mi">65</span><span class="cp">]</span><span class="o">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">rotate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="na">-a</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">translate(</span><span class="cp">[</span><span class="mi">0</span><span class="p">,</span><span class="mf">7.5</span><span class="p">,</span><span class="mi">0</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">scale(</span><span class="cp">[</span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="cp">]</span><span class="err">)</span><span class="w"> </span><span class="err">cylinder(h</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">5,</span><span class="w"> </span><span class="err">r</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">10,</span><span class="w"> </span><span class="err">center=true)</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="err">}</span>
</code></pre></div>
<p>Here is a picture of the final CAM design:</p>
<p><img alt="Rendered New Part" src="../images/ACCAM-Render.png"></p>
<p><a href="../images/AC-Cam.stl">Link to the generated STL File</a></p>
<p>And here is the CAM put back into the moter housing:</p>
<p><img alt="New CAM in place" src="../images/ACCAM-New.jpg"></p>
<p>I have included the files in case someone else has the same problem. The part number for the entire uint is "DURO DYNE C 005096". Let me know if you find this helpful.</p>Training Graduate Students Across Disciplines: Tools and Techniques for Developing Scientific Software2020-07-31T00:00:00-04:002020-07-31T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-07-31:/training-graduate-students-across-disciplines-tools-and-techniques-for-developing-scientific-software.html<p><img align="right," alt="US-RSE Research Software Engineering Association Logo" hspace="10" src="https://us-rse.org/assets/img/rse_logo_background.svg" width="200,"></p>
<p>On Friday during the last day of PEARC20 there was a wonderful Research Software Engineers Community Workshop (US-RSE) to discuss building and sustaining software development community. </p>
<p><a href="https://us-rse.org/events/2020/2020-07-pearc20">Workshop Website</a></p>
<p>I had the pleasure to be a part of one of the panels and gave a short presentation on my work on …</p><p><img align="right," alt="US-RSE Research Software Engineering Association Logo" hspace="10" src="https://us-rse.org/assets/img/rse_logo_background.svg" width="200,"></p>
<p>On Friday during the last day of PEARC20 there was a wonderful Research Software Engineers Community Workshop (US-RSE) to discuss building and sustaining software development community. </p>
<p><a href="https://us-rse.org/events/2020/2020-07-pearc20">Workshop Website</a></p>
<p>I had the pleasure to be a part of one of the panels and gave a short presentation on my work on my CMSE802 course "methods in computational modeling". This class is designed to teach Graduate Students from across campus the software endingeering tools they need to make robust and sustainable software. The slides for my talk can be found here:</p>
<p><a href="https://docs.google.com/presentation/d/189FfV-AGPwE6sPdEraXIXmlDQLFtmurqESWVIxfeLvA/edit#slide=id.g8e631fd34f_2_178">Link to Slides</a></p>CyberAmbassadors:Results from Pilot Testing a New Professional Skills Curriculum2020-07-28T00:00:00-04:002020-07-28T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-07-28:/cyberambassadorsresults-from-pilot-testing-a-new-professional-skills-curriculum.html<p><img alt="PEARC20 Banner" src="https://carcc.org/wp-content/uploads/2019/10/pearc20-1024x213.jpg"></p>
<p>I was pleased to present the most resent paper form our CyberAmbassador Project entitled <strong><em>"CyberAmbassadors: Results from Pilot Testing a New Professional Skills Curriculum" By Astri Briliyanti, Julie Wilson Rojewski, Katy Luchini-Colbry, and Dirk Colbry</em></strong>. This paper wwas given the best paper award in the PEARC 2020 “People involved in …</p><p><img alt="PEARC20 Banner" src="https://carcc.org/wp-content/uploads/2019/10/pearc20-1024x213.jpg"></p>
<p>I was pleased to present the most resent paper form our CyberAmbassador Project entitled <strong><em>"CyberAmbassadors: Results from Pilot Testing a New Professional Skills Curriculum" By Astri Briliyanti, Julie Wilson Rojewski, Katy Luchini-Colbry, and Dirk Colbry</em></strong>. This paper wwas given the best paper award in the PEARC 2020 “People involved in research computing – workforce development, diversity, and professionalization” Track.</p>
<blockquote>
<p>The award winners were selected by independent juries, separate ones for each conference track from candidates that were most highly-rated in the conference’s peer review process. The juries deemed the winning manuscripts to be comprehensively well rounded, reflective of deep understanding of the subject areas discussed, and of very high quality overall.</p>
</blockquote>
<p><a href="https://pearc.acm.org/pearc20/awards/">Click here for information about PEARC and the other award winners</a></p>
<p><a href="https://docs.google.com/presentation/d/1kNtmSjMErnVU4PnmgVyZucmP12yWZW4cFfGCp2F07R0/edit?usp=sharing">A link to the slides can be found here</a></p>Third Workshop on Strategies for Enhancing HPC Education and Training (SEHET20)2020-07-27T00:00:00-04:002020-07-27T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-07-27:/third-workshop-on-strategies-for-enhancing-hpc-education-and-training-sehet20.html<p><img alt="SigHPC Education Logo" src="http://sighpceducation.acm.org/images/sighpc77educhapterlogofw.jpg"></p>
<p>I presented an overview of my CMSE401 course (methods in parallel programming) at the <a href="https://sighpceducation.acm.org/events/SEHET20_Technical_Program.html">SEHET20 Workshop Program</a> put on by the <a href="https://sighpceducation.acm.org">SigHPC Education Chapter</a>.</p>
<p>The presentation ("The Design of a Practical Flipped Classroom Model for Teaching Parallel Programming to Undergraduates") focused mostly on how I designed the course using a …</p><p><img alt="SigHPC Education Logo" src="http://sighpceducation.acm.org/images/sighpc77educhapterlogofw.jpg"></p>
<p>I presented an overview of my CMSE401 course (methods in parallel programming) at the <a href="https://sighpceducation.acm.org/events/SEHET20_Technical_Program.html">SEHET20 Workshop Program</a> put on by the <a href="https://sighpceducation.acm.org">SigHPC Education Chapter</a>.</p>
<p>The presentation ("The Design of a Practical Flipped Classroom Model for Teaching Parallel Programming to Undergraduates") focused mostly on how I designed the course using a flipped classroom model and jupyter notebooks.</p>
<p><a href="https://docs.google.com/presentation/d/11V1Q9nmISe7Zn9kPd5t0XGQykIwcxlvx07_XOj3OiAk/edit?usp=sharing">A link to the slides can be found here</a></p>
<p><a href="https://github.com/colbrydi/CMSE401_Examples">A link to some example notebooks is here</a></p>Making a Landline Intercom2020-04-19T00:00:00-04:002020-04-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-04-19:/making-a-landline-intercom.html<p><img alt="Wire Diagram" src="../images/Phone_Diagram.png"></p>
<p>We don't have a landline but over the years I have collected a bunch of phone wiring, sockets, etc. My spouse asked their social network and we got three landline phones and the kids and I put together a really simple intercom project with a 9 volt battery. </p>
<p>All we …</p><p><img alt="Wire Diagram" src="../images/Phone_Diagram.png"></p>
<p>We don't have a landline but over the years I have collected a bunch of phone wiring, sockets, etc. My spouse asked their social network and we got three landline phones and the kids and I put together a really simple intercom project with a 9 volt battery. </p>
<p>All we did is put the 9volt in the red wire of the phone and connected them as shown in the diagram. The kids ran the phone with 25 feet of wire between the houses and were able to hear each other quite clearly from either garage. </p>
<p>The whole project didn't take more than 30 minutes even with me playing around with the soldering iron (which was probably not needed).</p>
<p>Phone lasted for quite a few days and the kids had a lot of fun. If we wanted to use this as a permeant setup I may have improved the circut but as a fun proof of concept it turned out great.</p>
<p><img alt="Child one on the phone" src="../images/Phone_J.JPG"></p>
<p><img alt="Child two on the phone" src="../images/Phone_wireing.JPG"></p>
<p><img alt="Electronics" src="../images/Phone_X.JPG"></p>Magical Staff2020-03-15T00:00:00-04:002020-03-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-03-15:/magical-staff.html<p><img alt="Magical Staff Drawing" src="../images/StaffDrawing.png"></p>
<p>My eldest is a big Harry Potter Fan and made a few drawings of a Magical Staff they wanted to construct. I was really excited by the drawing. We purchased the crystal and staff online and sat down in front of OpenSCAD and came up with a fairly close design …</p><p><img alt="Magical Staff Drawing" src="../images/StaffDrawing.png"></p>
<p>My eldest is a big Harry Potter Fan and made a few drawings of a Magical Staff they wanted to construct. I was really excited by the drawing. We purchased the crystal and staff online and sat down in front of OpenSCAD and came up with a fairly close design. Here is the OpenSCAD code:</p>
<div class="highlight"><pre><span></span><code><span class="n">difference</span><span class="p">(){</span>
<span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">-50</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">50</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">16</span><span class="p">,</span><span class="w"> </span><span class="n">center</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">true</span><span class="p">);</span>
<span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">-103</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">13</span><span class="p">,</span><span class="w"> </span><span class="n">center</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">true</span><span class="p">);</span>
<span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">-40</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">26</span><span class="p">,</span><span class="n">r1</span><span class="o">=</span><span class="mi">13</span><span class="p">,</span><span class="n">r2</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">center</span><span class="o">=</span><span class="nb">true</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">translate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">-50</span><span class="p">])</span><span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="n">z</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="o">:</span><span class="mi">4</span><span class="o">:+</span><span class="mi">350</span><span class="p">])</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">z</span><span class="p">])</span><span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">16</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">z</span><span class="o">/</span><span class="mi">6</span><span class="o">+</span><span class="mi">10</span><span class="p">])</span><span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">-30</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="mi">5</span><span class="p">);</span>
<span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="o">-</span><span class="n">z</span><span class="o">+</span><span class="mi">180</span><span class="p">])</span><span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">16</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">z</span><span class="o">/</span><span class="mi">6</span><span class="o">+</span><span class="mi">10</span><span class="p">])</span><span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">-30</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="mi">5</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">translate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">-40</span><span class="p">])</span><span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="n">z</span><span class="o">=</span><span class="p">[</span><span class="mi">-180</span><span class="o">:</span><span class="mi">6</span><span class="o">:</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">z</span><span class="p">])</span><span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">16</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">-30</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="mi">5</span><span class="p">);</span>
<span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="o">-</span><span class="n">z</span><span class="o">+</span><span class="mi">180</span><span class="p">])</span><span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">16</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">-30</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="mi">5</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">translate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">-50</span><span class="p">])</span><span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="n">z</span><span class="o">=</span><span class="p">[</span><span class="mi">350</span><span class="o">:</span><span class="mi">6</span><span class="o">:+</span><span class="mi">420</span><span class="p">])</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">z</span><span class="p">])</span><span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">16</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">350</span><span class="o">/</span><span class="mi">6</span><span class="o">+</span><span class="mi">10</span><span class="p">])</span><span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">-30</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="mi">5</span><span class="p">);</span>
<span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="o">-</span><span class="n">z</span><span class="o">+</span><span class="mi">180</span><span class="p">])</span><span class="w"> </span><span class="n">translate</span><span class="p">([</span><span class="mi">16</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">350</span><span class="o">/</span><span class="mi">6</span><span class="o">+</span><span class="mi">10</span><span class="p">])</span><span class="w"> </span><span class="n">rotate</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">-30</span><span class="p">,</span><span class="mi">0</span><span class="p">])</span><span class="w"> </span><span class="n">cylinder</span><span class="p">(</span><span class="n">h</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="mi">5</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
<p><img alt="OpenSCAD drawing of staff" src="../images/Staff_OpenSCAD.png"></p>
<p>The print came out well with minimal supports and we were happy with the design. The final project looked great.</p>
<p><img alt="Final image of the staff with Crystal" src="../images/Finished_Staff.png"></p>
<p>To get everything to stick together we used Surgu Moldable glue. We were very happy with the final product.</p>
<p><img alt="Final image of the staff in costume" src="../images/TLC_with_Staff.jpg"></p>Zoomin in the Classroom2020-03-11T00:00:00-04:002020-03-11T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-03-11:/zoomin-in-the-classroom.html<p>I was asked to put together a few videos to help transition to online teaching. Here is my first pass...</p>
<p>Zoom is an online video chat program with many useful features including:</p>
<ul>
<li>Face-to-face Communication</li>
<li>Desktop Sharing</li>
<li>Chat</li>
<li>Breakout rooms</li>
<li>Video Recording</li>
</ul>
<p>Using zoom requires the download and installation of an …</p><p>I was asked to put together a few videos to help transition to online teaching. Here is my first pass...</p>
<p>Zoom is an online video chat program with many useful features including:</p>
<ul>
<li>Face-to-face Communication</li>
<li>Desktop Sharing</li>
<li>Chat</li>
<li>Breakout rooms</li>
<li>Video Recording</li>
</ul>
<p>Using zoom requires the download and installation of an app and anyone with an MSU account can get the app and log into zoom at the following website:</p>
<p><a href="http://zoom.msu.edu/">http://zoom.msu.edu/</a></p>
<p>A basic introduction to zoom controls can be found here (video is a little last minute and I apologize for the quality):</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/_YMlLgIe9d0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>As we move our classes online, we can use zoom as a one size fits most solution. In this tutorial I will talk about a couple of solutions. All use zoom and have a lot of similar setups. First off is a concept of zoom rooms which are represented by a nine-digit address (think phone number). </p>
<p>Each account is associated with a personal zoom room. This room is tied to your ID and doesn’t change and does not need to be scheduled. You can use your personal zoom room when you are planning a meeting and do not have access to zoom.msu.edu. However, you can also create new zoom rooms. These are scheduled though the zoom interface. For class I recommend setting up a scheduled zoom room which you use for both office hours and in-class activities. IF you schedule this room far enough into the future you can use it for all your class related zoom meetings. The following video goes though scheduling your zoom room:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/47C21AX898Q" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Here is the scheduled meeting I set up in the video. Have a look and see what it is like from the student point of view (I may cancel this meeting at some point but feel free to try):</p>
<p><a href="https://msu.zoom.us/s/675712003">https://msu.zoom.us/s/675712003</a></p>
<p>Once you have a room you have a zoom URL and id which students can use to log into the room. I recommend using a different zoom room for office hours and for class. </p>
<h1>Online Office hours</h1>
<ol>
<li>Open your personal zoom session using the zoom link you created.</li>
<li>Check audio and video</li>
<li>Wait for students. </li>
<li>Have students share their screen if you are helping them debug problems.</li>
<li>Share your screen if you are showing them something. </li>
</ol>
<iframe width="560" height="315" src="https://www.youtube.com/embed/rXnwQbW0q2g" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<h1>Recording Video</h1>
<ol>
<li>Open your personal zoom session using the zoom link you created.</li>
<li>Check audio and video</li>
<li>Share your desktop</li>
<li>Maximize your slides</li>
<li>Click Record</li>
<li>Present your lecture</li>
<li>Click Stop</li>
<li>Select End Meeting (Zoom will convert your video into a useable format) </li>
<li>Upload video to YouTube(use unlisted setting). </li>
</ol>
<iframe width="560" height="315" src="https://www.youtube.com/embed/mswYtrcSRRI" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="560" height="315" src="https://www.youtube.com/embed/VScTgfeLARA" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<h1>Live Lecture</h1>
<ol>
<li>Open your personal zoom session using the zoom link you created.</li>
<li>Check audio and video</li>
<li>Share your desktop</li>
<li>Maximize your slides</li>
<li>Wait for students to join</li>
<li>Mute all students</li>
<li>Click Record (Optional) </li>
<li>Present your lecture</li>
<li>Make sure to check chatroom for questions.</li>
<li>Click Stop</li>
<li>Select End Meeting (Zoom will convert your video into a useable format) </li>
<li>Upload video to YouTube(use unlisted setting)</li>
</ol>
<iframe width="560" height="315" src="https://www.youtube.com/embed/-bI7qcojy0w" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="560" height="315" src="https://www.youtube.com/embed/VScTgfeLARA" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<h1>Recommendations for students</h1>
<ul>
<li>Although there are tablet/phone apps, whenever possible use a laptop or two screens. This helps with code along activities. </li>
<li>Calling in via phone can significantly improve sound quality.</li>
<li>Turning off student video can save on bandwidth and improve their experience. </li>
</ul>
<h1>Advanced Options</h1>
<p>I have been playing with a couple of more advanced setups when using zoom. Once you have mastered the basics, here are a few things to think about:</p>
<h2>Using Breakout Rooms:</h2>
<p>During live classes and maybe office hours you can click on the breakout room option to automatically add students to rooms. Students can join their breakout rooms or leave and join the “main” room. </p>
<h2>Live Closed Captioning:</h2>
<p>Although there is a closed captioning button on the zoom interface, It does not work without a third party program or someone type in words live. However, google slides has a close captioning option which works fairly well and can add to your zoom experience. Just run a zoom slide in presentation mode and one of the onscreen options is a close captioning button.</p>
<h2>Remote desktop control:</h2>
<p>Another advanced feature is remote desktop control. It does not work with breakout rooms enabled but will allow you to remote control a computer which may be especially helpful if you are trying to help a student debug a problem. I have not tried this feature yet but you could probably use your personal room and try it from there.</p>
<h2>Using a Tablet to record writing:</h2>
<p>Another nice option is you can use an Ipad to act as your whiteboard as you record a lecture. The tablet is really nice because it works a pen. Instead of sharing your screen you can plug in your ipad and share that. </p>Tensorflow and Keras/PyTorch on the HPCC2020-02-27T00:00:00-05:002020-02-27T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-02-27:/tensorflow-and-keraspytorch-on-the-hpcc.html<p><img alt="The Tensorflow Logo" src="//avatars0.githubusercontent.com/u/15658638?s=200&v=4"></p>
<p>Every few semesters I have a Tensorflow example I want to try on the HPCC. I'm not an expert on the software and between the HPCC, CUDA, Tensorflow, Keras, Python, Anaconda, etc. there are a lot of continuously changing and moving parts. Seems like when I figure out how to …</p><p><img alt="The Tensorflow Logo" src="//avatars0.githubusercontent.com/u/15658638?s=200&v=4"></p>
<p>Every few semesters I have a Tensorflow example I want to try on the HPCC. I'm not an expert on the software and between the HPCC, CUDA, Tensorflow, Keras, Python, Anaconda, etc. there are a lot of continuously changing and moving parts. Seems like when I figure out how to get it working something changes. This time around wasn't too bad so I thought I would share my latest install instructions.</p>
<h3>Step 1: Install Anaconda Python</h3>
<p>Here is a video I made a while ago about how to download and install Anaconda on the HPC.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/g0rGb6QqBPo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<h3>Step 2: Make a Tensor Conda Environment</h3>
<p>On one of the development nodes create a Conda environment using the following commands:</p>
<div class="highlight"><pre><span></span><code>conda<span class="w"> </span>create<span class="w"> </span>--prefix<span class="w"> </span>./tensorflow-env<span class="w"> </span>tensorflow-gpu<span class="w"> </span>keras<span class="w"> </span>pytorch
conda<span class="w"> </span>init<span class="w"> </span>bash
<span class="nb">exit</span>
</code></pre></div>
<h3>Step 3: Test Conda</h3>
<p>Now we need to test keras and tensorflow. I found an example on the internet and had to fix a few bugs. I am not sure how good an example is but it is good enough to run as a benchmark test. You can download the example file (<a href="//colbrydi.github.io/images/convolutional.py">convolutional.py</a>, or just use the following commands on a dev-node on the HPC:</p>
<div class="highlight"><pre><span></span><code>wget<span class="w"> </span>https://colbrydi.github.io/images/convolutional.py
conda<span class="w"> </span>activate<span class="w"> </span>./tensorflow-env
<span class="nb">time</span><span class="w"> </span>python<span class="w"> </span>convolution.py
</code></pre></div>
<p>Running the above took me about 8 minutes on dev-intel18 about 1.3 minutes on dev-intel16-k80 and about 2 minutes on dev-intel14-k20. I also ran on the special Volta node (nvl-001) which is only available to buy-in users. That thing only to about 30 seconds. Here are the commands I used (Requires buy-in access):</p>
<div class="highlight"><pre><span></span><code>salloc<span class="w"> </span>--gres<span class="o">=</span>gpu:1<span class="w"> </span>--nodelist<span class="o">=</span>nvl-001<span class="w"> </span>--time<span class="o">=</span><span class="m">00</span>:10:00<span class="w"> </span>--account<span class="o">=</span>cmse
conda<span class="w"> </span>activate<span class="w"> </span>./tensorflow-env
<span class="nb">time</span><span class="w"> </span>srun<span class="w"> </span>python<span class="w"> </span>convolutional.py
</code></pre></div>
<h3>Step 4: SLURM Script</h3>
<p>To run this in a SLURM script you need to request GPUs and use the srun command. Here is an example:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash --login</span>
<span class="c1">#SBATCH --time=01:00:00</span>
<span class="c1">#SBATCH -c 1</span>
<span class="c1">#SBATCH -n 1</span>
<span class="c1">#SBATCH --gres=gpu:1</span>
<span class="c1">#SBATCH --mem=4gb</span>
module<span class="w"> </span>load<span class="w"> </span>powertools
conda<span class="w"> </span>activate<span class="w"> </span>../tensorflow-env
<span class="nb">time</span><span class="w"> </span>srun<span class="w"> </span>python<span class="w"> </span>convolutional.py
<span class="c1">#Prints out job statistics</span>
js<span class="w"> </span><span class="si">${</span><span class="nv">SLURM_JOB_ID</span><span class="si">}</span>
</code></pre></div>
<p>This installation is probably not optimal but seems to work consistently. </p>Converting directory of ipynb to pdfs2020-02-02T00:00:00-05:002020-02-02T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-02-02:/converting-directory-of-ipynb-to-pdfs.html<p><img alt="Image Showing Icons for converting from Jupyter to PDF files" src="//colbrydi.github.io/images/Jupyter2PDF.png"></p>
<p>I wanted to share the content for one of my courses with some other instructors but it turns out not everyone uses Jupyter notebooks. </p>
<p><strong>Option 1:</strong> I decided to convert all of the notebooks to pdfs using <a href="https://nbconvert.readthedocs.io/en/latest/">nbconvert</a>. However, <code>nbconvert</code> isn't working on my MacBook. I think I have too …</p><p><img alt="Image Showing Icons for converting from Jupyter to PDF files" src="//colbrydi.github.io/images/Jupyter2PDF.png"></p>
<p>I wanted to share the content for one of my courses with some other instructors but it turns out not everyone uses Jupyter notebooks. </p>
<p><strong>Option 1:</strong> I decided to convert all of the notebooks to pdfs using <a href="https://nbconvert.readthedocs.io/en/latest/">nbconvert</a>. However, <code>nbconvert</code> isn't working on my MacBook. I think I have too many versions of latex and python installed and I can't quickly find the right magic to get the pdf converter working. </p>
<p><strong>Option 2:</strong> However, <code>nbconvert</code> works great if I convert to html. So then I had a thought to first convert to html and then print to pdf. However this option requires me to open and save each html file individually.</p>
<p><strong>Option 3:</strong> I discovered there are a bunch of tools that convert html to pdf. Most of them require similar installation I was having trouble with in Option 1. However, Chrome allows me to run in "Headless" mode which may make it easy to automate.</p>
<p>My final solution has the following steps. Note, this only works on MacOS but conceptually could be converted to work on any system:</p>
<ol>
<li>Use <code>nbconvert</code> to convert each ipynb in the current directory to html.</li>
<li>Use <code>Chrome</code> headless mode to convert html files to pdfs.</li>
<li>Use the <code>join</code> command to combine all of the pdfs into one file.</li>
</ol>
<p>Here is a bash script that seems to work:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
mkdir<span class="w"> </span>-p<span class="w"> </span>HTML
jupyter<span class="w"> </span>nbconvert<span class="w"> </span>--to<span class="w"> </span>html<span class="w"> </span>--no-prompt<span class="w"> </span>--allow-errors<span class="w"> </span>--output-dir<span class="w"> </span>HTML<span class="w"> </span>*.ipynb
mkdir<span class="w"> </span>-p<span class="w"> </span>PDF
<span class="k">for</span><span class="w"> </span>file<span class="w"> </span><span class="k">in</span><span class="w"> </span>./html/*.html
<span class="k">do</span>
<span class="w"> </span><span class="nv">filename</span><span class="o">=</span><span class="k">$(</span>basename<span class="w"> </span>--<span class="w"> </span><span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span><span class="k">)</span>
<span class="w"> </span><span class="nv">extension</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">filename</span><span class="p">##*.</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="nv">filename</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">filename</span><span class="p">%.*</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span>/Applications/Google<span class="se">\ </span>Chrome.app/Contents/MacOS/Google<span class="se">\ </span>Chrome<span class="w"> </span>--headless<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--virtual-time-budget<span class="o">=</span><span class="m">10000</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--crash-dumps-dir<span class="o">=</span>./html/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--disable-gpu<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--print-to-pdf<span class="o">=</span>./PDF/<span class="si">${</span><span class="nv">filename</span><span class="si">}</span>.pdf<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--no-margins<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>file://<span class="nv">$PWD</span>/<span class="nv">$file</span>
<span class="k">done</span>
<span class="s2">"/System/Library/Automator/Combine PDF Pages.action/Contents/Resources/join.py"</span><span class="w"> </span>-o<span class="w"> </span>CMSE314.pdf<span class="w"> </span>./PDF/*.pdf
open<span class="w"> </span>CMSE314.pdf
</code></pre></div>
<p>Once I got the bash script working I decided a Makefile would be better</p>
<div class="highlight"><pre><span></span><code><span class="nv">IPYNB_FILES</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">$(</span>wildcard<span class="w"> </span>*.ipynb<span class="k">)</span>
<span class="nv">PDF_FILES</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">$(</span>patsubst<span class="w"> </span>%.ipynb,<span class="w"> </span>PDF/%.pdf,<span class="w"> </span><span class="k">$(</span>IPYNB_FILES<span class="k">))</span>
<span class="nf">all</span><span class="o">:</span><span class="w"> </span><span class="n">CMSE</span>314.<span class="n">pdf</span>
<span class="w"> </span>open<span class="w"> </span>CMSE314.pdf
<span class="nf">HTML/%.html</span><span class="o">:</span><span class="w"> </span>%.<span class="n">ipynb</span>
<span class="w"> </span>@mkdir<span class="w"> </span>-p<span class="w"> </span><span class="s2">"</span><span class="k">$(</span>@D<span class="k">)</span><span class="s2">"</span>
<span class="w"> </span>jupyter<span class="w"> </span>nbconvert<span class="w"> </span>--to<span class="w"> </span>html<span class="w"> </span>--no-prompt<span class="w"> </span>--allow-errors<span class="w"> </span>--output-dir<span class="w"> </span>HTML<span class="w"> </span><span class="s2">"</span>$<span class="s2"><"</span>
<span class="nf">PDF/%.pdf</span><span class="o">:</span><span class="w"> </span><span class="n">HTML</span>/%.<span class="n">html</span>
<span class="w"> </span>@mkdir<span class="w"> </span>-p<span class="w"> </span><span class="s2">"</span><span class="k">$(</span>@D<span class="k">)</span><span class="s2">"</span>
<span class="w"> </span>/Applications/Google<span class="se">\ </span>Chrome.app/Contents/MacOS/Google<span class="se">\ </span>Chrome<span class="w"> </span>--headless<span class="w"> </span>--crash-dumps-dir<span class="o">=</span><span class="k">$(</span>@D<span class="k">)</span><span class="w"> </span>--disable-gpu<span class="w"> </span>--print-to-pdf<span class="w"> </span>file://<span class="k">$(</span>PWD<span class="k">)</span>/$<
<span class="w"> </span>mv<span class="w"> </span>output.pdf<span class="w"> </span><span class="nv">$@</span>
<span class="nf">CMSE314.pdf</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">PDF_FILES</span><span class="k">)</span>
<span class="w"> </span><span class="s2">"/System/Library/Automator/Combine PDF Pages.action/Contents/Resources/join.py"</span><span class="w"> </span>-o<span class="w"> </span>CMSE314.pdf<span class="w"> </span><span class="k">$(</span>PDF_FILES<span class="k">)</span>
<span class="nf">clean</span><span class="o">:</span>
<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span>HTML
<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span>PDF
</code></pre></div>
<p>Both seem to work and get the job done. The output for the script is a little easier to read but the makefile is nice because it can be restarted (although It can't be run with the -j parallel option because of the output.pdf temporary file).</p>
<p>Anyway, let me know if you find this helpful. I am sure there are better ways but I needed something quick.</p>Live Closed Captioning2020-01-26T00:00:00-05:002020-01-26T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-01-26:/live-closed-captioning.html<p><img alt="Picture of me writing on the white board with live closed captioning" src="//colbrydi.github.io/images/CC_example.png"></p>
<p>Many MSU classrooms have two overhead monitors that project on two screens at the front of class. Each room is equipped with a technology cart that has a desktop computer, overhead camera and laptop connections that can be used to project any of the content to either screen. Up until …</p><p><img alt="Picture of me writing on the white board with live closed captioning" src="//colbrydi.github.io/images/CC_example.png"></p>
<p>Many MSU classrooms have two overhead monitors that project on two screens at the front of class. Each room is equipped with a technology cart that has a desktop computer, overhead camera and laptop connections that can be used to project any of the content to either screen. Up until now I haven't really seen many learning benefit of having two screens in smaller classrooms. However, recently I have been using one screen for my standard presentation and the other screen with a live close caption. Here is a video with a few examples:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/fE1bveNGeR0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p style="text-align:right;">Special Thanks to my Graduate Teaching Assistant, Janez Krek, for taking these videos</p>
<p>https://youtu.be/fE1bveNGeR0
To get this to work I am using <a href="https://support.google.com/docs/answer/9109474?hl=en">Google Slide</a> presentations which have an option for live closed captioning. Here are my steps:</p>
<ol>
<li>Log into the classroom desktop with my MSU login/password</li>
<li>Open up Chrome</li>
<li>Log into <a href="http://googleapps.msu.edu">http://googleapps.msu.edu</a> drive using my MSU login/password</li>
<li>Open a new Google Slides file and set the background to black</li>
<li>Start the presentation and hit the 'CC' button which appears at the bottom of the screen when I move my mouse</li>
<li>Attach the portable room microphone to my lapel and turn it on</li>
<li>Adjust volume as appropriate for room and test the captioning</li>
</ol>
<p>Once I have closed captioning work on the right monitor, I plug in my laptop and use the technology cart interface to set the left screen and desktop monitor to the laptop input. Now I can control my lecture content from my laptop and the close captioning will work on a mostly black screen. </p>
<p>Obviously student's with a hearing disability (temporary and long term) may benefit from the technology but overall I have found many students appreciate the extra information. When I am excited I can talk fast and this helps students keep up with what I am saying. I also have had positive feedback from international students for which English is a second language. </p>
<p>Another drawback to this approach is that I can't figure out how to download a transcript from the lecture. I full transcript could be useful resource for me as well as students. </p>JJRobots B-Robot2020-01-18T00:00:00-05:002020-01-18T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-01-18:/jjrobots-b-robot.html<p><img alt="Picture of JJRobots And kids" src="../images/BRobot.jpeg"></p>
<p>A few years ago I purchased a kit from <a href="//www.jjrobots.com/">JJRobots</a>. The kit provides all non-plastic parts and then allows you to build a variety of robots. Up until now I only made the <a href="Sphere-O-Bot (JJRobots version)">eggbot</a> which tends to come out around Easter time, the kids love it.</p>
<p>Last week my niblets …</p><p><img alt="Picture of JJRobots And kids" src="../images/BRobot.jpeg"></p>
<p>A few years ago I purchased a kit from <a href="//www.jjrobots.com/">JJRobots</a>. The kit provides all non-plastic parts and then allows you to build a variety of robots. Up until now I only made the <a href="Sphere-O-Bot (JJRobots version)">eggbot</a> which tends to come out around Easter time, the kids love it.</p>
<p>Last week my niblets and children helped me put together a 3D printer. I used the printer to print the last few parts of the B-Robot. The bumpers came out backwards but serviceable. </p>
<p><img alt="Printer and Robot Parts" src="../images/Tronxy-B-Robot.jpeg"></p>
<p>This week we got together, took apart the eggbot and then assembled the b-robot. It was a lot of fun, the kids had a great time. To be honest only my nephue really had the pacients to stick it out and complete the robot but it was still a lot of fun.</p>
<p>The one big challenge for me was that the software did not work out of the box. it took me a while to figure out that the version of the kit I had (from a few years ago) was different than the kit currently on the <a href="//www.jjrobots.com/">JJRobots</a> website. Fortunately, once I figured out I was working from old hardware, it was fairly easy to search the site and find the old directions. </p>
<p>With the software installed all the kids had a great time driving it around. It is fun to see the robot balance on two wheels and the app works great. </p>
<p>Both projects (eggbot and B-robot) from <a href="//www.jjrobots.com/">JJRobots</a> were fun to build. I am looking forward to building the other projects with the kids.</p>Tronxy X1 Cura Settings2020-01-12T00:00:00-05:002020-01-12T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-01-12:/tronxy-x1-cura-settings.html<p><img alt="Completed Benchy" src="../images/Tronxy-Benchy.jpeg"></p>
<p>My new <a href="https://www.tronxy.com/x-series/tronxy-x1.html">Tronxy X1</a> has been a lot of fun to put together. The tricky part was finding the right setting for the slicer. Turns out it is not that difficult at all. I was able to download my favorite OpenSource slicer, <a href="https://ultimaker.com/software/ultimaker-cura">CURA</a> and needed to type in the specifications …</p><p><img alt="Completed Benchy" src="../images/Tronxy-Benchy.jpeg"></p>
<p>My new <a href="https://www.tronxy.com/x-series/tronxy-x1.html">Tronxy X1</a> has been a lot of fun to put together. The tricky part was finding the right setting for the slicer. Turns out it is not that difficult at all. I was able to download my favorite OpenSource slicer, <a href="https://ultimaker.com/software/ultimaker-cura">CURA</a> and needed to type in the specifications from the documentation. Here are some screen shots of my Cura settings:</p>
<p><img alt="Machine Settings - Printer" src="../images/Cura1.png">
<img alt="Machine Settings - Extruder" src="../images/Cura2.png">
<img alt="Machine Settings - Details" src="../images/Cura2_Details.png"></p>
<p>Everything seems to be working well. I highly recommend the Tronxy as a great kit to make your first 3D printer.</p>Tronxy X12020-01-11T00:00:00-05:002020-01-11T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2020-01-11:/tronxy-x1.html<p><img alt="Brother and the TronXY X1" src="../images/Tronxy-Frank.jpeg"></p>
<p>My "baby" brother is a new professor and designing an additive manufacturing course for his students. He found a fairly affordable (price of a typical textbook) 3D printer kit called a <a href="https://www.tronxy.com/x-series/tronxy-x1.html">Tronxy X1</a> for his students to put together and take home. </p>
<p>To get ready, he and I purchased our …</p><p><img alt="Brother and the TronXY X1" src="../images/Tronxy-Frank.jpeg"></p>
<p>My "baby" brother is a new professor and designing an additive manufacturing course for his students. He found a fairly affordable (price of a typical textbook) 3D printer kit called a <a href="https://www.tronxy.com/x-series/tronxy-x1.html">Tronxy X1</a> for his students to put together and take home. </p>
<p>To get ready, he and I purchased our own printers and we put them together this weekend with all of the kids. This is a really great kit. The printer has tons of amazing features for something so affordable:</p>
<ul>
<li>Interactive screen and SD card reader design that doesn't require an attached computer while it is printing</li>
<li>The controller board is obviously designed for a more advanced printer and allows for upgrades like additional fans and a heated printer bed</li>
<li>Did I mention this is affordable (~$90)</li>
</ul>
<p><img alt="Nephew and the TronXY X1" src="../images/Tronxy-Alex2.jpeg"></p>
<p>The build took us about 6 hours from unboxing to cleanup (after printing). It was a great way to pass a Saturday. We had to make a few tweaks to the build but overall I was pleased. Some things included:</p>
<ul>
<li>My kit was missing some spacers but I was able to replace them with a couple of extra nuts. I may print out new spacers but so far it is working fine.</li>
<li>The SD card only came with one simple gcode example. It was not a very impressive piece</li>
<li>I could not find instructions to set up the slicer. None of the common ones such as <a href="https://slic3r.org/">Slic3r</a> or <a href="https://ultimaker.com/software/ultimaker-cura">Cura</a> seemed to have TronXY X1 set up already. Turns out this wasn't too bad to fix.</li>
</ul>
<p>Overall it was a great kit and my Nephew in particular is excited to get on <a href="https://www.thingiverse.com/">thingaverise</a> to see if he can find models to download, slice and print.</p>Cylindrical Lithophans2019-12-25T00:00:00-05:002019-12-25T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-12-25:/cylindrical-lithophans.html<p><img alt="2019 Picture of kids" src="../images/Kids_2019.jpg"></p>
<p>Lithophans are one of my favorite things to 3D print. I have written software in MATLAB and python that can convert images to lithophans. Most years I design a lithophan and 3D printable frame to give away to family for Christmas and each year I like to try to do …</p><p><img alt="2019 Picture of kids" src="../images/Kids_2019.jpg"></p>
<p>Lithophans are one of my favorite things to 3D print. I have written software in MATLAB and python that can convert images to lithophans. Most years I design a lithophan and 3D printable frame to give away to family for Christmas and each year I like to try to do something different, creative and a little more challenging. </p>
<p>This year we had a picture of all 9 kids (my four, our brothers four and our exchange student) all standing in a line between our two houses. This picture works best as a long and thin image (See above). I thought it would be cool to print this on a cylinder with a light source on the inside and then have it rotate. It took me a while to get the mesh right but the results turned out quite good:</p>
<iframe width="200" height="315" align="center" src="https://www.youtube.com/embed/4I3ItcZOAjM" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>I also had fun with my time-lapse camera during the print. I added in an electronic tea light when the print head was high enough and I managed to show the lithophan as it is being printed.</p>
<iframe width="500" height="315" align="center" src="https://www.youtube.com/embed/-h8pF6psdp4?start=1" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>The last thing I did was make a "lid" for the cylinder so that I could include a picture of my niece who was the only grandchild not in the picture from the two houses. For this one I just made one of my standard square lithophans and then added a circular frame with a stepped lid so that It would fit snugly in the cylinder. It took a few iterations to get the lid part working but I just drilled a hole in the top and used the failures as ornaments for my brother and sister to give away.</p>CMSE Impact Video2019-12-14T00:00:00-05:002019-12-14T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-12-14:/cmse-impact-video.html<p><img alt="Picture of me" src="//colbrydi.github.io/images/3_horiz.png"></p>
<p>Xiaoxing (Adele) Han is a videographer who made some amazing videos to help promote research being done in our department. The following is the video she made of my research which I am very excited to share. I think she did a great job!</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/VdsKHhncXYw" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Thank you Adele!!!</p>Running ImageJ on the HPCC2019-12-04T00:00:00-05:002019-12-04T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-12-04:/running-imagej-on-the-hpcc.html<p><img align="right," alt="Fiji logo" hspace="10" src="//imagej.net/_images/a/ae/Fiji-icon.png" width="200,"></p>
<p>A group working over in Plant and Soil Sciences are experts on ImageJ but are relatively new to the HPCC. Together we put together a workflow for getting ImageJ up and running on the HPCC.</p>
<h3>Step 1: Log onto the HPCC with an X11 connection</h3>
<p>Log onto the HPC using …</p><p><img align="right," alt="Fiji logo" hspace="10" src="//imagej.net/_images/a/ae/Fiji-icon.png" width="200,"></p>
<p>A group working over in Plant and Soil Sciences are experts on ImageJ but are relatively new to the HPCC. Together we put together a workflow for getting ImageJ up and running on the HPCC.</p>
<h3>Step 1: Log onto the HPCC with an X11 connection</h3>
<p>Log onto the HPC using an X11 connection. You need x11 in order to test the install using the Graphical User Interface (GUI). If you are on windows, I recommend <a href="//mobaxterm.mobatek.net/">MobaXTerm</a> as an easy way to get started, if you are on Mac you may need to install <a href="https://www.xquartz.org/">XQuarts</a>.</p>
<p>If you don't have X11 working you can try logging in using the HPCC remote server and a "full linux desktop" by going to the following website:</p>
<p>https://webrdp.hpcc.msu.edu</p>
<p>You can test if graphics are working by typing the following on the command line:</p>
<div class="highlight"><pre><span></span><code>xeyes
</code></pre></div>
<h3>Step 2: Install ImageJ</h3>
<p>For this step I recommend installing <a href="https://fiji.sc/">Fiji</a> (Fiji is just ImageJ) which a little easier to use than ImageJ proper. For one thing it makes installing a few plug-ins much easier.</p>
<p>Download the 64-bit installer into your home directory on the HPCC. Go to the <a href="//imagej.net/Fiji/Downloads">Fiji</a> website to get a URL link to the latest version. Once you have the URL you can run the following command on an HPC Development node:</p>
<div class="highlight"><pre><span></span><code>wget<span class="w"> </span>https://downloads.imagej.net/fiji/latest/fiji-linux64.zip
</code></pre></div>
<p>After the file has downloaded you just need to unzip the file using the following command:</p>
<div class="highlight"><pre><span></span><code>unzip<span class="w"> </span>fiji-linux64.zip
</code></pre></div>
<p>Test ImageJ using the following command:</p>
<div class="highlight"><pre><span></span><code>~/Fiji.app/ImageJ-linux64
</code></pre></div>
<p>The first time running it will ask if you want to run the "updater." I would recommend doing this.</p>
<h3>Step 3: Install needed plugins</h3>
<p>Once you have Fiji running you shold be able to install most plugins from the user interface. You can also copy plugin jar files to the plugins folder inside the Fiji.app folder.</p>
<h3>Step 4: Write a Macro</h3>
<p>You can run for about 2 hours on any of the HPCC development nodes, however, to really take advantage of the HPCC you want to run in batch mode. Batch mode does not let you use the mouse and click on the graphics which means you need to create an ImageJ macro that runs from start to end without any user input. For now I will assume that you know how to make a macro, if not, you may have to look for some sort of online tutorial or read the <a href="https://imagej.nih.gov/ij/docs/guide/146-14.html">manual</a>,</p>
<p>If you want to just do a quick test I recmmend making a macro with just a "hello world" print command as follows:</p>
<div class="highlight"><pre><span></span><code>print(hello world);
</code></pre></div>
<p>Save the file as <code>myMacro.ijm</code></p>
<h3>Step 5: Submit the macro to the scheduler</h3>
<p>Let us assume that your macro name is <code>myMacro.ijm</code> then the following submission script should work to get your macro running in SLURM:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#SBATCH --mem=4gb</span>
<span class="c1">#SBATCH --time=04:00:00</span>
<span class="c1">#SBATCH -n 1</span>
<span class="c1">#SBATCH -c 1</span>
module<span class="w"> </span>load<span class="w"> </span>java
module<span class="w"> </span>load<span class="w"> </span>powertools
srun<span class="w"> </span>~/Fiji.app/ImageJ-linux64<span class="w"> </span>--headless<span class="w"> </span>--memory<span class="o">=</span>4000M<span class="w"> </span>-macro<span class="w"> </span>myMacro.ijm
js<span class="w"> </span><span class="nv">$SLURM_JOB_ID</span>
</code></pre></div>
<p>If we call the above script <code>runFiji.sb</code> then you can submit the job to the SLURM scheduler using the following command:</p>
<div class="highlight"><pre><span></span><code>sbatch<span class="w"> </span>runFiji.sb
</code></pre></div>
<p>That should be it. You can change the required memory or the amount of time it takes to run. </p>CMSE Presentation at SC192019-11-17T00:00:00-05:002019-11-17T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-11-17:/cmse-presentation-at-sc19.html<p><img alt="SC19 Logo" src="https://sc19.supercomputing.org/app/uploads/2018/11/ogimage_1200.png"></p>
<p>I just got out of giving a presentation about the CMSE department as part of the <a href="https://sc19.supercomputing.org/session/?sess=sess104">Sixth SC Workshop on Best Practices for HPC Training and Education (BPHTE19)</a> workshop at <a href="https://sc19.supercomputing.org/">SC19</a>. It was fun to show people what we are doing in CMSE and share our progress. <a href="//colbrydi.github.io/images/SC19_bphpcte_Colbrydi.pdf">Here is a …</a></p><p><img alt="SC19 Logo" src="https://sc19.supercomputing.org/app/uploads/2018/11/ogimage_1200.png"></p>
<p>I just got out of giving a presentation about the CMSE department as part of the <a href="https://sc19.supercomputing.org/session/?sess=sess104">Sixth SC Workshop on Best Practices for HPC Training and Education (BPHTE19)</a> workshop at <a href="https://sc19.supercomputing.org/">SC19</a>. It was fun to show people what we are doing in CMSE and share our progress. <a href="//colbrydi.github.io/images/SC19_bphpcte_Colbrydi.pdf">Here is a link for my slides</a>. </p>
<h2>Computational Mathematics, Science and Engineering (CMSE): Establishing an Academic Department: Dedicated to Scientific Computation as a Discipline</h2>
<p>By Dirk Colbry, Michael Murillo, Adam Alessio and Andrew Christlieb</p>
<h3>Abstract</h3>
<p>The Computational Mathematics, Science and Engineering (CMSE) department is one of the newest units at Michigan State University (MSU). Founded in 2015, CMSE recognizes computation as the “triple junction” of algorithm development and analysis, high performance computing, and applications to scientific and engineering modeling and data science (as illustrated in Figure 1). This approach is designed to engage with computation as a new integrated discipline, rather than a series of decentralized, isolated sub-specialties. In the four years since its inception, the department has grown and flourished; however, the pathway was sometimes arduous. This paper shares lessons learned during the department’s development and the initiatives it has taken on to support computational research and education across the university. By sharing these lessons, we hope to encourage and support the establishment of similar departments at other universities and grow this integrated approach to scientific computation as a discipline.</p>Surplus Workbench2019-10-04T00:00:00-04:002019-10-04T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-10-04:/surplus-workbench.html<p><img alt="Finished Workbench" src="../images/Finished_workbench.JPG"></p>
<p>Since moving into our new home I have been wanting to build/buy a workbench for my shop. I have been living with a piece of plywood on some saw horses (bowing in the middle). I wanted something more rugged but on casters so I could roll the bench out …</p><p><img alt="Finished Workbench" src="../images/Finished_workbench.JPG"></p>
<p>Since moving into our new home I have been wanting to build/buy a workbench for my shop. I have been living with a piece of plywood on some saw horses (bowing in the middle). I wanted something more rugged but on casters so I could roll the bench out into my main garage when I needed space or if I wanted to clean up. I also really liked the idea of having a workbench where I could swap out tools mounted to 2x2' boards. The idea is the tools could be stored in a shelf and then the 2x2' boards could easily be set into the bench when needed. It was mostly an idea in the back of my head but I was having fun doodling designs. </p>
<p><img alt="Bench at surplus store" src="../images/Surplus_pick.JPG"></p>
<p>I was passing though the <a href="https://msusurplusstore.com/">MSU surplus store</a> and came across the perfect bench. Actually I had been looking though the store specifically for something I could use as a workbench and/or something with parts I could up-cycle into a workbench (like casters) but I was on my way out the door disappointed. As I was leaving someone else was pushing a workbench out the door that looked exactly like what I wanted. I was kicking myself that I missed the bench and, as I walked to my car, I longingly looked over my solder to watch, whomever beat me to the bench, put it in their car. However, I realized that the person with the bench was not a customer but actually someone putting it outside on the store curb with other items for sale. I quickly grabbed the bench and brought it home.</p>
<p>The main working space is 6' x 2' so I cut it into 3 square 2x2' boards. I can now swap out any of the boards and the modular workspace is really nice. So far I have my router, my grinder and my vice mounted to 2x2' boards I can easily swap out. </p>
<p>It fits really nice in my tiny workshop and having a great bench has been a wonderful catalyst for me working in my shop on a variety of projects. </p>Sherpas of Supercomputing2019-09-15T00:00:00-04:002019-09-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-09-15:/sherpas-of-supercomputing.html<p><img alt="Cover picture of the September 2019 ASEE PRISM Magazine Featuring the Sherpas of Supercomputing article" src="http://www.asee-prism.org/wp-content/uploads/2019/10/Sherpas-of-Supercomputing.jpg"></p>
<p>I had the honor of recently being featured in an ASEE PRISM Magazine article about research facilitation. </p>
<blockquote>
<p>An essential, if not yet well-defined, field is emerging at the intersection of research and the high-performance machines that make sense of big data.
By Lucy Birmingham and Mark Matthews</p>
</blockquote>
<p><a href="http://www.asee-prism.org/sherpas-of-supercomputing/">See the full …</a></p><p><img alt="Cover picture of the September 2019 ASEE PRISM Magazine Featuring the Sherpas of Supercomputing article" src="http://www.asee-prism.org/wp-content/uploads/2019/10/Sherpas-of-Supercomputing.jpg"></p>
<p>I had the honor of recently being featured in an ASEE PRISM Magazine article about research facilitation. </p>
<blockquote>
<p>An essential, if not yet well-defined, field is emerging at the intersection of research and the high-performance machines that make sense of big data.
By Lucy Birmingham and Mark Matthews</p>
</blockquote>
<p><a href="http://www.asee-prism.org/sherpas-of-supercomputing/">See the full article here</a></p>Developing curriculum to train Graduate students how to utilize Raspberry Pis to Automate Research Labs2019-07-24T00:00:00-04:002019-07-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-07-24:/developing-curriculum-to-train-graduate-students-how-to-utilize-raspberry-pis-to-automate-research-labs.html<p><img alt="Picture of Shelby in front of her poster" src="//colbrydi.github.io/images/2019_ENSURE_Shelby.jpg"></p>
<p>Summer 2019 CMSE <a href="https://cmse.msu.edu/employment-opportunities/cmse-student-technical-support/">CMSE Undergraduate Intern</a> Poster presentation at MidSure by Shelby Santos</p>
<p>Raspberry Pis (RPis) are affordable microcomputers (approx. $35) that have the capability to revolutionize the accuracy and efficiency of data collection in basic laboratory environments. Recreationally, RPis can be used as a base for smart televisions, home …</p><p><img alt="Picture of Shelby in front of her poster" src="//colbrydi.github.io/images/2019_ENSURE_Shelby.jpg"></p>
<p>Summer 2019 CMSE <a href="https://cmse.msu.edu/employment-opportunities/cmse-student-technical-support/">CMSE Undergraduate Intern</a> Poster presentation at MidSure by Shelby Santos</p>
<p>Raspberry Pis (RPis) are affordable microcomputers (approx. $35) that have the capability to revolutionize the accuracy and efficiency of data collection in basic laboratory environments. Recreationally, RPis can be used as a base for smart televisions, home automation or video games. However, by utilizing basic robotics and circuits in the workplace, microcomputers such as RPis can be programmed to automate data collection (such as temperature measurements or time laps photography) and output sophisticated plots, create response mechanisms, and other mechanical or electronic tools to automate a research and save significant amounts of researcher time while producing more precise measurements. In this project we are exploring the capabilities of RPis for graduate laboratories and developing curriculum to teach graduate students how to take advantage of this highly flexible and affordable technology. </p>Investigation of the process of converting a C++ code into an implementable FPGA file2019-07-24T00:00:00-04:002019-07-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-07-24:/investigation-of-the-process-of-converting-a-c-code-into-an-implementable-fpga-file.html<p><img alt="Picture of Ana in front of her poster" src="//colbrydi.github.io/images/2019_ENSURE_Ana.jpg"></p>
<p>Summer 2019 <a href="https://www.egr.msu.edu/graduate/ensure">ENSURE Student</a> Poster presentation at MidSure by Ana Flavia Borges de Almeida Barreto</p>
<p>A Field Programmable Gate Arrays (FPGAs) is an integrated circuit that can be “rewired” to become other circuits. This ability makes FPGAs highly configurable and can significantly help speed up large scale computation used in …</p><p><img alt="Picture of Ana in front of her poster" src="//colbrydi.github.io/images/2019_ENSURE_Ana.jpg"></p>
<p>Summer 2019 <a href="https://www.egr.msu.edu/graduate/ensure">ENSURE Student</a> Poster presentation at MidSure by Ana Flavia Borges de Almeida Barreto</p>
<p>A Field Programmable Gate Arrays (FPGAs) is an integrated circuit that can be “rewired” to become other circuits. This ability makes FPGAs highly configurable and can significantly help speed up large scale computation used in scientific research. While useful, FPGAs are often underutilized because of the complexity of developing circuits using Hardware Descriptive Languages (HDLs). Software compilers exist (ex. OpenCL and Merlin) to make circuit design easier but these compilers require many hours to translate C++ into a variable circuit implementable file and can fail to even find a solution. Compiling a C++ code into an implementable file has three main steps. First, converting the C++ code into an HDL code, running the implementation of that code, and finally mapping the implementation for the specific FPGA. The first two steps are relatively quick to complete. However, the mapping part is what makes the whole process slow. It is difficult for the computer to take an arbitrary HDL circuit diagram and map it to the FPGA framework. The calculations is an optimization problem that requires searching though many different circuit pathways to find one that will work. The premise of this project is to explore methods to more efficiently converting C++ into FPGA applicable file. As this is a big problem to tackle, the first objective is to understand how the mapping is currently done (using Intel FPGA SDK), make a practical guide of installing and using the open source compiler and explore alternatives to the existing FPGA programming workflows. </p>Searching the Algorithm Space to Automate Scientific Image Analysis Workflows2019-07-24T00:00:00-04:002019-07-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-07-24:/searching-the-algorithm-space-to-automate-scientific-image-analysis-workflows.html<p><img alt="Picture of Noah in front of her poster" src="//colbrydi.github.io/images/2019_ENSURE_Noah.jpg"></p>
<p>Summer 2019 <a href="https://grad.msu.edu/srop">SROP student</a> poster presentation at MidSure by Noah Stolz</p>
<p>Scientific data is often gathered in the form of Images. Scientific Image analysis is the process of pulling specific measurements out of images. Which measures are important complete depend on the scientific question being asked. For this reason, there …</p><p><img alt="Picture of Noah in front of her poster" src="//colbrydi.github.io/images/2019_ENSURE_Noah.jpg"></p>
<p>Summer 2019 <a href="https://grad.msu.edu/srop">SROP student</a> poster presentation at MidSure by Noah Stolz</p>
<p>Scientific data is often gathered in the form of Images. Scientific Image analysis is the process of pulling specific measurements out of images. Which measures are important complete depend on the scientific question being asked. For this reason, there is no single image analysis software that scientists can use to analyze their data. Currently new software needs to be written for every problem. This research aims to automate the software development by focusing on common image analysis workflows. We are developing methods to formulate these workflows and search the “algorithm space” to find the specific workflow for a specific scientific problem.<br>
As an example and proof of concept, this research will look at the common workflow of Image Segmentation. Image segmentation is a process used in many fields of research, from analyzing cells in petri dishes to identifying different crops in fields. However, image segmentation is a time- consuming process. Although an individual picture may take a short amount of time for a researcher to manually segment, this task can become an overwhelming burden for large data sets. While there are many image segmentation algorithms for specific data sets, no single algorithm will generalize well across other data sets. In this project, we aim to create a tool that helps research segment any image based data set and try to find an algorithm that will automate the process for the particular data analysis task. The success of this tool is in reducing the amount of research time it takes to segment an image data set. In this prototype, we will use python’s skimage.segmentation library which provides a comprehensive set of image segmentation algorithms. Each algorithm in the skimage.segmentation has multiple parameters which leads to a large search space of different parameters and algorithms. We call this search space the “Algorithm Space”. We will apply genetic algorithms which are known to effectively evaluate large, non-differentiable search spaces to the image segemntaiton “Algorithm Space”. This tool will be tested against prelabeled datasets and measure amount of time the tool saves for the researcher to segment the images.</p>The Keyboard Trap: Making Jupyter Notebooks Accessible to All Students2019-07-24T00:00:00-04:002019-07-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-07-24:/the-keyboard-trap-making-jupyter-notebooks-accessible-to-all-students.html<p><img alt="Picture of Abudit in front of her poster" src="//colbrydi.github.io/images/2019_EnSURE_Abudit.jpg"></p>
<p>Summer 2019 <a href="https://cmse.msu.edu/employment-opportunities/cmse-student-technical-support/">CMSE Undergraduate Intern</a> Poster presentation at MidSure by Abudit Rai:</p>
<p>Jupyter Notebook is a relatively new environment that incorporates formatted text (using markdown), multimedia (video and pictures using html) and executable code where users can write and execute code within “cells” for languages such as Python, Julia, and …</p><p><img alt="Picture of Abudit in front of her poster" src="//colbrydi.github.io/images/2019_EnSURE_Abudit.jpg"></p>
<p>Summer 2019 <a href="https://cmse.msu.edu/employment-opportunities/cmse-student-technical-support/">CMSE Undergraduate Intern</a> Poster presentation at MidSure by Abudit Rai:</p>
<p>Jupyter Notebook is a relatively new environment that incorporates formatted text (using markdown), multimedia (video and pictures using html) and executable code where users can write and execute code within “cells” for languages such as Python, Julia, and R making it a particularly effective means of teaching students concepts that involve programming. Because notebooks help in communication, many instructors prefer to implement class content using Jupyter Notebooks. However, there are currently several accessibility issues with the current Jupyter Notebook software that prevents every user from having an equal opportunity to use this platform. This research investigates one of the largest problems; the issue of keyboard traps where keyboard navigation of a website becomes stuck after a certain point, effectively trapping a user who does not have the ability to use the mouse. By reviewing the Javascript code for Jupyter Notebooks, this project will look at key bindings and commands to see how the interface is set up, find the areas that cause this trap to occur, come up with ways to fix the potential solutions and provide the best fix to the OpenSource Jupyter Software Community Project.</p>Surplus Tool Cabinet2019-07-09T00:00:00-04:002019-07-09T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-07-09:/surplus-tool-cabinet.html<p><img alt="Picture of my Surplus tool Cabinet" src="../images/SurplusToolCabinet.JPG"></p>
<p>I found this really durable whiteboard conference room cabinet at the <a href="https://msusurplusstore.com/">MSU surplus store</a> and converted it to a tool cabinet. </p>
<p>The original cabinet had 3/4" whiteboard panels in the back and additional panels on the inside of each door:</p>
<p><img alt="Whiteboard on door" src="../images/Whiteboards.JPG"></p>
<p>These things were really heavy! I removed all of …</p><p><img alt="Picture of my Surplus tool Cabinet" src="../images/SurplusToolCabinet.JPG"></p>
<p>I found this really durable whiteboard conference room cabinet at the <a href="https://msusurplusstore.com/">MSU surplus store</a> and converted it to a tool cabinet. </p>
<p>The original cabinet had 3/4" whiteboard panels in the back and additional panels on the inside of each door:</p>
<p><img alt="Whiteboard on door" src="../images/Whiteboards.JPG"></p>
<p>These things were really heavy! I removed all of the whiteboards and added a pegboard. I added this to the wall of my garage next too my car. It looks great and fits nicely. </p>2019 Ensure Talk - An Engineers Autobiography2019-06-25T00:00:00-04:002019-06-25T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-06-25:/2019-ensure-talk-an-engineers-autobiography.html<p><img alt="EnSURE Logo" src="//www.egr.msu.edu/graduate/sites/default/files/content/ensure_logo.jpg"></p>
<p>I gave my annual autobiographical talk today to the <a href="//www.egr.msu.edu/graduate/ensure">EnSURE</a> (Engineering Summer Undergraduate Research Experience) students. I give this talk every year and it is noticeably different that most of my presentations because the EnSURE organizers asked me to talk about myself and how I got where I am (instead …</p><p><img alt="EnSURE Logo" src="//www.egr.msu.edu/graduate/sites/default/files/content/ensure_logo.jpg"></p>
<p>I gave my annual autobiographical talk today to the <a href="//www.egr.msu.edu/graduate/ensure">EnSURE</a> (Engineering Summer Undergraduate Research Experience) students. I give this talk every year and it is noticeably different that most of my presentations because the EnSURE organizers asked me to talk about myself and how I got where I am (instead of research or teaching). The first time I gave this talk it was really awkward. However, now I think it is kind of fun to tell stories about the choices I made throughout school and my carrier. I get a lot of questions when I am done so I like to think the student's enjoyed the talk as well. This year was a basic carbon copy of last years talk with a little design update to the slides. Here are a copy of my slides:</p>
<p><a href="./Images/20190625_An_Engineers_Autobiography.pdf">20190625_An_Engineers_Autobiography.pdf</a></p>Research Team Retreat2019-06-18T00:00:00-04:002019-06-18T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-06-18:/research-team-retreat.html<p><img alt="Research group in front of college van" src="//colbrydi.github.io/images/Group_Van.jpg"></p>
<p>This summer we managed to find a time to take my research team on a retreat (Road Trip!) to the lake (Noah, Abudit, Shelby, Lilly and Anna). It was a great opportunity to get to know each other and really bond as a team. We played games, had a boat …</p><p><img alt="Research group in front of college van" src="//colbrydi.github.io/images/Group_Van.jpg"></p>
<p>This summer we managed to find a time to take my research team on a retreat (Road Trip!) to the lake (Noah, Abudit, Shelby, Lilly and Anna). It was a great opportunity to get to know each other and really bond as a team. We played games, had a boat ride, went swimming and even got some work done. </p>
<p>Overall it was a great trip. I hope to be able to maintain this new tradition with future summer students.</p>
<p><img align="right," alt="Playing a little cards while Dirk is on a Phone call" hspace="10" src="//colbrydi.github.io/images/Cards.jpg" width="33%,"> <img align="right," alt="Dirk on a Phone call" hspace="10" src="//colbrydi.github.io/images/Work.jpg" width="33%,"> <img align="right," alt="Team taking a Boat Ride" hspace="10" src="//colbrydi.github.io/images/Boat.jpg" width="33%,"> <img align="right," alt="Big kids can play tether ball too" hspace="10" src="//colbrydi.github.io/images/TetherBall.jpg" width="33%,"><img align="right," alt="Another Group Photol" hspace="10" src="//colbrydi.github.io/images/Lake.jpg" width="66%,"></p>Workshop Community Building for High-Performance Computing Curriculum development2019-06-10T00:00:00-04:002019-06-10T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-06-10:/workshop-community-building-for-high-performance-computing-curriculum-development.html<p><img align="right," alt="National Computational Science Institute Logo" hspace="10" src="http://shodor.org/logos/logos/NCSI/thumb.jpg" width="200,"></p>
<p><img align="right," alt="XSEDE Logo" hspace="10" src="https://www.xsede.org/image/image_gallery?uuid=c0ae4cfa-fa0e-4546-8b02-3305bf2a99cc&groupId=10157&t=1369258426990" width="200,"></p>
<p>Last year, I attended a two-and-a-half-day community and curriculum building workshop. It could not have been more perfectly timed. I was able to come up with a solid plan for my CMSE401 parallel programming course which I ended up teaching this last spring. </p>
<p>This year I was not able to …</p><p><img align="right," alt="National Computational Science Institute Logo" hspace="10" src="http://shodor.org/logos/logos/NCSI/thumb.jpg" width="200,"></p>
<p><img align="right," alt="XSEDE Logo" hspace="10" src="https://www.xsede.org/image/image_gallery?uuid=c0ae4cfa-fa0e-4546-8b02-3305bf2a99cc&groupId=10157&t=1369258426990" width="200,"></p>
<p>Last year, I attended a two-and-a-half-day community and curriculum building workshop. It could not have been more perfectly timed. I was able to come up with a solid plan for my CMSE401 parallel programming course which I ended up teaching this last spring. </p>
<p>This year I was not able to attend the workshop in Tulsa OK (darn jury duty!):</p>
<p><a href="https://portal.xsede.org/course-calendar/-/training-user/class/1311/session/2613">link to CBHPCCD19</a></p>
<p>However, I did arrange with the organizer (Aaron Weeden) and was able to give a video conference presentation about my class. Hopefully this year's participants found my talk useful. Here is a link to my slides in case anyone is intertested:</p>
<p><a href="//colbrydi.github.io/images/20190612-MSU_HPC_CMSE401.pdf">Link to Slides</a></p>Do More, Faster: Utilizing Advanced Computational Resources in Your Research Team2019-05-21T00:00:00-04:002019-05-21T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-05-21:/do-more-faster-utilizing-advanced-computational-resources-in-your-research-team.html<p><img alt="INSciTS Logo" src="https://www.inscits.org/assets/site/inscits-logo.png"></p>
<p>Here are my slides from the <a href="https://www.inscits.org/">SciTS conference</a> last month. </p>
<blockquote>
<p>The SciTS conference is the annual international forum dedicated to SciTS, bringing together thought leaders from a broad range of disciplines and fields, including: communications, management, social and behavioral sciences, information technology, systems science, and translational research. It provides investigators …</p></blockquote><p><img alt="INSciTS Logo" src="https://www.inscits.org/assets/site/inscits-logo.png"></p>
<p>Here are my slides from the <a href="https://www.inscits.org/">SciTS conference</a> last month. </p>
<blockquote>
<p>The SciTS conference is the annual international forum dedicated to SciTS, bringing together thought leaders from a broad range of disciplines and fields, including: communications, management, social and behavioral sciences, information technology, systems science, and translational research. It provides investigators, academic administrators, and funders wit h state-of-the-art knowledge, strategies, and connections. SciTS scholars, scientists engaged in team-based research, institutional leaders who promote collaborative research, policymakers, and federal agency representatives will be in attendance.</p>
</blockquote>
<p><strong>Abstract</strong>
The use of technology in research is becoming ubiquitous as low cost programmable sensors and advanced computing (e.g., AI and big data) emerge in nearly every domain. Out of necessity, many scientists and domain experts have become Technology Users: individuals who need to employ advanced technology in their research, but who do not have broad expertise in topics such as engineering and programming. Modern laptop and desktop computers are extremely powerful, and most scientists can accomplish 90-95% of what they need with a computer in their office. However, much of today’s cutting edge science needs bigger, stronger and faster computers. High Performance Computing (HPC) systems (aka supercomputers) are designed to take science to the next level. Universities and national funding agencies (NSF, NIH, DOD, DOE, etc.) provide researchers access to large scale computing resources, often free of charge. The most common barrier to scientists who want to utilize these resources is knowledge: that these tools exist, and how to get started using them.</p>
<p>This talk introduced the world of large scale computing. We discussed how to gain access to both local and national resources, the common types of problems that can be solved using these resources, and training programs specifically designed to help new researchers leverage these technologies and use them effectively in their own research.</p>
<p><a href="//colbrydi.github.io/images/20190521-SciTS.pdf">Link to the slides</a></p>Automating Scientific Image Analysis Workflows2019-04-23T00:00:00-04:002019-04-23T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-04-23:/automating-scientific-image-analysis-workflows.html<p><img align="right," alt="GA Title Slide" hspace="10" src="//colbrydi.github.io/images/20190423_GA_Talk.png" width="200,"></p>
<p>I was asked to give a talk to Michael Murillo machine learning seminar group. Michael put the group together as a first step to gather topics/ideas for a graduate level machine learning class that CMSE will be teaching in the fall.</p>
<p>In this talk I introduced a difficult machine …</p><p><img align="right," alt="GA Title Slide" hspace="10" src="//colbrydi.github.io/images/20190423_GA_Talk.png" width="200,"></p>
<p>I was asked to give a talk to Michael Murillo machine learning seminar group. Michael put the group together as a first step to gather topics/ideas for a graduate level machine learning class that CMSE will be teaching in the fall.</p>
<p>In this talk I introduced a difficult machine learning problem in the area of Exploratory Image Analysis. I explained why this problem is unique and not easily solved using machine learning. Then I introduced my ideas on how we plan to try and solve the problem this summer using machine learning and, more specifically, Genetic Algorithms. </p>
<p><a href="//colbrydi.github.io/images/20190423_GA_Image_Segmentation.pdf">Link to the slides</a></p>Extracting Desire to learn (D2L) zip files into student folders2019-04-15T00:00:00-04:002019-04-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-04-15:/extracting-desire-to-learn-d2l-zip-files-into-student-folders.html<p><img align="right," alt="D2L Logo" hspace="10" src="https://info.d2l.com/hs-fs/hubfs/D2L-web-500.png?width=500&name=D2L-web-500.png" width="200,"></p>
<p>At MSU we use Desire to Learn for our classroom content management system. Students can go to D2L to get their assignments and turn in their assignments though the D2L dropbox. </p>
<p>When grading, I can download all of the student's submissions into a zip file such as the following:</p>
<p><code>Part …</code></p><p><img align="right," alt="D2L Logo" hspace="10" src="https://info.d2l.com/hs-fs/hubfs/D2L-web-500.png?width=500&name=D2L-web-500.png" width="200,"></p>
<p>At MSU we use Desire to Learn for our classroom content management system. Students can go to D2L to get their assignments and turn in their assignments though the D2L dropbox. </p>
<p>When grading, I can download all of the student's submissions into a zip file such as the following:</p>
<p><code>Part 1 - Software Evaluation Download Mar 23, 2019 609 AM.zip</code></p>
<p><code>HW1 - Matrix Transpose Download Feb 2, 2019 1001 AM.zip</code></p>
<p>If this folder is unzipped files similar to the following will appear:</p>
<div class="highlight"><pre><span></span><code><span class="mf">323771</span><span class="o">-</span><span class="mf">372818</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Sam</span><span class="w"> </span><span class="n">Smith</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Jan</span><span class="w"> </span><span class="mf">21</span><span class="p">,</span><span class="w"> </span><span class="mf">2019</span><span class="w"> </span><span class="mf">652</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">submission</span><span class="mf">.</span><span class="n">tgz</span>
<span class="mf">325034</span><span class="o">-</span><span class="mf">372818</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Nan</span><span class="w"> </span><span class="n">Carpenter</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Jan</span><span class="w"> </span><span class="mf">21</span><span class="p">,</span><span class="w"> </span><span class="mf">2019</span><span class="w"> </span><span class="mf">330</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mf">1</span><span class="n">d</span><span class="w"> </span><span class="n">wave</span><span class="w"> </span><span class="n">eqn</span><span class="mf">.</span><span class="n">pages</span>
<span class="mf">325034</span><span class="o">-</span><span class="mf">372818</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Nan</span><span class="w"> </span><span class="n">Carpenter</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Jan</span><span class="w"> </span><span class="mf">21</span><span class="p">,</span><span class="w"> </span><span class="mf">2019</span><span class="w"> </span><span class="mf">330</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">wave1d</span><span class="mf">.</span><span class="n">py</span>
<span class="mf">331582</span><span class="o">-</span><span class="mf">372818</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Joe</span><span class="w"> </span><span class="n">Taylor</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Jan</span><span class="w"> </span><span class="mf">19</span><span class="p">,</span><span class="w"> </span><span class="mf">2019</span><span class="w"> </span><span class="mf">315</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">HW1</span><span class="mf">.</span><span class="n">ipynb</span>
<span class="mf">331582</span><span class="o">-</span><span class="mf">372818</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Joe</span><span class="w"> </span><span class="n">Taylor</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Jan</span><span class="w"> </span><span class="mf">19</span><span class="p">,</span><span class="w"> </span><span class="mf">2019</span><span class="w"> </span><span class="mf">315</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">HW1_report</span><span class="mf">.</span><span class="n">pdf</span>
<span class="mf">331582</span><span class="o">-</span><span class="mf">372818</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Joe</span><span class="w"> </span><span class="n">Taylor</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Jan</span><span class="w"> </span><span class="mf">19</span><span class="p">,</span><span class="w"> </span><span class="mf">2019</span><span class="w"> </span><span class="mf">315</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">short</span><span class="mf">.</span><span class="n">mp4</span>
<span class="mf">331582</span><span class="o">-</span><span class="mf">372818</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Joe</span><span class="w"> </span><span class="n">Taylor</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Jan</span><span class="w"> </span><span class="mf">19</span><span class="p">,</span><span class="w"> </span><span class="mf">2019</span><span class="w"> </span><span class="mf">320</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">HW1</span><span class="mf">.</span><span class="n">tgz</span>
</code></pre></div>
<p>For many of the topics we cover (mostly programming) these filenames are not easy to use. In my classes student often submit files that need to work together with given file names (everything after the final dash). I wrote the following program to convert the directory of complex filenames into folders. A separate folder for each student with the files in the folder:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nv">files</span><span class="o">=</span><span class="sb">`</span>ls<span class="sb">`</span>
find<span class="w"> </span>.<span class="w"> </span>-iname<span class="w"> </span><span class="s2">"*"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="nb">read</span><span class="w"> </span>file
<span class="k">do</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$file</span>
<span class="w"> </span><span class="nv">folder</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$file</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>cut<span class="w"> </span>-d<span class="w"> </span><span class="s2">"-"</span><span class="w"> </span>-f<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="sb">`</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"creating folder </span><span class="nv">$folder</span><span class="s2">"</span>
<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span><span class="s2">"</span><span class="nv">$folder</span><span class="s2">"</span>
<span class="w"> </span><span class="nv">newname</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$file</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>cut<span class="w"> </span>-d<span class="w"> </span><span class="s2">"-"</span><span class="w"> </span>-f5-<span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="sb">`</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Renaming to </span><span class="nv">$newname</span><span class="s2">"</span>
<span class="w"> </span>mv<span class="w"> </span><span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span><span class="w"> </span><span class="s2">"./</span><span class="nv">$folder</span><span class="s2">/</span><span class="nv">$newname</span><span class="s2">"</span>
<span class="k">done</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>Sam Smith
Nan Carpenter
Joe Taylor
</code></pre></div>
<p>Each directory contains that student's files. For example the <code>Joe Taylor</code> folder will have:</p>
<div class="highlight"><pre><span></span><code>HW1.ipynb
HW1_report.pdf
short.mp4
HW1.tgz
</code></pre></div>
<p>I hope you find this script useful.</p>Sphere-O-Bot (JJRobots version)2019-04-13T00:00:00-04:002019-04-13T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-04-13:/sphere-o-bot-jjrobots-version.html<p><img align="right," alt="Sphere-O Robot" hspace="10" src="../images/Sphereo.jpg" width="200,"></p>
<p>So, the Sphere-O-Bot was my first (and so far only) <a href="https://www.jjrobots.com/sphere-o-bot-jjrobots-version/">JJRobots</a> project a few years ago (when I didn't have a blog). My 3D printer was not in the best shape but I was able to get everything assembled and working.</p>
<p>I dug out the Sphere-O-bot this week and after …</p><p><img align="right," alt="Sphere-O Robot" hspace="10" src="../images/Sphereo.jpg" width="200,"></p>
<p>So, the Sphere-O-Bot was my first (and so far only) <a href="https://www.jjrobots.com/sphere-o-bot-jjrobots-version/">JJRobots</a> project a few years ago (when I didn't have a blog). My 3D printer was not in the best shape but I was able to get everything assembled and working.</p>
<p>I dug out the Sphere-O-bot this week and after some trial and error I got it back up and running just in time for the Easter Holiday. One of the tricky parts was getting the software to work on an old windows 8 surface we had laying around. The software has three components:</p>
<ol>
<li><a href="https://www.arduino.cc/en/Main/Software">Arduino Leonardo Arduino drivers</a></li>
<li><a href="https://inkscape.org/release/0.91/">Inkscape Version 0.91</a> - Newer versions of Inkscape do not work.</li>
<li><a href="https://wiki.evilmadscientist.com/Installing_software">Eggbot Extention for Inkscape</a></li>
</ol>
<p>Running the installers for the above is just the first step. I had a heck of a time getting the software to connect to the Arduino. It took a lot of googling and multiple problems on top of each other which made it tricky. One key thing I had to change was the following line in the
<code>eggbot_scanwin32.py</code> file:</p>
<p><code>hKey = _winreg.OpenKey( hReg, r"SYSTEM\CurrentControlSet\Enum\USB\VID_2341&PID_0043" )</code></p>
<p>Turns out the VID and PID are unique to my system. You can figure these numbers out by using the Arduino software and looking at the board specifications or you can look them up in the device manager. Kind of tricky. </p>
<p>Anyway, once we got it working it turns out to be extremely fun. I purchased a bunch of ping-pong balls online:</p>
<p><a href="https://www.amazon.com/dp/B001QVA5A2/ref=cm_sw_em_r_mt_dp_U_OIFSCbTPRMWYN">STIGA 2-Star White No Print Table Tennis Balls (144-Count) by Amazon.com</a></p>
<p>These are reasonably priced and work well for printing. We have already gone though and printed a bunch of designs we download from the internet. </p>
<p><img align="right," alt="Example Printed Spheres" hspace="10" src="../images/FirstPlots.jpg" width="200,"></p>
<p>I also had my daughter draw a picture and put it on an egg:</p>
<p><img align="right," alt="Plot from a Drawing" hspace="10" src="../images/Drawing.jpg" width="200,"></p>
<p>There are many steps to getting this to work but they are all fairly easy:</p>
<ol>
<li>Draw a picture with only one color (to start) on white paper.</li>
<li>Open up the eggbot svg Inkscape template file from the examples directory.</li>
<li>Take a picture with my phone and email it to me.</li>
<li>Upload the picture into Inkscape.</li>
<li>Run the trace bitmap function in the menu. This will take some fiddling to get only the parts of the image you want.</li>
<li>Edit the resulting vectors as needed.</li>
<li>Highlight the entire bitmap such that all of the points are selected.</li>
<li>Run the object to path function in the menu.</li>
<li>Run the Eggbot Extension and print as normal. </li>
</ol>
<p>The above instructions are really just notes for myself the next time I want to do this activity with the kids. If anyone besides me is reading this blog, please feel free to put a comment below or drop me an email and I am happy to put together more detailed instructions for installation and/or printing a picture.</p>Accessibility of Jupyter for Impaired Individuals2019-04-05T00:00:00-04:002019-04-05T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-04-05:/accessibility-of-jupyter-for-impaired-individuals.html<p><img align="right," alt="Asha in front of her poster" hspace="10" src="//colbrydi.github.io/images/AshaPoster.jpeg" width="200,"></p>
<p>This poster presents work by CMSETech Intern, Ahsa Shekar and her work on finding ways to make Jupyter More accessible. </p>
<p><strong>Abstract</strong></p>
<p>Jupyter is an open-source programming environment that uses "literate programming" by combining documentation, coding and visualization into one file format. The Jupyter notebook format is also particularly well-suited for …</p><p><img align="right," alt="Asha in front of her poster" hspace="10" src="//colbrydi.github.io/images/AshaPoster.jpeg" width="200,"></p>
<p>This poster presents work by CMSETech Intern, Ahsa Shekar and her work on finding ways to make Jupyter More accessible. </p>
<p><strong>Abstract</strong></p>
<p>Jupyter is an open-source programming environment that uses "literate programming" by combining documentation, coding and visualization into one file format. The Jupyter notebook format is also particularly well-suited for learning environments, where the instructional content (text, links, videos, etc.) can be combined with executable code and student output (graphs, charts, solutions, feedback) in a single, easy-to-use, file format. Unfortunately, Jupyter software is not fully accessible. This means individuals with impaired senses and abilities may not be able to interact with Jupyter effectively. MSU is leading an initiative working towards making Jupyter more accessible. In this project, I investigated the different accessibility issues with Jupyter and identified key problems that, if fixed, would provide an improved experience for the greatest number of our students. This project made progress in solving two key issues; 1) automatically identify and measuring color contrast across the entire graphical user interface and 2) identify structural edits to the source code that can be made to improve the overall readability for screen readers.</p>
<p><a href="//colbrydi.github.io/images/UURAF_Jupyter_Accessibility.pdf">Link to the poster</a></p>The challenges of developing research tools in active and responsive open source communities2019-04-05T00:00:00-04:002019-04-05T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-04-05:/the-challenges-of-developing-research-tools-in-active-and-responsive-open-source-communities.html<p><img align="right," alt="Poster with Ty and Hattie" hspace="10" src="//colbrydi.github.io/images/2019_PA_Poster.jpg" width="200,"></p>
<p>This poster presents work by Professorial Assistance, Ty Buckly and Hattie Pimentel and their work on building an image annotation system inside of Jupyter. </p>
<p><strong>Abstract</strong></p>
<p>Scientists often gather observations of the world using images and need efficient image annotating tools to extract data from these images. Project Insight seeks to …</p><p><img align="right," alt="Poster with Ty and Hattie" hspace="10" src="//colbrydi.github.io/images/2019_PA_Poster.jpg" width="200,"></p>
<p>This poster presents work by Professorial Assistance, Ty Buckly and Hattie Pimentel and their work on building an image annotation system inside of Jupyter. </p>
<p><strong>Abstract</strong></p>
<p>Scientists often gather observations of the world using images and need efficient image annotating tools to extract data from these images. Project Insight seeks to make this process easier and quicker. The project consists of two parts: first, an image annotation package, the Graphical User Interface (GUI); and second, behind-the-scenes machine learning. When Project Insight is completed, data from the GUI will be sent to the machine learning component, which will attempt to anticipate the user’s next choices. The machine-learning component will search through algorithm space for the most suitable algorithm.</p>
<p>Jupyter Notebooks are an interface used by researchers to combines code, pictures, notes and other multimedia. Using Jupyter Notebooks, several image annotation tools were created. Currently, the image annotation tools are being moved to a future replacement of Jupyter Notebooks: JupyterLab. Citing security concerns, the JupyterLab team blocked key JavaScript execution in the beta version of JupyterLab. This means that many programs that worked in the original Jupyter Notebooks, including most of the Project Insight programs, are not functional in JupyterLab.</p>
<p>This presentation will cover multiple attempts to retaining functionality in JupyterLab and discuss why the JupyterLab’s security block hinders functionality and contrasts the accessible theme of JupyterLab.</p>
<p><a href="//colbrydi.github.io/images/2019_Jupyter_PA_Poster.pdf">Link to the poster</a></p>HPCC SLURM Buy-in Notes2019-03-28T00:00:00-04:002019-03-28T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-03-28:/hpcc-slurm-buy-in-notes.html<p><a href="https://slurm.schedmd.com/"><img align="right," alt="SLURM Logo" hspace="10" src="//upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Slurm_logo.svg/2000px-Slurm_logo.svg.png" width="200,"></a></p>
<p>This year, the HPCC moved over to SLURM. Overall, I think the new scheduler is nice but it required me to relearn a bunch of things I have gotten to know by reflex.</p>
<p>One nice new feature is that I can manage our own Buy-in account. The CMSE department has …</p><p><a href="https://slurm.schedmd.com/"><img align="right," alt="SLURM Logo" hspace="10" src="//upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Slurm_logo.svg/2000px-Slurm_logo.svg.png" width="200,"></a></p>
<p>This year, the HPCC moved over to SLURM. Overall, I think the new scheduler is nice but it required me to relearn a bunch of things I have gotten to know by reflex.</p>
<p>One nice new feature is that I can manage our own Buy-in account. The CMSE department has multiple buy-in nodes. For example, the command to <strong>show</strong> everyone on the CMSE account is:</p>
<div class="highlight"><pre><span></span><code><span class="nv">sacctmgr</span><span class="w"> </span><span class="k">show</span><span class="w"> </span><span class="nv">association</span><span class="w"> </span><span class="nv">account</span><span class="o">=</span><span class="nv">cmse</span>
</code></pre></div>
<p>If I want to <strong>add</strong> someone to the CMSE buy-in account I just do the following:</p>
<div class="highlight"><pre><span></span><code>sacctmgr add user account=cmse name=colbrydi
</code></pre></div>
<p>Then if I want to <strong>delete</strong> someone from he CMSE buy-in account is just as easy:</p>
<div class="highlight"><pre><span></span><code>sacctmgr delete user account=cmse name=colbrydi
</code></pre></div>
<p>The HPCC staff also provides the following useful powertools that users can access by running <code>module load powertools</code>:</p>
<p><strong>priority_status</strong> or <strong>buyin_status</strong> - equivalent commands that shows all of your buy-in nodes and who is currently running on them.</p>
<p>To use the buy-in account you need to specifically use the account option in the job script (I think). For example, the following SBATCH commands should request one of the Volta GPU cards on the cmse account:</p>
<div class="highlight"><pre><span></span><code>#SBATCH --gres=gpu:1
#SBATCH --nodelist=nvl-001
#SBATCH --time=08:00:00
#SBATCH --account=cmse
</code></pre></div>
<p>Many of our people want to use the Volta over Jupyterhub run though the <a href="https://webrdp.hpcc.msu.edu">webrdp</a> website. To do this, you would log into <a href="https://webrdp.hpcc.msu.edu">webrdp</a>, open a terminal on the <a href="https://webrdp.hpcc.msu.edu">webrdp</a> desktop, ssh to one of the dev nodes and then issue the following command:</p>
<div class="highlight"><pre><span></span><code><span class="nx">salloc</span><span class="w"> </span><span class="o">--</span><span class="nx">gres</span><span class="p">=</span><span class="nx">gpu</span><span class="p">:</span><span class="mi">1</span><span class="w"> </span><span class="o">--</span><span class="nx">x11</span><span class="w"> </span><span class="o">--</span><span class="nx">nodelist</span><span class="p">=</span><span class="nx">nvl</span><span class="o">-</span><span class="mi">001</span><span class="w"> </span><span class="o">--</span><span class="nx">time</span><span class="p">=</span><span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="o">--</span><span class="nx">mail</span><span class="o">-</span><span class="k">type</span><span class="p">=</span><span class="nx">BEGIN</span><span class="w"> </span><span class="o">--</span><span class="nx">account</span><span class="p">=</span><span class="nx">cmse</span>
</code></pre></div>
<p>Once the job starts, you would just type <code>jupyter notebook</code> assuming you have it installed.</p>
<p>Hope you find these notes useful.</p>
<ul>
<li>Dirk</li>
</ul>Understanding Research Technology Support and Where it Fails2019-03-26T00:00:00-04:002019-03-26T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-03-26:/understanding-research-technology-support-and-where-it-fails.html<p><img align="right," alt="Image of a Microscope" hspace="10" src="//colbrydi.github.io/images/Microscope.png" width="200,"></p>
<p>I was invited to give a talk to a newly formed IT Research Support group here on MSU campus. As the name suggests, the purpose of this group is to find the best ways to help researchers use technology to do their science. </p>
<p>This talk was a patchwork of many …</p><p><img align="right," alt="Image of a Microscope" hspace="10" src="//colbrydi.github.io/images/Microscope.png" width="200,"></p>
<p>I was invited to give a talk to a newly formed IT Research Support group here on MSU campus. As the name suggests, the purpose of this group is to find the best ways to help researchers use technology to do their science. </p>
<p>This talk was a patchwork of many of my other talks. I motivated the discussion by talking about the history of research computing and telling a few stories about how research IT is very different than Enterprise IT. The core of the talk was a group discussion around the idea that the most common source of failure in any project is in human communication.</p>
<p>Then I discussed both campus and national efforts to solve the "human problem" by training a workforce of CI professionals dedicated to supporting research technology. I ended my talk with a quick review of the many national efforts and a plug for our <a href="https://colbrydi.github.io/cyberambassadors/">CyberAmbassador</a> project.</p>
<p>A link to my slides can be found here:</p>
<p><a href="//colbrydi.github.io/images/20190326_IT_Research_support_Talk.pdf">Link to pdf of slides</a></p>Managing Files on the MSU Engineering Jupyterhub server2019-03-08T00:00:00-05:002019-03-08T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-03-08:/managing-files-on-the-msu-engineering-jupyterhub-server.html<p><img align="right," alt="Disk Icon" hspace="10" src="//colbrydi.github.io/images/Floppy_Disk.png" width="200,"></p>
<p>The MSU Engineering Jupyterhub server provides 2GB of disk storage space for each student. It can be helpful to learn some disk management so you can use this space effectively.</p>
<p>For example, pip install uses a temporary folder when downloading packages. Periodically, deleting this folder can significantly free up space …</p><p><img align="right," alt="Disk Icon" hspace="10" src="//colbrydi.github.io/images/Floppy_Disk.png" width="200,"></p>
<p>The MSU Engineering Jupyterhub server provides 2GB of disk storage space for each student. It can be helpful to learn some disk management so you can use this space effectively.</p>
<p>For example, pip install uses a temporary folder when downloading packages. Periodically, deleting this folder can significantly free up space. </p>
<p><code>rm -rf ~/.cache</code></p>
<p>Also, some functions in the scikit learn and seaboarn libraries download their own data folders. You may also want to have them delete those once they are no longer using those packages.</p>
<div class="highlight"><pre><span></span><code>rm -rf ~/scikit_learn_data
rm -rf ~/seaborn-data
</code></pre></div>
<p>Another good command to learn is the “dh” command. I like to run the following in my home directory to see which of my folders are taking up the most space:</p>
<p><code>du -sh ~/*</code></p>
<p>To include hidden folders use:</p>
<p><code>du -sch .[!.]* *</code></p>
<p>To also sort the results (may take a while):</p>
<p><code>du -sch .[!.]* * | sort -h</code></p>
<ul>
<li>Dirk</li>
</ul>Virtual Reality Talk at the MSU Library2019-02-26T00:00:00-05:002019-02-26T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-02-26:/virtual-reality-talk-at-the-msu-library.html<p><img align="right," alt="VR Image" hspace="10" src="//cmse.msu.edu/sites/_cmse/assets/Image/dsc00645.jpg" width="300,"></p>
<p>A few weeks ago I was asked by Terence O'Neill to give a talk to the VR/AR seminar group being held at the library. He wanted me to talk about our "Mobile beast" and show how we are using VR equipment in CMSE. </p>
<p>The VR equipment had a bug …</p><p><img align="right," alt="VR Image" hspace="10" src="//cmse.msu.edu/sites/_cmse/assets/Image/dsc00645.jpg" width="300,"></p>
<p>A few weeks ago I was asked by Terence O'Neill to give a talk to the VR/AR seminar group being held at the library. He wanted me to talk about our "Mobile beast" and show how we are using VR equipment in CMSE. </p>
<p>The VR equipment had a bug and we could not get the demo working. Oh well, I still think the talk went well and I was able to make a few new connections. </p>
<p>Below is a copy of my slides:</p>
<p><a href="//colbrydi.github.io/images/20190226-VR_Library.pdf">CMSE VR Presentation Slides</a></p>Quick Makefile Overview2019-02-06T00:00:00-05:002019-02-06T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-02-06:/quick-makefile-overview.html<p><img align="right," alt="Makefile as a Directed Acyclic Graph" hspace="10" src="//colbrydi.github.io/images/make.png" width="320,"></p>
<p><a href="//colbrydi.github.io/images//Makefile">Example Makefile</a></p>
<p>A former student recently emailed me asking for a good reference about makefiles. To be honest, I teach a lot about makefiles but I am not sure I have a go-to source. I could have googled something but instead I just tapped out this quick description. I thought …</p><p><img align="right," alt="Makefile as a Directed Acyclic Graph" hspace="10" src="//colbrydi.github.io/images/make.png" width="320,"></p>
<p><a href="//colbrydi.github.io/images//Makefile">Example Makefile</a></p>
<p>A former student recently emailed me asking for a good reference about makefiles. To be honest, I teach a lot about makefiles but I am not sure I have a go-to source. I could have googled something but instead I just tapped out this quick description. I thought it makes for a fairly quick introduction to Makefiles. Feel free to leave your favorite resource in the comments.</p>
<p>Makefiles are fairly simple in concept but are an entire programming language so can get very advanced. Basically a makefile consists of rules in the following format:</p>
<div class="highlight"><pre><span></span><code><span class="n">target</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">prerequisite_file1</span><span class="w"> </span><span class="n">prerequisite_file2</span><span class="w"> </span><span class="n">prequiste_file3</span><span class="w"> </span><span class="o">...</span>
<span class="w"> </span><span class="n">Recipe</span><span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="n">Recipe</span><span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="o">...</span>
</code></pre></div>
<p>When you type “make Target” on the command line you active the rule. Then the Make program recursively activates rules to make each of the prerequisites. If there is no rule to make a prerequisite it looks to see if the prerequisite file exists. If there is no rule and there is no file you will get an error. However, assuming you have all the prerequisite files you need for a rule, the Make program then runs the Recipe commands (things like gcc or whatever you want). It is assumed that the commands will take the prerequisites as inputs and generate the target file (this is not always the case).
That is basically it. Some more things to note:
* If you just type “make” (with no target) the Make command assumes you are trying to make the first rule in the file.
* Often the first rule target is called “all” with a bunch of prerequisites and no recipe steps. This is just a high level rule that tells make to build all of your main target files. There is nothing special about the keyword “all” it is just used as a convention.
* Often there is a rule at the end of a makefile called “clean” with no prerequisites. You typically have to explicitly activate this target by writing “make clean” (i.e. you normally do not make “clean” a prerequisite to a rule). Typically the Recipe steps are remove (rm) commands that delete all intermediate files you may have generated (ex exe and *.o files). Basically giving you a clean slate.
* Makefiles can have variables and wildcards. In these cases they can get fairly advanced.
* Make is fairly smart, it will rebuild a target if the targets modification date is older than the modification date of it’s prerequisites. This means that if you repeat the make command it will not automatically regenerate a file it thinks will not change.</p>
<p>That was fun for me. I hope you find this description a useful first step. </p>
<p>Above is a toy example of a makefile and a visual representation.</p>
<ul>
<li>Dirk</li>
</ul>Quick Graphviz Tutorial2019-01-02T00:00:00-05:002019-01-02T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-01-02:/quick-graphviz-tutorial.html<p><img alt="vim system diagram" src="//colbrydi.github.io/images/vim.jpg"></p>
<p>This quick tutorial shows provides basic instructions for generating the above graph using <a href="https://www.graphviz.org/">Graphviz</a>.</p>
<p><a href="https://www.graphviz.org/"><img alt="Graphviz" src="https://graphviz.gitlab.io/_pages/Resources/app.png"></a></p>
<p>If you already have Anaconda installed on your system, you can quickly install graphviz using the following command:</p>
<div class="highlight"><pre><span></span><code>conda install graphviz
</code></pre></div>
<p>Once graphviz is installed you need to create a text file with the connections. This …</p><p><img alt="vim system diagram" src="//colbrydi.github.io/images/vim.jpg"></p>
<p>This quick tutorial shows provides basic instructions for generating the above graph using <a href="https://www.graphviz.org/">Graphviz</a>.</p>
<p><a href="https://www.graphviz.org/"><img alt="Graphviz" src="https://graphviz.gitlab.io/_pages/Resources/app.png"></a></p>
<p>If you already have Anaconda installed on your system, you can quickly install graphviz using the following command:</p>
<div class="highlight"><pre><span></span><code>conda install graphviz
</code></pre></div>
<p>Once graphviz is installed you need to create a text file with the connections. This file is called a dot file. There are a lot of tutorials about the dot syntax but the basics is to make a list of nodes and their connections. For example, the following file (called test.dot) makes the figure above. Use the <code>-></code> to indicate a directed link and a <code>--</code> to indicate a bi-directional link:</p>
<div class="highlight"><pre><span></span><code><span class="n">digraph</span><span class="w"> </span><span class="n">G</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Linux</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Normal</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">label</span><span class="o">=</span><span class="s">"vim"</span><span class="w"> </span><span class="p">];</span>
<span class="w"> </span><span class="n">Normal</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Insert</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">label</span><span class="o">=</span><span class="s">"i/I/a/A/o/O"</span><span class="w"> </span><span class="p">];</span>
<span class="w"> </span><span class="n">Insert</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Normal</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">label</span><span class="o">=</span><span class="s">"Esc"</span><span class="p">];</span>
<span class="w"> </span><span class="n">Normal</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Command</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">label</span><span class="o">=</span><span class="s">": (colon)"</span><span class="p">];</span>
<span class="w"> </span><span class="n">Command</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Normal</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">label</span><span class="o">=</span><span class="s">"w"</span><span class="p">];</span>
<span class="w"> </span><span class="n">Command</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Linux</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">label</span><span class="o">=</span><span class="s">"q/q!/wq/x"</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<p>Once you have the file you can make an image using the following <code>dot</code> command (dot is one of the commands installed by Graphviz)</p>
<div class="highlight"><pre><span></span><code>dot -Tpng test.dot -ovim.png
</code></pre></div>
<p>Just for fun, I tried my dot file with the other tools provided with Graphviz (twopim, sfdp, neato, osage, fdp, circo and patchwork). They don't look as good as dot (most of them mess up the line labels) however i could see that they may be useful (and easy) for other figures:</p>
<p><img alt="twopim" src="//colbrydi.github.io/images/twopi.jpg">
<img alt="sfdp" src="//colbrydi.github.io/images/sfdp.jpg">
<img alt="neato" src="//colbrydi.github.io/images/neato.jpg">
<img alt="osage" src="//colbrydi.github.io/images/osage.jpg">
<img alt="fdp" src="//colbrydi.github.io/images/fdp.jpg">
<img alt="circo" src="//colbrydi.github.io/images/circo.jpg">
<img alt="patchwork" src="//colbrydi.github.io/images/patchwork.jpg"></p>
<p>Hope you find this useful.</p>Three and Four way switch status lights2019-01-01T00:00:00-05:002019-01-01T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2019-01-01:/three-and-four-way-switch-status-lights.html<p><img align="right," alt="Indicator light with lights on" hspace="10" src="../images/Indicator_Light2.jpg" width="200,"></p>
<p>There are two light switches in our mudroom that control the garage lights and the lights on the front of the house. Unfortunately I can not see these lights when standing next to the switches and sense there are multiple switches for each light you can not know if the …</p><p><img align="right," alt="Indicator light with lights on" hspace="10" src="../images/Indicator_Light2.jpg" width="200,"></p>
<p>There are two light switches in our mudroom that control the garage lights and the lights on the front of the house. Unfortunately I can not see these lights when standing next to the switches and sense there are multiple switches for each light you can not know if the lights are actually on or off. I decided to install an indicator light next to the switch to tell if the lights are on or off. </p>
<p>I did some web searching and there are some three way switches with "pilot" or indicator lights. However, most of these are lit when the switch is off and I wanted them to be lit when on. I also didn't really like the look of most of them.</p>
<h3>Attempt 1 - 120v LEDs (Spoiler, this Didn't work)</h3>
<p>I found some really cool <a href="https://www.amazon.com/dp/B000K2IKPK/ref=cm_sw_em_r_mt_dp_U_DU5kCbE2CCCXM">120V LEDs on Amazon</a>. My thought was to wire one in series with my lights. However, once I got them and tried them out the LEDs worked fine but the outside lights didn't work at all. I slapped my head realizing that the "D" in LED stands for "diode" and I was chopping off the A/C signal. </p>
<p><img align="right," alt="120V LED" hspace="10" src="https://images-na.ssl-images-amazon.com/images/I/31CimWUcxwL.jpg" width="10%,"></p>
<h3>Attempt 2 - two 120v LEDs (This didn't work either)</h3>
<p>The obvious solution (to me) was to just use two of the 120V LEDs with going in opposite directions this way current would always be able to flow though them. That didn't work either. Not exactly sure why... my best guess is that the LEDs dropped the current or voltage in such a way that the outside lights no longer could work. To be honest I didn't spend much time debugging the problem because I came up with another solution...</p>
<h3>Attempt 3 - Remote Current Indicator (This one worked nicely)</h3>
<p>For my third attempt I found some <a href="https://www.amazon.com/dp/B005CWHD42/ref=cm_sw_em_r_mt_dp_U_h15kCbGJ8DPMQ">Low Cost Remote Current Indicators</a>. </p>
<p><img alt="Current Indicators" src="https://images-na.ssl-images-amazon.com/images/I/31Y%2B-8BHU7L.jpg"></p>
<p>What is nice about these is that they get their power though induction instead of directly being wired in. The only trick was to ensure that enough current is going through them to get the attached LEDs to work. Fortunately wrapping the wires around the doughnut increases the current. </p>
<p>The trick was to figure out how to get them to work properly with three way and four way switches. The three way solution is fairly strait-forward, all you need to do is wrap the wire going to the light fixture around a few times and the indicator turns on anytime the light turns on. Here are a few diagrams I quickly made to show you how it works.</p>
<p><img align="right," alt="Three way switch Circuit diagram" hspace="10" src="../images/ThreeWaySwitch.png" width="25%,">
<img align="right," alt="Three way switch wiring" hspace="10" src="../images/ThreeWayWire.png" width="25%,"></p>
<p>I didn't realize until I had it all apart the third time that the garage ligth is a four way switch and not just a three way switch. This caused some head scratching since I could not know for sure which wires would be hot. The solution was to pass all the wires though as shown in the following figure. This only gave me one "loop" but that proved to be enough to get the LED to light up.</p>
<p><img align="right," alt="Four way switch Circuit diagram" hspace="10" src="../images/FourWaySwitch.png" width="25%,">
<img align="right," alt="Four way switch wiring" hspace="10" src="../images/FourWayWire.png" width="25%,"></p>
<p>Anyways, fun little project and now I can tell if my lights are on.
!</p>Using an X11 Virtual Frame Buffer to run GUI jobs in batch mode on the HPC.2018-12-20T00:00:00-05:002018-12-20T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-12-20:/using-an-x11-virtual-frame-buffer-to-run-gui-jobs-in-batch-mode-on-the-hpc.html<p><img align="right," alt="X11 Logo" hspace="10" src="//upload.wikimedia.org/wikipedia/commons/a/ab/X11.png" width="20%,"></p>
<p>I have an example program that came with <a href="http://bccd.net/">BCCD</a> called Pandemic which I wanted to run on our local HPCC. Unfortunately Pandemic requires X11 to run and I would get a segmentation fault every time I ran it in the batch system.</p>
<p>This blog post shows how I used the …</p><p><img align="right," alt="X11 Logo" hspace="10" src="//upload.wikimedia.org/wikipedia/commons/a/ab/X11.png" width="20%,"></p>
<p>I have an example program that came with <a href="http://bccd.net/">BCCD</a> called Pandemic which I wanted to run on our local HPCC. Unfortunately Pandemic requires X11 to run and I would get a segmentation fault every time I ran it in the batch system.</p>
<p>This blog post shows how I used the X11 Virtual Frame Buffer (Xvfb) to enable X11 in batch mode. This example uses SLURM running in CentOS7. This trick can come in really handing when you are using MATLAB to because last I checked it needed X11 to run in order to generate and save figures. This will let you do that even in batch mode.</p>
<p>First, here is the batch script for the OpenMP version (Should work for serial jobs as well).</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#SBATCH --time=00:10:00</span>
<span class="c1">#SBATCH --ntasks=1</span>
<span class="c1">#SBATCH --cpus-per-task=16</span>
<span class="c1">#SBATCH --mem=10G</span>
<span class="c1">#SBATCH --job-name Pandemic-OpenMP</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">SLURM_SUBMIT_DIR</span><span class="si">}</span>
<span class="c1"># Have X11 find an open display number and communicate the number</span>
<span class="c1"># though a temporary file to set the environment variable.</span>
<span class="nv">display_file</span><span class="o">=</span>.tmp_display.txt
Xvfb<span class="w"> </span>-displayfd<span class="w"> </span><span class="m">1</span><span class="w"> </span>-auth<span class="w"> </span>/dev/null<span class="w"> </span><span class="m">1</span>><span class="nv">$display_file</span><span class="w"> </span><span class="m">2</span>><span class="w"> </span>/dev/null<span class="w"> </span><span class="p">&</span>
sleep<span class="w"> </span><span class="m">5</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span>:<span class="sb">`</span>cat<span class="w"> </span><span class="nv">$display_file</span><span class="sb">`</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"DISPLAY set to </span><span class="nv">$DISPLAY</span><span class="s2">"</span>
rm<span class="w"> </span><span class="nv">$display_file</span>
<span class="c1"># Benchmark program with different numbers of processes</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">OMP_NUM_THREADS</span><span class="o">=</span><span class="m">16</span>
<span class="nb">time</span><span class="w"> </span>./Pandemic.c-openmp
<span class="nb">export</span><span class="w"> </span><span class="nv">OMP_NUM_THREADS</span><span class="o">=</span><span class="m">8</span>
<span class="nb">time</span><span class="w"> </span>./Pandemic.c-openmp
<span class="nb">export</span><span class="w"> </span><span class="nv">OMP_NUM_THREADS</span><span class="o">=</span><span class="m">4</span>
<span class="nb">time</span><span class="w"> </span>./Pandemic.c-openmp
<span class="nb">export</span><span class="w"> </span><span class="nv">OMP_NUM_THREADS</span><span class="o">=</span><span class="m">2</span>
<span class="nb">time</span><span class="w"> </span>./Pandemic.c-openmp
<span class="nb">export</span><span class="w"> </span><span class="nv">OMP_NUM_THREADS</span><span class="o">=</span><span class="m">1</span>
<span class="nb">time</span><span class="w"> </span>./Pandemic.c-openmp
<span class="nb">echo</span><span class="w"> </span><span class="k">done</span>
<span class="c1"># Report job statistics</span>
scontrol<span class="w"> </span>show<span class="w"> </span>job<span class="w"> </span><span class="nv">$SLURM_JOB_ID</span>
</code></pre></div>
<p>And here is the job I got working with MPI. Notice it is basically the same but I need to pass the DISPLAY variable though MPI for it to work. </p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#SBATCH --time=00:10:00</span>
<span class="c1">#SBATCH --ntasks=16</span>
<span class="c1">#SBATCH --cpus-per-task=1</span>
<span class="c1">#SBATCH --mem=10G</span>
<span class="c1">#SBATCH --job-name Pandemic-MPI</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">SLURM_SUBMIT_DIR</span><span class="si">}</span>
module<span class="w"> </span>swap<span class="w"> </span>GNU<span class="w"> </span>Intel
module<span class="w"> </span>load<span class="w"> </span>OpenMPI
<span class="c1"># Have X11 find an open display number and communicate the number</span>
<span class="c1"># though a temporary file to set the environment variable.</span>
<span class="nv">display_file</span><span class="o">=</span>.tmp_display.txt
Xvfb<span class="w"> </span>-displayfd<span class="w"> </span><span class="m">1</span><span class="w"> </span>-auth<span class="w"> </span>/dev/null<span class="w"> </span><span class="m">1</span>><span class="nv">$display_file</span><span class="w"> </span><span class="m">2</span>><span class="w"> </span>/dev/null<span class="w"> </span><span class="p">&</span>
sleep<span class="w"> </span><span class="m">5</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span>:<span class="sb">`</span>cat<span class="w"> </span><span class="nv">$display_file</span><span class="sb">`</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"DISPLAY set to </span><span class="nv">$DISPLAY</span><span class="s2">"</span>
rm<span class="w"> </span><span class="nv">$display_file</span>
<span class="c1"># Benchmark program with different numbers of processes</span>
<span class="nb">time</span><span class="w"> </span>mpirun<span class="w"> </span>-n<span class="w"> </span><span class="m">16</span><span class="w"> </span>-x<span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span><span class="nv">$HOSTNAME</span>:<span class="nv">$DISPLAY</span><span class="w"> </span>./Pandemic.c-mpi
<span class="nb">time</span><span class="w"> </span>mpirun<span class="w"> </span>-n<span class="w"> </span><span class="m">8</span><span class="w"> </span>-x<span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span><span class="nv">$HOSTNAME$DISPLAY</span><span class="w"> </span>./Pandemic.c-mpi
<span class="nb">time</span><span class="w"> </span>mpirun<span class="w"> </span>-n<span class="w"> </span><span class="m">4</span><span class="w"> </span>-x<span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span><span class="nv">$HOSTNAME$DISPLAY</span><span class="w"> </span>./Pandemic.c-mpi
<span class="nb">time</span><span class="w"> </span>mpirun<span class="w"> </span>-n<span class="w"> </span><span class="m">2</span><span class="w"> </span>-x<span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span><span class="nv">$HOSTNAME$DISPLAY</span><span class="w"> </span>./Pandemic.c-mpi
<span class="nb">time</span><span class="w"> </span>mpirun<span class="w"> </span>-n<span class="w"> </span><span class="m">1</span><span class="w"> </span>-x<span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span><span class="nv">$HOSTNAME$DISPLAY</span><span class="w"> </span>./Pandemic.c-mpi
<span class="c1"># Report job statistics</span>
scontrol<span class="w"> </span>show<span class="w"> </span>job<span class="w"> </span><span class="nv">$SLURM_JOB_ID</span>
</code></pre></div>
<p>Hope you find this useful.</p>How do we know what we don't know we don't know?2018-12-07T00:00:00-05:002018-12-07T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-12-07:/how-do-we-know-what-we-dont-know-we-dont-know.html<p><img align="right," alt="Education2035" hspace="10" src="//colbrydi.github.io/images/Education2035.jpg" width="100%,"></p>
<p>Faculty at MSU conducted a workshop to brainstorm and discuss ways we can try and foresee how education will be changed by technology in an effort to get ahead of the changes and make sure our teaching goals and values stay intact.</p>
<p>My presentation was about the difficulty in predicting …</p><p><img align="right," alt="Education2035" hspace="10" src="//colbrydi.github.io/images/Education2035.jpg" width="100%,"></p>
<p>Faculty at MSU conducted a workshop to brainstorm and discuss ways we can try and foresee how education will be changed by technology in an effort to get ahead of the changes and make sure our teaching goals and values stay intact.</p>
<p>My presentation was about the difficulty in predicting what technology will look like in 17 years.</p>
<ul>
<li><a href="//colbrydi.github.io/images/20181207-Education_2035_colbrydi.pdf">Link to the Presentation Slides</a></li>
</ul>Alexa Dot Holder2018-09-16T00:00:00-04:002018-09-16T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-09-16:/alexa-dot-holder.html<p><img align="right," alt="Alexa Dot Wall Mount" hspace="10" src="../images/Alexa.jpg" width="200,"></p>
<p>My brother got me an Alexa Dot for Christmas a few years ago. It turns out to be a lot of fun. We put it in the kitchen and the kids play music. My wife and I also use it to add things to our shopping list (very convenient!) Anyway …</p><p><img align="right," alt="Alexa Dot Wall Mount" hspace="10" src="../images/Alexa.jpg" width="200,"></p>
<p>My brother got me an Alexa Dot for Christmas a few years ago. It turns out to be a lot of fun. We put it in the kitchen and the kids play music. My wife and I also use it to add things to our shopping list (very convenient!) Anyway, I like to explore <a href="https://www.thingiverse.com/">Thingaverse.com</a> by tying in the names of gadgets I have to see what kinds of modifications you can find. When I typed in Alexa dot I came across this really cool plug holder which would get it off my kitchen counter. </p>
<p><a href="https://www.thingiverse.com/thing:2300667">https://www.thingiverse.com/thing:2300667</a></p>
<p>The design is by <a href="https://www.thingiverse.com/cptskippy/designs">Cpt Skippy</a> and I am really pleased with how it turned out. </p>
<p>I may post a new picture when I get a smaller USB cable that doesn't need to wrap around so much.</p>3D printed Switch Guard2018-08-15T00:00:00-04:002018-08-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-08-15:/3d-printed-switch-guard.html<p><img align="right," alt="Picture of the switch" hspace="10" src="../images/Switch2.jpg" width="200,"></p>
<p>We recently moved into a new house and the kids are having fun running around turning on and off all the lights. Unfortunately, some of the light switches are not for lights. One goes to our garbage disposal, another goes to the gas fireplace and another turns off the furnace …</p><p><img align="right," alt="Picture of the switch" hspace="10" src="../images/Switch2.jpg" width="200,"></p>
<p>We recently moved into a new house and the kids are having fun running around turning on and off all the lights. Unfortunately, some of the light switches are not for lights. One goes to our garbage disposal, another goes to the gas fireplace and another turns off the furnace. I decided I wanted to design a 3D printed magnetic light switch cover that would connect to the standard bolts that hold on a light switch cover plate. The cover would discourage inadvertent switching of the lights but also easily allow us to use the switch when we need too.</p>
<p>As with many of my projects I guessed I was not the first to have this idea so I searched existing designs on <a href="https://www.thingiverse.com/">Thingiverse.com</a> and <a href="https://www.youmagine.com/">Youmagine.com/</a>. Although I could not find designs that used magnets, there were a number of interesting options that used the same screw holes. The one I thought looked the best was by <a href="https://www.thingiverse.com/jwolle1/about">jwolle1</a>:</p>
<p><a href="https://www.thingiverse.com/thing:2512884"><img align="right," alt="Inspiration for my light switch cover" hspace="10" src="https://cdn.thingiverse.com/renders/2d/8e/71/79/04/799bad5a3b514f096e69bbc4a7896cd9_preview_featured.jpg" width="200,"></a></p>
<p>So I started with this design and wrote a simple <a href="http://www.openscad.org/">OpenSCAD</a> program to load in the design as a starting point and then modify it to add holes where I could add some standard (8mm x 3mm) cylindrical magnets i got from Home Depot. </p>
<p><img align="right," alt="Magnets" hspace="10" src="../images/Magnets.jpg" width="200,"></p>
<p>The final product turned out great. I ended up printing three of them for myself and one for my brother. Here are some pictures:</p>
<p><img align="right," alt="Picture of the switch" hspace="10" src="../images/Switch1.jpg" width="200,"></p>
<p><img align="right," alt="Picture of the switch" hspace="10" src="../images/Switch3.jpg" width="200,"></p>
<p>Download the STL file for this design here:</p>
<p><a href="https://www.youmagine.com/designs/magnetic-light-switch-cover">Youmagine.com</a></p>
<iframe src="https://www.youmagine.com/designs/magnetic-light-switch-cover/embed" width="735" height="219" frameborder="0"></iframe>Emerging Technologies (FPGAs) @ 2018 ACI-REF VR2018-08-10T00:00:00-04:002018-08-10T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-08-10:/emerging-technologies-fpgas-2018-aci-ref-vr.html<p><img align="right," alt="Picture from Panel" hspace="10" src="//colbrydi.github.io/images/Emerging_Technologies.jpg" width="20%,"></p>
<p>Today is the last day of the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a> workshop here in Norman Ok. It was a great week with a lot of great people. I highly recommend the workshop for anyone in the area of Advanced Computing Instruction especially those do some Research and Education Facilitation in …</p><p><img align="right," alt="Picture from Panel" hspace="10" src="//colbrydi.github.io/images/Emerging_Technologies.jpg" width="20%,"></p>
<p>Today is the last day of the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a> workshop here in Norman Ok. It was a great week with a lot of great people. I highly recommend the workshop for anyone in the area of Advanced Computing Instruction especially those do some Research and Education Facilitation in their roles. </p>
<p>Thank you Aaron Bergstrom (U North Dakota) for again facilitating the panel discussion on <strong>Emerging Technologies</strong> at the . My fellow panel members included:</p>
<ul>
<li>David Chin, Drexel U re Hadoop/Spark/MapReduce</li>
<li>Shawn Doughty, Tufts U re OnDemand and Jupyter/Rstudio</li>
<li>Anita Schwartz, U Delaware re Singularity</li>
<li>Mariya Vyushkova, U Notre Dame re Quantum Computing</li>
</ul>
<p>I was invited to this panel to share based on the work we are doing at the CMSE department related to FPGAs. </p>
<ul>
<li>Purchased 2 nodes with Altera FPGAs</li>
<li>Started a FPGA Taskforce for interested researchers</li>
<li>Working with a Graduate Student (Ahmed Yousif) on an independent study related to an optimization problem (function fitting) on FPGAs</li>
<li>This fall CMSE plans to conduct a Graduate Course to benchmark and test the 7 dwarves algorithms</li>
</ul>
<p>Here is a direct link to my FPGA slides:</p>
<p><a href="//colbrydi.github.io/images/20180810_ACI_REF_VR_FPGAs.pdf">FPGA only slides</a></p>
<p>Follow the workshop on <a href="https://www.facebook.com/OUHPC/">Facebook</a></p>Deciding Which Technologies to Adopt, and When @ 2018 ACI-REF VR2018-08-08T00:00:00-04:002018-08-08T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-08-08:/deciding-which-technologies-to-adopt-and-when-2018-aci-ref-vr.html<p><img align="right," alt="Image used in presentation representing the technology adoption wave" hspace="10" src="//colbrydi.github.io/images/Wave.jpg" width="30%,"></p>
<p>I had fun leading a discussion on <strong>Deciding Which Technologies to Adopt, and When</strong> at the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a> event in Norman Oklahoma. Since many of the participants were virtual I experimented with doing a Brainstorming exercise over zoom. We had local volunteers on typing in comments from our …</p><p><img align="right," alt="Image used in presentation representing the technology adoption wave" hspace="10" src="//colbrydi.github.io/images/Wave.jpg" width="30%,"></p>
<p>I had fun leading a discussion on <strong>Deciding Which Technologies to Adopt, and When</strong> at the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a> event in Norman Oklahoma. Since many of the participants were virtual I experimented with doing a Brainstorming exercise over zoom. We had local volunteers on typing in comments from our local participants and remote participants typing their comments directly into the zoom group chat. I tried my best to use typical brainstorming techniques and repeat all of the comments so everyone could hear them. This was especially important since remote participates could not hear the local participants.</p>
<p>See a video of the discussion here:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/HR1_OZnKqn0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>If you are interested, you can find a copy of my slides can be found here:</p>
<p><a href="http://www.oscer.ou.edu/acirefvirtres2018_talk_techadoption_colbry_20180808.pdf">PDF of Slides</a></p>Teams of CI Professionals: Recruitment & Retention, Management, Team-building, and Motivation Panel @ 2018 ACI-REF VR2018-08-07T00:00:00-04:002018-08-07T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-08-07:/teams-of-ci-professionals-recruitment-retention-management-team-building-and-motivation-panel-2018-aci-ref-vr.html<p><img align="right," alt="Image from Panel" hspace="10" src="//colbrydi.github.io/images/Audience.jpg" width="20%,"></p>
<p>Thank you Aaron Bergstrom (U North Dakota) for facilitating the panel discussion on <strong>Teams of CI Professionals: Recruitment & Retention, Management, Team-building, and Motivation</strong> at the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a>. My fellow panel members included:</p>
<ul>
<li>Jerry Perez, U Texas Dallas</li>
<li>Derek Leydig, Pennsylvania State U</li>
<li>Claire Mizumoto, U California San Diego …</li></ul><p><img align="right," alt="Image from Panel" hspace="10" src="//colbrydi.github.io/images/Audience.jpg" width="20%,"></p>
<p>Thank you Aaron Bergstrom (U North Dakota) for facilitating the panel discussion on <strong>Teams of CI Professionals: Recruitment & Retention, Management, Team-building, and Motivation</strong> at the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a>. My fellow panel members included:</p>
<ul>
<li>Jerry Perez, U Texas Dallas</li>
<li>Derek Leydig, Pennsylvania State U</li>
<li>Claire Mizumoto, U California San Diego</li>
<li>Robert Ping, Indiana U</li>
</ul>
<p>I thought the panel went really well with both remote and local panelist and both local and remote participants. We discussed all types of topics related to finding good people, mentoring students, professional development and institutional politics. Go to the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a> and they should link to a video if you want to watch.</p>
<p>Follow the workshop on <a href="https://www.facebook.com/OUHPC/">Facebook</a></p>Leading and Listening in Complex CI Conversations @ 2018 ACI-REF VR2018-08-06T00:00:00-04:002018-08-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-08-06:/leading-and-listening-in-complex-ci-conversations-2018-aci-ref-vr.html<p><img align="right," alt="Graphical Depiction of two people having a complex conversation" hspace="10" src="//www.leamcleod.com/wp-content/uploads/2014/06/Communication-challenge.jpg" width="20%,"></p>
<p>I just got done trying out our latest CyberAmbassador Curriculum developed specifically for the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a>.</p>
<p>We had some growing pains related to getting zoom breakout rooms working with a large group of people calling in. Some didn't have working mics, others work in a group room and …</p><p><img align="right," alt="Graphical Depiction of two people having a complex conversation" hspace="10" src="//www.leamcleod.com/wp-content/uploads/2014/06/Communication-challenge.jpg" width="20%,"></p>
<p>I just got done trying out our latest CyberAmbassador Curriculum developed specifically for the <a href="http://www.oscer.ou.edu/acirefvirtres2018.php">2018 ACI-REF Virtual residency</a>.</p>
<p>We had some growing pains related to getting zoom breakout rooms working with a large group of people calling in. Some didn't have working mics, others work in a group room and could not talk, others just didn't feel like participating. Overall though I think it worked out well especially during the second breakout when we had a little more time. Check out the slides on the CyberAmbassador website:</p>
<p><a href="//colbrydi.github.io/cyberambassadors/practice-examples-for-complex-communication.html">Link to ACI-REF 2018 Complex Communications Files</a></p>Progress Update on the Development and Implementation of the Advanced CI-REF VR Program2018-07-25T00:00:00-04:002018-07-25T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-07-25:/progress-update-on-the-development-and-implementation-of-the-advanced-ci-ref-vr-program.html<p><img align="right," alt="ACI-Ref Buiding Picture" hspace="10" src="https://aciref.org/wp-content/uploads/2017/09/FullSizeRender-2-1024x765.jpg" width="20%,"></p>
<p>Henry Neeman did a great job presenting our paper on <a href="https://pearc18.conference-program.com/?page_id=10&id=pap144&sess=sess164">Progress Update for the CI-REF VR Program</a> at <a href="//www.pearc18.pearc.org/">PEARC18</a>.</p>
<p>Writing of this paper was truly a unique experience for me. The entire paper was written virtually over weekly video conference meetings and included the thirteen sited (maximum) authors and I …</p><p><img align="right," alt="ACI-Ref Buiding Picture" hspace="10" src="https://aciref.org/wp-content/uploads/2017/09/FullSizeRender-2-1024x765.jpg" width="20%,"></p>
<p>Henry Neeman did a great job presenting our paper on <a href="https://pearc18.conference-program.com/?page_id=10&id=pap144&sess=sess164">Progress Update for the CI-REF VR Program</a> at <a href="//www.pearc18.pearc.org/">PEARC18</a>.</p>
<p>Writing of this paper was truly a unique experience for me. The entire paper was written virtually over weekly video conference meetings and included the thirteen sited (maximum) authors and I think dozens of other contributors. Henry does an amazing job coordinating this army of people, soliciting their input and giving them opportunities to learn. I am looking forward to the ACI-Ref Virtual Residency workshop in a couple of weeks so I can see everyone again in person.</p>
<p>Below is a summary:</p>
<p><strong> Description: </strong>
Cyberinfrastructure facilitation -- working directly with researchers to advance the computing-intensive and data-intensive aspects of their investigations, especially via large-scale and advanced computing -- has emerged as a crucial component of research computing processes. However, because no national formal education curriculum exists to cultivate the needed workforce, informal education is the most viable approach. The Advanced Cyberinfrastructure Research and Education Facilitator (ACI-REF) Virtual Residency (VR) program, in operation since summer 2015, is a national informal education program that trains Cyberinfrastructure (CI) Facilitators on effective methods for serving their research and education constituents, especially focusing on professional (soft) skills (because much of the technical content that facilitators need is available via other opportunities). The VR consists primarily of (a) summer weeklong intensive workshops that provide both content and experiences in facilitation, (b) biweekly conference calls that expand upon both the kinds of topics introduced in the workshops and additional content and experiences, (c) meetings at national conferences and (d) a grant proposal writing apprenticeship. Future plans focus on extending the VR model to intermediate and advanced levels, as well as disseminating a "train-the-trainers-of-trainers" approach to teaching other organizations to conduct their own VR activities within their own CI Facilitator communities, in order to expand the VR's scale, scope, and reach. The VR program and its offshoots have already served 364 CI professionals at 188 institutions across the US and internationally. Virtual Residents are positioned to advance to the intermediate level and beyond, and ultimately to become institutional and national CI leaders.</p>Fitting iCE-Cube Neutrino Path models using Neural Networks2018-07-24T00:00:00-04:002018-07-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-07-24:/fitting-ice-cube-neutrino-path-models-using-neural-networks.html<p><img align="right," alt="Mohammed and his poster" hspace="10" src="//colbrydi.github.io/images/Mohammed.jpg" width="30%,"></p>
<p>This summer Mohammed Salih worked with me as part of the Engineering <a href="https://www.egr.msu.edu/graduate/ensure">ENSURE</a> Program. He presented his work with Jessie Micallef and me as a poster at <a href="https://urca.msu.edu/mid-sure">MidSURE</a>.</p>
<p><strong>Poster Abstract</strong>: Neutrinos are small particles with a mass close to zero. It’s rare interaction with normal matter makes it difficult …</p><p><img align="right," alt="Mohammed and his poster" hspace="10" src="//colbrydi.github.io/images/Mohammed.jpg" width="30%,"></p>
<p>This summer Mohammed Salih worked with me as part of the Engineering <a href="https://www.egr.msu.edu/graduate/ensure">ENSURE</a> Program. He presented his work with Jessie Micallef and me as a poster at <a href="https://urca.msu.edu/mid-sure">MidSURE</a>.</p>
<p><strong>Poster Abstract</strong>: Neutrinos are small particles with a mass close to zero. It’s rare interaction with normal matter makes it difficult to determine its precise mass. Researchers in the the Ice Cube project are working on determining neutrinos precise mass. The Ice Cube project is a sub particle detector in the south pole that records the interactions of neutrinos. The detector is a 150 KM grid with a hexagonal shape 1 km under Ice. Each hole in the grid has a 1 km string, each with about 60 light sensors. Since a neutrino is faster than the speed of light in ice, it emits light in ice. Therefore, the emitted photon is detected by the sensors in the detector. Using the measurements from the detector, we can determine angle, origin, speed, and energy using analytics and machine learning. The interest in this project is to identify neutrinos have come from the center of the galaxy. To check if machine learning is an appropriate method for this project, we create a virtual ice cube simulator using phantom data generation method to simplify and control the data set. Then we use machine learning on this data to determine if it is an appropriate method for this problem.</p>Campus Champions 101 Panel at PEARC182018-07-23T00:00:00-04:002018-07-23T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-07-23:/campus-champions-101-panel-at-pearc18.html<p><img align="right," alt="Campus Champion Logo Logo" hspace="10" src="https://www.xsede.org/wwwteragrid/archive/image/image_gallery%3Fuuid=554fecca-1a37-44d0-826f-afad9470153d&groupId=298192&t=1291845274821" width="20%,"></p>
<p>I had an opportunity to participate on the "Researcher Facing" Panel as part of the <a href="https://www.xsede.org/community-engagement/campus-champions">Campus Champion 101</a> workshop at <a href="https://www.pearc18.pearc.org/">PEARC18</a>. </p>
<p><a href="//docs.google.com/document/d/1OsOCUOeWAk1IfJzEi8LuO5YnIiKkU2AO-cxzY0F0gzk/edit">Link to Notes For the Panel can be found here</a></p>
<p>The <a href="https://pearc18.conference-program.com/?page_id=10&id=work108&sess=sess123">Campus Champions 101 workshop</a> explores campus research computing roles. The workshop builds on the Campus Champions’ history and …</p><p><img align="right," alt="Campus Champion Logo Logo" hspace="10" src="https://www.xsede.org/wwwteragrid/archive/image/image_gallery%3Fuuid=554fecca-1a37-44d0-826f-afad9470153d&groupId=298192&t=1291845274821" width="20%,"></p>
<p>I had an opportunity to participate on the "Researcher Facing" Panel as part of the <a href="https://www.xsede.org/community-engagement/campus-champions">Campus Champion 101</a> workshop at <a href="https://www.pearc18.pearc.org/">PEARC18</a>. </p>
<p><a href="//docs.google.com/document/d/1OsOCUOeWAk1IfJzEi8LuO5YnIiKkU2AO-cxzY0F0gzk/edit">Link to Notes For the Panel can be found here</a></p>
<p>The <a href="https://pearc18.conference-program.com/?page_id=10&id=work108&sess=sess123">Campus Champions 101 workshop</a> explores campus research computing roles. The workshop builds on the Campus Champions’ history and leverages the recent Research Computing and Data Professionals Job Elements and Career Guide that was developed by the March 2018 NSF-sponsored Campus Research Computing Consortium (CaRCC) workshop. Opening sessions will cover (a) the Cyberinfrastructure ecosystem and (b) research computing roles and responsibilities, introducing four broad job families: researcher-facing roles, systems-facing roles, sponsor/stakeholder-facing roles, and software/data-facing roles. Short talks, panels and roundtables will cover three of these four roles: researcher-facing, sponsor/stakeholder-facing, and systems-facing.</p>Bringing Professional Skills Training to CI - PEARC18 Birds of a Feather (BOF)2018-07-22T00:00:00-04:002018-07-22T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-07-22:/bringing-professional-skills-training-to-ci-pearc18-birds-of-a-feather-bof.html<p><img align="right," alt="PEARC18 Logo" hspace="10" src="http://insidehpc.com/wp-content/uploads/2017/10/pearc18.jpg" width="20%,"></p>
<p>The <a href="//colbrydi.github.io/cyberambassadors">CyberAmbassador</a> a Birds of a Feather Event at the 2018 PEARC Conference. Please Join us on Wednesday, July 25th from 5pm - 6pm. More information can be found here:</p>
<p><a href="//pearc18.conference-program.com/?page_id=10&id=bof122&sess=sess203">Link to BOF Schedule</a></p>
<p><strong> Description: </strong>
A number of professional skills training programs and resources have been developed in recent years, in …</p><p><img align="right," alt="PEARC18 Logo" hspace="10" src="http://insidehpc.com/wp-content/uploads/2017/10/pearc18.jpg" width="20%,"></p>
<p>The <a href="//colbrydi.github.io/cyberambassadors">CyberAmbassador</a> a Birds of a Feather Event at the 2018 PEARC Conference. Please Join us on Wednesday, July 25th from 5pm - 6pm. More information can be found here:</p>
<p><a href="//pearc18.conference-program.com/?page_id=10&id=bof122&sess=sess203">Link to BOF Schedule</a></p>
<p><strong> Description: </strong>
A number of professional skills training programs and resources have been developed in recent years, in part due to the efforts of the NSF CyberTraining Initiative. The goal of this BOF is to bring these groups together and explore ways to share information and coordinate efforts.</p>
<p>As the integration of CI in research continues, CI Professionals find themselves tackling problems and consulting on projects that are increasingly complex and collaborative. In order to respond to these various requests, CI Professionals need both the expertise to solve computational challenges and the professional skills to work effectively with individuals and teams who have diverse backgrounds, experiences, and goals.</p>
<p>The organizers of this BOF will facilitate conversations and brainstorming activities within this interactive session, with the goal of identifying opportunities to build on each other’s success and develop ongoing collaborations. For example, authors of open-source professional skills curricula might meet experienced facilitators with an interest in bringing their content to new audiences. Or content developers might identify gaps in available resources and develop new partnerships to address these needs.</p>
<p>This BOF is proposed by a team that has received NSF CyberTraining funds to develop open-source, professional skills training in communication, teamwork and leadership for CI Professionals along with a “train the trainers” module to facilitate national access to these materials.</p>Pelican Github.io Tutorial2018-07-17T00:00:00-04:002018-07-17T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-07-17:/pelican-githubio-tutorial.html<p><img align="right," alt="Pelican Github.io Logo" hspace="10" src="//docs.google.com/a/msu.edu/drawings/d/slwvmwORPg9obcema4qeYEg/image?parent=e/2PACX-1vRGU8icBbrWq1sHjWR3oDeCjeVVY_xgl2-aq1T3iTFxFMYiBdrv73kwax9A-P4pXXV_Wy6RwVtcG1Cz&rev=6&h=200&w=182&ac=1" width="20%,"></p>
<p>My student's and I figured out how to use <a href="https://blog.getpelican.com/">Pelican</a> with <a href="https://pages.github.com/">Github.io</a> pages to create some websites for work. We tried to type up the instructions as a tutorial and build a pelican template. Here is a link to the tutorial:</p>
<p><a href="//colbrydi.github.io/msupelican">Pelican / Github.io Tutorial</a></p>
<p>Some of our websites …</p><p><img align="right," alt="Pelican Github.io Logo" hspace="10" src="//docs.google.com/a/msu.edu/drawings/d/slwvmwORPg9obcema4qeYEg/image?parent=e/2PACX-1vRGU8icBbrWq1sHjWR3oDeCjeVVY_xgl2-aq1T3iTFxFMYiBdrv73kwax9A-P4pXXV_Wy6RwVtcG1Cz&rev=6&h=200&w=182&ac=1" width="20%,"></p>
<p>My student's and I figured out how to use <a href="https://blog.getpelican.com/">Pelican</a> with <a href="https://pages.github.com/">Github.io</a> pages to create some websites for work. We tried to type up the instructions as a tutorial and build a pelican template. Here is a link to the tutorial:</p>
<p><a href="//colbrydi.github.io/msupelican">Pelican / Github.io Tutorial</a></p>
<p>Some of our websites currently using this method of publishing include:</p>
<ul>
<li><a href="//colbrydi.github.io">http://colbrydi.github.io</a></li>
<li><a href="//nolanfeeny.github.io/">http://nolanfeeny.github.io/</a></li>
<li><a href="//colbrydi.github.io/msupelican">http://colbrydi.github.io/msupelican</a></li>
<li><a href="//colbrydi.github.io/cyberambassadors">http://colbrydi.github.io/cyberambassadors</a></li>
</ul>
<p>Feel free to comment below (or email me) if you have any questions.</p>
<p><strong> Dirk </strong></p>Installing RStudio on the MSU HPCC2018-07-06T00:00:00-04:002018-07-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-07-06:/installing-rstudio-on-the-msu-hpcc.html<p><img alt="Rstudio" src="//www.rstudio.com/wp-content/uploads/2016/09/RStudio-Logo-Blue-Gray-250.png"></p>
<p>These instructions are for installing RStudio on the HPCC. RStudio provides a variety of installers on their website <a href="https://www.rstudio.com/products/rstudio/download2/">https://www.rstudio.com/products/rstudio/download2/</a>. However, the HPCC uses an older version of Linux and the precompiled binaries are not compatible. Trying to install RStudio from the source code is …</p><p><img alt="Rstudio" src="//www.rstudio.com/wp-content/uploads/2016/09/RStudio-Logo-Blue-Gray-250.png"></p>
<p>These instructions are for installing RStudio on the HPCC. RStudio provides a variety of installers on their website <a href="https://www.rstudio.com/products/rstudio/download2/">https://www.rstudio.com/products/rstudio/download2/</a>. However, the HPCC uses an older version of Linux and the precompiled binaries are not compatible. Trying to install RStudio from the source code is an option but is also difficult due to the large numbers of libraries and compile dependancies. I basically gave up trying to install RStudio on until I realized that a version is included as an option in Anaconda. The following instructions show how to install RStudio using Anaconda.</p>
<h2><strong>Step 0:</strong> Connect to the HPC.</h2>
<p>Use ssh to connect to the HPCC and make sure you have a working X11 server (ex. MobaXterm on windows or XQuarts on Mac). More information about X11 can be found here:</p>
<ul>
<li><a href="//wiki.hpcc.msu.edu/display/hpccdocs/Installing+an+X-server+on+Windows">Old HPCC Windows instructions</a></li>
<li><a href="https://wiki.hpcc.msu.edu/display/hpccdocs/Installing+an+X-server++for+Macs">Old HPCC Mac Instructions</a></li>
</ul>
<h2><strong>Step 1:</strong> Download the Linux Anaconda installers</h2>
<p>Find the latest Linux installer on the <a href="//www.anaconda.com/download/#linux">Anaconda website</a>. Right click on the download button for version 3.x and select "Copy Link". Use the <code>wget</code> command on the hpcc to download the script file by typing wget and then pasting the url. For Example:</p>
<div class="highlight"><pre><span></span><code>wget https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh
</code></pre></div>
<h2><strong>Step 2:</strong> Install anaconda using the defaults</h2>
<p>Run the downloaded script in bash using the following command:</p>
<div class="highlight"><pre><span></span><code>bash ./Anaconda3-5.2.0-Linux-x86_64.sh
</code></pre></div>
<p>The last question will ask you to add the anaconda folder to your bashrc file. Say yes...</p>
<h2><strong>Step 3:</strong> Install RStudio</h2>
<p>Use the conda install command to install RStudio. Note you may need to run <code>source ~/.bashrc</code> first to update your path:</p>
<div class="highlight"><pre><span></span><code>conda install rstudio
</code></pre></div>
<p>That should be it, RStudio is now installed in your home directory on the HPCC. The only other thing you will need is a X11 server. Assuming you have connected to the hpcc with an X11 connection you can run rstudio from the command line by typing:</p>
<div class="highlight"><pre><span></span><code>rstudio
</code></pre></div>Managing Inconsistent Runtimes on the HPCC2018-06-29T00:00:00-04:002018-06-29T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-06-29:/managing-inconsistent-runtimes-on-the-hpcc.html<p>One of our postdocs stopped by my office today to talk about a problem with inconsistant runtimes on our <a href="//icer.msu.edu">HPCC</a>. This user is running a lot of jobs with 512 cores (current cpu limit is 520). The problem is that it is really difficult for the user to estimate the …</p><p>One of our postdocs stopped by my office today to talk about a problem with inconsistant runtimes on our <a href="//icer.msu.edu">HPCC</a>. This user is running a lot of jobs with 512 cores (current cpu limit is 520). The problem is that it is really difficult for the user to estimate the walltimes. There is a 2x difference in walltimes on the HPCC. The code has been tested on other large scale systems with much less variation. The user would prefer more consistent runtimes and did not mind waiting in the queue (although less queue time is better). </p>
<p>The problem is that there are many things that can impact the runtime of a job. The HPCC runs at over 95% utilization most of the time so many of the resources are in contention. Without additional information, my best guess is that the bottleneck is MPI communication over the high speed network. If this is the case, I hypothesize that moving more processes to the same nodes (instead of spread across the cluster) will help reduce network competition with other users and result in more consistent walltimes. Here are some things I suggested the user try:</p>
<ul>
<li>Open a ticket with iCER and see if you can set up a consulting appointment. They may be able to get much better idea about what resources are in contention and causing the bottleneck: <a href="https://contact.icer.msu.edu/contact">https://contact.icer.msu.edu/contact</a></li>
<li>Add “qstat -f ${PBS_JOBID}” to the end of the jobscript. This will provide some stats that may help us understand differences between jobs that run fast and slow. </li>
<li>Try different core and node ratios (ex. nodes=128:ppn=4, nodes=64:ppn=8, nodes=32:ppn=8 etc.). Although increasing the ppn will likely make queue times longer, i believe the overall runtime will be more consistent.</li>
<li>Start the process to get DOE and XSEDE allocations. Create XSEDE Portal account here: <a href="https://portal.xsede.org/">https://portal.xsede.org/</a> and send me your login name when you get it so we can add you to the MSU campus champion account.</li>
</ul>
<p>Anyone else have a similar problem? Other suggestions that I may have missed?</p>
<p><strong> Dirk </strong></p>Getting dlvr.it to work with Pelican generated website2018-06-28T14:00:00-04:002018-06-28T14:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-06-28:/getting-dlvrit-to-work-with-pelican-generated-website.html<p>Back when I worked for <a href="//icer.msu.edi">iCER</a> I had my blog set up to automatically Tweet when I posted something new using a service called Twitter Feed. Unfortunately, Twitter Feed is no longer around and their website recommended <a href="//dlvr.it">dlvr.it</a> as an alternative. </p>
<p>So far it was fairly strait forward to …</p><p>Back when I worked for <a href="//icer.msu.edi">iCER</a> I had my blog set up to automatically Tweet when I posted something new using a service called Twitter Feed. Unfortunately, Twitter Feed is no longer around and their website recommended <a href="//dlvr.it">dlvr.it</a> as an alternative. </p>
<p>So far it was fairly strait forward to set up an account and I followed the menus to connect my <a href="//docs.getpelican.com/en/3.6.3/">Pelican</a> RSS feed to my <a href="//twitter.com/colbrydi">Titter</a> account. At first it didn't seem to do anything but then I noticed a message that said something like "Last checked 56 seconds ago" and I went back to my Twitter account and all of my blog posts had been tweeted. Which is what I wanted so Yeah!</p>
<p>However, there were a lot of options that I didn't understand. For example, I thought it would only tweet the latest blog post. It didn't, it tweeted them all. Which is fine, this is where I wanted to go but I was hoping to test it first before blasting all of my twitter followers (I do not think I have many). </p>
<p>The second problem was that it only tweeted the title and the URL which is not very detailed. I don't want to everyone to be required to go to my blog post. I found an option to also include the body of the post. We will see if that does anything. </p>
<p>I plan to use this post as a test.</p>
<p><strong>Dirk</strong></p>
<p><strong>Update 1:</strong> There was a setting in dlvr.it that I turned on to "Post Body". This let more come though the twitter feed and nicely includes an image. This makes the behavior act as expected so I am really excited. Oddly, this particular post is not seen on Twitter. This is because I went over the 10 post limit on dlvr.it. I am going to change the post date to today and see if that helps things out.</p>
<p><strong>Update 2:</strong> Changing the date in the post didn't seem to trigger dlvr.it to post this message on twitter. I am now changing the filename inside of pelican to see if that changes the RSS feed enough to make dlvr.it think this is a new post. </p>
<p><strong>Update 3:</strong> Changing the pelican filename didn't do anything either. I looked though the RSS feed and now I think maybe changing the post title will make a difference.</p>Ensure Talk - An Engineers Autobiography2018-06-27T00:00:00-04:002018-06-27T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-06-27:/ensure-talk-an-engineers-autobiography.html<p><img alt="EnSURE Logo" src="//www.egr.msu.edu/graduate/sites/default/files/content/ensure_logo.jpg"></p>
<p>I gave an autobiographical talk today to the <a href="//www.egr.msu.edu/graduate/ensure">EnSURE</a> (Engineering Summer Undergraduate Research Experience) students. I have given this talk a few times and it is noticeably different that most of my presentations because the EnSURE organizers asked me to talk about myself and how I got where I am …</p><p><img alt="EnSURE Logo" src="//www.egr.msu.edu/graduate/sites/default/files/content/ensure_logo.jpg"></p>
<p>I gave an autobiographical talk today to the <a href="//www.egr.msu.edu/graduate/ensure">EnSURE</a> (Engineering Summer Undergraduate Research Experience) students. I have given this talk a few times and it is noticeably different that most of my presentations because the EnSURE organizers asked me to talk about myself and how I got where I am (instead of research or teaching). The first time I gave this talk it was really awkward. However, now I think it is kind of fun to tell stories about the choices I made throughout school and my carrier. I get a lot of questions when I am done so I like to think the student's enjoyed the talk as well. Here are a copy of my slides:</p>
<p><a href="./Images/20180628_An_Engineers_Autobiography.pdf">0180628_An_Engineers_Autobiography.pdf</a></p>
<p><strong> Dirk </strong></p>Commit Early and Often2018-06-26T12:00:00-04:002018-06-26T12:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-06-26:/commit-early-and-often.html<p><strong>Ugh! Rookie mistake!</strong> I just spent all morning working on my website and blog. I was really happy with the improvements. I was making some modifications to the Makefile in the pelican folder to streamline my workflow when... all of my files disappeared.</p>
<p><strong>Rookie Mistake #1:</strong> I unintentionally did a …</p><p><strong>Ugh! Rookie mistake!</strong> I just spent all morning working on my website and blog. I was really happy with the improvements. I was making some modifications to the Makefile in the pelican folder to streamline my workflow when... all of my files disappeared.</p>
<p><strong>Rookie Mistake #1:</strong> I unintentionally did a <code>rm -rf *</code> in the wrong folder and lost of of my work.</p>
<p><strong>Rookie Mistake #2:</strong> I had not committed any of my changes to the repository. All was lost. </p>
<p>Sigh. Now I need to try to go back and remember what I did all morning. </p>
<p>Lessons relearned:</p>
<ul>
<li>Commit early and Often</li>
<li>Be respectful of <code>rm -rf *</code> especially when it is in a <code>Makefile</code></li>
</ul>
<p><strong> Dirk </strong></p>Planning a new Course2018-06-10T00:00:00-04:002018-06-10T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-06-10:/planning-a-new-course.html<p>This post describes a technique I use to plan an organize a course from scratch. I learned this technique from Dr. Brian O'Shea. </p>
<p>At Michigan State University (MSU) my official title is curriculum specialist. I have the unique opportunity to help develop entirely new courses for a brand new department …</p><p>This post describes a technique I use to plan an organize a course from scratch. I learned this technique from Dr. Brian O'Shea. </p>
<p>At Michigan State University (MSU) my official title is curriculum specialist. I have the unique opportunity to help develop entirely new courses for a brand new department (Computational Mathematics, Science and Engineering CMSE, EST 2015). </p>
<p>My colleague (Dr. Brian O'Shea) showed me a nice trick to organize my semester and brainstorm ideas to do a better job with backward design and matching up my lessons to my learning goals. I make a grid of 3 rows with 5 columns on a whiteboard. Each box represents one week of the semester (15 total). I then use post-it notes to write down ideas and place them on the board. This is a brainstorming stage where all ideas are equal. If I do this early enough I can leave the board up and move the sticky notes around as I think about my learning goals.</p>
<p><img alt="MarkerBoard Grid" src="./Images/markdown-img-paste-20180610141148377.png"></p>
<p>I recently moved away from the whiteboard and use a blank wall in my office. I created the grid with painters tape. This leaves my whiteboard open for other ideas and lets me put up a board really early. For example, it is June and I have a board up for my Spring Course on Parallel programming.</p>
<p><img alt="Painters Tape Wall" src="./Images/markdown-img-paste-20180610140831494.png"></p>
<p>Since I have been desinging many courses I may just leave the painters tape up. </p>Spark mail2018-06-06T00:00:00-04:002018-06-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-06-06:/spark-mail.html<p><img align="right," alt="Spark Logo" hspace="10" src="//sparkmailapp.com/img/spark2/common/spark.svg" width="20%,">
I am weird that I like email. It is my preferred form of communication on projects (maybe I am just getting old). I trust that I will get to my email and that I will not loose things. I realize that this trust is often not true but it is …</p><p><img align="right," alt="Spark Logo" hspace="10" src="//sparkmailapp.com/img/spark2/common/spark.svg" width="20%,">
I am weird that I like email. It is my preferred form of communication on projects (maybe I am just getting old). I trust that I will get to my email and that I will not loose things. I realize that this trust is often not true but it is true enough for me to keep productive. For the longest time Gmail was my favorite email application. However, work changed policy from "we highly recommend you don't use Gmail" to "don't use gmail". So I chose to switch to the university Office 365 accounts. </p>
<p>I tried the built-in Mac mail program. It didn't thread my email the way I prefer. I also tried outlook which had its own problems. I don't really feel like bashing this software, it just didn't work in a way that fit my Gmail workflow.</p>
<p>As I move into a new laptop I stubbled across the spark email client. About three weeks into using it I am very satisfied. The laptop and phone versions work nicely together. I really like having both my personal Gmail account and work accounts connected in one application. There are a lot of things I like but probably the best is that since using Spark I have gotten my inbox to zero almost every day. This is big news. I am a big fan of emptying my inbox but this is often a challenge. I like the fact that Spark will allow me to show badges with the number of messages in my inbox instead of the number of unread messages. For some reason this is a real motivator for getting my email where they need to be. It also helps me keep my Gmail clean which is mostly news feeds, advertisements, etc. </p>Inverse Problems Seminar2018-06-04T00:00:00-04:002018-06-04T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-06-04:/inverse-problems-seminar.html<p>I was invited to give a talk at the inverse seminars symposium:</p>
<p><a href="//inverseproblems2018.org/">https://inverseproblems2018.org/</a></p>
<p>At first I was a little nervous since I don't consider myself an expert in inverse problems so I did not fully understand what they wanted me to present. However after talking to the organizers …</p><p>I was invited to give a talk at the inverse seminars symposium:</p>
<p><a href="//inverseproblems2018.org/">https://inverseproblems2018.org/</a></p>
<p>At first I was a little nervous since I don't consider myself an expert in inverse problems so I did not fully understand what they wanted me to present. However after talking to the organizers (Kirk and James), I discovered what they wanted was a high level talk and overview of advanced computing. This was something I could do! I put together snippets from a variety of my previous talks. Here is a link to the slides:</p>
<p><a href="./Images/20180604-InverseProblems.pdf">20180604-InverseProblems</a></p>
<p>Originally I planned for a 60 minute talk but only ended up with 45 minutes (at most) of content. Luckly for me the organizers only really wanted a 30 Minutes talk and in the end I was relaxed and able to get though my slides at a reasonable pace in 20 minutes which led to 10 minutes of questions which turned out to be perfect. </p>
<p>Overall the Inverse Problems Symposium was a great event and I learned a lot. I was honored to be invited to give a talk. I was even considering attending next year at Notre Dame.</p>
<p><strong> Dirk </strong></p>FarmBot2018-05-27T00:00:00-04:002018-05-27T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-27:/farmbot.html<p><img align="right," alt="FarmBot Logo" hspace="10" src="https://cdn.shopify.com/s/files/1/2040/0289/files/FarmBot.io_Trimmed_Logo_Gray_on_Transparent_1_434x200.png?v=1525220371" width="200,"></p>
<p>I have been coveting a <a href="//farm.bot/">FarmBot</a>. At $2500 to $4000 these farming robots are a little out of my price range. However, I guess if I had a big enough yard I would still be looking to purchase one of these. They are so cool, they will plant, water, weed …</p><p><img align="right," alt="FarmBot Logo" hspace="10" src="https://cdn.shopify.com/s/files/1/2040/0289/files/FarmBot.io_Trimmed_Logo_Gray_on_Transparent_1_434x200.png?v=1525220371" width="200,"></p>
<p>I have been coveting a <a href="//farm.bot/">FarmBot</a>. At $2500 to $4000 these farming robots are a little out of my price range. However, I guess if I had a big enough yard I would still be looking to purchase one of these. They are so cool, they will plant, water, weed your garden for you. So much fun! And you get to eat the product when you are done.</p>
<p>The company uses open source hardware so I guess I could try to build it on my own and save a little money. All of the tools are built using 3D Printing so there is that. </p>
<p>This project would go well with my robotic composting project I want to build.</p>White Noise and Teaching2018-05-23T00:00:00-04:002018-05-23T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-23:/white-noise-and-teaching.html<p>It can be difficult to be the first student to talk in a quite classroom. I have been experimenting with the use of white noise to raise the overall din in the room and encourage taking more quickly.</p>
<p>When I teach I try to encourage peer and group discussions. Activities …</p><p>It can be difficult to be the first student to talk in a quite classroom. I have been experimenting with the use of white noise to raise the overall din in the room and encourage taking more quickly.</p>
<p>When I teach I try to encourage peer and group discussions. Activities such as think-pair-share, brainstorming and just group problem solving happen in of my classes. I have started pumping white noise into the room during these work times. I do this more often when class size is low, there is low energy (such as after lunch) or the activity starts with a coding exercise where students work individually. </p>
<p>I feel that by using the white noise it encourages talking. A quite room can
<a href="//coffitivity.com/">Coffitivity.com</a></p>JJRobotics2018-05-19T00:00:00-04:002018-05-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-19:/jjrobotics.html<p><img align="right," alt="JJRobotics Logo" hspace="10" src="https://www.jjrobots.com/wp-content/uploads/2018/10/jjrobots-logo-500x242.png" width="200,"></p>
<p>A great purchase for anyone with a 3D printer and looking for projects With the kids.</p>
<p>About a year ago I stubbled across <a href="//www.jjrobots.com/">JJRobots</a> and I just love the idea of this company. They sell a kit of electronic parts to maker hobbyists with access to 3D printers. The idea …</p><p><img align="right," alt="JJRobotics Logo" hspace="10" src="https://www.jjrobots.com/wp-content/uploads/2018/10/jjrobots-logo-500x242.png" width="200,"></p>
<p>A great purchase for anyone with a 3D printer and looking for projects With the kids.</p>
<p>About a year ago I stubbled across <a href="//www.jjrobots.com/">JJRobots</a> and I just love the idea of this company. They sell a kit of electronic parts to maker hobbyists with access to 3D printers. The idea is that you use your 3D printer to make the items and everything you can't print is in the kit provided by JJRobots. They spend their time developing new products that you can print with the same kit. For example, the kit I purchased can be used to build the following:</p>
<ul>
<li>Marker board writer</li>
<li>Eggbot</li>
<li>Robotic arm</li>
<li>Motorized Camera slider</li>
<li>Mini Robotic</li>
<li>Air Hocky Robot</li>
</ul>
<p>So far I have only built the Eggbot but it was awesome! We spent last easter painting egg shells.</p>
<p><<Pictures here>></p>
<p>Unfortunately that print was tricky and my printer has been on the fritz lately (see other posts) and I haven't had the time/space to fix it. However, my family and I are moving to a bigger house with a dedicated maker space so I hope to start working on projects again. </p>Remember the Milk2018-05-19T00:00:00-04:002018-05-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-19:/remember-the-milk.html<p>I have been a paid subscriber and dedicated customer to Remember the Milk (RTM) for many years (I have lost count). RTM is a todo list application. It is in the same category of something like Wonderlist and ToDoist (There are many others). RTM fits my personality and needs quite …</p><p>I have been a paid subscriber and dedicated customer to Remember the Milk (RTM) for many years (I have lost count). RTM is a todo list application. It is in the same category of something like Wonderlist and ToDoist (There are many others). RTM fits my personality and needs quite nicely. My workflow usese a lot of ideas from David Allen's "Getting things Done" (GTD) and I mostly utilize RTM with insparation from the following blog post:</p>
<p><a href="blog.rememberthemilk.com/post/116665489183/guest-post-advanced-gtd-with-remember-the-milk">Remember the Milk</a></p>
<p>Over the years I have thought about switching to one of the many different product. However, I like RTM, I like the company and it works for me. I have my "Smart lists" setup with my basic workflows.</p>
<p>The smart lists allow me to "Program" RTM and make it into the app I want. Sure there are a lot of features I would request but RTM is the tool for me.</p>Back to Blogging2018-05-18T00:00:00-04:002018-05-18T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-18:/back-to-blogging.html<p>I like the idea of blogging but often have difficulty in the delivery. A few years ago I maintained a blog when I worked for the Institute for Cyber-Enabled Research (iCER) at Michigan State University (MSU). At iCER I did a lot of High Performance Computing (HPC) User support. This …</p><p>I like the idea of blogging but often have difficulty in the delivery. A few years ago I maintained a blog when I worked for the Institute for Cyber-Enabled Research (iCER) at Michigan State University (MSU). At iCER I did a lot of High Performance Computing (HPC) User support. This involved mostly helping researchers figure out how to use advanced computing systems in their research. I got into a habit of writing up a blog post when I learned something new or figured out a solution to a practical technical problem. I also found the blog useful for posting slides and examples from presentations I would often make at workshops and conferences. Many of these old posts are still see a little traffic. Check them out here:</p>
<p><a href="//wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/Dirk%27s+iCER+wiki+homepage">iCER Wiki</a></p>
<p>I also did some blogging related to my kids and our "maker" hobby.</p>
<p><a href="//apprenticemaker.blogspot.com/">Maker Wiki</a></p>
<p>Over the years I moved away from working directly for iCER and have not done a good job keeping up with my maker blog. I am going to try to bring the blogging habit back into my workflow. I am currently in the middle of quite a few projects relating to education and large scale computing.</p>
<p>I have a couple of goals for writing a blog. These include:</p>
<ul>
<li>Getting over my reluctance to share and communicate in online communities</li>
<li>Helping me organize and be accountable for my projects</li>
<li>Connecting with people with similar interests to my own</li>
<li>Providing informations that others may find helpful</li>
</ul>
<p>To reach these goals I am going to try to get into a daily habit of writing. I don't think I will post a blog everyday but the writing habit is important to me.</p>CNC Mill Picking2018-05-18T00:00:00-04:002018-05-18T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-18:/cnc-mill-picking.html<p>I am interested in purchasing a CNC machine for the shop space in our new house. I think I would like to stick to something around $1000 but would be willing to invest as much as $2000 if I can find the right machine. There are a lot of choices …</p><p>I am interested in purchasing a CNC machine for the shop space in our new house. I think I would like to stick to something around $1000 but would be willing to invest as much as $2000 if I can find the right machine. There are a lot of choices in this price range for hobby CNC machines. Some other selection criteria include:</p>
<ul>
<li>Big cut area (I would love 4'x8' but that seems a lot) - I am interested in trying out thigns you can do with furniture.</li>
<li>Small detailed work - I also have a few small projects that require detail. I realize it will be difficult to have a machine that does both big and small.</li>
<li>Open source hardware (I just like these projects) - Something I can easily modify with a rich user community.</li>
</ul>
<p>During my initial search I came across the following systems and communities.</p>
<p><a href="//sienci.com/product/sienci-mill-one-kit/">Sienci</a>
<a href="//diylilcnc.org/bigshoulders/">DiyLilcnc</a>
<a href="//carbide3d.com/shapeoko/">Shapeoko</a>
<a href="//www.inventables.com/technologies/x-carve/">X-Carve</a>
<a href="//openbuilds.com/?category=cartesian-style-cnc&id=287">Openbuild community</a></p>
<p>I just stumbled across the <a href="//www.maslowcnc.com/">Maslow</a> and got really excited. The large format capabilities (4'x8') with the really low price ($350) would leave more than enough in my budget to buy two machines. In fact, I may be able to buy a Maslow and then use it to make the parts I would need another project like the DiyLilcnc (Although I just realized the kickstart for that is relly old 2011). There are probably other projects.</p>Picking Blog Software2018-05-18T00:00:00-04:002018-05-18T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-18:/picking-blog-software.html<p>When I was at iCER we used confluence for our <a href="//wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/Dirk%27s+iCER+wiki+homepage">blogging software</a> and when I did a (personal blog](//apprenticemaker.blogspot.com/) I used google blogger. On of the barriers for me getting back into blogging (besides my own personal hangups) is that I was never really satisified with either …</p><p>When I was at iCER we used confluence for our <a href="//wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/Dirk%27s+iCER+wiki+homepage">blogging software</a> and when I did a (personal blog](//apprenticemaker.blogspot.com/) I used google blogger. On of the barriers for me getting back into blogging (besides my own personal hangups) is that I was never really satisified with either solution. I wanted to "own" my blog information and didn't really feel comfortable with advertisers making money directly from my content (this is it's own argument). Here are some criteria for what I was looking for:</p>
<ul>
<li>Easy to edit/ add new content</li>
<li>Easily backed up to my own computer and ported to a different system.</li>
<li>Ability to modify and add my own tweaks/ features.</li>
<li>Uses markdown or some similar text based file format to make it easy to use linux commands</li>
<li>Comments</li>
<li>Tags</li>
<li>Usage statistics</li>
</ul>
<p>After some digging and talking to people I decided that I wanted to try out a static website generation tool and the obvious one that seemed to fit my interests and criteria is Pelican and a git repository.</p>
<p>Unfortunately, Pelican was not as easy as I had hoped. On two different occasions over the last couple of years I have made a deliberate effort to build a new website using Pelican. It "worked" but there was some missing feature or bug that would get in my way actually making me adopt the format. I also think that my reluctance to post things makes it harder for me to get started.</p>
<p>In fact, as I write this blog post (in markdown) I do not actually have a blog to post it on. My hope is that having some posts already waiting "in the hopper" will make my use of the software easier. </p>
<p>With the help of one of my summer students, I am making an effort to try to come up with a useable version of a blog using Pelican. My feeling is that active open source projects are constantly evolving and therefore getting better. Maybe third time is a charm? If Pelican still does not work out I may need to explore a different platform. </p>
<p>In the end I would like a workflow that not only works for my blog but also for some of other projects that need websites. I also would like to try to make a kind of Pelican template repository that I can give to some of my peers and make it quick and easy for them to start their own website/blog.</p>Jupyter Accessibility Project2018-05-17T00:00:00-04:002018-05-17T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2018-05-17:/jupyter-accessibility-project.html<p>My university (Michigan State University) plans to scale up the use of Jupyter in our classrooms but we need to address accessibility issues before we can adopt the platform for wide scale use. On behalf of MSU I had an external group conduct a preliminary audit of our current jupyterhub …</p><p>My university (Michigan State University) plans to scale up the use of Jupyter in our classrooms but we need to address accessibility issues before we can adopt the platform for wide scale use. On behalf of MSU I had an external group conduct a preliminary audit of our current jupyterhub server. Based on this audit we determined that finding ways to improve Jupyter for the following user communities would be a good start.</p>
<ul>
<li>Individuals who have some one of the more common types of colorblindness</li>
<li>Individuals who use screen readers to help with understanding (ex. Dyslexia)</li>
<li>Individuals with dexterity issues who cannot effectively use a mouse</li>
</ul>
<p>For part of my Jupyter hub project I am going to try to address these issues. My hope is that some people are already working on these issues and I can just integrate existing plugins or patches. However, If these communities can not be found I will try to build one myself. I am currently talking with leaders in the Jupyter community to see the best way to build such a community. Feel free to send me an email (colbrydi@msu.edu) and I would be happy to include you in on the conversation once I figure it out.</p>Intrducing a 3D printer to my daughter's Kindergarten class2015-05-09T00:00:00-04:002015-05-09T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2015-05-09:/intrducing-a-3d-printer-to-my-daughters-kindergarten-class.html<p><img align="right," alt="./images/Photo5.jpg" hspace="10" src="../images/Photo5.jpg" width="200,"></p>
<p>We had fun showing off my 3D printer to the Kindergarten class. Although the
printer is fun to watch, kids tend to get board with it in just a few minutes.
I like to come up with classroom projects that let the kids use things that
are printed with the …</p><p><img align="right," alt="./images/Photo5.jpg" hspace="10" src="../images/Photo5.jpg" width="200,"></p>
<p>We had fun showing off my 3D printer to the Kindergarten class. Although the
printer is fun to watch, kids tend to get board with it in just a few minutes.
I like to come up with classroom projects that let the kids use things that
are printed with the printer. Since it is so cheep it is also fun to print
something ahead of time that they can take home. </p>
<p>For the kindergarten class we printed some flowers that the kids could use to
put together a cut spring time flower pots just in time for mother's day: </p>
<p><img align="right," alt="./images/Photo1.jpg" hspace="10" src="../images/Photo1.jpg" width="200,"></p>
<p>It even lights up:</p>
<p><img align="right," alt="./images/Photo6.jpg" hspace="10" src="../images/Photo6.jpg" width="200,"></p>
<p>The kids had a great time. Here are some pictures </p>
<p><img align="right," alt="./images/Photo3.jpg" hspace="10" src="../images/Photo3.jpg" width="200,"></p>
<p><img align="right," alt="./images/Photo4.jpg" hspace="10" src="../images/Photo4.jpg" width="200,"></p>
<p><img align="right," alt="./images/Photo2.jpg" hspace="10" src="../images/Photo2.jpg" width="200,"></p>
<p>Turned out to be a lot of fun. If you are interested in making your own here is a list of instructions on Instructables:</p>
<p><a href="http://www.instructables.com/id/Spring-kindergarten-Class-Project-with-3D-Printer/">http://www.instructables.com/id/Spring-kindergarten-Class-Project-with-3D-Printer/</a> </p>
<p>And the 3D files on YouMagine:</p>
<p><a href="https://www.youmagine.com/designs/light-up-flower">https://www.youmagine.com/designs/light-up-flower</a></p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2015/05/showing-off-3d-printer-to-daughters.html">Blogger</a> using custom python script. Comment on errors below.</p>Electric box extender2015-05-02T00:00:00-04:002015-05-02T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2015-05-02:/electric-box-extender.html<p><img align="right," alt="./images/blogger-image--470762115.jpg" hspace="10" src="../images/blogger-image--470762115.jpg" width="200,"></p>
<p>I have been having trouble with my 3D printer over the last couple of months.
It turns out that the fans got disconnected. Once I fixed the problem my
printer is working better than ever. I was itching for something to print so I
came up with this simple spacer …</p><p><img align="right," alt="./images/blogger-image--470762115.jpg" hspace="10" src="../images/blogger-image--470762115.jpg" width="200,"></p>
<p>I have been having trouble with my 3D printer over the last couple of months.
It turns out that the fans got disconnected. Once I fixed the problem my
printer is working better than ever. I was itching for something to print so I
came up with this simple spacer. </p>
<p>I installed a motion sensor in my basement because the light switch is in an
inconvenient place and most of the time we go in this part of the basement my
hands seem to be full. The motion sensor fixes this problem and even found a
outlet plate on thingiverse that fit perfectly.</p>
<p><a href="http://www.thingiverse.com/thing:457658">http://www.thingiverse.com/thing:457658</a></p>
<p>The only problem is that the installed switch box is too small for the new
sensor and my printed play sticks out a good 15 mm from the box.</p>
<p><img align="right," alt="./images/blogger-image-2136501427.jpg" hspace="10" src="../images/blogger-image-2136501427.jpg" width="200,"></p>
<p>A few minutes with OpenScad and a two hour print later I have a nice looking
solution.</p>
<p><img align="right," alt="./images/blogger-image--906009313.jpg" hspace="10" src="../images/blogger-image--906009313.jpg" width="200,"></p>
<p>Here is a link to the OpenSCAD file if you are interested in printing your
own: </p>
<p><a href="https://www.youmagine.com/designs/wall-switch-spacer">https://www.youmagine.com/designs/wall-switch-spacer</a> </p>
<p>Easy and fun! Nice to have my printer back!</p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2015/05/electric-box-extender.html">Blogger</a> using custom python script. Comment on errors below.</p>Ultimaker 2, cleaning a clogged nozzle2015-05-02T00:00:00-04:002015-05-02T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2015-05-02:/ultimaker-2-cleaning-a-clogged-nozzle.html<p><img align="right," alt="./images/IMG_2048.jpg" hspace="10" src="../images/IMG_2048.jpg" width="200,"></p>
<p>So my Ultimaker2 extruding head got clogged, really bad. Something got jammed
while printing and I could not get the filament to feed. This had happened
once in the past when I tried using some really cheap filament but I am not
sure the real root of the problem this …</p><p><img align="right," alt="./images/IMG_2048.jpg" hspace="10" src="../images/IMG_2048.jpg" width="200,"></p>
<p>So my Ultimaker2 extruding head got clogged, really bad. Something got jammed
while printing and I could not get the filament to feed. This had happened
once in the past when I tried using some really cheap filament but I am not
sure the real root of the problem this time. In any case, I found using the
"atomic" method for getting a clean nozzle. This method is named after
MostlyAtomicBob who described the method for the first time in the following
post:</p>
<p>http://umforum.ultimaker.com/index.php?/topic/3561-ultimaker-2-extruder-nozzle-blocked/#entry28940</p>
<p>Here is a summary of the steps that I used:</p>
<p>1) Remove the boden tube from the printer head</p>
<p>2) Heat up the head to full temperature (around 210 degrees)</p>
<p>3) Push in a piece of spare filament until I can no longer push or it starts
coming out the end.</p>
<p>4) Reduce temperature to around 90 degrees. Filament should solidify but stay
a little soft.</p>
<p>5) Pull the filament out of the print head.</p>
<p>6) Repeat steps 2-5 until I get completely clean filament</p>
<p>You can see the result of my cleaning in the picture below. The pieces are a
little backward and ordered from right to left. Note the really dark char
slowly going away and the pieces getting longer as they nest better and better
in the print head.</p>
<p>Here is the same procedure when it clogged a second time. This time I
remembered to order them from left to right. </p>
<p><img align="right," alt="./images/blogger-image-193838586.jpg" hspace="10" src="../images/blogger-image-193838586.jpg" width="200,"></p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/12/ultimaker-2-cleaning-clogged-nozzle.html">Blogger</a> using costom python script. Comment on errors below.</p>April 1st Brownies2015-04-03T00:00:00-04:002015-04-03T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2015-04-03:/april-1st-brownies.html<p>An oldie but a goodie with a modern twist. This year I printed out the letter
E on my 3D printer using Brown PLA plastic. </p>
<p><img align="right," alt="" hspace="10" src="../images/blogger-image--2074689448.jpg" width="200,"></p>
<p>Then I put the Brown E's in a baking pan:</p>
<p><img alt="" src="../images/blogger-image--458341699.jpg"></p>
<p>Cover with aluminum foil and tell the kids I made brownies for them.</p>
<p><img align="right," alt="../images/blogger-image-1911830558.jpg" hspace="10" src="../images/blogger-image-1911830558.jpg" width="200,"></p>
<p>The were …</p><p>An oldie but a goodie with a modern twist. This year I printed out the letter
E on my 3D printer using Brown PLA plastic. </p>
<p><img align="right," alt="" hspace="10" src="../images/blogger-image--2074689448.jpg" width="200,"></p>
<p>Then I put the Brown E's in a baking pan:</p>
<p><img alt="" src="../images/blogger-image--458341699.jpg"></p>
<p>Cover with aluminum foil and tell the kids I made brownies for them.</p>
<p><img align="right," alt="../images/blogger-image-1911830558.jpg" hspace="10" src="../images/blogger-image-1911830558.jpg" width="200,"></p>
<p>The were a little disappointed but then quickly got into the spirit of the day
and played the same joke on their mom and nanny. I even took some "sugar
free" brownies into the dentist with me and gave some to my brother and his
house mates. </p>
<p>Happy April fools day.</p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2015/04/april-1st-brownies.html">Blogger</a> using custom python script. Comment on errors below.</p>Hack to automatically restart programs that stall during inicialization2015-01-16T00:00:00-05:002015-01-16T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2015-01-16:/hack-to-automatically-restart-programs-that-stall-during-inicialization.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Sometimes we get jobs that stall out right at the beginning but do not error
out until the walltime for the job has been exceeded. Users get an email
saying their job "exceeds walltime …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Sometimes we get jobs that stall out right at the beginning but do not error
out until the walltime for the job has been exceeded. Users get an email
saying their job "exceeds walltime" but when they check the output nothing (or
very little) seems to have happened. The cause of this problem is highly
dependent on what the job is doing. However, in some cases a simple resubmit
of the job gets it working. The following scripts check to see if the program
is running and automatically re-submits the job if their seems to be a
problem.</p>
<h2><strong>file_flag_example.qsub</strong></h2>
<div class="highlight"><pre><span></span><code><span class="w"> </span>#!/bin/bash<span class="w"> </span>-l
<span class="w"> </span>#PBS<span class="w"> </span>-l<span class="w"> </span>mem=100gb,nodes=100:ppn=1,walltime=03:59:59
<span class="w"> </span>#PBS<span class="w"> </span>-m<span class="w"> </span>a
<span class="w"> </span>#PBS<span class="w"> </span>-j<span class="w"> </span>oe
<span class="w"> </span>#<span class="w"> </span>This<span class="w"> </span>method<span class="w"> </span>checks<span class="w"> </span>to<span class="w"> </span>see<span class="w"> </span>if<span class="w"> </span>a<span class="w"> </span>file<span class="w"> </span>is<span class="w"> </span>created<span class="w"> </span>at<span class="w"> </span>the<span class="w"> </span>beginning<span class="w"> </span>of<span class="w"> </span>a<span class="w"> </span>run.<span class="w"> </span>If<span class="w"> </span>the<span class="w"> </span>file<span class="w"> </span>is<span class="w"> </span>not<span class="w"> </span>created<span class="w"> </span>then<span class="w"> </span>the<span class="w"> </span>job<span class="w"> </span>is<span class="w"> </span>killed<span class="w"> </span>and<span class="w"> </span>restarted.
<span class="w"> </span>#<span class="w"> </span>This<span class="w"> </span>method<span class="w"> </span>assumes<span class="w"> </span>that<span class="w"> </span>the<span class="w"> </span>program<span class="w"> </span>stalls<span class="w"> </span>before<span class="w"> </span>it<span class="w"> </span>has<span class="w"> </span>a<span class="w"> </span>chance<span class="w"> </span>to<span class="w"> </span>generate<span class="w"> </span>a<span class="w"> </span>file.<span class="w"> </span>This<span class="w"> </span>method<span class="w"> </span>is<span class="w"> </span>useful<span class="w"> </span>if<span class="w"> </span>the<span class="w"> </span>code<span class="w"> </span>does<span class="w"> </span>self<span class="w"> </span>checkpointing<span class="w"> </span>and<span class="w"> </span>the<span class="w"> </span>failure<span class="w"> </span>state<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>busy<span class="w"> </span>wait.
<span class="w"> </span>cd<span class="w"> </span><span class="cp">${</span><span class="n">PBS_O_WORKDIR</span><span class="cp">}</span>
<span class="w"> </span>
<span class="w"> </span>#Run<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
<span class="w"> </span>(<span class="w"> </span>
<span class="w"> </span>mpirun<span class="w"> </span>mycommand<span class="w"> </span>
<span class="w"> </span>)<span class="w"> </span><span class="err">&</span>
<span class="w"> </span>PID=$!
<span class="w"> </span>#<span class="w"> </span>Sleep<span class="w"> </span>for<span class="w"> </span>enough<span class="w"> </span>time<span class="w"> </span>to<span class="w"> </span>see<span class="w"> </span>if<span class="w"> </span>the<span class="w"> </span>job<span class="w"> </span>is<span class="w"> </span>running
<span class="w"> </span>sleep<span class="w"> </span>300
<span class="w"> </span>if<span class="w"> </span>[<span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span>testfile.flag<span class="w"> </span>]
<span class="w"> </span>then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>Seems<span class="w"> </span>to<span class="w"> </span>have<span class="w"> </span>stalled.<span class="w"> </span>Killing<span class="w"> </span>and<span class="w"> </span>restarting"
<span class="w"> </span>kill<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>qsub<span class="w"> </span>$0
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>stats<span class="w"> </span>for<span class="w"> </span>debugging"
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>exit<span class="w"> </span>1
<span class="w"> </span>fi
<span class="w"> </span>wait<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>RET=$?
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>#return<span class="w"> </span>the<span class="w"> </span>output<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
<span class="w"> </span>exit<span class="w"> </span><span class="nv">$RET</span>
</code></pre></div>
<p><strong>output_monitor_example.qsub</strong></p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>#!/bin/bash<span class="w"> </span>-l
<span class="w"> </span>#PBS<span class="w"> </span>-l<span class="w"> </span>mem=100gb,nodes=100:ppn=1,walltime=03:59:59
<span class="w"> </span>#PBS<span class="w"> </span>-m<span class="w"> </span>a
<span class="w"> </span>#PBS<span class="w"> </span>-j<span class="w"> </span>oe
<span class="w"> </span>#<span class="w"> </span>This<span class="w"> </span>method<span class="w"> </span>monitors<span class="w"> </span>job<span class="w"> </span>output<span class="w"> </span>and<span class="w"> </span>stops<span class="w"> </span>if<span class="w"> </span>the<span class="w"> </span>output<span class="w"> </span>doesn't<span class="w"> </span>change.<span class="w"> </span>
<span class="w"> </span>#<span class="w"> </span>This<span class="w"> </span>method<span class="w"> </span>assumes<span class="w"> </span>that<span class="w"> </span>the<span class="w"> </span>program<span class="w"> </span>continuously<span class="w"> </span>generates<span class="w"> </span>output<span class="w"> </span>at<span class="w"> </span>regular<span class="w"> </span>intervals.<span class="w"> </span>
<span class="w"> </span>cd<span class="w"> </span><span class="cp">${</span><span class="n">PBS_O_WORKDIR</span><span class="cp">}</span>
<span class="w"> </span>testfile=`testfile.flag`
<span class="w"> </span>
<span class="w"> </span>#Run<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
<span class="w"> </span>(<span class="w"> </span>
<span class="w"> </span>mpirun<span class="w"> </span>mycommand<span class="w"> </span>><span class="w"> </span><span class="nv">$testfile</span>
<span class="w"> </span>)<span class="w"> </span><span class="err">&</span>
<span class="w"> </span>PID=$!
<span class="w"> </span>#<span class="w"> </span>Sleep<span class="w"> </span>for<span class="w"> </span>enough<span class="w"> </span>time<span class="w"> </span>to<span class="w"> </span>start<span class="w"> </span>generating<span class="w"> </span>output
<span class="w"> </span>sleep<span class="w"> </span>300
<span class="w"> </span>linecount1=`cat<span class="w"> </span><span class="nv">$testfile</span><span class="w"> </span>|<span class="w"> </span>wc<span class="w"> </span>-l`
<span class="w"> </span>
<span class="w"> </span>#<span class="w"> </span>Sleep<span class="w"> </span>enough<span class="w"> </span>for<span class="w"> </span>more<span class="w"> </span>output
<span class="w"> </span>sleep<span class="w"> </span>100
<span class="w"> </span>linecount2=`cat<span class="w"> </span><span class="nv">$testfile</span><span class="w"> </span>|<span class="w"> </span>wc<span class="w"> </span>-l`
<span class="w"> </span>if<span class="w"> </span>[<span class="w"> </span>"<span class="nv">$linecount1</span>"<span class="w"> </span>==<span class="w"> </span>"<span class="nv">$linecount2</span>"<span class="w"> </span>]
<span class="w"> </span>then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>Seems<span class="w"> </span>to<span class="w"> </span>have<span class="w"> </span>stalled.<span class="w"> </span>Killing<span class="w"> </span>and<span class="w"> </span>restarting"
<span class="w"> </span>kill<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>qsub<span class="w"> </span>$0
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>stats<span class="w"> </span>for<span class="w"> </span>debugging"
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>exit<span class="w"> </span>1
<span class="w"> </span>fi
<span class="w"> </span>wait<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>RET=$?
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>#return<span class="w"> </span>the<span class="w"> </span>output<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
<span class="w"> </span>exit<span class="w"> </span><span class="nv">$RET</span>
</code></pre></div>
<h2><strong>qstat_monitor_example.qsub</strong></h2>
<div class="highlight"><pre><span></span><code><span class="w"> </span>#!/bin/bash<span class="w"> </span>-l
<span class="w"> </span>#PBS<span class="w"> </span>-l<span class="w"> </span>mem=100gb,nodes=100:ppn=1,walltime=03:59:59
<span class="w"> </span>#PBS<span class="w"> </span>-m<span class="w"> </span>a
<span class="w"> </span>#PBS<span class="w"> </span>-j<span class="w"> </span>oe
<span class="w"> </span>#<span class="w"> </span>This<span class="w"> </span>method<span class="w"> </span>uses<span class="w"> </span>the<span class="w"> </span>same<span class="w"> </span>idea<span class="w"> </span>as<span class="w"> </span>the<span class="w"> </span>previous<span class="w"> </span>but<span class="w"> </span>instead<span class="w"> </span>of<span class="w"> </span>relying<span class="w"> </span>on<span class="w"> </span>output<span class="w"> </span>it<span class="w"> </span>uses<span class="w"> </span>the<span class="w"> </span>cput<span class="w"> </span>stat<span class="w"> </span>generated<span class="w"> </span>by<span class="w"> </span>qstat.
<span class="w"> </span>#<span class="w"> </span>This<span class="w"> </span>solution<span class="w"> </span>will<span class="w"> </span>not<span class="w"> </span>work<span class="w"> </span>if<span class="w"> </span>the<span class="w"> </span>job<span class="w"> </span>is<span class="w"> </span>in<span class="w"> </span>a<span class="w"> </span>busy<span class="w"> </span>wait<span class="w"> </span>state.<span class="w"> </span>
<span class="w"> </span>cd<span class="w"> </span><span class="cp">${</span><span class="n">PBS_O_WORKDIR</span><span class="cp">}</span>
<span class="w"> </span>#Run<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
<span class="w"> </span>(<span class="w"> </span>
<span class="w"> </span>mpirun<span class="w"> </span>mycommand<span class="w"> </span>
<span class="w"> </span>)<span class="w"> </span><span class="err">&</span>
<span class="w"> </span>PID=$!
<span class="w"> </span>#<span class="w"> </span>Sleep<span class="w"> </span>for<span class="w"> </span>enough<span class="w"> </span>time<span class="w"> </span>to<span class="w"> </span>start<span class="w"> </span>generating<span class="w"> </span>output
<span class="w"> </span>sleep<span class="w"> </span>300
<span class="w"> </span>cpu1=`qstat<span class="w"> </span>-f<span class="w"> </span><span class="nv">$PBS_JOBID</span><span class="w"> </span>|<span class="w"> </span>grep<span class="w"> </span>resources_used.cput`
<span class="w"> </span>
<span class="w"> </span>#<span class="w"> </span>Sleep<span class="w"> </span>enough<span class="w"> </span>for<span class="w"> </span>more<span class="w"> </span>output
<span class="w"> </span>sleep<span class="w"> </span>100
<span class="w"> </span>cpu2=`qstat<span class="w"> </span>-f<span class="w"> </span><span class="nv">$PBS_JOBID</span><span class="w"> </span>|<span class="w"> </span>grep<span class="w"> </span>resources_used.cput`
<span class="w"> </span>if<span class="w"> </span>[<span class="w"> </span>"<span class="nv">$cpu1</span>"<span class="w"> </span>==<span class="w"> </span>"<span class="nv">$cpu2</span>"<span class="w"> </span>]
<span class="w"> </span>then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>Seems<span class="w"> </span>to<span class="w"> </span>have<span class="w"> </span>stalled.<span class="w"> </span>Killing<span class="w"> </span>and<span class="w"> </span>restarting"
<span class="w"> </span>kill<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>qsub<span class="w"> </span>$0
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>stats<span class="w"> </span>for<span class="w"> </span>debugging"
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>exit<span class="w"> </span>1
<span class="w"> </span>fi
<span class="w"> </span>wait<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>RET=$?
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>#return<span class="w"> </span>the<span class="w"> </span>output<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
<span class="w"> </span>exit<span class="w"> </span><span class="nv">$RET</span>
</code></pre></div>
<p>These solutions are nice work arounds because, if it works, the scripts just
restarts your job until it runs and gets the research done. However, using
this hack does not get at the root of the problem. Actually there are two
problems:</p>
<ol>
<li>Something is broken causing the job to hang. This could be a race condition in the code, a bad node, bad file I/O, bad network connections, etc. All depends on what the code is doing.</li>
<li>Code hangs insteads of quitting and reporting an error. Well engineered code should not hang. For example, file and network access should have timeouts so that code is not running forever. </li>
</ol>
<p>Researchers, should first notify the HPCC if they are using this hack so we
can try to track down problems with the nodes. Researchers should also work to
modify their code to report an error if something hanges. This will also help
track down the problem.</p>
<ul>
<li>Dirk</li>
</ul>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2015/01/16/Hack+to+automatically+restart+programs+that+stall+during+inicialization">ICER Wiki</a> using custom python script. Comment on errors below.</p>3D Cookie Cutter app comparison2014-12-25T00:00:00-05:002014-12-25T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-12-25:/3d-cookie-cutter-app-comparison.html<p>My oldest daughter is in her second year of Brownies. My wife and I
volunteered to lead a couple of their meetings with a maker theme. I thought
the 3D printer is always a hit and what better project for girl scouts than
cookies (yes I realize this is cliche …</p><p>My oldest daughter is in her second year of Brownies. My wife and I
volunteered to lead a couple of their meetings with a maker theme. I thought
the 3D printer is always a hit and what better project for girl scouts than
cookies (yes I realize this is cliche but it is still fun). I did some
searching on the internet and came up with a variety of ways the girl scouts
could design cookie cutters. However, I had a checklist of desires and none
of them quite made the "cut" (I will stop now). Here is what I was looking
for: </p>
<ol>
<li>An option to upload and trace a picture. This is necessary because I do not have enough computers for the group. I saw a similar activity at the Detroit Maker Faire and I thought hand drawn images would be more natural.</li>
<li>Ability to add both internal and external features on the cookies. What I would really like is two layers so the cutters could imprint on the cookie as well as cut through the cookie.</li>
<li>Ideally an online app so I do not have to download anything. </li>
<li>Obviously an easy way to output stl files I could use with my printer.</li>
</ol>
<p><strong>CookieCaster:</strong><a href="http://www.cookiecaster.com/#editor">http://www.cookiecaster.com/#editor</a><br>
Online app for drawing outline of cookie cutters. This makes for a fun group
project because you can scan and upload hand drawings into the editor and
trace them. The downside is that this editor only does a cookie outline and
not the insides. Here is an example we did: </p>
<p><img align="right," alt="./images/CookieCaster_screen.tiff" hspace="10" src="../images/CookieCaster_screen.tiff" width="200,"></p>
<p><strong>Cookie Cutter Editor:</strong> <a href="http://www.local-guru.net/blog/pages/cookiecutter-editor">http://www.local-guru.net/blog/pages/cookiecutter-editor</a><br>
Downloadable cookie cutter editor. Uses a programming language called
processing (written in java). I could not get the MacOS stand alone version
to work so I tried to download processing and run the source directly. Still
was missing some libraries so I gave up on it. Looking at the examples I do
not think it would work much better than CookieCaster and could not get any of
the internal details I wanted anyway. </p>
<p><strong>Thingiverse:</strong></p>
<p><img alt="http://www.thingiverse.com/apps/customizer/run?thing_id=116042&code=2633da7762519236fd52412f6c8bf143" src="http://www.thingiverse.com/apps/customizer/run?thing_id=116042&code=2633da7762519236fd52412f6c8bf143"> </p>
<p>Thingiverse has one of their customizable apps. This one was really nice
because it could do some internal details. Seems to only have one height but
having details is better than the other two. However, this app also only had
an option for online editing and no way to upload a picture. It was also a
difficult interface to master. For example, it does not handle overlapping
editing very well and was particularly hard to get right with my laptop's
touchpad. Here is an examples we made: </p>
<p><a href="http://www.thingiverse.com/apps/customizer/run?thing_id=116042">http://www.thingiverse.com/apps/customizer/run?thing_id=116042</a> </p>
<p>Although none of the apps seem like they would work, none of them really fit
all of my criteria. I am going to have the girl scouts use Cookie Caster
since it is the one that works with pictures. Please stay tuned for future
blog posts with details highlights from the girl scout projects. </p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/12/3d-cookie-cutter-app-comparison.html">Blogger</a> using custom python script. Comment on errors below.</p>2014-12-16 HPCC workshop slides and handouts2014-12-15T00:00:00-05:002014-12-15T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-12-15:/2014-12-16-hpcc-workshop-slides-and-handouts.html<p>Blog post <strong>edited</strong> by Anonymous</p>
<p>I will be teaching my bi-annual Introductory and Advanced HPCC workshops
tomorrow. Below are links to my updated slides and handouts. Registration
looks lite so feel free to drop in if you have the time. These workshops are
being provided as part of IT Services …</p><p>Blog post <strong>edited</strong> by Anonymous</p>
<p>I will be teaching my bi-annual Introductory and Advanced HPCC workshops
tomorrow. Below are links to my updated slides and handouts. Registration
looks lite so feel free to drop in if you have the time. These workshops are
being provided as part of IT Services two day offering of no-charge seminars
to faculty and graduate students on technology topics on December 16 and 17.
More information and registration can be done at the following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu/events/tech-seminars/</a></p>
<p><strong>Introduction to the HPCC: A Hands-On Introduction to High Performance
Computing at MSU</strong></p>
<ul>
<li>Tuesday, December 16th 8:30am-11:30am in 403 of the Computer Center</li>
<li>During this workshop, participants will learn through hands-on examples how to get started with the MSU High Performance Computing Center (HPCC). Topics include: connecting to the HPCC; copying files to your home directories; navigating the command line interface; accessing available software; testing and running programs interactively; editing files; writing job scripts; submitting jobs to the queue; and monitoring jobs running on the system.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help you get started and diagnose problems.</li>
</ul>
<p><strong>12:00-1:15pm Lunch is provided in the MSU Ballroom (registered participants
only)</strong></p>
<p><strong>Making Your Research Go Faster: Advance Topics in Getting the Most Out of
the MSU HPCC</strong></p>
<ul>
<li>Tuesday, December 16th 1:30pm-4:30pm in 403 of the Computer Center</li>
<li>During this workshop, participants already familiar with using the HPCC systems will be shown advanced techniques on how to use the system more effectively. Topics include: techniques for effective scheduling of pleasantly parallel jobs; shared memory jobs and shared network jobs; running jobs longer than a week; and making jobs fault tolerant.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help get you started and diagnose problems.</li>
</ul>
<p>SLIDES:</p>
<ul>
<li>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/2014-12-16%20Introduction%20to%20HPCC.pdf?version=1&modificationDate=1418669575000&api=v2">2014-12-16 Introduction to HPCC.pdf</a></p>
</li>
<li>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/2014-12-16_Advanced%20HPCC.pdf?version=1&modificationDate=1418669589000&api=v2">2014-12-16_Advanced HPCC.pdf</a></p>
</li>
</ul>
<p>HANDOUTS:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/2014-12-16%20[HPCC_WORKSHOP_COMMANDS.pdf](./images/HPCC_WORKSHOP_COMMANDS.pdf)?version=1&modificationDate=1418670002000&api=v2">2014-12-16 [HPCC_WORKSHOP_COMMANDS.pdf](./images/HPCC_WORKSHOP_COMMANDS.pdf)</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/HPCC_Command_Summary.pdf?version=1&modificationDate=1418669599000&api=v2">HPCC_Command_Summary.pdf</a></li>
<li><a href="http://files.fosswire.com/2007/08/fwunixref.pdf">http://files.fosswire.com/2007/08/fwunixref.pdf</a></li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/12/15/2014-12-16+HPCC+workshop+slides+and+handouts">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/12/15/2014-12-16+HPCC+workshop+slides+and+handouts">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-12-05 Western Michigan University, Introduction to iCER slides2014-12-04T00:00:00-05:002014-12-04T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-12-04:/2014-12-05-western-michigan-university-introduction-to-icer-slides.html<p>Blog post <strong>edited</strong> by Anonymous</p>
<p>Here are a copy of my slides and the handout for our two hour introductory
talk at Western Michigan University:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411650/2014-12-05%20WMU%20iCER%20Talk.pdf?version=1&modificationDate=1417709173000&api=v2">2014-12-05 WMU iCER Talk.pdf</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411650/2014-12-05%20WMU%20iCER%20Talk%20Commands.pdf?version=1&modificationDate=1417709861000&api=v2">2014-12-05 WMU iCER Talk Commands.pdf</a></li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/12/04/2014-12-05+Western+Michigan+University%2C+Introduction+to+iCER+slides">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/12/04/2014-12-05+Western+Michigan+University%2C+Introduction+to+iCER+slides">ICER Wiki</a> using custom python script. Comment on errors …</p><p>Blog post <strong>edited</strong> by Anonymous</p>
<p>Here are a copy of my slides and the handout for our two hour introductory
talk at Western Michigan University:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411650/2014-12-05%20WMU%20iCER%20Talk.pdf?version=1&modificationDate=1417709173000&api=v2">2014-12-05 WMU iCER Talk.pdf</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411650/2014-12-05%20WMU%20iCER%20Talk%20Commands.pdf?version=1&modificationDate=1417709861000&api=v2">2014-12-05 WMU iCER Talk Commands.pdf</a></li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/12/04/2014-12-05+Western+Michigan+University%2C+Introduction+to+iCER+slides">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/12/04/2014-12-05+Western+Michigan+University%2C+Introduction+to+iCER+slides">ICER Wiki</a> using custom python script. Comment on errors below.</p>Fairy House2014-11-28T00:00:00-05:002014-11-28T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-11-28:/fairy-house.html<p><img align="right," alt="./images/DSC_6664.JPG" hspace="10" src="../images/DSC_6664.JPG" width="200,"></p>
<p>Over the summer we spent some time up at our Aunt and Uncle's cottage. Kids
were picking up a lot of nature detritus to "keep" and my wife got the idea to
make a Fairy House. Turned out to be a really fun project. I particularly
like the paper lantern …</p><p><img align="right," alt="./images/DSC_6664.JPG" hspace="10" src="../images/DSC_6664.JPG" width="200,"></p>
<p>Over the summer we spent some time up at our Aunt and Uncle's cottage. Kids
were picking up a lot of nature detritus to "keep" and my wife got the idea to
make a Fairy House. Turned out to be a really fun project. I particularly
like the paper lantern made from an old hornet's nest and the ladder. The
trick seems to be start with the big stuff and keep filling in the details
until it looked right. </p>
<p>- Dirk</p>
<p><img align="right," alt="./images/DSC_6675.JPG" hspace="10" src="../images/DSC_6675.JPG" width="200,"></p>
<p><img align="right," alt="./images/DSC_6680.JPG" hspace="10" src="../images/DSC_6680.JPG" width="200,"></p>
<p><img align="right," alt="./images/DSC_6645.JPG" hspace="10" src="../images/DSC_6645.JPG" width="200,"></p>
<p><img align="right," alt="./images/DSC_6682.JPG" hspace="10" src="../images/DSC_6682.JPG" width="200,"></p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/11/fairy-house.html">Blogger</a> using custom python script. Comment on errors below.</p>Learning to use a sewing and embroidering machine2014-11-28T00:00:00-05:002014-11-28T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-11-28:/learning-to-use-a-sewing-and-embroidering-machine.html<p><img align="right," alt="./images/blogger-image-206748741.jpg" hspace="10" src="../images/blogger-image-206748741.jpg" width="200,"></p>
<p>We are trying to turn our basement into a mini makerspace for the kids. Quite
a few years ago my wife bought a fancy sewing machine that does embroidery
which I thought would make a fun addition. My eldest daughter and I got
playing with the machine and it is …</p><p><img align="right," alt="./images/blogger-image-206748741.jpg" hspace="10" src="../images/blogger-image-206748741.jpg" width="200,"></p>
<p>We are trying to turn our basement into a mini makerspace for the kids. Quite
a few years ago my wife bought a fancy sewing machine that does embroidery
which I thought would make a fun addition. My eldest daughter and I got
playing with the machine and it is quite amazing. In no time at all we were
able to embroider a bow and make a cute little pillow. I look forward to
seeing what we will make next time.</p>
<p>- Dirk</p>
<p><img align="right," alt="./images/blogger-image-1769496472.jpg" hspace="10" src="../images/blogger-image-1769496472.jpg" width="200,"></p>
<p><img align="right," alt="./images/blogger-image-967993881.jpg" hspace="10" src="../images/blogger-image-967993881.jpg" width="200,"></p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/11/learning-to-use-sewing-and-embroidering.html">Blogger</a> using custom python script. Comment on errors below.</p>Lithophanes2014-11-28T00:00:00-05:002014-11-28T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-11-28:/lithophanes.html<p><img align="right," alt="Snowflake" hspace="10" src="../images/il_570xN.688476899_eapl.jpg" width="200,"></p>
<p>I saw my first 3D printer when I was visiting the University of Notre Dame.
It was over 10 years ago, it was fairly expensive to operate and it printed in
a kind of expensive wax. </p>
<p>Anyway, this was the first time I saw a lithophane. It was a picture …</p><p><img align="right," alt="Snowflake" hspace="10" src="../images/il_570xN.688476899_eapl.jpg" width="200,"></p>
<p>I saw my first 3D printer when I was visiting the University of Notre Dame.
It was over 10 years ago, it was fairly expensive to operate and it printed in
a kind of expensive wax. </p>
<p>Anyway, this was the first time I saw a lithophane. It was a picture of
Marilyn Monroe that just popped out if you held it so up to a light so that it
is backlit. It was just so cool that I really wanted to make some Lithophanes
when I got my 3D printer. </p>
<p>Once I finally found some time I did some research and there were a few
programs on the web which would make lithophanes for me. </p>
<p>http://maker.dongguk.edu/webclient/cylinder_litho_index.html<br>
http://www.thingiverse.com/thing:74322 </p>
<p>I tried a few but nothing excited me so I just wrote one myself. I am very
pleased with the results: </p>
<p>If you just want a lithophane, we added them to my wife's etsy store: </p>
<p><a href="https://www.etsy.com/shop/familyheART">https://www.etsy.com/shop/familyheART</a> </p>
<p>If you already have a printer I am happy to send you the MATLAB code to
generate these for yourself. However, I must warn you that it is not very
robust and I am reluctant to post it until I can add a bunch of error checking
and a little better user interface. It's also written in MATLAB, ideally I
would like to change to something like Python but I am just quicker working
with MATLAB.</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/11/lithophanes.html">Blogger</a> using custom python script. Comment on errors below.</p>Snowflake Lithophane2014-11-28T00:00:00-05:002014-11-28T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-11-28:/snowflake-lithophane.html<p><img align="right," alt="./images/Snowflake.png" hspace="10" src="../images/Snowflake.png" width="200,"></p>
<p>There is something magical about looking at a lithophane. I like handing them
to people with the light coming from the front. Most of the time they are
polite and say, "that's nice." Then I have them hold the lithophane to let
the light come in from the back and …</p><p><img align="right," alt="./images/Snowflake.png" hspace="10" src="../images/Snowflake.png" width="200,"></p>
<p>There is something magical about looking at a lithophane. I like handing them
to people with the light coming from the front. Most of the time they are
polite and say, "that's nice." Then I have them hold the lithophane to let
the light come in from the back and the image pops out. "Wow," "cool" they
say and the expressions on their faces are ones of awe and surprise. </p>
<p>For Christmas this year, my wife came up with the idea of putting a picture of
the kids inside of a lithophane inside of a 3D printed snowflake. </p>
<p><img align="right," alt="./images/il_570xN.688476899_eapl.jpg" hspace="10" src="../images/il_570xN.688476899_eapl.jpg" width="200,"></p>
<p>Took a while to figure out but got much easier when I found an openscad
program to generate snowflakes I found on Thingiverse: </p>
<p><a href="http://www.thingiverse.com/thing:74322">http://www.thingiverse.com/thing:74322</a> </p>
<p>I picked a random seed that fit well with my idea and then I tried importing
my the stl file from my lithophane program. I could not quite git it to work.
So, instead I decided to make the print in two parts and glue them together. I
did some minor modifications to the openscad file add an area for the
lithophane, the hole for a ribbon and in latter modifications a way to add the
year.</p>
<p>You can download the stl file for the frame on youmagine:</p>
<p><a href="https://www.youmagine.com/designs/snowflake-ornament-frame">https://www.youmagine.com/designs/snowflake-ornament-frame</a></p>
<p>Now I just print the lithophane separate from the frame and it turns out quite
well. Here is a video:</p>
<p>I am happy to share any of the code or you can just upload a photo and order
one of these on our etsy store: </p>
<p><a href="https://www.etsy.com/shop/familyheART">https://www.etsy.com/shop/familyheART</a> </p>
<p>I wife has challenged me to do something similar with a heart for valentines
day. That should be fun. </p>
<p>- Dirk </p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/11/snowflake-lithophane.html">Blogger</a> using custom python script. Comment on errors below.</p>zsh job number autocomplete2014-11-16T00:00:00-05:002014-11-16T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-11-16:/zsh-job-number-autocomplete.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>We do not directly support zsh users on our system. However, many of our more
advanced users enjoy some of the modern and advanced features provided by zsh.
One of these users shared a …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>We do not directly support zsh users on our system. However, many of our more
advanced users enjoy some of the modern and advanced features provided by zsh.
One of these users shared a code snippet that he uses in his ~/.zshrc file to
autocomplete job id numbers. He is letting us share the code for use to our
users who prefer zsh over bash.</p>
<p><strong>.zshrc</strong></p>
<div class="highlight"><pre><span></span><code><span class="nv">_jobs_list</span><span class="ss">()</span><span class="w"> </span>{
<span class="nv">qstat</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">grep</span><span class="w"> </span>$<span class="nv">USER</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">cut</span><span class="w"> </span><span class="o">-</span><span class="nv">d</span><span class="s1">' '</span><span class="w"> </span><span class="o">-</span><span class="nv">f1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">cut</span><span class="w"> </span><span class="o">-</span><span class="nv">d</span><span class="s1">'.'</span><span class="w"> </span><span class="o">-</span><span class="nv">f1</span>
}
<span class="nv">_jshow_complete</span><span class="ss">()</span><span class="w"> </span>{
<span class="k">if</span><span class="w"> </span><span class="ss">((</span><span class="w"> </span><span class="nv">CURRENT</span><span class="w"> </span><span class="o">==</span><span class="mi">2</span><span class="ss">))</span><span class="c1">; then</span>
<span class="nv">jjobs</span><span class="o">=</span><span class="ss">(</span><span class="w"> </span>$<span class="ss">(</span><span class="nv">_jobs_list</span><span class="ss">))</span>
<span class="nv">_multi_parts</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="nv">jjobs</span>
<span class="k">else</span>
<span class="nv">_files</span>
<span class="nv">fi</span>
}
<span class="nv">compdef</span><span class="w"> </span><span class="nv">_jshow_complete</span><span class="w"> </span><span class="nv">showstart</span><span class="w"> </span><span class="nv">jdel</span><span class="w"> </span><span class="nv">checkjob</span><span class="w"> </span><span class="nv">qstat</span>
</code></pre></div>
<p>This code should work on any system that used PBS Torque.</p>
<p>I hope you find it useful,</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/11/16/zsh+job+number+autocomplete">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/11/16/zsh+job+number+autocomplete">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-05-07: Workshop on Managing, Sharing and Moving Big Data2014-10-28T00:00:00-04:002014-10-28T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-10-28:/2014-05-07-workshop-on-managing-sharing-and-moving-big-data.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>This is a new workshop being provided as part of IT Services two day offering
of no-charge seminars to faculty and graduate students on technology topics on
May 7. More information and registration can be done at the following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu …</a></p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>This is a new workshop being provided as part of IT Services two day offering
of no-charge seminars to faculty and graduate students on technology topics on
May 7. More information and registration can be done at the following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu/events/tech-seminars/</a></p>
<p>SLIDES:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411599/Managing_Sharing_Data.pdf?version=1&modificationDate=1399466439000&api=v2">Managing_Sharing_Data.pdf</a></li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/05/07/2014-05-07%3A+Workshop+on+Managing%2C+Sharing+and++Moving+Big+Data">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/05/07/2014-05-07%3A+Workshop+on+Managing%2C+Sharing+and++Moving+Big+Data">ICER Wiki</a> using custom python script. Comment on errors below.</p>Restart Stalled Programs2014-10-28T00:00:00-04:002014-10-28T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-10-28:/restart-stalled-programs.html<p>Page <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a> - "Migration of unmigrated content due to installation of a new plugin"</p>
<p><a href="/display/~colbrydi@msu.edu"> <img alt="User icon: colbrydi@msu.edu" src="/images/icons/profilepics/default.svg"> </a> <a href="/display/~colbrydi@msu.edu/2015/01/16/Hack+to+automatically+restart+programs+that+stall+during+inicialization">Hack to automatically restart programs that stall during
inicialization</a></p>
<p>Unknown User (colbrydi@msu.edu) posted on Jan 16, 2015</p>
<p>Sometimes we get jobs that stall out right at the beginning but do not error …</p><p>Page <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a> - "Migration of unmigrated content due to installation of a new plugin"</p>
<p><a href="/display/~colbrydi@msu.edu"> <img alt="User icon: colbrydi@msu.edu" src="/images/icons/profilepics/default.svg"> </a> <a href="/display/~colbrydi@msu.edu/2015/01/16/Hack+to+automatically+restart+programs+that+stall+during+inicialization">Hack to automatically restart programs that stall during
inicialization</a></p>
<p>Unknown User (colbrydi@msu.edu) posted on Jan 16, 2015</p>
<p>Sometimes we get jobs that stall out right at the beginning but do not error
out until the walltime for the job has been exceeded. Users get an email
saying their job "exceeds walltime" but when they check the output nothing (or
very little) seems to have happened. The cause of this problem is highly
dependent on what the job is doing. However, in some cases a simple resubmit
of the job gets it working. The following scripts check to see if the program
is running and automatically re-submits the job if their seems to be a
problem.</p>
<p><strong>file_flag_example.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -l</span>
<span class="c1">#PBS -l mem=100gb,nodes=100:ppn=1,walltime=03:59:59</span>
<span class="c1">#PBS -m a</span>
<span class="c1">#PBS -j oe</span>
<span class="c1"># This method checks to see if a file is created at the beginning of a run. If the file is not created then the job is killed and restarted.</span>
<span class="c1"># This method assumes that the program stalls before it has a chance to generate a file. This method is useful if the code does self checkpointing and the failure state is a busy wait.</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="w"> </span>
<span class="c1">#Run the main program</span>
<span class="o">(</span><span class="w"> </span>
<span class="w"> </span>mpirun<span class="w"> </span>mycommand
</code></pre></div></td></tr></table></div>
<p>) &
PID=$!</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span>Sleep<span class="w"> </span>for<span class="w"> </span>enough<span class="w"> </span>time<span class="w"> </span>to<span class="w"> </span>see<span class="w"> </span>if<span class="w"> </span>the<span class="w"> </span>job<span class="w"> </span>is<span class="w"> </span>running
sleep<span class="w"> </span>300
if<span class="w"> </span>[<span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span>testfile.flag<span class="w"> </span>]
then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>Seems<span class="w"> </span>to<span class="w"> </span>have<span class="w"> </span>stalled.<span class="w"> </span>Killing<span class="w"> </span>and<span class="w"> </span>restarting"
<span class="w"> </span>kill<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>qsub<span class="w"> </span>$0
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>stats<span class="w"> </span>for<span class="w"> </span>debugging"
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>exit<span class="w"> </span>1
fi
wait<span class="w"> </span><span class="nv">$PID</span>
RET=$?
qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
#return<span class="w"> </span>the<span class="w"> </span>output<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
exit<span class="w"> </span><span class="nv">$RET</span>
</code></pre></div>
<p><strong>output_monitor_example.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -l</span>
<span class="c1">#PBS -l mem=100gb,nodes=100:ppn=1,walltime=03:59:59</span>
<span class="c1">#PBS -m a</span>
<span class="c1">#PBS -j oe</span>
<span class="c1"># This method monitors job output and stops if the output doesn't change. </span>
<span class="c1"># This method assumes that the program continuously generates output at regular intervals. </span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="nv">testfile</span><span class="o">=</span><span class="sb">`</span>testfile.flag<span class="sb">`</span>
<span class="w"> </span>
<span class="c1">#Run the main program</span>
<span class="o">(</span><span class="w"> </span>
<span class="w"> </span>mpirun<span class="w"> </span>mycommand<span class="w"> </span>><span class="w"> </span><span class="nv">$testfile</span>
</code></pre></div></td></tr></table></div>
<p>) &
PID=$!</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span>Sleep<span class="w"> </span>for<span class="w"> </span>enough<span class="w"> </span>time<span class="w"> </span>to<span class="w"> </span>start<span class="w"> </span>generating<span class="w"> </span>output
sleep<span class="w"> </span>300
linecount1=`cat<span class="w"> </span><span class="nv">$testfile</span><span class="w"> </span>|<span class="w"> </span>wc<span class="w"> </span>-l`
<span class="w"> </span>
#<span class="w"> </span>Sleep<span class="w"> </span>enough<span class="w"> </span>for<span class="w"> </span>more<span class="w"> </span>output
sleep<span class="w"> </span>100
linecount2=`cat<span class="w"> </span><span class="nv">$testfile</span><span class="w"> </span>|<span class="w"> </span>wc<span class="w"> </span>-l`
if<span class="w"> </span>[<span class="w"> </span>"<span class="nv">$linecount1</span>"<span class="w"> </span>==<span class="w"> </span>"<span class="nv">$linecount2</span>"<span class="w"> </span>]
then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>Seems<span class="w"> </span>to<span class="w"> </span>have<span class="w"> </span>stalled.<span class="w"> </span>Killing<span class="w"> </span>and<span class="w"> </span>restarting"
<span class="w"> </span>kill<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>qsub<span class="w"> </span>$0
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>stats<span class="w"> </span>for<span class="w"> </span>debugging"
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>exit<span class="w"> </span>1
fi
wait<span class="w"> </span><span class="nv">$PID</span>
RET=$?
qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
#return<span class="w"> </span>the<span class="w"> </span>output<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
exit<span class="w"> </span><span class="nv">$RET</span>
</code></pre></div>
<p><strong>qstat_monitor_example.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -l</span>
<span class="c1">#PBS -l mem=100gb,nodes=100:ppn=1,walltime=03:59:59</span>
<span class="c1">#PBS -m a</span>
<span class="c1">#PBS -j oe</span>
<span class="c1"># This method uses the same idea as the previous but instead of relying on output it uses the cput stat generated by qstat.</span>
<span class="c1"># This solution will not work if the job is in a busy wait state.</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="c1">#Run the main program</span>
<span class="o">(</span><span class="w"> </span>
<span class="w"> </span>mpirun<span class="w"> </span>mycommand
</code></pre></div></td></tr></table></div>
<p>) &
PID=$!</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span>Sleep<span class="w"> </span>for<span class="w"> </span>enough<span class="w"> </span>time<span class="w"> </span>to<span class="w"> </span>start<span class="w"> </span>generating<span class="w"> </span>output
sleep<span class="w"> </span>300
cpu1=`qstat<span class="w"> </span>-f<span class="w"> </span><span class="nv">$PBS_JOBID</span><span class="w"> </span>|<span class="w"> </span>grep<span class="w"> </span>resources_used.cput`
<span class="w"> </span>
#<span class="w"> </span>Sleep<span class="w"> </span>enough<span class="w"> </span>for<span class="w"> </span>more<span class="w"> </span>output
sleep<span class="w"> </span>100
cpu2=`qstat<span class="w"> </span>-f<span class="w"> </span><span class="nv">$PBS_JOBID</span><span class="w"> </span>|<span class="w"> </span>grep<span class="w"> </span>resources_used.cput`
if<span class="w"> </span>[<span class="w"> </span>"<span class="nv">$cpu1</span>"<span class="w"> </span>==<span class="w"> </span>"<span class="nv">$cpu2</span>"<span class="w"> </span>]
then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>Seems<span class="w"> </span>to<span class="w"> </span>have<span class="w"> </span>stalled.<span class="w"> </span>Killing<span class="w"> </span>and<span class="w"> </span>restarting"
<span class="w"> </span>kill<span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span>qsub<span class="w"> </span>$0
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>stats<span class="w"> </span>for<span class="w"> </span>debugging"
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
<span class="w"> </span>exit<span class="w"> </span>1
fi
wait<span class="w"> </span><span class="nv">$PID</span>
RET=$?
qstat<span class="w"> </span>-f<span class="w"> </span><span class="cp">${</span><span class="n">PBS_JOBID</span><span class="cp">}</span>
#return<span class="w"> </span>the<span class="w"> </span>output<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>main<span class="w"> </span>program
exit<span class="w"> </span><span class="nv">$RET</span>
</code></pre></div>
<p>These solutions are nice work arounds because, if it works, the scripts just
restarts your job until it runs and gets the research done. However, using
this hack does not get at the root of the problem. Actually there are two
problems:</p>
<ol>
<li>Something is broken causing the job to hang. This could be a race condition in the code, a bad node, bad file I/O, bad network connections, etc. All depends on what the code is doing.</li>
<li>Code hangs insteads of quitting and reporting an error. Well engineered code should not hang. For example, file and network access should have timeouts so that code is not running forever.</li>
</ol>
<p>Researchers, should first notify the HPCC if they are using this hack so we
can try to track down problems with the nodes. Researchers should also work to
modify their code to report an error if something hanges. This will also help
track down the problem.</p>
<ul>
<li>
<p>Dirk</p>
</li>
<li>
<p><a href="/label/~colbrydi@msu.edu/dirk">dirk</a></p>
</li>
</ul>
<p>[ <img alt="User icon: colbrydi@msu.edu" src="/images/icons/profilepics/default.svg"> ](
/display/~colbrydi@msu.edu</p>
<p>) <a href="/display/~colbrydi@msu.edu/2014/12/15/2014-12-16+HPCC+workshop+slides+and+handouts">2014-12-16 HPCC workshop slides and
handouts</a></p>
<p>Unknown User (colbrydi@msu.edu) posted on Dec 15, 2014</p>
<p>I will be teaching my bi-annual Introductory and Advanced HPCC workshops
tomorrow. Below are links to my updated slides and handouts. Registration
looks lite so feel free to drop in if you have the time. These workshops are
being provided as part of IT Services two day offering of no-charge seminars
to faculty and graduate students on technology topics on December 16 and 17.
More information and registration can be done at the following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu/events/tech-seminars/</a></p>
<p><strong>Introduction to the HPCC: A Hands-On Introduction to High Performance
Computing at MSU</strong></p>
<ul>
<li>Tuesday, December 16th 8:30am-11:30am in 403 of the Computer Center</li>
<li>During this workshop, participants will learn through hands-on examples how to get started with the MSU High Performance Computing Center (HPCC). Topics include: connecting to the HPCC; copying files to your home directories; navigating the command line interface; accessing available software; testing and running programs interactively; editing files; writing job scripts; submitting jobs to the queue; and monitoring jobs running on the system.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help you get started and diagnose problems.</li>
</ul>
<p><strong>12:00-1:15pm Lunch is provided in the MSU Ballroom (registered participants
only)</strong></p>
<p><strong>Making Your Research Go Faster: Advance Topics in Getting the Most Out of
the MSU HPCC</strong></p>
<ul>
<li>Tuesday, December 16th 1:30pm-4:30pm in 403 of the Computer Center</li>
<li>During this workshop, participants already familiar with using the HPCC systems will be shown advanced techniques on how to use the system more effectively. Topics include: techniques for effective scheduling of pleasantly parallel jobs; shared memory jobs and shared network jobs; running jobs longer than a week; and making jobs fault tolerant.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help get you started and diagnose problems.</li>
</ul>
<p>SLIDES:</p>
<ul>
<li>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/2014-12-16%20Introduction%20to%20HPCC.pdf?version=1&modificationDate=1418669575000&api=v2">2014-12-16 Introduction to HPCC.pdf</a></p>
</li>
<li>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/2014-12-16_Advanced%20HPCC.pdf?version=1&modificationDate=1418669589000&api=v2">2014-12-16_Advanced HPCC.pdf</a></p>
</li>
</ul>
<p>HANDOUTS:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/2014-12-16%20[HPCC_WORKSHOP_COMMANDS.pdf](./images/HPCC_WORKSHOP_COMMANDS.pdf)?version=1&modificationDate=1418670002000&api=v2">2014-12-16 [HPCC_WORKSHOP_COMMANDS.pdf](./images/HPCC_WORKSHOP_COMMANDS.pdf)</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411653/HPCC_Command_Summary.pdf?version=1&modificationDate=1418669599000&api=v2">HPCC_Command_Summary.pdf</a></li>
<li>
<p><a href="http://files.fosswire.com/2007/08/fwunixref.pdf">http://files.fosswire.com/2007/08/fwunixref.pdf</a></p>
</li>
<li>
<p><a href="/label/~colbrydi@msu.edu/slides">slides</a></p>
</li>
<li><a href="/label/~colbrydi@msu.edu/dirk">dirk</a></li>
</ul>
<p>[ <img alt="User icon: colbrydi@msu.edu" src="/images/icons/profilepics/default.svg"> ](
/display/~colbrydi@msu.edu</p>
<p>) <a href="/display/~colbrydi@msu.edu/2014/12/04/2014-12-05+Western+Michigan+University%2C+Introduction+to+iCER+slides">2014-12-05 Western Michigan University, Introduction to iCER
slides</a></p>
<p>Unknown User (colbrydi@msu.edu) posted on Dec 04, 2014</p>
<p>Here are a copy of my slides and the handout for our two hour introductory
talk at Western Michigan University:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411650/2014-12-05%20WMU%20iCER%20Talk.pdf?version=1&modificationDate=1417709173000&api=v2">2014-12-05 WMU iCER Talk.pdf</a></li>
<li>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411650/2014-12-05%20WMU%20iCER%20Talk%20Commands.pdf?version=1&modificationDate=1417709861000&api=v2">2014-12-05 WMU iCER Talk Commands.pdf</a></p>
</li>
<li>
<p><a href="/label/~colbrydi@msu.edu/slides">slides</a></p>
</li>
<li><a href="/label/~colbrydi@msu.edu/dirk">dirk</a></li>
</ul>
<p>[ <img alt="User icon: colbrydi@msu.edu" src="/images/icons/profilepics/default.svg"> ](
/display/~colbrydi@msu.edu</p>
<p>) <a href="/display/~colbrydi@msu.edu/2014/11/16/zsh+job+number+autocomplete">zsh job number
autocomplete</a></p>
<p>Unknown User (colbrydi@msu.edu) posted on Nov 16, 2014</p>
<p>We do not directly support zsh users on our system. However, many of our more
advanced users enjoy some of the modern and advanced features provided by zsh.
One of these users shared a code snippet that he uses in his ~/.zshrc file to
autocomplete job id numbers. He is letting us share the code for use to our
users who prefer zsh over bash.</p>
<p><strong>.zshrc</strong></p>
<div class="highlight"><pre><span></span><code><span class="nv">_jobs_list</span><span class="ss">()</span><span class="w"> </span>{
<span class="nv">qstat</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">grep</span><span class="w"> </span>$<span class="nv">USER</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">cut</span><span class="w"> </span><span class="o">-</span><span class="nv">d</span><span class="s1">' '</span><span class="w"> </span><span class="o">-</span><span class="nv">f1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">cut</span><span class="w"> </span><span class="o">-</span><span class="nv">d</span><span class="s1">'.'</span><span class="w"> </span><span class="o">-</span><span class="nv">f1</span>
}
<span class="nv">_jshow_complete</span><span class="ss">()</span><span class="w"> </span>{
<span class="k">if</span><span class="w"> </span><span class="ss">((</span><span class="w"> </span><span class="nv">CURRENT</span><span class="w"> </span><span class="o">==</span><span class="mi">2</span><span class="ss">))</span><span class="c1">; then</span>
<span class="nv">jjobs</span><span class="o">=</span><span class="ss">(</span><span class="w"> </span>$<span class="ss">(</span><span class="nv">_jobs_list</span><span class="ss">))</span>
<span class="nv">_multi_parts</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="nv">jjobs</span>
<span class="k">else</span>
<span class="nv">_files</span>
<span class="nv">fi</span>
}
<span class="nv">compdef</span><span class="w"> </span><span class="nv">_jshow_complete</span><span class="w"> </span><span class="nv">showstart</span><span class="w"> </span><span class="nv">jdel</span><span class="w"> </span><span class="nv">checkjob</span><span class="w"> </span><span class="nv">qstat</span>
</code></pre></div>
<p>This code should work on any system that used PBS Torque.</p>
<p>I hope you find it useful,</p>
<ul>
<li>
<p>Dirk</p>
</li>
<li>
<p><a href="/label/~colbrydi@msu.edu/example">example</a></p>
</li>
<li><a href="/label/~colbrydi@msu.edu/dirk">dirk</a></li>
</ul>
<p>[ <img alt="User icon: colbrydi@msu.edu" src="/images/icons/profilepics/default.svg"> ](
/display/~colbrydi@msu.edu</p>
<p>) <a href="/display/~colbrydi@msu.edu/2014/10/23/2014-10-23+Advanced+High+Performance+Computing">2014-10-23 Advanced High Performance
Computing</a></p>
<p>Unknown User (colbrydi@msu.edu) posted on Oct 23, 2014</p>
<p>Here are the slides for the Advanced HPC class:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411677/2014-10-24_CI-
Days%20Advanced%20HPCC.pdf?version=1&modificationDate=1414073145000&api=v2">2014-10-24_CI-Days Advanced
HPCC.pdf</a></p>
<p>And here is the handout:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411677/2014-10-24_CI-
Days%20Advanced%20HPCC%20Handout.pdf?version=1&modificationDate=1414077081000&api=v2">2014-10-24_CI-Days Advanced HPCC
Handout.pdf</a></p>
<ul>
<li>
<p>Dirk</p>
</li>
<li>
<p><a href="/label/~colbrydi@msu.edu/slides">slides</a></p>
</li>
<li><a href="/label/~colbrydi@msu.edu/dirk">dirk</a></li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/Blog">View Online</a> *
<a href="https://wiki.hpcc.msu.edu/pages/diffpagesbyversion.action?pageId=5411630&revisedVersion=14&originalVersion=13">View Changes
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/Blog">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-10-23 Advanced High Performance Computing2014-10-23T00:00:00-04:002014-10-23T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-10-23:/2014-10-23-advanced-high-performance-computing.html<p>Blog post <strong>edited</strong> by Anonymous</p>
<p>Here are the slides for the Advanced HPC class:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411677/2014-10-24_CI-
Days%20Advanced%20HPCC.pdf?version=1&modificationDate=1414073145000&api=v2">2014-10-24_CI-Days Advanced HPCC.pdf</a></p>
<p>And here is the handout:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411677/2014-10-24_CI-Days%20Advanced%20HPCC%20Handout.pdf?version=1&modificationDate=1414077081000&api=v2">2014-10-24_CI-Days Advanced HPCC Handout.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/10/23/2014-10-23+Advanced+High+Performance+Computing">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/10/23/2014-10-23+Advanced+High+Performance+Computing">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-08-20: EDAMAME Workshop at Kellogg Biological Center2014-10-20T00:00:00-04:002014-10-20T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-10-20:/2014-08-20-edamame-workshop-at-kellogg-biological-center.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>Attached are copies the slides I am planning to present at the EDAMAME
workshop. Information about the class can be found here:</p>
<p><a href="http://edamame-course.org/">http://edamame-course.org/</a></p>
<p>This presentation is a little different than my previous ones since it more
for researchers outside of MSU and …</p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>Attached are copies the slides I am planning to present at the EDAMAME
workshop. Information about the class can be found here:</p>
<p><a href="http://edamame-course.org/">http://edamame-course.org/</a></p>
<p>This presentation is a little different than my previous ones since it more
for researchers outside of MSU and we are only really using the HPCC as an
example of the types of resources that they may be able to find at their own
institutions.</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411568/2014-08-20%20ClusterComputing_EDAMAME.pdf?version=1&modificationDate=1408472523000&api=v2">2014-08-20 ClusterComputing_EDAMAME.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/08/19/2014-08-20%3A+EDAMAME+Workshop+at+Kellogg+Biological+Center">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/08/19/2014-08-20%3A+EDAMAME+Workshop+at+Kellogg+Biological+Center">ICER Wiki</a> using custom python script. Comment on errors below.</p>CSE 891 Section 1: Parallel Computing: Fundamentals and Applications2014-09-10T00:00:00-04:002014-09-10T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-09-10:/cse-891-section-1-parallel-computing-fundamentals-and-applications.html<p>Blog post <strong>added</strong> by Anonymous</p>
<p>I was asked to give a talk about the HPCC and my research. Here are the slides
if anyone is interested:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411579/2014-09-10%20Parallel%20Programming%20Class.pdf?version=1&modificationDate=1410370985000&api=v2">2014-09-10 Parallel Programming Class.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/09/10/CSE+891+Section+1%3A+Parallel+Computing%3A+Fundamentals+and+Applications">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/09/10/CSE+891+Section+1%3A+Parallel+Computing%3A+Fundamentals+and+Applications">ICER Wiki</a> using custom python script. Comment on errors below.</p>Fish Tank Kids Coat Rack2014-09-01T00:00:00-04:002014-09-01T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-09-01:/fish-tank-kids-coat-rack.html<p><img align="right," alt="./images/blogger-image-627755271.jpg" hspace="10" src="../images/blogger-image-627755271.jpg" width="200,"></p>
<p>This one was a lot of fun. We have a coat area down by the back door landing.
I used those plastic hooks with the special double sided tape that allows you
to remove the hooks without damaging the wall. Unfortunate, the kids managed
to pull the hooks off taking …</p><p><img align="right," alt="./images/blogger-image-627755271.jpg" hspace="10" src="../images/blogger-image-627755271.jpg" width="200,"></p>
<p>This one was a lot of fun. We have a coat area down by the back door landing.
I used those plastic hooks with the special double sided tape that allows you
to remove the hooks without damaging the wall. Unfortunate, the kids managed
to pull the hooks off taking some paint and drywall with them. </p>
<p>Anyway my plan was to print put some new hooks using my 3D printer. I thought
this would be the "cheap" way to go. I let my imagination get away with me
and ended up with something a little more expensive but also something unique.</p>
<p><img align="right," alt="./images/blogger-image--608522330.jpg" hspace="10" src="../images/blogger-image--608522330.jpg" width="200,"></p>
<p>The design is quite simple and consists of two hooks I found on thingiverse; a
fish:</p>
<p><a href="http://www.thingiverse.com/thing:242736">http://www.thingiverse.com/thing:242736</a></p>
<p><img align="right," alt="./images/blogger-image--674318040.jpg" hspace="10" src="../images/blogger-image--674318040.jpg" width="200,"></p>
<p>And a round hook or bubble: </p>
<p><a href="http://www.thingiverse.com/thing:52764">http://www.thingiverse.com/thing:52764</a> </p>
<p>The overall affect is a fish tank with a bunch of fish skeletons swimming
around.</p>
<p><img align="right," alt="./images/blogger-image-1257835475.jpg" hspace="10" src="../images/blogger-image-1257835475.jpg" width="200,"></p>
<p>The design is a little odd bit visually appealing and has a ton of places for
the kids to hand stuff.</p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/09/fish-rank-kids-coat-rack.html">Blogger</a> using custom python script. Comment on errors below.</p>2014-2015 New Faculty Orientation2014-08-25T00:00:00-04:002014-08-25T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-08-25:/2014-2015-new-faculty-orientation.html<p>Blog post <strong>added</strong> by Anonymous</p>
<p>Here are the slides for introducing new faculty to iCER. Thanks to Ben Ong for
editing and updating these slides for me.</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411587/2014-08-25-faculty_orientation.pdf?version=1&modificationDate=1408988146000&api=v2">2014-08-25-faculty_orientation.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/08/25/2014-2015+New+Faculty+Orientation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/08/25/2014-2015+New+Faculty+Orientation">ICER Wiki</a> using custom python script. Comment on errors below.</p>Vacation fort2014-08-04T00:00:00-04:002014-08-04T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-08-04:/vacation-fort.html<p><img align="right," alt="./images/blogger-image-559300753.jpg" hspace="10" src="../images/blogger-image-559300753.jpg" width="200,"></p>
<p>We are spending time with the kids at our aunt and uncle's cabin in northern
Michigan. This is an old hunting cabin but it is not exactly roughing it.
There is air conditioning and internet. </p>
<p>I am worried that the kids are spending too much time inside on the devices …</p><p><img align="right," alt="./images/blogger-image-559300753.jpg" hspace="10" src="../images/blogger-image-559300753.jpg" width="200,"></p>
<p>We are spending time with the kids at our aunt and uncle's cabin in northern
Michigan. This is an old hunting cabin but it is not exactly roughing it.
There is air conditioning and internet. </p>
<p>I am worried that the kids are spending too much time inside on the devices.
At some point yesterday I banned the devices and told my eldest to go out and
build a fort.</p>
<hr>
<p>She found some old ladders ready for the burn pile and built from there.</p>
<p><img align="right," alt="./images/blogger-image-1964742467.jpg" hspace="10" src="../images/blogger-image-1964742467.jpg" width="200,"></p>
<p>The fort turned out really well. It is surprisingly strong and all of the
kids are proud of what they built.</p>
<p><img alt="./images/blogger-image--1836650834.jpg" src="../images/blogger-image--1836650834.jpg"> </p>
<p><img alt="./images/blogger-image-988089399.jpg" src="../images/blogger-image-988089399.jpg"></p>
<p>It is also a fun place to pick-nick lunch out of the hot afternoon sun. </p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/08/vacation-fort.html">Blogger</a> using custom python script. Comment on errors below.</p>BackBlog - Teleporter video2014-07-19T00:00:00-04:002014-07-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-07-19:/backblog-teleporter-video.html<p>We through a Star Trek themed surprise party for my wife and I thought it
would be fun to see if I could make a transporter effect. </p>
<p><img align="right," alt="./images/small_DSC_0858.JPG" hspace="10" src="../images/small_DSC_0858.JPG" width="200,"></p>
<p>To start we took two pictures, one with everyone in an away team formation and
one with just the background </p>
<p><img align="right," alt="./images/small_DSC_0859.JPG" hspace="10" src="../images/small_DSC_0859.JPG" width="200,"></p>
<p>I then wrote …</p><p>We through a Star Trek themed surprise party for my wife and I thought it
would be fun to see if I could make a transporter effect. </p>
<p><img align="right," alt="./images/small_DSC_0858.JPG" hspace="10" src="../images/small_DSC_0858.JPG" width="200,"></p>
<p>To start we took two pictures, one with everyone in an away team formation and
one with just the background </p>
<p><img align="right," alt="./images/small_DSC_0859.JPG" hspace="10" src="../images/small_DSC_0859.JPG" width="200,"></p>
<p>I then wrote a simple program to add the two images together using a weighted
sum. This technique looked okay but I also wanted some noise. I used the
matlab imnoise function to create some sparkles. Inicially just a few, grow to
a lot about halfway and then disparate as the people materialize. </p>
<p>The trick is to get the speckles to only appear where there are people. I
could have tried some clever image subtraction trick to find the region that
changed but it was just as quick for me to use GIMP to cut the people out and
make a simple flag array using the following Matlab code. I had to also use
GIMP to paint over any white spots in the image. Fortunately for my image
there was very little white. I ended up with the following (I guess I could
also have just select the white and set everything else to black). </p>
<p><img align="right," alt="./images/small_flag.JPG" hspace="10" src="../images/small_flag.JPG" width="200,"></p>
<p>Now, potting All the components together and looping from 0 to 100 I was able
to create a directory of images that I wanted and stitched the images together
using ffmpeg. </p>
<p>To top it off, I found a transporter like sound on the net and used my copy
of Camtasia (<a href="http://www.techsmith.com/camtasia.html">http://www.techsmith.com/camtasia.html</a>) to put the audio and
video together. </p>
<p>Here is my final product: </p>
<iframe allowfullscreen="allowfullscreen" class="b-hbp-video b-uploaded" frameborder="0" height="266" id="BLOGGER-video-7bcf554431118cd5-5264" mozallowfullscreen="mozallowfullscreen" src="https://www.blogger.com/video.g?token=AD6v5dz_ZfGT92ivLvCO99r6vtIQwsrCdTFS4HnpnE9gn-b0I56TmP1sT3yoEIwLWpJkgZjeMUzDCGuOum9y10XB2-JDX2_13biU5k9nkIy8h4sMe0_u3vw76NatscksCqtwmJsiy98" webkitallowfullscreen="webkitallowfullscreen" width="320"></iframe>
<ul>
<li>Dirk</li>
</ul>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/07/backblog-teleporter-video.html">Blogger</a> using custom python script. Comment on errors below.</p>Bug bots2014-07-19T00:00:00-04:002014-07-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-07-19:/bug-bots.html<p><img align="right," alt="./images/blogger-image-1543028749.jpg" hspace="10" src="../images/blogger-image-1543028749.jpg" width="200,"></p>
<p>I like the simple fun of brushbots; little robots, typically made with
toothbrush heads, that use a cell phone vibrating motor to scoot around a
table (you can buy kits here: <a href="http://www.makershed.com/products/brushbots">http://www.makershed.com/products/brushbots</a>).
I decided that this would be a great project for me to do …</p><p><img align="right," alt="./images/blogger-image-1543028749.jpg" hspace="10" src="../images/blogger-image-1543028749.jpg" width="200,"></p>
<p>I like the simple fun of brushbots; little robots, typically made with
toothbrush heads, that use a cell phone vibrating motor to scoot around a
table (you can buy kits here: <a href="http://www.makershed.com/products/brushbots">http://www.makershed.com/products/brushbots</a>).
I decided that this would be a great project for me to do with my kids. Step
one was to find the parts. </p>
<ol>
<li>Cell phone motors (pack of 50).</li>
<li>Batteries. Large pack ordered on amazon: <a href="http://amzn.com/B0043SAFK0">http://amzn.com/B0043SAFK0</a></li>
<li>Glue</li>
<li>Double sided tape</li>
<li>Toothbrushes</li>
</ol>
<p>Between the motors and batteries, the price per brushbot was getting
expensive. So Instead of buying a bunch of toothbrushes I decided to see if I
could print the body instead. I came across the following design on
thingiverse that did exactly what I wanted:</p>
<p><a href="http://www.thingiverse.com/thing:68968">http://www.thingiverse.com/thing:68968</a></p>
<p>I ended up using only the "legs" and the body. I also think it was also
better to assemble the body upside down. In any case, I was able to print out
these three parts and just let the kids assemble them using superglue:</p>
<p><img align="right," alt="./images/blogger-image--1230627461.jpg" hspace="10" src="../images/blogger-image--1230627461.jpg" width="200,">
<img align="right," alt="./images/blogger-image-1497897705.jpg" hspace="10" src="../images/blogger-image-1497897705.jpg" width="200,">
<img align="right," alt="./images/blogger-image-1759121257.jpg" hspace="10" src="../images/blogger-image-1759121257.jpg" width="200,"></p>
<p>remove the sticker cover on the motor to stick it to the "bug". </p>
<p><img align="right," alt="./images/blogger-image--1230627461.jpg" hspace="10" src="../images/blogger-image--1230627461.jpg" width="200,">
<img align="right," alt="./images/blogger-image-1701867005.jpg" hspace="10" src="../images/blogger-image-1701867005.jpg" width="200,"></p>
<p>Add small pieces of double sided tape and lay down one of the wires on the
tape. Then have the kids stick the battery on top of the tape and use a second
piece of tape to secure the top wire.</p>
<p><img align="right," alt="./images/blogger-image--1230627461.jpg" hspace="10" src="../images/blogger-image--1230627461.jpg" width="200,">
<img align="right," alt="./images/blogger-image-714541735.jpg" hspace="10" src="../images/blogger-image-714541735.jpg" width="200,"></p>
<p>Overall this activity was easily complete in 15 minutes and the kids had
something to take home with them. Special thanks to
<img align="right," alt="./images/blogger-image--1230627461.jpg" hspace="10" src="../images/blogger-image--1230627461.jpg" width="200,">
(http://www.thingiverse.com/ekaggrat/overview) at thingiverse for
posting his 3D designs which made this project possible. </p>
<p>- Dirk </p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/07/bug-bots.html">Blogger</a> using custom python script. Comment on errors below.</p>Coffee table drawer2014-07-19T00:00:00-04:002014-07-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-07-19:/coffee-table-drawer.html<p><img align="right," alt="https://lh3.googleusercontent.com/-l_ZWXvKvKq0/U8rPgSLHSwI/AAAAAAAAC3E/HCGGQf68jo4/s640/blogger-image-676460031.jpg" hspace="10" src="https://lh3.googleusercontent.com/-l_ZWXvKvKq0/U8rPgSLHSwI/AAAAAAAAC3E/HCGGQf68jo4/s640/blogger-image-676460031.jpg" width="200,"></p>
<p><img align="right," alt="https://lh5.googleusercontent.com/-wvPyGQ8cWk0/U8rPhQtRyQI/AAAAAAAAC3M/J9N6D2fVY9s/s640/blogger-image-1119305401.jpg" hspace="10" src="https://lh5.googleusercontent.com/-wvPyGQ8cWk0/U8rPhQtRyQI/AAAAAAAAC3M/J9N6D2fVY9s/s640/blogger-image-1119305401.jpg" width="200,"></p>
<p>The kids have really done a number on our living room coffee table. The top
has been "distressed" by biting, thrown toys and lots of drumming. </p>
<p><img align="right," alt="./images/blogger-image-1898114810.jpg" hspace="10" src="../images/blogger-image-1898114810.jpg" width="200,"></p>
<p>The drawers have also been used as steps to get up on top of the table. Most
of the hardware components have broken under …</p><p><img align="right," alt="https://lh3.googleusercontent.com/-l_ZWXvKvKq0/U8rPgSLHSwI/AAAAAAAAC3E/HCGGQf68jo4/s640/blogger-image-676460031.jpg" hspace="10" src="https://lh3.googleusercontent.com/-l_ZWXvKvKq0/U8rPgSLHSwI/AAAAAAAAC3E/HCGGQf68jo4/s640/blogger-image-676460031.jpg" width="200,"></p>
<p><img align="right," alt="https://lh5.googleusercontent.com/-wvPyGQ8cWk0/U8rPhQtRyQI/AAAAAAAAC3M/J9N6D2fVY9s/s640/blogger-image-1119305401.jpg" hspace="10" src="https://lh5.googleusercontent.com/-wvPyGQ8cWk0/U8rPhQtRyQI/AAAAAAAAC3M/J9N6D2fVY9s/s640/blogger-image-1119305401.jpg" width="200,"></p>
<p>The kids have really done a number on our living room coffee table. The top
has been "distressed" by biting, thrown toys and lots of drumming. </p>
<p><img align="right," alt="./images/blogger-image-1898114810.jpg" hspace="10" src="../images/blogger-image-1898114810.jpg" width="200,"></p>
<p>The drawers have also been used as steps to get up on top of the table. Most
of the hardware components have broken under this level of used. </p>
<p><img align="right," alt="./images/blogger-image-676460031.jpg" hspace="10" src="../images/blogger-image-676460031.jpg" width="200,"></p>
<p>I decided that this would be a fun project for my 3D printer. There are a lot
of drawer guides in thingiverse. I downloaded a few drawer guild stl files but
none of them really fit my dawers:</p>
<p><a href="http://www.thingiverse.com/thing:250839">http://www.thingiverse.com/thing:250839</a></p>
<p><a href="http://www.thingiverse.com/thing:39023">http://www.thingiverse.com/thing:39023</a></p>
<p>I ended up having to design my own using a pair of calipers and OpenScad. I
think they turned out great:</p>
<p><img align="right," alt="./images/Drawer_guild.png)" hspace="10" src="../images/Drawer_guild.png" width="200,"></p>
<p><img align="right," alt="./images/blogger-image-543254614.jpg" hspace="10" src="../images/blogger-image-543254614.jpg" width="200,"></p>
<p>I also replaced 3 of the four drawer stops using another design developed in
OpenScad:</p>
<p><img align="right," alt="./images/DrawerStop.png)" hspace="10" src="../images/DrawerStop.png" width="200,">
<img align="right," alt="./images/blogger-image-1595648182.jpg" hspace="10" src="../images/blogger-image-1595648182.jpg" width="200,"></p>
<p>After the flat part was printed I put it in a warm pan on the stove to soften
the plastic a little so I could put an appropriate bend in it. Here is a link
to my files if you are interested in working with them:</p>
<p><a href="https://github.com/colbrydi/3D-Drawer">https://github.com/colbrydi/3D-Drawer</a></p>
<p>I also posted them on YouMagine.org</p>
<p><a href="https://www.youmagine.com/designs/coffee-table-drawer-guild-and-stopper">https://www.youmagine.com/designs/coffee-table-drawer-guild-and-stopper</a></p>
<p>Overall, I am happy with the results. And I am comforted in knowing that I
can replace the parts again if the kids get destructive again.</p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/07/coffee-table-drawer.html">Blogger</a> using custom python script. Comment on errors below.</p>Gears with logos2014-06-22T00:00:00-04:002014-06-22T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-06-22:/gears-with-logos.html<p><img align="right," alt="./images/blogger-image--1703746521.jpg" hspace="10" src="../images/blogger-image--1703746521.jpg" width="200,"></p>
<p>The theme for the kids VBS (Vacation Bible School) this year is "Workshop of
Wonders" by <a href="http://2014.cokesburyvbs.com/">Cokesbury</a>. I participate every
year with my kids but was extremely enthusiastic that the theme this year was
for makers so my goal is to show off the 3D printer and make some things …</p><p><img align="right," alt="./images/blogger-image--1703746521.jpg" hspace="10" src="../images/blogger-image--1703746521.jpg" width="200,"></p>
<p>The theme for the kids VBS (Vacation Bible School) this year is "Workshop of
Wonders" by <a href="http://2014.cokesburyvbs.com/">Cokesbury</a>. I participate every
year with my kids but was extremely enthusiastic that the theme this year was
for makers so my goal is to show off the 3D printer and make some things for
the 50 or so kid participants. It's no fun watching a 3D printer without
having something to touch and even take home. However, I am fairly new to 3D
print design so this was only my second endeavor to design something of my
own. </p>
<p>I was trying to think of a project where each student would get a gismo to
take home but we could put them all together to build something as a group.
This fit in with the theme quite nicely however most of my ideas were a little
too complex to get off the ground. </p>
<p>I finally settled on building gears with logos and holes on one end so they
could attach them to their backpacks. I thought I could assemble all of the
gears together to make one big machine the kids could play with that all
worked together. With this plan in mind I searched though thingiverse and
youmagin for some designs I could use as a starting point. I was really
excited when I found this one by <a href="http://www.thingiverse.com/thing:40410">http://www.thingiverse.com/thing:40410</a> that
can be used to make gears that fit into Lego technics. I have a ton of lego
technics and I really liked the idea that the motivated students to integrate
their gear with their own legos. </p>
<p>After some iteration, I ultimately ended up doing is importing the images into
Inkscape and converting them to a dfx file which is a vector map using
instructions I found on a couple of website. Here is an example image I loaded
into Inkscape </p>
<p><img align="right," alt="./images/icer.png" hspace="10" src="../images/icer.png" width="200,">
<img align="right," alt="./images/example-dfx-gear.png" hspace="10" src="../images/example-dfx-gear.png" width="200,"></p>
<p>Using this technique I was able to make a variety of gears and they worked
well with lego. Here is a video of one of the gears working with my old first
generation mindstorm: </p>
<p>This turned out to be a really flexible design and I was also able to someone
elses openscad files to make other gears with different projects I found on
thingiverse. For example here is one with a robot in a openscad file from
thingiverse <a href="http://www.thingiverse.com/thing:53285">http://www.thingiverse.com/thing:53285</a>: </p>
<p><img alt="./images/example-scad-include-gear.png" src="../images/example-scad-include-gear.png"></p>
<p>Here is another one from an stl file I found on youmagin that originally was a
keychain modification for the ultimaker
(<a href="https://www.youmagine.com/designs/ultimaker-key-fob-flat">https://www.youmagine.com/designs/ultimaker-key-fob-flat</a>): </p>
<p><img alt="./images/example-stl-import-gear.png" src="../images/example-stl-import-gear.png"></p>
<p>I really like this design because it is easy for me to swap out different
logos based on the event. I uploaded my source files and these examples to
github: </p>
<p><a href="https://github.com/colbrydi/gear-givaways">https://github.com/colbrydi/gear-givaways</a> </p>
<p>And posted a few on Youmagine.org </p>
<p><a href="https://www.youmagine.com/designs/gear-givaways">https://www.youmagine.com/designs/gear-givaways</a> </p>
<p>Please let me know if you find this helpful and email me links to gears that
you have made.</p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/06/gears-with-logos.html">Blogger</a> using custom python script. Comment on errors below.</p>Milk Jug Bird Feeder2014-06-22T00:00:00-04:002014-06-22T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-06-22:/milk-jug-bird-feeder.html<p><img align="right," alt="./images/Evernote+Snapshot+20140309+162615.jpg" hspace="10" src="../images/Evernote+Snapshot+20140309+162615.jpg" width="200,"></p>
<p>My eldest daughter wanted to make a birdhouse/feeder. It was the middle of
winter and I think she was thinking of building it out of wood. However, I
realized what she really wanted to do was decorate it using some extra
stickers she had. So, we compromised and I …</p><p><img align="right," alt="./images/Evernote+Snapshot+20140309+162615.jpg" hspace="10" src="../images/Evernote+Snapshot+20140309+162615.jpg" width="200,"></p>
<p>My eldest daughter wanted to make a birdhouse/feeder. It was the middle of
winter and I think she was thinking of building it out of wood. However, I
realized what she really wanted to do was decorate it using some extra
stickers she had. So, we compromised and I cut out a hole in a washed out
milkjug. </p>
<p>She spent the afternoon decorating and had a lot of fun.</p>
<p><img align="right," alt="./images/Evernote+Snapshot+20140309+160605.jpg" hspace="10" src="../images/Evernote+Snapshot+20140309+160605.jpg" width="200,"></p>
<p><img align="right," alt="./images/Evernote+Snapshot+20140309+160639.jpg" hspace="10" src="../images/Evernote+Snapshot+20140309+160639.jpg" width="200,"></p>
<p>Add a little birdseed and the project was complete. </p>
<p><img align="right," alt="./images/Evernote+Snapshot+20140309+162652.jpg" hspace="10" src="../images/Evernote+Snapshot+20140309+162652.jpg" width="200,"></p>
<p>Easy way to entertain a fidgety girl on a beautiful but cold afternoon. The
seeds ended up on the ground although I don't ever think I saw a bird. </p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/06/milk-jug-bird-feeder.html">Blogger</a> using customs python script. Comment on errors below.</p>Diet Coke and Mentos2014-06-16T00:00:00-04:002014-06-16T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-06-16:/diet-coke-and-mentos.html<p><img align="right," alt="./images/blogger-image-1819023175.jpg" hspace="10" src="../images/blogger-image-1819023175.jpg" width="200,"></p>
<p>As any true geek dad, I have always wanted to do the Diet Coke and mentos
experiment. However it is messy and the logistics of getting the mentos in
the coke without getting crazy sticky is challenging. I did a quick search of
thingiverse and found a couple of wonderful …</p><p><img align="right," alt="./images/blogger-image-1819023175.jpg" hspace="10" src="../images/blogger-image-1819023175.jpg" width="200,"></p>
<p>As any true geek dad, I have always wanted to do the Diet Coke and mentos
experiment. However it is messy and the logistics of getting the mentos in
the coke without getting crazy sticky is challenging. I did a quick search of
thingiverse and found a couple of wonderful solutions: </p>
<p><a href="http://www.thingiverse.com/thing:100563">http://www.thingiverse.com/thing:100563</a><br>
<a href="https://www.thingiverse.com/thing:36187">https://www.thingiverse.com/thing:36187</a> </p>
<p>A short 3D print of the second one and all we needed was Diet Coke, Mentos,
string, paperclip and space: </p>
<p><img align="right," alt="./images/blogger-image-1583448446.jpg" hspace="10" src="../images/blogger-image-1583448446.jpg" width="200,"></p>
<p>So much fun :)</p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/06/diet-coke-and-mentos.html">Blogger</a> using custom python script. Comment on errors below.</p>Tripod Quick release mount2014-06-13T00:00:00-04:002014-06-13T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-06-13:/tripod-quick-release-mount.html<p>I got a tripod for my birthday this year. It is a really nice one, but I lost
the quick release mount. Here is a picture with a quick release mount from
another tripod. Obviously, it doesn't fit: </p>
<p><img align="right," alt="./images/Tripod.jpg" hspace="10" src="../images/Tripod.jpg" width="200,"></p>
<p>After some shopping online this one looks fairly good, but possibly not …</p><p>I got a tripod for my birthday this year. It is a really nice one, but I lost
the quick release mount. Here is a picture with a quick release mount from
another tripod. Obviously, it doesn't fit: </p>
<p><img align="right," alt="./images/Tripod.jpg" hspace="10" src="../images/Tripod.jpg" width="200,"></p>
<p>After some shopping online this one looks fairly good, but possibly not the
right size. </p>
<p><a href="http://www.tripodquickrelease.com/Acme_Lite_Quick_Releases.htm">http://www.tripodquickrelease.com/Acme_Lite_Quick_Releases.htm</a> </p>
<p>So, I took some measurements and came up with this drawing </p>
<p><img align="right," alt="./images/quickreleasemount.jpg" hspace="10" src="../images/quickreleasemount.jpg" width="200,"></p>
<p>My goal is to see if I can print this out using a 3D printer. When I started
this project I was planning to use the one in the engineering department where
I work, but now I have an Ultimaker 2. I know I need an stl file but I was
not sure the best way to generate one. After some quick internet searching I
found openscad. </p>
<p><a href="http://www.openscad.org/">http://www.openscad.org/</a> </p>
<p>I downloaded the software and found the wiki quick start page. It did not
take long to figure out the simple language. I started by making a cube and
then creating the top. Then I made objects to cut out the inside and the
bevels. Here is the resulting model. </p>
<p><img align="right," alt="./images/QuickRelease.png" hspace="10" src="../images/QuickRelease.png" width="200,">
<img align="right," alt="./images/QuickRelease2.png" hspace="10" src="../images/QuickRelease2.png" width="200,"></p>
<p>The hardest part was to get the surface normals for the wedge shape object to
face in the right direction. However, once I drew out my points and used the
right hand rule things settled in nicely. Openscad outputs stl files so all I
did was import it into the cura program to generate the gcode for my printer
and save it to the SD card to print. The first print was a little too close
to my tolerances and I could not get it to fit inside the tripod. However,
with some simple modifications to the openscad file I was able to generate a
quick release mount that I think looks quite nice and works well: </p>
<p><img align="right," alt="./images/blogger-image-138421032.jpg" hspace="10" src="../images/blogger-image-138421032.jpg" width="200,"></p>
<p>A future design would fix the bottom to the mount somehow. I posted the STL
and OpenScad files on YouMagin.org if you are interested in makeing your own: </p>
<p><a href="https://www.youmagine.com/designs/tripod-quick-release-mount">https://www.youmagine.com/designs/tripod-quick-release-mount</a> </p>
<p>- Dirk </p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/06/tripod-quick.html">Blogger</a> using custom python script. Comment on errors below.</p>Party Piñata2014-06-12T00:00:00-04:002014-06-12T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-06-12:/party-pinata.html<p><img align="right," alt="./images/blogger-image--2094275764.jpg" hspace="10" src="../images/blogger-image--2094275764.jpg" width="200,"></p>
<p>I like to watch Curious George with my Kids. That little monkey is a bit of a
maker. I really like the episodes where he tries to build things. In most
cases he doesn't get his design right and has to debug and iterate on the
design until he has …</p><p><img align="right," alt="./images/blogger-image--2094275764.jpg" hspace="10" src="../images/blogger-image--2094275764.jpg" width="200,"></p>
<p>I like to watch Curious George with my Kids. That little monkey is a bit of a
maker. I really like the episodes where he tries to build things. In most
cases he doesn't get his design right and has to debug and iterate on the
design until he has what he wants. It is a very good lesson for our young
makers. </p>
<p>In one episode George decides to make a piñata. My youngest daughter LOVEs
this episode and has been asking my wife and I to make a piñata for weeks. A
few weeks ago my wife relented and found a really nice youtube tutorial on-
line: </p>
<p>We got the supplies suggested by the video and my wife and kids started paper
machine a star shaped balloon and party hats. The family ended up making two
since the overall costs were about the same. </p>
<p>We filled them with candy and put one of them to the test this weekend at a
family party: </p>
<p>The only problem we had was keeping the piñata attached to the string.
Ultimately, I had to tie the string around the whole thing to get it to stay.
I probably need to follow George's example by iterating and engineering a
better process for the second one. </p>
<p>- Dirk </p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/06/party-pinata.html">Blogger</a> using costom python script. Comment on errors below.</p>Ann Arbor Mini Maker Faire2014-05-24T00:00:00-04:002014-05-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-05-24:/ann-arbor-mini-maker-faire.html<p>A few weeks ago we caravanned a group of family and friends down to the Ann
Arbor Mini Maker Faire. </p>
<p><img align="right," alt="./images/aammf2014v1_rfs.jpg" hspace="10" src="../images/aammf2014v1_rfs.jpg" width="200,"></p>
<p>It was everything I had
hoped: there were plenty of robots, 3d manufacturing devices and other things
to touch and play with. My oldest daughter (with help from her papa …</p><p>A few weeks ago we caravanned a group of family and friends down to the Ann
Arbor Mini Maker Faire. </p>
<p><img align="right," alt="./images/aammf2014v1_rfs.jpg" hspace="10" src="../images/aammf2014v1_rfs.jpg" width="200,"></p>
<p>It was everything I had
hoped: there were plenty of robots, 3d manufacturing devices and other things
to touch and play with. My oldest daughter (with help from her papa) even got
to do her first soldering using one of the "learn to solder badges" from the
maker shed
(<a href="http://www.makershed.com/Learn_to_Solder_Skill_Badge_Kit_p/mkls01.htm">http://www.makershed.com/Learn_to_Solder_Skill_Badge_Kit_p/mkls01.htm</a>). I
heard a lot about this kit and it was better than I thought it would be: </p>
<p><img alt="./images/small_DSC_6122.JPG" src="../images/small_DSC_6122.JPG"></p>
<p>I really like the size and the enthusiasm at the event and am looking forward
to the Detroit Maker Faire. </p>
<p><img alt="./images/small_DSC_6128.JPG)" src="../images/small_DSC_6128.JPG"></p>
<p><img alt="./images/small_DSC_6131.JPG)" src="../images/small_DSC_6131.JPG"></p>
<p>Can't wait until the next one. </p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/05/ann-arbor-mini-maker-faire.html">Blogger</a> using costom python script. Comment on errors below.</p>Garden Fortress2014-05-24T00:00:00-04:002014-05-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-05-24:/garden-fortress.html<p><img align="right," alt="./images/blogger-image--276711787.jpg" hspace="10" src="../images/blogger-image--276711787.jpg" width="200,"></p>
<p>I love the idea of gardening. Specifically vegetable gardens where we can
grow, harvest and eat some of our food. I've had a few gardens before, but in our current house it is difficult to have one because our beautiful four legged neighbors (deer) like to snack in our yard …</p><p><img align="right," alt="./images/blogger-image--276711787.jpg" hspace="10" src="../images/blogger-image--276711787.jpg" width="200,"></p>
<p>I love the idea of gardening. Specifically vegetable gardens where we can
grow, harvest and eat some of our food. I've had a few gardens before, but in our current house it is difficult to have one because our beautiful four legged neighbors (deer) like to snack in our yard. </p>
<p>I got this idea from a community garden just down the road. They used an old
glass door as a small green house. I am using the same idea but as a
screened-in space for a little garden. </p>
<p>I got the door from our local Habitat resale store and it works great. </p>
<p>Here is a picture of the final product. Note, that the door opens for easy top
access and the back can be easily removed also by lifting it off the screws I
use as hooks. </p>
<p>Here are a few pictures of my kids and me making the garden. </p>
<p><img alt="./images/small_garden1.jpeg" src="../images/small_garden1.jpeg"></p>
<p><img alt="./images/small_garden2.jpeg" src="../images/small_garden2.jpeg"></p>
<p><img alt="./images/small_garden3.jpeg" src="../images/small_garden3.jpeg"></p>
<p><img alt="./images/small_garden5.jpeg" src="../images/small_garden5.jpeg"></p>
<p><img alt="./images/garden.jpg" src="../images/garden.jpg"></p>
<p>I also recently posted this as my first project on instructables: </p>
<p><a href="http://www.instructables.com/id/Garden-Fortress-to-stop-deer-and-other-vegetable-p">http://www.instructables.com/id/Garden-Fortress-to-stop-deer-and-other-vegetable-p</a> </p>
<p>Send me pics if you make your own.</p>
<p>- Dirk</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/05/garden-fortress.html">Blogger</a> using custom python script. Comment on errors below.</p>Scavenging Parts2014-05-24T00:00:00-04:002014-05-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-05-24:/scavenging-parts.html<p>I found this printer at a garage sale for $2. </p>
<p><img align="right," alt="./images/blogger-image-881123768.jpg" hspace="10" src="../images/blogger-image-881123768.jpg" width="200,"></p>
<p>I like to find old mechanical/electrical devices and take them apart with my
kids. </p>
<p><img align="right," alt="./images/blogger-image-272658734.jpg" hspace="10" src="../images/blogger-image-272658734.jpg" width="200,"></p>
<p>In addition to just having fun and learning how things work, we were able to
salvage the stepper motors. I want to try hooking them …</p><p>I found this printer at a garage sale for $2. </p>
<p><img align="right," alt="./images/blogger-image-881123768.jpg" hspace="10" src="../images/blogger-image-881123768.jpg" width="200,"></p>
<p>I like to find old mechanical/electrical devices and take them apart with my
kids. </p>
<p><img align="right," alt="./images/blogger-image-272658734.jpg" hspace="10" src="../images/blogger-image-272658734.jpg" width="200,"></p>
<p>In addition to just having fun and learning how things work, we were able to
salvage the stepper motors. I want to try hooking them up to the Arduino
(<a href="http://www.arduino.cc/">http://www.arduino.cc/</a>) that my brother gave me for Christmas. If it
works, we are going to try and make a reprap (<a href="http://reprap.org/">http://reprap.org/</a>) using
parts printed from my Ultimaker 2 (<a href="https://www.ultimaker.com/">https://www.ultimaker.com/</a>). </p>
<p>- Dirk </p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/05/scavaging-parts.html">Blogger</a> using costom python script. Comment on errors below.</p>Light Box2014-05-11T00:00:00-04:002014-05-11T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-05-11:/light-box.html<p><img align="right," alt="Light Box" hspace="10" src="../images/small_Light4.jpeg" width="200,"></p>
<p>In addition to making, I love to take pictures. Photography has been a passion
of mine since I was my daughter's age. I remember one vacation in Lake
Michigan with my family: my Aunt Pat gave me a 110 cartridge camera and
basically just let me loose. I was hooked …</p><p><img align="right," alt="Light Box" hspace="10" src="../images/small_Light4.jpeg" width="200,"></p>
<p>In addition to making, I love to take pictures. Photography has been a passion
of mine since I was my daughter's age. I remember one vacation in Lake
Michigan with my family: my Aunt Pat gave me a 110 cartridge camera and
basically just let me loose. I was hooked. The only problem with my habit was
the cost of developing film, which I tried to pay for in high school by
working at a one hour developing store and in college by working on the
newspaper and yearbook. Finally, even though the resolution was poor, digital
cameras allowed me to feed my habit - and as they continuously get better I
haven't looked back. </p>
<p>I found the plans for this light box on the internet (<a href="http://digital-photography-school.com/how-to-make-a-inexpensive-light-tent">http://digital-photography-school.com/how-to-make-a-inexpensive-light-tent</a>) and thought it
would make a good project for me and my kids. A light box will also come in
handy if we need to take pictures of the things we are making. </p>
<p>The first step was to measure and cut the box. This is where we got stuck
because the cardboard paper I had was a little old and yellowish. I wanted
something cleaner so I had to set this project aside until I could by some
better cardboard (and cloth, while I was at it). </p>
<p><a align="right," href="../images/small_LIght1.jpeg" hspace="10" width="200,"><img alt="" src="../images/small_LIght1.jpeg"></a></p>
<p>I ended up buying Bristol board as recommend in the blog post. It has a nice
smooth finish and I think it will work well. </p>
<p><a align="right," href="../images/small_Light2.jpeg" hspace="10" width="200,"><img alt="" src="../images/small_Light2.jpeg"></a></p>
<p>My daughter and I cut out strips of the board and glued them to the box. </p>
<p><a align="right," href="../images/small_Light3.jpeg" hspace="10" width="200,"><img alt="" src="../images/small_Light3.jpeg"></a></p>
<p>Once glued, we added the cloth and tested it out on my android phone. This looks okay, although I think I need some better (and brighter) lights -
but this looks to be a good start. </p>
<p><a align="right," href="../images/small_Light5.jpeg" hspace="10" width="200,"><img alt="" src="../images/small_Light5.jpeg"></a></p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/05/light-box.html">Blogger</a> using costom python script. Comment on errors below.</p>Reverse Making2014-05-11T00:00:00-04:002014-05-11T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-05-11:/reverse-making.html<p><img align="right," alt="./images/small_Drill4.jpeg" hspace="10" src="../images/small_Drill4.jpeg" width="200,"></p>
<p>I paid a visit to our university's surplus store. This is a wonderful place
where the university can recycle and reuse some of its vast equipment and
supplies. It turns out to be a big junk store. My goal was to find something
useful that I could take apart with …</p><p><img align="right," alt="./images/small_Drill4.jpeg" hspace="10" src="../images/small_Drill4.jpeg" width="200,"></p>
<p>I paid a visit to our university's surplus store. This is a wonderful place
where the university can recycle and reuse some of its vast equipment and
supplies. It turns out to be a big junk store. My goal was to find something
useful that I could take apart with my kids; I came across this $3 drill that
someone had already modified. You can tell by the after market trigger switch. </p>
<p><img align="right," alt="./images/small_Drill1.jpeg" hspace="10" src="../images/small_Drill1.jpeg" width="200,"></p>
<p>My daughters and I took this drill apart to see how it works. The girls loved
the activity. </p>
<p><img align="right," alt="./images/small_Drill2.jpeg" hspace="10" src="../images/small_Drill2.jpeg" width="200,"></p>
<p><img align="right," alt="./images/small_Drill3.jpeg" hspace="10" src="../images/small_Drill3.jpeg" width="200,"></p>
<p>We had some trouble getting the chuck off, but overall it was a great way to
spend an hour on Saturday. I was just going to throw away the drill, but I
think the motor and gearbox may come in handy for a future project. </p>
<p><img align="right," alt="./images/small_Drill5.jpeg" hspace="10" src="../images/small_Drill5.jpeg" width="200,"></p>
<p>I like the idea of reverse making. This is an activity well-suited for younger
kids. I plan to hit the garage sales this summer looking for things we can
take apart and recycle or reuse for parts. Maybe we will build a robot!</p>
<p>Blogpost migrated from <a href="https://apprenticemaker.blogspot.com/2014/05/i-paid-visit-to-our-universities.html">Blogger</a> using costom python script. Comment on errors below.</p>2014-05-06: HPCC Workshops2014-05-06T00:00:00-04:002014-05-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-05-06:/2014-05-06-hpcc-workshops.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>These workshops are being provided as part of IT Services two day offering of
no-charge seminars to faculty and graduate students on technology topics on
May 6 and 7. More information and registration can be done at the following
website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu …</a></p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>These workshops are being provided as part of IT Services two day offering of
no-charge seminars to faculty and graduate students on technology topics on
May 6 and 7. More information and registration can be done at the following
website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu/events/tech-seminars/</a></p>
<p><strong>Introduction to the HPCC: A Hands-On Introduction to High Performance
Computing at MSU</strong></p>
<ul>
<li>Tuesday, May 6th 8:30am-11:30am in 403 of the Computer Center</li>
<li>During this workshop, participants will learn through hands-on examples how to get started with the MSU High Performance Computing Center (HPCC). Topics include: connecting to the HPCC; copying files to your home directories; navigating the command line interface; accessing available software; testing and running programs interactively; editing files; writing job scripts; submitting jobs to the queue; and monitoring jobs running on the system.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help you get started and diagnose problems.</li>
</ul>
<p><strong>12:00-1:15pm Lunch is provided in the MSU Ballroom (registered participants
only)</strong></p>
<p><strong>Making Your Research Go Faster: Advance Topics in Getting the Most Out of
the MSU HPCC</strong></p>
<ul>
<li>Tuesday, May 6th 1:30pm-4:30pm in 403 of the Computer Center</li>
<li>During this workshop, participants already familiar with using the HPCC systems will be shown advanced techniques on how to use the system more effectively. Topics include: techniques for effective scheduling of pleasantly parallel jobs; shared memory jobs and shared network jobs; running jobs longer than a week; and making jobs fault tolerant.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help get you started and diagnose problems.</li>
</ul>
<p>SLIDES:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411596/2014-05-06%20Introduction%20to%20HPCC.pdf?version=2&modificationDate=1399379083000&api=v2">2014-05-06 Introduction to HPCC.pdf</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411596/2014-05-06_Advanced%20HPCC.pdf?version=1&modificationDate=1399319879000&api=v2">2014-05-06_Advanced HPCC.pdf</a> </li>
</ul>
<p>HANDOUTS:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411596/2014-05-06%20HPCC_WORKSHOP_COMMANDS.pdf?version=1&modificationDate=1399319960000&api=v2">2014-05-06 HPCC_WORKSHOP_COMMANDS.pdf</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411588/HPCC_WORKSHOP_COMMANDS.pdf?version=1&modificationDate=1387215144000&api=v2">HPCC_WORKSHOP_COMMANDS.pdf</a></li>
<li><a href="http://files.fosswire.com/2007/08/fwunixref.pdf">http://files.fosswire.com/2007/08/fwunixref.pdf</a> </li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/05/05/2014-05-06%3A+HPCC+Workshops">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/05/05/2014-05-06%3A+HPCC+Workshops">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-02-27 HPCC Traveling Roadshow - Quantitative Fisheries Center2014-02-27T00:00:00-05:002014-02-27T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-02-27:/2014-02-27-hpcc-traveling-roadshow-quantitative-fisheries-center.html<p>Blog post <strong>added</strong> by Anonymous</p>
<p>I talked to the Quantitative Fisheries Center today and had a lively
discussion on how to utilize HPCC resources better. Below are links to my
slides as well as a copy of the submission scripts that we worked on as a
group.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411557/2014-02-27%20Fisheries%20Introduction%20to%20HPCC.pdf?version=1&modificationDate=1393540050000&api=v2">2014-02-27 Fisheries …</a></p><p>Blog post <strong>added</strong> by Anonymous</p>
<p>I talked to the Quantitative Fisheries Center today and had a lively
discussion on how to utilize HPCC resources better. Below are links to my
slides as well as a copy of the submission scripts that we worked on as a
group.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411557/2014-02-27%20Fisheries%20Introduction%20to%20HPCC.pdf?version=1&modificationDate=1393540050000&api=v2">2014-02-27 Fisheries Introduction to
HPCC.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411557/SampleQFC.qsub?version=1&modificationDate=1393540050000&api=v2">SampleQFC.qsub</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/02/27/2014-02-27+HPCC+Traveling+Roadshow+-+Quantitative+Fisheries+Center">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/02/27/2014-02-27+HPCC+Traveling+Roadshow+-+Quantitative+Fisheries+Center">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-01-31 Introduction to MATLAB Workshop2014-01-27T00:00:00-05:002014-01-27T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-01-27:/2014-01-31-introduction-to-matlab-workshop.html<p>Blog post <strong>edited</strong> by Anonymous</p>
<p>I am trying something new this time. The following are links to a dataset we
will use as an example in the class:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/GROUP_YEARLY_2013.txt?version=1&modificationDate=1390832160000&api=v2">GROUP_YEARLY_2013.txt</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/GROUP_YEARLY_2013_v2.txt?version=1&modificationDate=1390832173000&api=v2">GROUP_YEARLY_2013_v2.txt</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/GROUP_YEARLY_2013_v3.csv?version=1&modificationDate=1390832184000&api=v2">GROUP_YEARLY_2013_v3.csv</a></li>
</ul>
<p>Here is an example Function:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/imagecrop.m?version=1&modificationDate=1390832201000&api=v2">imagecrop.m</a></li>
</ul>
<p>Here are the slides:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/2014-01-31%20Introduction%20to%20Matlab.pdf?version=1&modificationDate=1390832462000&api=v2">2014-01-31 Introduction to Matlab.pdf …</a></li></ul><p>Blog post <strong>edited</strong> by Anonymous</p>
<p>I am trying something new this time. The following are links to a dataset we
will use as an example in the class:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/GROUP_YEARLY_2013.txt?version=1&modificationDate=1390832160000&api=v2">GROUP_YEARLY_2013.txt</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/GROUP_YEARLY_2013_v2.txt?version=1&modificationDate=1390832173000&api=v2">GROUP_YEARLY_2013_v2.txt</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/GROUP_YEARLY_2013_v3.csv?version=1&modificationDate=1390832184000&api=v2">GROUP_YEARLY_2013_v3.csv</a></li>
</ul>
<p>Here is an example Function:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/imagecrop.m?version=1&modificationDate=1390832201000&api=v2">imagecrop.m</a></li>
</ul>
<p>Here are the slides:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/2014-01-31%20Introduction%20to%20Matlab.pdf?version=1&modificationDate=1390832462000&api=v2">2014-01-31 Introduction to Matlab.pdf</a></li>
</ul>
<p>Here are some other useful handouts:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411670/2014-01-31_Introduction_to_Matlab_Handout.pdf?version=1&modificationDate=1390832521000&api=v2">2014-01-31_Introduction_to_Matlab_Handout.pdf</a></li>
<li><a href="http://web.mit.edu/18.06/www/MATLAB/matlab-cheatsheet.pdf">http://web.mit.edu/18.06/www/MATLAB/matlab-cheatsheet.pdf</a></li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/01/27/2014-01-31+Introduction+to+MATLAB+Workshop">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/01/27/2014-01-31+Introduction+to+MATLAB+Workshop">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-01-22: Using Cloud Resources in your Research2014-01-22T00:00:00-05:002014-01-22T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-01-22:/2014-01-22-using-cloud-resources-in-your-research.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>Here are the slides from our round table discussion on research in the cloud.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411646/2014-01-22%20Cloud%20Computing.pdf?version=1&modificationDate=1390418520000&api=v2">2014-01-22 Cloud
Computing.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/01/22/2014-01-22%3A+Using+Cloud+Resources+in+your+Research">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/01/22/2014-01-22%3A+Using+Cloud+Resources+in+your+Research">ICER Wiki</a> using custom python script. Comment on errors below.</p>2014-01-21: CSE 491 - Intro to Parallel Programming2014-01-21T00:00:00-05:002014-01-21T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2014-01-21:/2014-01-21-cse-491-intro-to-parallel-programming.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>Here is my "Pleasantly Parallel" presentation designed for a computer science
audience:</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411674/2014-01-21%20Parallel%20Paradigms.pdf?version=1&modificationDate=1390764984000&api=v2">2014-01-21 Parallel
Paradigms.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/01/26/2014-01-21%3A+CSE+491+-+Intro+to+Parallel+Programming">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2014/01/26/2014-01-21%3A+CSE+491+-+Intro+to+Parallel+Programming">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013-12-18: HPCC Workshops2013-12-18T00:00:00-05:002013-12-18T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-12-18:/2013-12-18-hpcc-workshops.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>These workshops are being provided as part of IT Services two day offering of
no-charge seminars to faculty and graduate students on technology topics on
December 17 and 18. More information and registration can be done at the
following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu …</a></p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>These workshops are being provided as part of IT Services two day offering of
no-charge seminars to faculty and graduate students on technology topics on
December 17 and 18. More information and registration can be done at the
following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu/events/tech-seminars/</a></p>
<p><strong>Introduction to the HPCC: A Hands-On Introduction to High Performance
Computing at MSU</strong></p>
<ul>
<li>Wednesday, December 18th 8:30am-11:30am in 403 of the Computer Center</li>
<li>During this workshop, participants will learn through hands-on examples how to get started with the MSU High Performance Computing Center (HPCC). Topics include: connecting to the HPCC; copying files to your home directories; navigating the command line interface; accessing available software; testing and running programs interactively; editing files; writing job scripts; submitting jobs to the queue; and monitoring jobs running on the system.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help you get started and diagnose problems.</li>
</ul>
<p><strong>12:00-1:15pm Lunch is provided in the MSU Ballroom (registered participants
only)</strong></p>
<p><strong>Making Your Research Go Faster: Advance Topics in Getting the Most Out of
the MSU HPCC</strong></p>
<ul>
<li>Wednesday, December 18th 1:30pm-4:30pm in 403 of the Computer Center</li>
<li>During this workshop, participants already familiar with using the HPCC systems will be shown advanced techniques on how to use the system more effectively. Topics include: techniques for effective scheduling of pleasantly parallel jobs; shared memory jobs and shared network jobs; running jobs longer than a week; and making jobs fault tolerant.</li>
<li>This is an interactive workshop held in a computer lab, although participants are encouraged to bring their research laptops if applicable. Multiple instructors will be available to help get you started and diagnose problems.</li>
</ul>
<p>SLIDES:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411588/2013-12-18%20Introduction%20to%20HPCC.pdf?version=1&modificationDate=1387215093000&api=v2">2013-12-18 Introduction to HPCC.pdf</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411588/2013-12-18%20Advanced%20HPCC.pdf?version=1&modificationDate=1387215109000&api=v2">2013-12-18 Advanced HPCC.pdf</a></li>
</ul>
<p>HANDOUTS:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411588/HPCC_Command_Summary.pdf?version=1&modificationDate=1387215131000&api=v2">HPCC_Command_Summary.pdf</a></li>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411588/HPCC_WORKSHOP_COMMANDS.pdf?version=1&modificationDate=1387215144000&api=v2">HPCC_WORKSHOP_COMMANDS.pdf</a></li>
<li><a href="http://files.fosswire.com/2007/08/fwunixref.pdf">http://files.fosswire.com/2007/08/fwunixref.pdf</a> </li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/12/16/2013-12-18%3A+HPCC+Workshops">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/12/16/2013-12-18%3A+HPCC+Workshops">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013-12-17: Introduction to MATLAB Workshop2013-12-17T00:00:00-05:002013-12-17T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-12-17:/2013-12-17-introduction-to-matlab-workshop.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>This workshops is being provided as part of IT Services two day offering of
no-charge seminars to faculty and graduate students on technology topics on
December 17 and 18. More information and registration can be done at the
following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu …</a></p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~archerc5@msu.edu">Camille Archer</a></p>
<p>This workshops is being provided as part of IT Services two day offering of
no-charge seminars to faculty and graduate students on technology topics on
December 17 and 18. More information and registration can be done at the
following website:</p>
<p><a href="http://tech.msu.edu/events/tech-seminars/">http://tech.msu.edu/events/tech-seminars/</a></p>
<p><strong>Introduction to MATLAB</strong><br>
This workshop will introduce you to MATLAB as a tool for doing scientific
research. The MATLAB programming language is particularly well suited for use
a research tool. MATLAB can be used in all aspects of the research process,
including: data gathering, data organization, data analysis, model simulation,
and visualization of results. This hands-on workshop is appropriate for
researchers who are new to programming and/or are not familiar with MATLAB.</p>
<p>SLIDES:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411585/2013-12-17%20Introduction%20to%20Matlab.pdf?version=1&modificationDate=1387214738000&api=v2">2013-12-17 Introduction to Matlab.pdf</a></li>
</ul>
<p>HANDOUTS:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/download/attachments/5411585/Introduction_to_Matlab.pdf?version=1&modificationDate=1387214772000&api=v2">Introduction_to_Matlab.pdf</a></li>
<li><a href="http://web.mit.edu/18.06/www/MATLAB/matlab-cheatsheet.pdf">http://web.mit.edu/18.06/www/MATLAB/matlab-cheatsheet.pdf</a></li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/12/16/2013-12-17%3A+Introduction+to+MATLAB+Workshop">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/12/16/2013-12-17%3A+Introduction+to+MATLAB+Workshop">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013-10-23 CI-Days Crash Course2013-10-24T00:00:00-04:002013-10-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-10-24:/2013-10-23-ci-days-crash-course.html<p>Blog post <strong>edited</strong> by Anonymous</p>
<p>Welcome to CI-Days Crash Course workshop on High Performance Computing at
Michigan State University. The following are slides and a link to the feedback
form so you can follow along:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411710/2013-10-24%20HPCC_Crash_Coarse.pdf?version=1&modificationDate=1382618799000&api=v2">2013-10-24
HPCC_Crash_Coarse.pdf</a></p>
<p><a href="https://docs.google.com/forms/d/1NVE7TYF310vSw0t-6ULPB0f6cmi8cBKuCiSih5Oe4aQ/viewform">FeedBack
Form</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/24/2013-10-23+CI-
Days+Crash+Course">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/24/2013-10-23+CI-Days+Crash+Course">ICER Wiki</a> using custom python …</p><p>Blog post <strong>edited</strong> by Anonymous</p>
<p>Welcome to CI-Days Crash Course workshop on High Performance Computing at
Michigan State University. The following are slides and a link to the feedback
form so you can follow along:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411710/2013-10-24%20HPCC_Crash_Coarse.pdf?version=1&modificationDate=1382618799000&api=v2">2013-10-24
HPCC_Crash_Coarse.pdf</a></p>
<p><a href="https://docs.google.com/forms/d/1NVE7TYF310vSw0t-6ULPB0f6cmi8cBKuCiSih5Oe4aQ/viewform">FeedBack
Form</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/24/2013-10-23+CI-
Days+Crash+Course">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/24/2013-10-23+CI-Days+Crash+Course">ICER Wiki</a> using custom python script. Comment on errors below.</p>Using X11 Virtual Frame Buffer (Xvfb) in a submission script2013-10-08T00:00:00-04:002013-10-08T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-10-08:/using-x11-virtual-frame-buffer-xvfb-in-a-submission-script.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~rahmenic@msu.edu">Nicholas Anthony Rahme</a></p>
<p>Some programs require access to a display in order to run properly. For
example, MATLAB will provide an error if you try to create an AVI file from a
image frame in -nodisplay mode. This can be an annoying problem on the HPCC …</p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~rahmenic@msu.edu">Nicholas Anthony Rahme</a></p>
<p>Some programs require access to a display in order to run properly. For
example, MATLAB will provide an error if you try to create an AVI file from a
image frame in -nodisplay mode. This can be an annoying problem on the HPCC
because all of our batch compute nodes do not have displays attached.
Fortunately, the X virtual Frame Buffer (Xvfb) is a program that simulates an
attached display and can allow programs that require Graphical User Interfaces
(X11) displays to run.</p>
<p>The following is a typical submission script for running a MATLAB program :</p>
<p><strong>makeAVI.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#PBS -l nodes=1:ppn=1,mem=2gb,feature=gbe</span>
<span class="c1">#PBS -l walltime=01:00:00</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -W x=gres:MATLAB</span>
<span class="c1">#Change directory to the qsub submission directory</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="c1">#Run matlab script (replace this with your X11 code)</span>
matlab<span class="w"> </span>-nodisplay<span class="w"> </span>-r<span class="w"> </span><span class="s2">"makeAVI"</span>
<span class="c1">#Print out job usage</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
</code></pre></div></td></tr></table></div>
<p>Running this script will give me an error that says "Display not found." The
following script is modified to start Xvfb at the beginning, run the same
basic command and then terminate Xvfb at the end:</p>
<p><strong>makeAVI.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#PBS -l nodes=1:ppn=1,mem=2gb,feature=gbe</span>
<span class="c1">#PBS -l walltime=01:00:00</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -W x=gres:MATLAB</span>
<span class="c1">#Change directory to the qsub submission directory</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="c1">##### Specify the display, start the Xvfb server, and save the process ID.</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">DISPLAY</span><span class="o">=</span><span class="s2">":1"</span>
rm<span class="w"> </span>-f<span class="w"> </span>/tmp/.X11-unix
rm<span class="w"> </span>-f<span class="w"> </span>/tmp/.X11-lock
Xvfb<span class="w"> </span><span class="nv">$DISPLAY</span><span class="w"> </span>-auth<span class="w"> </span>/dev/null/<span class="w"> </span><span class="p">&</span>
<span class="nv">XVFB_PID</span><span class="o">=</span><span class="nv">$!</span>
<span class="c1">#Give system time to spin up X11 display (Probably not needed)</span>
sleep<span class="w"> </span><span class="m">5</span>
<span class="c1">####</span>
<span class="c1">#Run matlab script</span>
matlab<span class="w"> </span>-r<span class="w"> </span><span class="s2">"makeAVI"</span>
<span class="c1">##### Stop the Xvfb server and remove the files it created.</span>
<span class="nb">kill</span><span class="w"> </span>-9<span class="w"> </span><span class="nv">$XVFB_PID</span>
rm<span class="w"> </span>-rf<span class="w"> </span>/tmp/.X11-unix
rm<span class="w"> </span>-f<span class="w"> </span>/tmp/.X11-lock
<span class="c1">####</span>
<span class="c1">#Print out job usage</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
</code></pre></div></td></tr></table></div>
<p>Please note that the -nodisplay option was removed from the MATLAB command
line in the second script</p>
<p>An working example of this script can be downloaded on the HPCC using the
getexample command. Type the following on any development node to run the
example:</p>
<div class="highlight"><pre><span></span><code><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">powertools</span>
<span class="n">getexample</span>
<span class="n">getexample</span><span class="w"> </span><span class="n">MATLAB_movie</span>
<span class="n">cd</span><span class="w"> </span><span class="n">MATLAB_movie</span>
<span class="n">qsub</span><span class="w"> </span><span class="n">makeAVI</span><span class="o">.</span><span class="n">qsub</span>
</code></pre></div>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/08/Using+X11+Virtual+Frame+Buffer+%28Xvfb%29+in+a+submission+script">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/08/Using+X11+Virtual+Frame+Buffer+%28Xvfb%29+in+a+submission+script">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013-10-03 Making your research go faster - ChE_MSE 802 Presentation2013-10-05T00:00:00-04:002013-10-05T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-10-05:/2013-10-03-making-your-research-go-faster-che_mse-802-presentation.html<p>Blog post <strong>added</strong> by Anonymous</p>
<p>Below are the slides I gave at a presentation for ChE_MSE 802. Most of the
students seemed interested in either fluid dynamics or molecular dynamics
problems. I presented some new background slides which talked about the
pillars of science.</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411654/2013-10-03%20ChE_MSE%20802.pdf?version=1&modificationDate=1380979652000&api=v2">2013-10-03 ChE_MSE
802.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/05/2013-10-03+Making+your+research+go+faster+-+ChE_MSE+802+Presentation">View …</a></p><p>Blog post <strong>added</strong> by Anonymous</p>
<p>Below are the slides I gave at a presentation for ChE_MSE 802. Most of the
students seemed interested in either fluid dynamics or molecular dynamics
problems. I presented some new background slides which talked about the
pillars of science.</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411654/2013-10-03%20ChE_MSE%20802.pdf?version=1&modificationDate=1380979652000&api=v2">2013-10-03 ChE_MSE
802.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/05/2013-10-03+Making+your+research+go+faster+-+ChE_MSE+802+Presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/10/05/2013-10-03+Making+your+research+go+faster+-+ChE_MSE+802+Presentation">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013-2014 New Faculty Orientation2013-08-22T00:00:00-04:002013-08-22T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-08-22:/2013-2014-new-faculty-orientation.html<p>Blog post <strong>added</strong> by Anonymous</p>
<p>Below are the slides I presented to the New Faculty Orientation held on August
20 and 26.</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411611/2013%20New%20Faculty%20Orientation.pdf?version=1&modificationDate=1377179022000&api=v2">2013 New Faculty
Orientation.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/08/22/2013-2014+New+Faculty+Orientation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/08/22/2013-2014+New+Faculty+Orientation">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013 Introduction to iCER - Buell Lab Seminar2013-08-01T00:00:00-04:002013-08-01T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-08-01:/2013-introduction-to-icer-buell-lab-seminar.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are slides for a seminar I am put together for C. Robin Buell's
Bioinformatics workshop. This talk gives an overview of iCER and our
reseources focusing on bioinformatics. I also introduced the concepts of
plesently parallel jobs and give …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are slides for a seminar I am put together for C. Robin Buell's
Bioinformatics workshop. This talk gives an overview of iCER and our
reseources focusing on bioinformatics. I also introduced the concepts of
plesently parallel jobs and give an example.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2013-08-01_iCER_Bioinformatics_Slides.pdf">2013-08-01 iCER Bioinformatics Slides.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/08/01/2013+Introduction+to+iCER+-+Buell+Lab+Seminar">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/08/01/2013+Introduction+to+iCER+-+Buell+Lab+Seminar">ICER Wiki</a> using custom python script. Comment on errors below.</p>Running ANSYS CFX5 solver on the HPCC2013-07-30T00:00:00-04:002013-07-30T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-07-30:/running-ansys-cfx5-solver-on-the-hpcc.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>The following job submission script should get a parallel MPI solve of the
cfx5solver working. In order for this to work on our system I needed to change
the ANSYS module file to include …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>The following job submission script should get a parallel MPI solve of the
cfx5solver working. In order for this to work on our system I needed to change
the ANSYS module file to include setting CFX5RSH to /usr/bin/ssh</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -l walltime=00:10:00,nodes=8:ppn=1,mem=4gb</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
module<span class="w"> </span>load<span class="w"> </span>ANSYS
<span class="c1">#Hack to create comma seperated list of hosts for this job</span>
<span class="nb">echo</span><span class="w"> </span><span class="sb">`</span>cat<span class="w"> </span><span class="nv">$PBS_NODEFILE</span><span class="sb">`</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s2">"s/ /,/g"</span><span class="w"> </span>><span class="w"> </span>Hosts.file
<span class="nb">read</span><span class="w"> </span>HOSTS<span class="w"> </span><<span class="w"> </span>Hosts.file
cfx5solve<span class="w"> </span>-def<span class="w"> </span>mymodel.def<span class="w"> </span>-start-method<span class="w"> </span><span class="s1">'Platform MPI Distributed Parallel'</span><span class="w"> </span>-par-dist<span class="w"> </span><span class="nv">$HOSTS</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="nv">$PBS_JOBID</span>
</code></pre></div></td></tr></table></div>
<p>Users should consider changing the <strong>walltime</strong> , <strong>nodes</strong> and <strong>def file</strong>
to get this to work for their program.</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/07/30/Running+ANSYS+CFX5+solver+on+the+HPCC">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/07/30/Running+ANSYS+CFX5+solver+on+the+HPCC">ICER Wiki</a> using custom python script. Comment on errors below.</p>Presentation Slides from ASEE 2013, Atlanta GA2013-07-29T00:00:00-04:002013-07-29T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-07-29:/presentation-slides-from-asee-2013-atlanta-ga.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are links to my slides for ASEE 2013 presentations:</p>
<ul>
<li>
<p>Katy Luchini-Colbry and Dirk Colbry. “ <strong>Gadget Avalanche: A Technology Literacy Course for Novice Adults.</strong> ” ASEE Annual Conference Atlanta, GA, June 2013. <a href="http://www.asee.org/file_server/papers/attachment/file/0003/4344/ASEE2013-Gadget-FINAL.pdf">[paper]</a> [slides]</p>
</li>
<li>
<p>Dirk Colbry and Katy Luchini-Colbry. “ <strong>STEM inSight …</strong></p></li></ul><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are links to my slides for ASEE 2013 presentations:</p>
<ul>
<li>
<p>Katy Luchini-Colbry and Dirk Colbry. “ <strong>Gadget Avalanche: A Technology Literacy Course for Novice Adults.</strong> ” ASEE Annual Conference Atlanta, GA, June 2013. <a href="http://www.asee.org/file_server/papers/attachment/file/0003/4344/ASEE2013-Gadget-FINAL.pdf">[paper]</a> [slides]</p>
</li>
<li>
<p>Dirk Colbry and Katy Luchini-Colbry. “ <strong>STEM inSight: Developing a Research Skills Course for First and Second Year Students.</strong> ” ASEE Annual Conference Atlanta, GA, June 2013. <a href="http://www.asee.org/file_server/papers/attachment/file/0003/3803/6188.pdf">[paper]</a> [slides]</p>
</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/07/29/Presentation+Slides+from+ASEE+2013%2C+Atlanta+GA">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/07/29/Presentation+Slides+from+ASEE+2013%2C+Atlanta+GA">ICER Wiki</a> using custom python script. Comment on errors below.</p>Presentation Slides from XSEDE 2013, San Diego CA2013-07-29T00:00:00-04:002013-07-29T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-07-29:/presentation-slides-from-xsede-2013-san-diego-ca.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are links to my slides for XSEDE 2013 presentations:</p>
<ul>
<li>
<p>Dirk Colbry, William Punch, Wolfgang Bauer, “The Institute for Cyber-Enabled Research: Regional Organization to Promote Computation in Science.” XSEDE 2013 Conference San Diego, CA, July 2013. [slides]</p>
</li>
<li>
<p>Dirk</p>
</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/07/29/Presentation+Slides+from+XSEDE+2013%2C+San+Diego+CA">View
Online …</a></p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are links to my slides for XSEDE 2013 presentations:</p>
<ul>
<li>
<p>Dirk Colbry, William Punch, Wolfgang Bauer, “The Institute for Cyber-Enabled Research: Regional Organization to Promote Computation in Science.” XSEDE 2013 Conference San Diego, CA, July 2013. [slides]</p>
</li>
<li>
<p>Dirk</p>
</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/07/29/Presentation+Slides+from+XSEDE+2013%2C+San+Diego+CA">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/07/29/Presentation+Slides+from+XSEDE+2013%2C+San+Diego+CA">ICER Wiki</a> using custom python script. Comment on errors below.</p>Running the XSEDE MPI workshop examples on the HPCC2013-06-18T00:00:00-04:002013-06-18T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-06-18:/running-the-xsede-mpi-workshop-examples-on-the-hpcc.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Differences in examples to get them to run on the HPCC.</p>
<p>Step 1:<br>
Download these examples buy loading the power tools module and using the get
example command:</p>
<div class="highlight"><pre><span></span><code><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">powertools</span>
<span class="n">getexample</span>
<span class="n">getexample</span><span class="w"> </span><span class="n">XSEDE_MPI_WORKSHOP …</span></code></pre></div><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Differences in examples to get them to run on the HPCC.</p>
<p>Step 1:<br>
Download these examples buy loading the power tools module and using the get
example command:</p>
<div class="highlight"><pre><span></span><code><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">powertools</span>
<span class="n">getexample</span>
<span class="n">getexample</span><span class="w"> </span><span class="n">XSEDE_MPI_WORKSHOP</span>
</code></pre></div>
<p>Step 2:<br>
Compile the code using "mpicc" or "mpif90":</p>
<div class="highlight"><pre><span></span><code>mpicc ex1.c
mpif90 ex1.f
</code></pre></div>
<p>Step 3:<br>
Run the code in the command line using same command:</p>
<div class="highlight"><pre><span></span><code>mpirun -np 4 ./a.out
</code></pre></div>
<p>Step 4:<br>
Write a submission script and run the job on the cluster:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span>
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -l walltime=05:00,nodes=8:ppn=1</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
mpirun<span class="w"> </span>./a.out
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
</code></pre></div></td></tr></table></div>
<p>Step 5:<br>
Submit the job using the following command:</p>
<div class="highlight"><pre><span></span><code>qsub myjob.qsub
</code></pre></div>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/06/17/Running+the+XSEDE+MPI+workshop+examples+on+the+HPCC">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/06/17/Running+the+XSEDE+MPI+workshop+examples+on+the+HPCC">ICER Wiki</a> using custom python script. Comment on errors below.</p>Spring 2013 Faculty Seminar Workshops2013-05-07T00:00:00-04:002013-05-07T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-05-07:/spring-2013-faculty-seminar-workshops.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here are the slides for the Intro to HPCC, Intro to MATLAB and Advanced HPCC
Classes.</p>
<p><a href="./images/2013-05-07_Introduction_to_Matlab.pdf">2013-05-07 Introduction to Matlab.pdf</a></p>
<p><a href="./images/2013-05-08_Introduction_to_HPCC.pdf">2013-05-08 Introduction to HPCC.pdf</a><br>
<a href="./images/2013-05-08_Advanced_HPCC.pdf">2013-05-08 Advanced HPCC.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/05/06/Spring+2013+Faculty+Seminar+Workshops">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/05/06/Spring+2013+Faculty+Seminar+Workshops">ICER Wiki</a> using custom …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here are the slides for the Intro to HPCC, Intro to MATLAB and Advanced HPCC
Classes.</p>
<p><a href="./images/2013-05-07_Introduction_to_Matlab.pdf">2013-05-07 Introduction to Matlab.pdf</a></p>
<p><a href="./images/2013-05-08_Introduction_to_HPCC.pdf">2013-05-08 Introduction to HPCC.pdf</a><br>
<a href="./images/2013-05-08_Advanced_HPCC.pdf">2013-05-08 Advanced HPCC.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/05/06/Spring+2013+Faculty+Seminar+Workshops">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/05/06/Spring+2013+Faculty+Seminar+Workshops">ICER Wiki</a> using custom python script. Comment on errors below.</p>Working with ImageJ plugins on the HPCC2013-05-06T00:00:00-04:002013-05-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-05-06:/working-with-imagej-plugins-on-the-hpcc.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Post written by Dan Perez (iCER Intern)</p>
<h3>Errors</h3>
<p>One of the iCER users has had problems getting ImageJ - ImageViewer scripts
and plugins working on the HPCC. The ImageViewer classes, plugins, and scripts
are in …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Post written by Dan Perez (iCER Intern)</p>
<h3>Errors</h3>
<p>One of the iCER users has had problems getting ImageJ - ImageViewer scripts
and plugins working on the HPCC. The ImageViewer classes, plugins, and scripts
are in a directory called MyMacro. When I execute RunJavaOnly.sh I get:</p>
<div class="highlight"><pre><span></span><code><span class="n">cd</span><span class="w"> </span><span class="n">MyMacro</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">Java</span>
<span class="o">./</span><span class="n">RunJavaOnly</span><span class="o">.</span><span class="n">sh</span>
<span class="n">File</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">Open</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">30</span><span class="n">_1_DE2_C2crop</span><span class="o">/</span><span class="n">temp</span><span class="o">/</span><span class="n">mask0001</span><span class="o">.</span><span class="n">tif</span><span class="w"> </span><span class="p">(</span><span class="n">this</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="mi">8</span><span class="o">-</span><span class="n">bit</span><span class="w"> </span><span class="n">image</span><span class="p">)</span>
<span class="n">Plugins</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">Analyze</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">Particle</span><span class="w"> </span><span class="n">Analyser</span>
<span class="n">Error</span><span class="p">:</span>
<span class="n">Plugin</span><span class="w"> </span><span class="n">org</span><span class="o">.</span><span class="n">double</span><span class="o">.</span><span class="n">bonej</span><span class="o">.</span><span class="n">ParticleCounter</span><span class="w"> </span><span class="n">did</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">find</span><span class="w"> </span><span class="n">required</span><span class="w"> </span><span class="k">class</span><span class="p">:</span><span class="w"> </span><span class="n">javax</span><span class="o">/</span><span class="n">media</span><span class="o">/</span><span class="n">j3d</span><span class="o">/</span><span class="n">Shape3D</span>
</code></pre></div>
<p>And when I execute ./runmacro.sh I get:</p>
<div class="highlight"><pre><span></span><code>./runmacro.sh
Error:
Unrecognized command: 3D Fast Filters
</code></pre></div>
<h3>Fixes</h3>
<h5>RunJavaOnly.sh</h5>
<p>I've had some success getting this to work by installing the latest version of
ImageJ (v1.46r) and adding class paths and a shared object path:</p>
<p>1. Download the latest version of ImageJ. This will create a directory called
ImageJ in you home directory.</p>
<div class="highlight"><pre><span></span><code><span class="n">cd</span><span class="w"> </span><span class="o">~</span>
<span class="n">wget</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">rsb</span><span class="o">.</span><span class="n">info</span><span class="o">.</span><span class="n">nih</span><span class="o">.</span><span class="n">gov</span><span class="o">/</span><span class="n">ij</span><span class="o">/</span><span class="n">download</span><span class="o">/</span><span class="n">linux</span><span class="o">/</span><span class="n">ij146</span><span class="o">-</span><span class="n">linux64</span><span class="o">.</span><span class="n">tar</span><span class="o">.</span><span class="n">gz</span>
<span class="n">tar</span><span class="w"> </span><span class="o">-</span><span class="n">zxf</span><span class="w"> </span><span class="n">ij146</span><span class="o">-</span><span class="n">linux64</span><span class="o">.</span><span class="n">tar</span><span class="o">.</span><span class="n">gz</span>
</code></pre></div>
<p>2. Delete the old version of ImageJ and copy the new version to your MyMacro
directory. Assuming that MyMacro is also in your home directory:</p>
<div class="highlight"><pre><span></span><code>rm MyMacro/ij-1.45q.jar
cp ImageJ/ij.jar MyMacro/ij.jar
</code></pre></div>
<p>3. Edit the RunJavaOnly.sh script. Java needs to know where to find the
libj3dcore-ogl.so shared object file (it's in the
/opt/software/Java3D/lib/amd64 directory). You'll also need to update the
script to use the new version of ImageJ you just installed. The script should
look like:</p>
<p>Post created by Daniel Perez (ICER Student Intern):</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash</span>
java<span class="w"> </span>-Xmx8000m<span class="w"> </span>-Dplugins.dir<span class="o">=</span>./plugins/<span class="w"> </span>-Djava.library.path<span class="o">=</span>/opt/software/Java3D/lib/amd64<span class="w"> </span>-jar<span class="w"> </span>./ij.jar<span class="w"> </span><span class="p">&</span>
</code></pre></div></td></tr></table></div>
<p>4. Edit the MyMacro/ij.jar/META-INF/MANIFEST.MF file. Java needs to know
where to find Java3D classes. Add the following line to the end of the file:</p>
<div class="highlight"><pre><span></span><code>Class-Path: /opt/software/Java3D/lib/ext/j3dcore.jar /opt/software/Java3D/lib/ext/j3dutils.jar
</code></pre></div>
<p>If this continues to create problems, you can also try adding
/opt/software/Java3D/lib/ext/vecmath.jar to the Class-Path. At this point I
think RunJavaOnly.sh works. However, I don't know what this program is
supposed to do, so Dirk should test this.</p>
<h5>runmacro.sh</h5>
<p>Works better using the plugins directory (instead of plugins2):</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
java<span class="w"> </span>-Xmx8000m<span class="w"> </span>-Dplugins.dir<span class="o">=</span>./plugins/<span class="w"> </span>-jar<span class="w"> </span>./ij.jar<span class="w"> </span>-macro<span class="w"> </span>./SplitLargePiece.txt
</code></pre></div></td></tr></table></div>
<p>Runs for awhile. Don't get the 3D Fast Filters error. Get the following errors
instead:</p>
<div class="highlight"><pre><span></span><code>No window with the title "particleID_parts" found.
This command requires a stack.
</code></pre></div>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/05/06/Working+with+ImageJ+plugins+on+the+HPCC">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/05/06/Working+with+ImageJ+plugins+on+the+HPCC">ICER Wiki</a> using custom python script. Comment on errors below.</p>Talk for visiting University of Duhok - Iraq2013-04-26T00:00:00-04:002013-04-26T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-04-26:/talk-for-visiting-university-of-duhok-iraq.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I gave a talk today to some visiting Faculty from the visiting of Duhok, Iraq.
We had an interesting discussion about the types of research problems we solve
and the types of hardware we use to solve the problems. Here …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I gave a talk today to some visiting Faculty from the visiting of Duhok, Iraq.
We had an interesting discussion about the types of research problems we solve
and the types of hardware we use to solve the problems. Here are the slides
from my talk.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2013-04-26_Duhok_Talk.pdf">2013-04-26 Duhok Talk.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/04/26/Talk+for+visiting+University+of+Duhok+-+Iraq">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/04/26/Talk+for+visiting+University+of+Duhok+-+Iraq">ICER Wiki</a> using custom python script. Comment on errors below.</p>Parallel Programming Paradigms2013-03-20T00:00:00-04:002013-03-20T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-03-20:/parallel-programming-paradigms.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the slides I gave as a guest lecturer to Pang-Ning Tan's CSE 491/891
class: "Computational Techniques for Large-Scale Data Analysis". I was
particularly excited about putting these slides together because the
summarized many concepts that I have …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the slides I gave as a guest lecturer to Pang-Ning Tan's CSE 491/891
class: "Computational Techniques for Large-Scale Data Analysis". I was
particularly excited about putting these slides together because the
summarized many concepts that I have given in a lot of my other presentations.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2013-03-20_Parallel_Paradigms.pdf">2013-03-20 Parallel Paradigms.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/03/20/Parallel+Programming+Paradigms">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/03/20/Parallel+Programming+Paradigms">ICER Wiki</a> using custom python script. Comment on errors below.</p>Using Python virtualenv on the HPCC2013-03-06T00:00:00-05:002013-03-06T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-03-06:/using-python-virtualenv-on-the-hpcc.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Python has a lot of packages, modules and libraries that researchers may want
to use. However, it is difficult for iCER and the HPCC to keep up with and
install all of the different …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Python has a lot of packages, modules and libraries that researchers may want
to use. However, it is difficult for iCER and the HPCC to keep up with and
install all of the different libraries, versions, and conflicts between them.
Users who need Python software packages installed on the HPCC have the
following choices:</p>
<ul>
<li>Try to install their own copy of Python and the modules in their home directories. (gives researchers full control but can be difficult and time consuming).</li>
<li>Ask HPCC staff to install the software in the base Python installation (This option could easily be rejected due to software conflicts)</li>
<li>Ask HPCC staff to install the software in a separate system module (This option is more forgiving of different versions but still requires the user to wait on HPCC staff to complete and test the install)</li>
<li>Use virtualenv (this works best if the software uses easy_install or pip and has the advantage of putting control in the hands of the researcher, This option also give the researcher the ability to explore which is integral to the research process)</li>
</ul>
<p>The remainder of this post will explain how to use virtualenv on the HPCC.
Documentation for virtualenv can be found at the following website:</p>
<p><a href="http://pypi.python.org/pypi/virtualenv">http://pypi.python.org/pypi/virtualenv</a></p>
<p><strong>Step 1:</strong> make sure you are on a developer node and use system modules to
load the Python version you want to use. You can see the default version of
Python by typing the following:</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="n">list</span>
</code></pre></div>
<p>As of writing this post, my default was set to Python/2.7.2. If you need
another vision you can see what is available by typing the following:</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="n">spider</span> <span class="n">Python</span>
</code></pre></div>
<p>To load a different version of python use the "module unload" and "module
load" commands. For example the following will switch from the default version
of Python to 2.6.7:</p>
<div class="highlight"><pre><span></span><code><span class="n">module</span><span class="w"> </span><span class="n">unload</span><span class="w"> </span><span class="n">Python</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">Python</span><span class="o">/</span><span class="mf">2.6</span><span class="o">.</span><span class="mi">7</span>
</code></pre></div>
<p><strong>Step 2:</strong> use system modules to load existing Python related software. Many
Python packages are already installed and can be loaded using the module
system. Use the following command to list all available modules:</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="n">avail</span>
</code></pre></div>
<p>Since most of the Python modules have the word "py" in them so you can use the
spider command to see what is installed:</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="n">spider</span> <span class="n">py</span>
</code></pre></div>
<p>Once you know what is available, you can load the modules using "module load"
command. For example:</p>
<div class="highlight"><pre><span></span><code><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">NumPy</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">SciPy</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">PIL</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">matplotlib</span>
</code></pre></div>
<p><strong>Step 3:</strong> at this point you can make your own virtual Python environment.
This environment lets you install packages using easy_install and pip in your
how directory. You can also have multiple different virtual Python
environments, each one with a different name. For this example, I am going to
call my virtual Python environment "myPy". Use the following commands to
create and activate a virtual environment:</p>
<div class="highlight"><pre><span></span><code>virtualenv myPy
source myPy/bin/activate
</code></pre></div>
<p>After running the source command, you should see your normal command prompt
with a (myPy) in front of it. This tells you, that you are running in the myPy
virtual environment.</p>
<p><strong>Step 4:</strong> Now that you are running in your virtual environment you can run
easy_install or pip commands to install almost all Python modules. For
example, the following command installed a redditanalysis package:</p>
<div class="highlight"><pre><span></span><code>easy_install redditanalysis
</code></pre></div>
<p><strong>Step 5:</strong> Once you have everything installed test out your program. If you
need to leave the Python virtual environment just type the following command:</p>
<div class="highlight"><pre><span></span><code>deactivate
</code></pre></div>
<p>If you log out of the system and log back in, you can reactivate the virtual
environment as before. Just make sure you load the modules you need:</p>
<div class="highlight"><pre><span></span><code><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">NumPy</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">SciPy</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">PIL</span>
<span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">matplotlib</span>
<span class="n">source</span><span class="w"> </span><span class="n">myPy</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">activate</span>
</code></pre></div>
<p><strong>Submission Scripts</strong><br>
Now that you have your own version of Python running, you will want to use it
in your submission scripts. The following job submission script can be used as
a template for running virtualenv Python jobs (note this job assumes that your
myPy folder is in your home directory):</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash --login</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -l walltime=03:00:00,mem=2gb,nodes=1:ppn=1</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
module<span class="w"> </span>load<span class="w"> </span>NumPy
module<span class="w"> </span>load<span class="w"> </span>SciPy
module<span class="w"> </span>load<span class="w"> </span>PIL
module<span class="w"> </span>load<span class="w"> </span>matplotlib
<span class="nb">source</span><span class="w"> </span>~/myPy/bin/activate
python<span class="w"> </span>mypythonscript.py
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
</code></pre></div></td></tr></table></div>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/03/06/Using+Python+virtualenv+on+the+HPCC">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/03/06/Using+Python+virtualenv+on+the+HPCC">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013-02-19 Campus Champion Fellows Presentation2013-02-19T00:00:00-05:002013-02-19T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-02-19:/2013-02-19-campus-champion-fellows-presentation.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I am presenting the attached slides to the monthly campus champions conference
call. The call describes my experience as a campus champion fellow and my work
on the Gordon Large Scale Video Analytics project:</p>
<p>Here is a link to the …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I am presenting the attached slides to the monthly campus champions conference
call. The call describes my experience as a campus champion fellow and my work
on the Gordon Large Scale Video Analytics project:</p>
<p>Here is a link to the slides:</p>
<ul>
<li><a href="./images/2013-02-19_LSVA_Fellows_talk.pdf">2013-02-19 LSVA Fellows talk.pdf</a></li>
</ul>
<p>Here are some links in the presentation:</p>
<ul>
<li><a href="http://www.xsede.org/ccfellows-projects">http://www.xsede.org/ccfellows-projects</a></li>
<li><a href="http://medici.ncsa.illinois.edu/">http://medici.ncsa.illinois.edu/</a></li>
<li><a href="http://wiki.hpcc.msu.edu/x/X5Xh">http://wiki.hpcc.msu.edu/x/X5Xh</a></li>
</ul>
<p>- Dirk</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/02/19/2013-02-19+Campus+Champion+Fellows+Presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/02/19/2013-02-19+Campus+Champion+Fellows+Presentation">ICER Wiki</a> using custom python script. Comment on errors below.</p>Movie Color Visualization2013-02-04T00:00:00-05:002013-02-04T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-02-04:/movie-color-visualization.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>As part of my work with the Campus Champion Fellowship I have been asked to
look into methods for displaying color for an entire movie similar to the
following:</p>
<p><a href="http://www.garethwilliamsportfolio.com/?page_id=55">ColourSpectrograms</a><br>
<a href="http://moviebarcode.tumblr.com/">MovieBarCode</a></p>
<p>I tried installing the ColourSpectrograms java software but had …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>As part of my work with the Campus Champion Fellowship I have been asked to
look into methods for displaying color for an entire movie similar to the
following:</p>
<p><a href="http://www.garethwilliamsportfolio.com/?page_id=55">ColourSpectrograms</a><br>
<a href="http://moviebarcode.tumblr.com/">MovieBarCode</a></p>
<p>I tried installing the ColourSpectrograms java software but had some trouble.
So, I tried implimenting an algorithm myself. The following are my results so
far:</p>
<p>Average Image|HSV Video Colormap|Average Bar|Link to video </p>
<p>---|---|---|--- </p>
<p><img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/GlobusOnline_mean.png?version=1&modificationDate=1359575480000&api=v2">|<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/GlobusOnline_bar.png?version=1&modificationDate=1359575480000&api=v2">|<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/GlobusOnline_Abar.png?version=1&modificationDate=1359600408000&api=v2">|<a href="https://wiki.hpcc.msu.edu/display/ITH/Video+Tutorial+-+GlobusOnline.org">Video Tutorial -
GlobusOnline.org</a> </p>
<p><img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/HPCCUSB_mean.png?version=1&modificationDate=1359575480000&api=v2">|<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/HPCCUSB_Bar.png?version=1&modificationDate=1359575480000&api=v2">|<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/HPCCUSB_ABar.png?version=1&modificationDate=1359600413000&api=v2">|Video Tutorial - HPCCUSB </p>
<p><img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/ChamB_mean.png?version=1&modificationDate=1359575480000&api=v2">|<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/ChamB_Bar.png?version=1&modificationDate=1359575480000&api=v2">|<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/ChamB_ABar.png?version=1&modificationDate=1359600403000&api=v2">|No video available on-line </p>
<p>Currently the algorithm is very slow and written in Python. The colors are
based on the HSV colorspace which can be visually understood using the
following image:</p>
<p><img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411567/images.jpeg?version=1&modificationDate=1359575868000&api=v2"> Note: Image obtained from <a href="http://www.mathworks.com/help/images/converting-color-data-between-color-spaces.html">http://www.mathworks.com/help/images/converting-color-data-between-color-spaces.html</a>I use the V (value) channel to determine if a color is black then I use the H
(Hue) channel to histogram the remaining colors for each image and display the
histogram linerally by color. I think this is working but it doesn't really
show the variations in color that you see with the MovieBarCode. Most notably
is that since I do not use the S (saturation) channel there is no white or
offwhite colors in the graph. I think this is a good start but will have to
talk though the best way to proceed with the rest of the team.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/30/Movie+Color+Visualization">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/30/Movie+Color+Visualization">ICER Wiki</a> using custom python script. Comment on errors below.</p>Speeding up Scientific Imaging Workflows- Design of Automated Image Annotation Tool2013-01-16T00:00:00-05:002013-01-16T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-01-16:/speeding-up-scientific-imaging-workflows-design-of-automated-image-annotation-tool.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the slides that I presented at the <a href="http://www.openvl.org/UCCV/">User Centered Computer
Vision</a> workshop.</p>
<p><strong>Abstract:</strong><br>
Low cost digital cameras have transformed the process of collecting data.
Researchers can now generate massive datasets and analyze the images later,
either manually or …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the slides that I presented at the <a href="http://www.openvl.org/UCCV/">User Centered Computer
Vision</a> workshop.</p>
<p><strong>Abstract:</strong><br>
Low cost digital cameras have transformed the process of collecting data.
Researchers can now generate massive datasets and analyze the images later,
either manually or with the assistance of software tools for processing and
annotating images. However, it remains time-consuming and expensive to develop
custom software analysis tools for a specific research problem or domain – and
often these custom tools cannot scale to larger datasets or adapt to new
research questions. Existing image analysis tools also work best with well-
defined research projects, where the researchers know what information to
extract from each image. Yet, for new projects, it is especially difficult to
build useful software, where researchers have not yet determined what
information is “interesting” within the images. One way to increase the
efficiency of the research is to improve the workflow of this exploration
process. This paper presents one approach for improving exploratory image
analysis workflows using point-based image annotation. We describe a landmark
labeling system that (1) assists researchers in identifying interesting
features and annotating images, (2) evolves over time to automate the
annotation process, and (3) can be readily scaled and adapted to explore new
problems and new domains. We describe both a proof-of-concept system in
current use, and ongoing work to develop a generalizable software tool to
support fully automated image annotation, with the ultimate goal of allowing
researchers to explore data faster and significantly reduces the mean time to
science.</p>
<p><a href="./images/2013-01-15_UCCV.pdf">2013-01-15 UCCV.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/16/Speeding+up+Scientific+Imaging+Workflows-+Design+of+Automated+Image+Annotation+Tool">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/16/Speeding+up+Scientific+Imaging+Workflows-+Design+of+Automated+Image+Annotation+Tool">ICER Wiki</a> using custom python script. Comment on errors below.</p>2013-01-03 Central Michigan University HPCC workshop2013-01-02T00:00:00-05:002013-01-02T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2013-01-02:/2013-01-03-central-michigan-university-hpcc-workshop.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>The following slides are for the CMU HPCC Training workshop being conducted on
January 3, 2013 in room 230 Engineering and Technology Building.</p>
<p>Introduction to HPCC - 9:00-12:00 p.m.<br>
Lunch (Provided if you register) - 12:00-1:00 p …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>The following slides are for the CMU HPCC Training workshop being conducted on
January 3, 2013 in room 230 Engineering and Technology Building.</p>
<p>Introduction to HPCC - 9:00-12:00 p.m.<br>
Lunch (Provided if you register) - 12:00-1:00 p.m.<br>
Advanced HPCC - 1:00-4:00 p.m.</p>
<h3>Two Sessions</h3>
<p><strong>Introduction to the HPCC: A Hands-On Introduction to High-Performance
Computing at MSU</strong><br>
Topics include connecting to the HPCC, copying files to your home directories,
navigating the command line interface, accessing available software, testing
and running programs interactively, editing files, writing job scripts,
submitting jobs to the queue and monitoring jobs running on the system.</p>
<p><a href="./images/2013-01-03_Introduction_to_HPCC.pdf">2013-01-03 Introduction to HPCC.pdf</a></p>
<p><strong>Making Your Research Go Faster: Advanced Topics in Getting the Most Out of
the MSU HPCC Hardware</strong><br>
Topics include techniques for effective scheduling of pleasantly paralleled
jobs, shared memory jobs and shared network jobs, running jobs longer than a
week and making jobs fault tolerant.</p>
<p><a href="./images/2013-01-03_Advanced_HPCC.pdf">2013-01-03 Advanced HPCC.pdf</a></p>
<h3>Call the College of Science and Technology Dean’s Office @ (989) 774-1870</h3>
<p>to register</p>
<p>• CMU faculty and graduate students can register for the Introduction to HPCC,
Advanced HPCC or both</p>
<p><a href="./images/HPCC-Training-2013.pdf">HPCC-Training-2013.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/02/2013-01-03+Central+Michigan+University+HPCC+workshop">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/02/2013-01-03+Central+Michigan+University+HPCC+workshop">ICER Wiki</a> using custom python script. Comment on errors below.</p>2012-11-29 CHE MSE 802 Research Methods class2012-11-29T00:00:00-05:002012-11-29T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-11-29:/2012-11-29-che-mse-802-research-methods-class.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Attached are the slides I presented to the CHE/MSE 802 Research methods class.</p>
<p><a href="./images/2012-11-29_ChemE_iCER_Talk.pdf">2012-11-29 ChemE iCER Talk.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/16/2012-11-29+CHE+MSE+802+Research+Methods+class">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/16/2012-11-29+CHE+MSE+802+Research+Methods+class">ICER Wiki</a> using custom python script. Comment on errors below.</p>Building a VM suitable for distribution to a class on a USB drive2012-11-15T00:00:00-05:002012-11-15T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-11-15:/building-a-vm-suitable-for-distribution-to-a-class-on-a-usb-drive.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>I am teaching a class to first and second year undergrads. Since we will be
using many different types of software I was discouraged by the idea of
installing all of the software they …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>I am teaching a class to first and second year undergrads. Since we will be
using many different types of software I was discouraged by the idea of
installing all of the software they would need on all of their computers.
Although installing software is a very useful exercise, the many different
platforms and laptops make this task prohibitively hard to manage.</p>
<p>In the past I have used "<a href="http://portableapps.com/">Portable Apps</a>" to deliver
software to windows users. This works great and eliminates the need to install
a lot of software. However, typical portable apps do not work on macs and more
than half of my class use macs.</p>
<p>So, my next thought was to use a lightweight Virtual Machine (VM). I had some
supply funds for the students so I bought them all 16GB USB drives and my plan
is to install the VM on the drives and use the drives to also pass around the
data files the students need for their projects. I would just put all the data
files on the VM but each student needs a different set of files and they would
make the VM much bigger than the 16GB that I had available.</p>
<p>My plan is to create a USB drive with a standard Virtual Machine Image and
then Individual project data for each student. Here is the process I went
though to make these drives:</p>
<h3>Choosing a VM platform and building the machine:</h3>
<p>I created a <a href="http://lubuntu.net/">Lubuntu</a> VM using
<a href="http://www.vmware.com/">VMWare</a> and installed all of the software I wanted
the students to have. This included; Python (with a lot of packages), ImageJ,
FFmpeg and others. Then I found out that VMWare Player doesn’t run on a mac so
I had to switch to <a href="https://www.virtualbox.org/">VirtualBox</a>. I used the
following instructions to convert the VM:</p>
<p><a href="http://www.ubuntugeek.com/howto-convert-vmware-image-to-virtualbox-image.html">http://www.ubuntugeek.com/howto-convert-vmware-image-to-virtualbox-image.html</a></p>
<p><strong>Take away</strong> : Use VirtualBox, it works the best cross platform and is
available for free</p>
<h3>Copy the VM to the USB drive (first attempt):</h3>
<p>Once I had a working VM I copied over the file to the USB drive and tried to
run it. Unfortunately, trying to boot the VM gave me all types of errors and I
ended up realizing that the USB drives are formatted using FAT32 and can not
contain files larger than 2gb
<img alt="(sad)" src="https://wiki.hpcc.msu.edu/s/en_US/8100/4410012ac87e845516b70bc69b6f7a893eabaa5a/_/images/icons/emoticons/sad.svg"></p>
<p><strong>Take away</strong> : Can't use FAT32 to save large VM drive files</p>
<h3>Reformat and copy the VM to the USB drive (second attempt):</h3>
<p>So, I did some research and both mac and windows support exFAT a new files
system from Microsoft specifically designed for larger files on USB drives.
Great! I formatted the USB drive to use exFAT, copied over the VM and then
individually copied the data files each student needed onto their USB drive.</p>
<p>Problem 1: exFAT drive formatted on a MAC</p>
<p>When I passed the drives out in class today they would not be recognized by
the windows machine. It seems that exFAT only works in one direction. When I
format the drive on my MAC and try to read it on windows it doesn’t work.
However, if I format exFAT on windows my MAC can read it fine.</p>
<p>Problem 2: Missing Linux USB Drivers</p>
<p>Another problem I did not consider is that the exFAT drive can not be
recognized in the Lubuntu VM. So, although the VM worked fine on the MACs, the
students could not read their data.</p>
<h3>Fix limitations of exFAT (third attempt):</h3>
<p>We did some brainstorming and came up with the following possible solutions
that cover both problems:</p>
<p><strong>Option 1</strong>. Format exFAT on windows machines (so they will work on both MAC
and windows). Figure out how to install exFAT drivers in Lubuntu.</p>
<p>This option ended up not being too difficult. I used the following commands to
install exFAT on the Lubuntu machine:</p>
<div class="highlight"><pre><span></span><code>sudo apt-add-repository ppa:relan/exFAT
sudo apt-get update
sudo apt-get install fuse-exFAT exFAT-utils
</code></pre></div>
<p><strong>Option 2</strong>. Duel partition the USB drive with one exFAT drive (Formatted on
windows) and one FAT32 drive for the data.</p>
<p>Seems like a lot of work and I would need to switch to the disk duplicate tool
(dd) to make the copies.</p>
<p><strong>Option 3</strong>. Format the entire drive back to FAT32 and break the VM down into
smaller files.</p>
<p>I can't find a way to break the large files apart in Virtual box. I could do
it with some third party splitting software but then I would need to figure
out how to merge the files on each individual computer. Not really a step I
prefer to take.</p>
<p>The VMDK files in VMWear are already broken up into 2gb or less files. I may
be able to convert back to the VMDK files and load the VM in Virtual box
directly.</p>
<p><strong>Option 4</strong>. Try a different large-scale file system such as NTSF and get
that to work on all of the platforms.</p>
<p>I was told that this could be tricky especially if the students to not
properly eject their USB drives. So far I have avoided this option.</p>
<p><strong>Final Take Aways</strong> : Option 1 worked for me. Use VirtualBox and an exFAT
formatted USB drive. Make sure the exFAT drive is formatted in Windows and
install exFAT drivers on your Linux VM before distribution. </p>
<p>- Dirk</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/15/Building+a+VM+suitable+for+distribution+to+a+class+on+a+USB+drive">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/15/Building+a+VM+suitable+for+distribution+to+a+class+on+a+USB+drive">ICER Wiki</a> using custom python script. Comment on errors below.</p>CI-Days presentation - From Images to Data- Scaling and Streamlining Research Workflows2012-11-02T00:00:00-04:002012-11-02T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-11-02:/ci-days-presentation-from-images-to-data-scaling-and-streamlining-research-workflows.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the slides from my <a href="http://tech.msu.edu/CI-Days/">MSU CI-Days</a>
presentation.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2012-10-26_Colbry_CI-Days_Talk.pdf">2012-10-26 Colbry CI-Days Talk.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/02/CI-
Days+presentation+-+From+Images+to+Data-+Scaling+and+Streamlining+Research+Workflows">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/02/CI-Days+presentation+-+From+Images+to+Data-+Scaling+and+Streamlining+Research+Workflows">ICER Wiki</a> using custom python script. Comment on errors below.</p>HPCC Hardware presentation to BEACON Graduate students and Postdocs2012-11-02T00:00:00-04:002012-11-02T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-11-02:/hpcc-hardware-presentation-to-beacon-graduate-students-and-postdocs.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Attached is the presentation I gave to BEACON students about the HPC and it's
hardware. After the seminar we had a tour of the HPC server room.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2012-10-31_HPCC_Hardware.pdf">2012-10-31 HPCC Hardware.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/02/HPCC+Hardware+presentation+to+BEACON+Graduate+students+and+Postdocs">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/02/HPCC+Hardware+presentation+to+BEACON+Graduate+students+and+Postdocs">ICER Wiki</a> using …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Attached is the presentation I gave to BEACON students about the HPC and it's
hardware. After the seminar we had a tour of the HPC server room.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2012-10-31_HPCC_Hardware.pdf">2012-10-31 HPCC Hardware.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/02/HPCC+Hardware+presentation+to+BEACON+Graduate+students+and+Postdocs">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/02/HPCC+Hardware+presentation+to+BEACON+Graduate+students+and+Postdocs">ICER Wiki</a> using custom python script. Comment on errors below.</p>ffmpeg to select video sequences2012-11-01T00:00:00-04:002012-11-01T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-11-01:/ffmpeg-to-select-video-sequences.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>ffmpeg can be used to crop the start and end times of a video sequece. Use the
following command to read though the ffmpeg help message in the terminal
window:</p>
<div class="highlight"><pre><span></span><code> man ffmpeg
</code></pre></div>
<p>Use arrow …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>ffmpeg can be used to crop the start and end times of a video sequece. Use the
following command to read though the ffmpeg help message in the terminal
window:</p>
<div class="highlight"><pre><span></span><code> man ffmpeg
</code></pre></div>
<p>Use arrow keys to move the text up and down and use 'q' key to quit. The two
options we are interested in are as follows:</p>
<div class="highlight"><pre><span></span><code>-i filename
input file name
-ss position
Seek to given time position in seconds. "hh:mm:ss[.xxx]" syntax is also supported.
-t duration
Restrict the transcoded/captured video sequence to the duration specified in seconds.
"hh:mm:ss[.xxx]" syntax is also supported.
</code></pre></div>
<p>The following command should trim out the interesting area of the video:</p>
<div class="highlight"><pre><span></span><code> ffmpeg -i ChamC_Jul12_2011_LB3.MOV -ss "00:01:40" -t 23 ChamC_Jul12_2011_LB3.mpg
</code></pre></div>
<p>That worked for me but the quality is not right. I did some more searching of
the ffmpeg man page and came up with the following item that will impact
quality:</p>
<div class="highlight"><pre><span></span><code><span class="o">-</span><span class="nx">sameq</span>
<span class="w"> </span><span class="nx">Use</span><span class="w"> </span><span class="nx">same</span><span class="w"> </span><span class="nx">video</span><span class="w"> </span><span class="nx">quality</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nx">source</span><span class="w"> </span><span class="p">(</span><span class="nx">implies</span><span class="w"> </span><span class="nx">VBR</span><span class="p">).</span>
</code></pre></div>
<p>This seems to work better:</p>
<div class="highlight"><pre><span></span><code> ffmpeg -i ChamC_Jul12_2011_LB3.MOV -ss "00:01:40" -t 23 -sameq ChamC_Jul12_2011_LB3.mpg
</code></pre></div>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/01/ffmpeg+to+select+video+sequences">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/11/01/ffmpeg+to+select+video+sequences">ICER Wiki</a> using custom python script. Comment on errors below.</p>Image Phenomics- The Development of an Image Grammar for High-Throughput Phenotyping Using Biological Images2012-10-01T00:00:00-04:002012-10-01T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-10-01:/image-phenomics-the-development-of-an-image-grammar-for-high-throughput-phenotyping-using-biological-images.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I was excited to give a talk as part of the BEACON seminar series. Our
research group has been working on this topic for a while and it is exciting
to get all of the ideas down into one clear …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I was excited to give a talk as part of the BEACON seminar series. Our
research group has been working on this topic for a while and it is exciting
to get all of the ideas down into one clear presentation. I plan on giving a
similar talk to CI-Days later in October.</p>
<ul>
<li>Dirk</li>
</ul>
<p>Slides</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/10/01/Image+Phenomics-+The+Development+of+an+Image+Grammar+for+High-
Throughput+Phenotyping+Using+Biological+Images">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/10/01/Image+Phenomics-+The+Development+of+an+Image+Grammar+for+High-Throughput+Phenotyping+Using+Biological+Images">ICER Wiki</a> using custom python script. Comment on errors below.</p>2012-09-06 Electrical and Computer Engineering, Faculty Seminar2012-09-06T00:00:00-04:002012-09-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-09-06:/2012-09-06-electrical-and-computer-engineering-faculty-seminar.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here is a link to the talk I gave to the Electrical and Computer Engineering
Faculty at their Weekly Seminar. This is a variation on my talk about how to
utilize advanced computational resources to make your research go faster …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here is a link to the talk I gave to the Electrical and Computer Engineering
Faculty at their Weekly Seminar. This is a variation on my talk about how to
utilize advanced computational resources to make your research go faster.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/10/01/2012-09-06+Electrical+and+Computer+Engineering%2C+Faculty+Seminar">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/10/01/2012-09-06+Electrical+and+Computer+Engineering%2C+Faculty+Seminar">ICER Wiki</a> using custom python script. Comment on errors below.</p>Choosing more than one architecture type2012-08-28T00:00:00-04:002012-08-28T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-08-28:/choosing-more-than-one-architecture-type.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Although all of our nodes are binary compatible, the HPCC is a heterogeneous
cluster based built from many different types of nodes. The node types are
based on the type of architecture that the …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Although all of our nodes are binary compatible, the HPCC is a heterogeneous
cluster based built from many different types of nodes. The node types are
based on the type of architecture that the node is built on and the year the
nodes where purchased. Here is a current list:</p>
<p>Cluster name</p>
<p>|</p>
<p>Development node</p>
<p>|</p>
<p>Chip Manufacturer</p>
<p>|</p>
<p>Year Purchased</p>
<p>|</p>
<p>Node Prefix </p>
<p>---|---|---|---|--- </p>
<p>amd05</p>
<p>|</p>
<p>dev-amd05</p>
<p>|</p>
<p>AMD</p>
<p>|</p>
<p>2005</p>
<p>|</p>
<p>amd </p>
<p>intel07</p>
<p>|</p>
<p>dev-intel07</p>
<p>|</p>
<p>Intel</p>
<p>|</p>
<p>2007</p>
<p>|</p>
<p>scw </p>
<p>amd09</p>
<p>|</p>
<p>dev-amd09</p>
<p>|</p>
<p>AMD</p>
<p>|</p>
<p>2009</p>
<p>|</p>
<p>afn </p>
<p>gfx10</p>
<p>|</p>
<p>dev-gfx10</p>
<p>|</p>
<p>Intel with nVidia Graphics Card</p>
<p>|</p>
<p>2010</p>
<p>|</p>
<p>nvx </p>
<p>intel10</p>
<p>|</p>
<p>dev-intel10</p>
<p>|</p>
<p>Intel</p>
<p>|</p>
<p>2010</p>
<p>|</p>
<p>icx </p>
<p>intel11</p>
<p>|</p>
<p>n/a</p>
<p>|</p>
<p>intel</p>
<p>|</p>
<p>2011</p>
<p>|</p>
<p>ifi </p>
<p>You can specify a specific architechture in your submission script using
feature= command. For example the following will only run on the intel10 node:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash --login</span>
<span class="c1">#PBS -l nodes=1:ppn=1,mem=1gb,feature=intel10,walltime=03:59:00</span>
<span class="c1">#PBS -j oe</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="c1">#Your stuff here...</span>
</code></pre></div></td></tr></table></div>
<p>You can also specifiy multipule architechtures using the PIPE (OR) charicture.
For example the following will run on the intel07 or the intel10 nodes:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash --login</span>
<span class="c1">#PBS -l nodes=1:ppn=1,mem=500mb,feature=intel07|intel10,walltime=00:05:00</span>
<span class="c1">#PBS -j oe</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="c1">#Your stuff here...</span>
</code></pre></div></td></tr></table></div>
<p>Unfortunately, as far as I can tell, the gfx10 nodes will not work using the
OR syntax. This is probably because the gfx10 nodes also use a different
network fabric (gbe) and I have not figured out the way use more complex logic
with the feature= resource.</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/08/28/Choosing+more+than+one+architecture+type">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/08/28/Choosing+more+than+one+architecture+type">ICER Wiki</a> using custom python script. Comment on errors below.</p>Altering jobs that are in the queue2012-08-15T00:00:00-04:002012-08-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-08-15:/altering-jobs-that-are-in-the-queue.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Recently a user came by my office to complain that his jobs where not running
in the queue. This is a reasonable complainte because many of our nodes are
being drained to reboot and …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Recently a user came by my office to complain that his jobs where not running
in the queue. This is a reasonable complainte because many of our nodes are
being drained to reboot and avoid the 208 day bug. Fortunately this user had a
lot of single threaded jobs that could run on the gfx10 nodes. To run on these
nodes the user needs to change the jobs to add the feature=gbe option
(Gigabyte either-net). First I tried the following command:</p>
<div class="highlight"><pre><span></span><code>qalter -l feature=gbe $PBS_JOBID
</code></pre></div>
<p>Unfortunately,the qalter command modifies jobs in the resource manager but the
job is waiting on the scheduler. I had hoped that the scheduler would talk to
the resouce manager and update the properties but this did not happen. After
talking with others in the office we found the command to alter jobs in the
scheduler:</p>
<div class="highlight"><pre><span></span><code> mjobctl -m feature=gbe $PBS_JOBID
</code></pre></div>
<p>This worked great and propagated all changes to the resource manager. I wrote
a quick script to change all nodes currently pending in the queue to accept
add this change:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -l</span>
<span class="k">for</span><span class="w"> </span>job<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span>qselect<span class="w"> </span>-s<span class="w"> </span>Q<span class="w"> </span>-u<span class="w"> </span><span class="nv">$USER</span><span class="sb">`</span><span class="p">;</span>
<span class="k">do</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>mjobctl<span class="w"> </span>-m<span class="w"> </span><span class="nv">feature</span><span class="o">=</span>gbe<span class="w"> </span><span class="nv">$job</span>
<span class="w"> </span>mjobctl<span class="w"> </span>-m<span class="w"> </span><span class="nv">feature</span><span class="o">=</span>gbe<span class="w"> </span><span class="nv">$job</span>
<span class="k">done</span>
</code></pre></div></td></tr></table></div>
<p>I added a qalter_all_gbe command to powertools to help users do the same thing
on the hpcc.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/08/15/Altering+jobs+that+are+in+the+queue">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/08/15/Altering+jobs+that+are+in+the+queue">ICER Wiki</a> using custom python script. Comment on errors below.</p>Research Software Design Tradeoffs2012-08-02T00:00:00-04:002012-08-02T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-08-02:/research-software-design-tradeoffs.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>After the successes (and setbacks) of our five summer undergraduate research
assistants, we finally have a useable ChamView prototype that is based on my
concept of an image grammar. My work over the last week or so has been to …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>After the successes (and setbacks) of our five summer undergraduate research
assistants, we finally have a useable ChamView prototype that is based on my
concept of an image grammar. My work over the last week or so has been to
integrate the students’ work, streamline the software and write a paper
highlighting the grammar. The software is being designed for three purposes:</p>
<ul>
<li>to create an immediately useable application for observational scientists in domains such as Biology and Psychology</li>
<li>to provide a programming interface for other researchers to utilize the image grammar framework in their own areas of study</li>
<li>to support ongoing research experiments by me and my students.</li>
</ul>
<p>To address the needs of these different audiences, the ChamView software is
being developed with the following goals in mind:</p>
<ul>
<li>developing programs that are easily usable by scientists with little to no experience in computation</li>
<li>developing libraries and a framework that are useful, easy to use and easy to expand upon</li>
<li>developing robust software that can be used to run experiments and generate (publishable) results</li>
</ul>
<p>Although these goals are not mutually exclusive, they are time consuming and
the students have left for the summer. Therefore, I need to prioritize. For
instance, I could easily spend a lot of time designing elegant software – and
well-designed software should pay off in the long run. However, I also know
that software design can be a never-ending process, and at some point it is
better to grab at the lowest hanging fruit and work toward a publication.</p>
<p>Of course, a major difficult with constantly chasing after the lowest hanging
fruit in order to write the next paper is that I never really get the time to
develop the research software into a tool that, I believe, could be
foundational to advances in many different areas. My solution is to publish
software early and often, while keeping the overarching design goals in mind.</p>
<p>By focusing on developing the next evolution of the software in order to move
my experiments forward, I will have the opportunity to use the software and
hopefully improve the design over time. With this approach, I need to be
willing to rewrite core components when it becomes apparent that they are no
long useful – although this is tricky since I also need to establish and
maintain a fairly consistent standard for other users to work with.</p>
<p>We will see how well this goes,</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/08/02/Research+Software+Design+Tradeoffs">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/08/02/Research+Software+Design+Tradeoffs">ICER Wiki</a> using custom python script. Comment on errors below.</p>Summer Undergraduate Research Students2012-07-31T00:00:00-04:002012-07-31T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-07-31:/summer-undergraduate-research-students.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I had a wonderful opportunity this summer to work with five undergraduate
students on my research. The students were part of a 10 week research program
and worked on different components of the <em>ChamView</em> Project. This was the
largest group …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I had a wonderful opportunity this summer to work with five undergraduate
students on my research. The students were part of a 10 week research program
and worked on different components of the <em>ChamView</em> Project. This was the
largest group of students I have mentored at one time, and it was challenging
to coordinate all of their projects in a way that was valuable to them but
also helpful in moving the research forward. On Tuesday of this week, the
students presented their work at the Summer Undergraduate Research Forum here
at Michigan State:</p>
<ul>
<li>Brea Myers. “ <strong>The Variation of Accuracy and Precision in Ground Truth Points.</strong> ” Presentation at the MSU Summer Undergraduate Research Forum, July 2012.</li>
<li>Jeremy Martin. “ <strong>Automated Color Space Exploration in Image Processing</strong> ” Poster presentation at the MSU Summer Undergraduate Research Forum, July 2012.</li>
<li>Patrick Korth. “ <strong>Using reinforcement Learning with SIFT to Track Objects in Videos.</strong> ” Poster presentation at the MSU Summer Undergraduate Research Forum, July 2012.</li>
<li>Shiloh Jordan. “ <strong>Automated Image Measurements.</strong> ” Presentation at the MSU Summer Undergraduate Research Forum, July 2012.</li>
<li>Jassiem Ifill. “ <strong>Aiding Manual Image Annotation using A Kinematic Model.</strong> ” Presentation at the MSU Summer Undergraduate Research Forum, July 2012.</li>
</ul>
<p>Funding for these students was provided by several MSU affiliates: the
Institute for Cyber-Enabled Research; EnSURE (Engineering Summer Undergraduate
Research Experience); BEACON, an NSF Center for the study of Evolution in
Action; and SROP (Summer Research Opportunities Program).</p>
<p>Overall, I found the summer experience to be energizing and productive, and I
hope the students also found value in their research experiences. We made a
lot of good progress on ChamView, and I hope to summarize the results into a
paper or two. </p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/07/31/Summer+Undergraduate+Research+Students">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/07/31/Summer+Undergraduate+Research+Students">ICER Wiki</a> using custom python script. Comment on errors below.</p>Drosophila Tour2012-06-15T00:00:00-04:002012-06-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-06-15:/drosophila-tour.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"We had the opportunity last week to stop by the Dworkin lab for a quick tour.Dr. Ian Dworkin (<a href="https://www.msu.edu/~idworkin/index.html">link toDworkinlab</a>) is doing someinteresting work studying evolution and drosophila wings (fruit flies). He hadrows of cages, each filled with different …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"We had the opportunity last week to stop by the Dworkin lab for a quick tour.Dr. Ian Dworkin (<a href="https://www.msu.edu/~idworkin/index.html">link toDworkinlab</a>) is doing someinteresting work studying evolution and drosophila wings (fruit flies). He hadrows of cages, each filled with different lines of experiments and controls.<img alt="" src="https://wiki.hpcc.msu.edu/download/thumbnails/5411658/2012-06-07%2015.41.35.jpg?version=1&modificationDate=1339765079000&api=v2"><img alt="" src="https://wiki.hpcc.msu.edu/download/thumbnails/5411658/2012-06-07%2015.41.46.jpg?version=1&modificationDate=1339765079000&api=v2">We have been collaborating together for almost a year, and a few of my SummerUndergraduate Research students are working on projects related to Ian'simages. In some of the experiments, the researchers take images of fly wingsand try to understand how the fly’s genes impact wing growth.<img alt="" src="https://wiki.hpcc.msu.edu/download/thumbnails/5411658/2012-06-07%2015.44.58.jpg?version=1&modificationDate=1339765079000&api=v2">The following is an example of an image:<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411658/wingexample.jpg?version=1&modificationDate=1339765079000&api=v2">Here’s a brief summary of our research goals for this collaboration (takenfrom a funding proposal):<strong>Drosophila Wing Images</strong> are two-dimensional representations of afunctionally important anatomical structure. Variation in wing shape isinfluenced by sex, genotype, and rearing environments; are representative forrelatively two dimensional structures, but similar tools are easily extendedto three dimensional (e.g., most other body parts) structures. Wing shape hasbeen widely used to study the genetic architecture of complex traits (Dworkinet al 2005; Dworkin and Gibson 2006; Mezey et al 2006).Our objective is to find the underlying structure of variation in thephenotype—and to associate this variation with variables of interest tobiologists (genotype, sex, fitness)—directly from the images themselves,rather than predetermining through the selection of landmarks which aspects ofthe phenotype are “important” to be captured. Two major challenges in thisproject are (1) automatic (and thus high throughput) extraction of featuresthat captures the majority of variation in the pixel-level wing images; and(2) analysis and classification of the features on high-dimensional predictorvariables such as genotype. Thus the data will tell us which are thebiologically important features, and why they are biologically important.<a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/06/15/Drosophila+Tour">ViewOnline</a>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/06/15/Drosophila+Tour">ICER Wiki</a> using custom python script. Comment on errors below.</p>Microscope Demonstration2012-06-06T00:00:00-04:002012-06-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-06-06:/microscope-demonstration.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Our research team had an opportunity to see a demonstration of an <a href="http://www.microscopy.olympus.eu/microscopes/Life_Science_Microscopes_FluoView_FV1000MPE_-
_Multiphoton_Laser_Scanning_Microscope_.htm">Olympus
two-photon microscope
</a>.<br>
<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411725/pp_MPE_rdax_250x138.jpg?version=1&modificationDate=1338990010000&api=v2"> This impressive piece of hardware can take focused depth images into a sample.These types of scopes are used specifically with biological samples that canbe …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Our research team had an opportunity to see a demonstration of an <a href="http://www.microscopy.olympus.eu/microscopes/Life_Science_Microscopes_FluoView_FV1000MPE_-
_Multiphoton_Laser_Scanning_Microscope_.htm">Olympus
two-photon microscope
</a>.<br>
<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411725/pp_MPE_rdax_250x138.jpg?version=1&modificationDate=1338990010000&api=v2"> This impressive piece of hardware can take focused depth images into a sample.These types of scopes are used specifically with biological samples that canbe fluorescently stained. This is not a simple field microscope; the entiresystem was mounted on a large stabilized table with a fairly powerful laserand the sensitivity of the microscope required the room to be in completedarkness.Much of the presentation was outside my area of expertise. However, ofparticular interest to me was the ease at which the technician was able togenerate beautiful, high resolution images using multiple modalities. It would be fun to have a project that would allow us to play with one so Icould learn more about the technology and how it works. I hope the EngineeringSummer Undergraduate Research Experience(<a href="https://sites.google.com/a/msu.edu/egr-summer-research/">EnSURE</a>) students
found the demonstration motivating and not too technical.</p>
<p>The following is an image generated with the microscope of a rabbit tongue:<br>
<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411725/rabbit1z.png?version=1&modificationDate=1338990010000&api=v2"> Some additional info from the demonstration technician: "Red is fluorescence staining with Eosin. Green and Cyan are second harmonicgeneration imaging of collagen in the forward and reflected directions,respectively (and simultaneously with fluorescence). Excitation wavelength was
860nm and the emission was in the violet channels (430nm for SHG) (forward and
reflected) and red (reflected)."</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/06/06/Microscope+Demonstration">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/06/06/Microscope+Demonstration">ICER Wiki</a> using custom python script. Comment on errors below.</p>Introduction to ICER, Buell Lab Presentation2012-05-30T00:00:00-04:002012-05-30T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-30:/introduction-to-icer-buell-lab-presentation.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>The following are links to slides for a presentation that John Johnston and I
are giving to Dr. Robin's Buell's lab tomorrow. This presentation is a an
introduction to iCER and the HPCC that focuses on Bioinformatics and
Pleasantly Parallel …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>The following are links to slides for a presentation that John Johnston and I
are giving to Dr. Robin's Buell's lab tomorrow. This presentation is a an
introduction to iCER and the HPCC that focuses on Bioinformatics and
Pleasantly Parallel Jobs. I am providing the slides in a couple of different
formats depending on the needs of the participants.</p>
<p>2 slides per page for easier reading<br>
3 slides to a page with area for notes<br>
6 slides to a page to save paper</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/30/Introduction+to+ICER%2C+Buell+Lab+Presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/30/Introduction+to+ICER%2C+Buell+Lab+Presentation">ICER Wiki</a> using custom python script. Comment on errors below.</p>Image Stitching2012-05-29T00:00:00-04:002012-05-29T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-29:/image-stitching.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"I have a video stitching program that I wrote in MATLAB to combine twooverlapping videos into one (See figure below):<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411766/ChamStitch.png?version=1&modificationDate=1338313379000&api=v2">Combining two overlapping videos into one image is an important step in ourscientific workflow for studying chameleons. Without image stitching …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"I have a video stitching program that I wrote in MATLAB to combine twooverlapping videos into one (See figure below):<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411766/ChamStitch.png?version=1&modificationDate=1338313379000&api=v2">Combining two overlapping videos into one image is an important step in ourscientific workflow for studying chameleons. Without image stitching,researchers need to measure distances between points in different videos.Here is a link to the current MATLAB version.<a href="./images/moviestitch.m">moviestitch.m</a>The user selects a corresponding point in both images and then “stitches” themtogether using a linearly-weighted mixture of pixel values. We will call theimage in the upper left A and the image in the lower right B. The mixtureregion is the region of overlap between A and B and is the only region we needto adjust. First we calculate the shortest distance in pixels from the upperleft image and the shortest distance in pixels from the lower right image forthe overlap region. We will call these overlap shortest distances SA and SB.<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411766/MixtureA.png?version=1&modificationDate=1338313379000&api=v2"><img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411766/MixtureB.png?version=1&modificationDate=1338313379000&api=v2">Next, we sum the corresponding distance values together from SA and SB to getthe total shortest distance. This will be used to normalize the distances andreturn a value between zero and one. Summing these two images together, we getthe following:<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411766/totaldistance.png?version=1&modificationDate=1338313379000&api=v2">Now, to get the weighted ratio WA and WB, we divide SA and SB by the totaldistance. If done correctly, we should be able to add any pixel in WA with thecorresponding pixel in WB and get 1.<img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411766/AWeight.png?version=1&modificationDate=1338313379000&api=v2"><img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411766/BWeight.png?version=1&modificationDate=1338313379000&api=v2">Finally, we calculate the new pixel value for each pixel in the overlap regionby summing the original pixel values (IA and IB) to the corresponding weightedregion and adding them together: IA<em>WB + IB</em>WA.<a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/29/Image+Stitching">ViewOnline</a>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/29/Image+Stitching">ICER Wiki</a> using custom python script. Comment on errors below.</p>video size and bitrates2012-05-24T00:00:00-04:002012-05-24T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-24:/video-size-and-bitrates.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>A question that comes up in my research periodically is how much hardware is
needed to store and transmit video data? Generally, I get away with the off-
the-cuff estimate of 1 hour of video needing 1GB of space. Recently …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>A question that comes up in my research periodically is how much hardware is
needed to store and transmit video data? Generally, I get away with the off-
the-cuff estimate of 1 hour of video needing 1GB of space. Recently, I needed
some better estimates for a proposal so I did some quick digging. The
following table (found
<a href="http://www.ezs3.com/public/What_bitrate_should_I_use_when_encoding_my_video_How_do_I_optimize_my_video_for_the_web.cfm">here</a>)
is a starting point to estimate file sizes and bit rates for MP4 files.</p>
<p>Output size, Bitrate, FileSize<br>
320x240 pixels, 400 kbps, 3MB / minute<br>
480x270 pixels, 700 kbps, 5MB / minute<br>
1024x576 pixels, 1500 kbps, 11MB / minute<br>
1280x720 pixels, 2500 kbps, 19MB / minute<br>
1920x1080 pixels, 4000 kbps, 30MB / minute</p>
<p>My 1gb/hour estimate doesn't specify the resolution of the video, the audio
content, the compression algorithm used, nor how much change is occurring in
the video (which affects the amount of compression that can be done). Using
the table above, we can see that 1gb/hour comes out to a file size of about 17
MB/sec, which is about the middle point in the table - making my original 1gb-
per-hour-of-video estimate fairly reasonable.</p>
<p>Based on this table, a single camera could easily generate between 4GB and
43GB of data per day. Many research projects gather data using systems of
multiple cameras with data piped directly into a computer.</p>
<p>For example, I am working on one (small) project that has two cameras, each
running about 5 minutes for three trials a day. While this initial setup only
generates about 510MB of data per day, we would like to eventually expand the
system to 5 cameras, generating about 1GB of data per day. If we also wanted
to process the data "in real time" (i.e., as quickly as possible), then we
would need to maintain a transfer rate of around 10MB/second. Even if we
bought high-end cameras, our transfer rate would probably top out around
20MB/second when using five cameras for this project.</p>
<p>This type of estimating is useful when writing proposals or designing
experiments, because it gives a realistic idea of the type of network that
will be required to handle the data transfer in “real time.”</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/24/video+size+and+bitrates">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/24/video+size+and+bitrates">ICER Wiki</a> using custom python script. Comment on errors below.</p>keepalive script - solution to work around automounter problems2012-05-23T00:00:00-04:002012-05-23T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-23:/keepalive-script-solution-to-work-around-automounter-problems.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Sometimes the automounter does not properly mount the home directories on our
hpcc system. This is not a problem when the job first starts because there is
a epilogue script that runs before each …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Sometimes the automounter does not properly mount the home directories on our
hpcc system. This is not a problem when the job first starts because there is
a epilogue script that runs before each job that verifies the home directory
is mounted before the job starts.</p>
<p>The problem comes when a job is running for a long time with no activity in
the home directory. This may cause the automounter to automatically unmount
the drive unintentionally. Then when the drive is used again sometimes the
mount fails and then the job fails. I wrote the following script that keeps
the directory alive thought the computation and it seems to work.</p>
<p>This script is a part of our Powertools distribution</p>
<ul>
<li>
<p>Dirk</p>
<h1>!/bin/bash</h1>
<h1>DESCRIPTION Script to keep alive directory mounts</h1>
<h1>LABEL PowerJobs</h1>
<h1>Script to keep alive directory mounts</h1>
<h1>Written by Dirk Colbry</h1>
<h1></h1>
<h1>USAGE:</h1>
<h1>source keepalive PATH</h1>
<h1>... Do Something ...</h1>
<h1>kill -9 ${KEEPALIVE}</h1>
<h1></h1>
<h1>USAGE 2:</h1>
<h1>alias keepalive="source keepalive"</h1>
<h1>keepalive PATH</h1>
<h1>... Do Something ...</h1>
<h1>kill -9 ${KEEPALIVE}</h1>
<h1></h1>
<h1>USAGE 1 ON THE MSU HPCC:</h1>
<h1>module load powertools</h1>
<h1>source /opt/software/powertools/doc/keepalive</h1>
<h1>... Do Something ...</h1>
<h1>kill -9 ${KEEPALIVE}</h1>
<h1></h1>
<h1>USAGE 2 ON THE MSU HPCC:</h1>
<h1>shopt -s expand_aliases</h1>
<h1>module load powertools</h1>
<h1>keepalive PATH</h1>
<h1>... Do Something ...</h1>
<h1>kill -9 ${KEEPALIVE}</h1>
<h1></h1>
<h1>NOTE: Users are Responsible for Killing the KEEPALIVE Process when their script finishes.</h1>
<h1></h1>
<h1>Set default PATH to current PATH</h1>
<p>dir=$1
if [ "${dir}" == "" ]
then
dir="."
fi</p>
<h1>This is a two step program. The first step checks to make sure the directory can be</h1>
<h1>mounted for each node. If the directory can not be mounted then the command returns with</h1>
<h1>an error.</h1>
<p>pbsdsh -u ${PTOOLS_ROOT}/share/KeepAlive/waitformount $dir
ret=$?
echo "pbsdsh exited with ${ret}"
if [ "${ret}" -ne "0" ]
then
echo "ERROR- Unable to mount Directory"
exit 1
fi</p>
<h1>The second step puts a process on each node that cds to the directory and goes to sleep.</h1>
<h1>This extra process is designed to ensure that the node stays mounted.</h1>
<p>pbsdsh -u ${PTOOLS_ROOT}/KeepAlive/dirsleep $dir &
KEEPALIVE="${KEEPALIVE} $!" </p>
</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/23/keepalive+script+-+solution+to+work+around+automounter+problems">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/23/keepalive+script+-+solution+to+work+around+automounter+problems">ICER Wiki</a> using custom python script. Comment on errors below.</p>Custom Build Systems2012-05-15T00:00:00-04:002012-05-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-15:/custom-build-systems.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I really get frustrated with Custom Build System (CBS), especially with
software for scientific research. I define a CBS as a set of scripts/makefiles
that are written by the software developers to install their software. In my
experience, writing …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I really get frustrated with Custom Build System (CBS), especially with
software for scientific research. I define a CBS as a set of scripts/makefiles
that are written by the software developers to install their software. In my
experience, writing your own build system is never a good idea. It may work
for you and your closest friends, but chances are the build will break when
you move to a larger system or when someone wants to do something that you did
not expect (which is always the case). There's a reason they call these
"custom" systems: even if your build works great, it is customized to your
systems (hardware, software, file structure, naming conventions, libraries,
etc.). Rarely do CBS include sufficient documentation and in order to get it
to work, an experienced system administrator has to learn your "custom" way of
doing things.</p>
<p>For example, I am currently trying to install some software written for
Physicists. When I first looked at the project webpage, I noticed that the
build system was nonstandard and required its own specialized instructions for
installation. This always makes me cringe, and when the opportunity arose I
asked one of the developers why didn't they use a standard build system? The
response was that the group had an outstanding, experienced programmer who
made a custom build system, complete with a secondary installer to make
installing the support libraries even easier. The developer assured me that
"it just works."</p>
<p>This all sounded good, so I gave it a try. The first problem I encountered is
that the installer focused on single user installs (Unbuntu), and I was asked
to install the software on the HPC in a nonstandard directory. There is no
list of required libraries; instead, they want me to use the support library
installer. The problem here is that our system probably already has many of
the support components installed and it required sudo (super user) access to
the system - which I do not, and should not, use for user-level software
installs.</p>
<p>Despite my reservations, I decided to try just downloading it and following
the directions:</p>
<p>./clean ; ./install</p>
<p>This command results in a bunch of errors, ending with a big FAILED
notification. However, there is no indication from the CBS about how to fix
the problem. So, I now need to read though the CVS scripts and see if I can
figure out what is wrong. If the developers had used a standard build system,
such as autoconfig or cmake, I could have used standard debugging techniques
to try to determine the cause of the errors.</p>
<p>Fixing the larger problem, of CBS that are not robust, requires re-thinking
how we train both computer programmers and research scientists. Basic
<a href="http://software-carpentry.org/">software carpentry</a> techniques are an
essential skill for modern researchers, and professional programmers need to
be taught WHY following standards and developing robust installation packages
is important. In many cases, the software is the science - and good sciences
is all about reproducibility.</p>
<p>My advice: stick with standard build systems, such as autoconfig or
<a href="http://www.cmake.org/">cmake</a>. Follow standard build and install practices
whenever possible. In the long run, it will mean less work for you and a more
robust and powerful build system than anything custom solution.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/15/Custom+Build+Systems">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/15/Custom+Build+Systems">ICER Wiki</a> using custom python script. Comment on errors below.</p>Solution to BLCR Segmentation Bug2012-05-15T00:00:00-04:002012-05-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-15:/solution-to-blcr-segmentation-bug.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>After a few months since our install of RHEL6 and my original post reporting
the problem with BLCR:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/x/c6LT">https://wiki.hpcc.msu.edu/x/c6LT</a></li>
</ul>
<p>We have found and tested a solution that works on our system. Here is a …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>After a few months since our install of RHEL6 and my original post reporting
the problem with BLCR:</p>
<ul>
<li><a href="https://wiki.hpcc.msu.edu/x/c6LT">https://wiki.hpcc.msu.edu/x/c6LT</a></li>
</ul>
<p>We have found and tested a solution that works on our system. Here is a link
to a detailed description of the Prelink problem/BLCR and how to fix it:</p>
<ul>
<li><a href="http://en.wikipedia.org/wiki/Prelink">http://en.wikipedia.org/wiki/Prelink</a></li>
<li><a href="https://upc-bugs.lbl.gov/blcr/doc/html/FAQ.html#prelink">https://upc-bugs.lbl.gov/blcr/doc/html/FAQ.html#prelink</a></li>
</ul>
<p>Note, for a while I was not convinced that Prelinking was a problem or the
only problem because my BLCR script used the --save-all option. Using this
option seems to have also messed up our system. Removing the --save-all option
is not a problem on our system since all of our compute nodes have same image
and directory structure.</p>
<p>This has already made a lot of users on our HPC system happy because it allows
them to get around our walltime limits and makes their jobs more robust to
node failure.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/15/Solution+to+BLCR+Segmentation+Bug">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/15/Solution+to+BLCR+Segmentation+Bug">ICER Wiki</a> using custom python script. Comment on errors below.</p>Video Processing and Enthought Python2012-05-15T00:00:00-04:002012-05-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-15:/video-processing-and-enthought-python.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>One of the reasons for choosing to work with the <a href="http://www.enthought.com/">Enthought version of Python
</a> is that future users of software that we develop
do not need to download and figure out how to install all of the different
Python modules …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>One of the reasons for choosing to work with the <a href="http://www.enthought.com/">Enthought version of Python
</a> is that future users of software that we develop
do not need to download and figure out how to install all of the different
Python modules that they may need to work with. This is particularly important
to me because I want to distribute our code to researchers that may not have
expertise using Python. I was particularly excited to learn that the Enthought
distribution of Python came with the <a href="http://scikits-
image.org/">scikits-image</a> module and the <a href="http://www.pythonware.com/products/pil/">Python Image Library (PIL)
</a>. Initially, I hoped that these
packages would contain everything I need right out of the box.</p>
<p>Unfortunately, I just learned that Enthought Python is missing some libraries
that are essential to my research. In particular, scikits-image works with the
OpenCV library but does not actually require OpenCV, and OpenCV is not
included in the Enthought install. Part of my goal for this summer is to
extend the functionality of the ChamView software which is written in Python
and requires OpenCV to read and write to video files. This limitation is
frustrating because video is particular important and can be difficult to work
with. I also remember that there were different procedures for installing
OpenCV on different OS platforms.</p>
<p>Since OpenCV is not available by default in the Enthought installation, I did
some quick web searching to see if I could find any alternatives. So far I
have come up with the following possibilities:</p>
<ul>
<li>pyffmpeg <a href="http://code.google.com/p/pyffmpeg/">http://code.google.com/p/pyffmpeg/</a>.</li>
<li>PyMedia <a href="http://pymedia.org/tut/">http://pymedia.org/tut/</a></li>
<li>VTK <a href="http://www.vtk.org/">http://www.vtk.org/</a></li>
<li>pyglet <a href="http://www.pyglet.org/">http://www.pyglet.org/</a></li>
</ul>
<p>Unfortunately, these first two options are also not installed by default in
the <a href="http://www.enthought.com/products/epdlibraries.php">Enthought distribution
</a>, so they are no better
than OpenCV. VTK and pyglet are installed but I am having trouble finding
documentation and getting an example to work. It may be that my best bet is to
install OpenCV. I will probably need to logically separate my code so as to
isolate functions that require OpenCV, in case it is not available when users
are installing and using our software elsewhere.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/15/Video+Processing+and+Enthought+Python">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/15/Video+Processing+and+Enthought+Python">ICER Wiki</a> using custom python script. Comment on errors below.</p>Choosing Python2012-05-14T00:00:00-04:002012-05-14T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-14:/choosing-python.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>This summer I have some undergraduate students working with me to help
jumpstart the image phenomics research projects. Much of what we are doing
will be experimental, and we will work on separate projects that are closely
integrated. Managing this …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>This summer I have some undergraduate students working with me to help
jumpstart the image phenomics research projects. Much of what we are doing
will be experimental, and we will work on separate projects that are closely
integrated. Managing this process requires a lot of planning, and one of the
first decisions I needed to make was selecting a programming language. The
following are a few of the selection criteria I considered:</p>
<ul>
<li>Available image processing tools and libraries</li>
<li>Video decoding libraries</li>
<li>Small learning curve</li>
<li>Easy to install on new system</li>
<li>Within my research budget ($300)</li>
<li>Large support community</li>
<li>Visualization tools</li>
</ul>
<p>Given these constraints, I explored the following options:</p>
<ul>
<li>Java</li>
<li>Matlab and/or Octave</li>
<li>C/C++</li>
<li>Python</li>
</ul>
<p>Ultimately, I choose Python for a number of reasons. A few points that won me
over include:</p>
<ul>
<li>Interprative language, enabling fast software development</li>
<li><a href="http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html">IPython notebook </a>, for teaching and group work</li>
<li><a href="http://www.enthought.com/">Enthought integrated installer </a> (free to academic users), enabling a common interface and baseline scientific modules that we can all work from</li>
<li>Wide range of useful packages in image processing and software carpentry <a href="http://www.enthought.com/products/epdlibraries.php">http://www.enthought.com/products/epdlibraries.php</a></li>
<li>My general interest in understanding python better</li>
<li>Open source</li>
</ul>
<p>Although I have used Python in the past, I am not a Python expert. Some of the
reservations I have moving to Python include:</p>
<ul>
<li>WhiteSpace delimination. Python fans don't see this as a problem, but I have some reservations. However, the only way for me to know is to jump in and try it out.</li>
<li>Module loading and libraries. One of the great benefits of MATLAB is the ease of installing and loading useful research libraries. My hope is the Enthought product will get me where I want to go with Python.</li>
<li>Limited and ever changing standards. Although i assume that Python is better than C/C++, there are very few standards and every module development group is doing something different. I need reproducible science, and I worry that I will constantly be needing to check the programming packages because of constant upgrades and changes.</li>
<li>Other installation difficulties. I need a system that is easy to install on a new system because I need to be able to give the software to researchers with little or no programming experience. I do not know an easy way to package an executable that researchers can readily distribute to their students' computers.</li>
</ul>
<p>After making the decision to use Python for our summer research group, I
talked to a number of people who also voiced an interest in learing more about
Python this summer. It looks like we may even put together a weekly discussion
group to talk about all things related to scientific Python. My hope is that
by the end of the summer I will have a much better understanding of the
benefits and limitations of Python for Image Phenomics research.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/14/Choosing+Python">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/14/Choosing+Python">ICER Wiki</a> using custom python script. Comment on errors below.</p>Scientific Imaging Research Group2012-05-14T00:00:00-04:002012-05-14T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-14:/scientific-imaging-research-group.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Historically, experimental observations in science have been limited to
handwritten logs and field diaries. However, the recent influx of low cost
digital cameras allows researchers who rely on visual observations to
digitally record experiments. This has increased the amount of …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Historically, experimental observations in science have been limited to
handwritten logs and field diaries. However, the recent influx of low cost
digital cameras allows researchers who rely on visual observations to
digitally record experiments. This has increased the amount of data available
to researchers and allows researchers to time-shift their observational
processes, so that multiple cameras can continually record experiments and
researchers can review and re-review the data at their leisure. Such low-cost
cameras can gather a tremendous amount of digital data, but there is no
simple, automated method for examining this data and extracting the
information necessary for scientific measurements. Thus, researchers typically
use man-hours (hire students) to manually annotate video data frame-by-frame,
which is an extremely slow process subject to variations in quality and
detail.</p>
<p>The problem is that the amount of data produced by existing digital cameras is
many orders of magnitude larger than the scientific observations needed by the
researchers. Our research team is developing new methodologies to facilitate
the scientific process and make it more affordable to filter large amounts of
image data into observations that can be used to test research hypotheses. The
goal of this research is to find ways to improve scientific workflow so that
researchers can scale-up faster and minimize their mean time to science.</p>
<p>This summer, we are going to focus on two domains centered around image
phenomics:</p>
<ul>
<li>Working with Dr. Fred Dyer from Zoology to extend prior work we have done to develop ChamView.</li>
<li>Working with Dr. Ian Dworkin from Zoology to try to automate his work with images of fly wings.</li>
</ul>
<p>As part of these projects, we are also collaborating with Dr. Yang Wang from
Mathematics and Dr. Lifeng Weng from Statistics, who are helping to develop
new learning algorithms for these image phenomics projects.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/14/Scientific+Imaging+Research+Group">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/14/Scientific+Imaging+Research+Group">ICER Wiki</a> using custom python script. Comment on errors below.</p>Summer 2012 Research Blog Experiment2012-05-10T00:00:00-04:002012-05-10T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-10:/summer-2012-research-blog-experiment.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>During the summer of 2012, I am going to conduct a personal experiment to
evaluate the benefits of maintaining an online research blog. My goals for
this summer blogging project include:</p>
<ul>
<li>Getting back into the habit of writing down and …</li></ul><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>During the summer of 2012, I am going to conduct a personal experiment to
evaluate the benefits of maintaining an online research blog. My goals for
this summer blogging project include:</p>
<ul>
<li>Getting back into the habit of writing down and organizing my thoughts about my research.</li>
<li>Connecting with other researchers with similar interests.</li>
<li>Helping my students understand the research process at a meta-level.</li>
<li>Evaluating the benefits of research blogging to see if I will continue it past the summer.</li>
<li>Becoming more confident writing in online communities.</li>
<li>Learn more about the technology and mechanics of maintaining an online journal and explore/evaluate the different online communities such as facebook, twitter and others.</li>
</ul>
<p>In the past I have had very positive experiences keeping a research journal.
For example, I have many engineering and research notebooks from many projects
as early as middle school. During the writing of my dissertation, I also found
the book <a href="http://www.amazon.com/Writing-Your-Dissertation-Fifteen-
Minutes/dp/080504891X">“Writing your dissertation in 15 minutes a day” by Joan Bolker
</a> to be extremely valuable. Following advice from Dr.
Bolker made writing my dissertation not only bearable but also extremely
enjoyable. I recommend this book to all my students at any stage of their
writing process but especially early on in their graduate careers to help them
build the habits they need to be good writers.</p>
<p>However, since finishing my dissertation, it has been hard for me to get back
into the habit of writing and I miss it. I have started personal research
journals using different programs such as Microsoft Word, LaTeX, Evernote and
others. The problem is not the technology but my motivation to keep writing
and stay in the habit of writing. My hope is that by maintaining a research
blog this summer, I will re-establish a writing habit and perhaps be of
benefit to others who might read the blog. Some things that I am worried about
this project:</p>
<ul>
<li>I have a mild form of dyslexia and it makes me nervous to publish anything that has not been proof-read by someone else, and it’s embarrassing when people point out spelling and grammatical mistakes that I really can’t see.</li>
<li>I am curious about being scooped. Some experienced colleges have cautioned that I should keep my research ideas secret until they are published, to preserve my intellectual property. I guess I have more faith in the scientific community, but it will be interesting to see whether this proves to be a naïve perspective (or if my ideas are even worth stealing!).</li>
<li>How much time will this take? Where is the balance between gaining from time spent blogging about research, and wasting time that could be spent on doing research?</li>
<li>What are the consequences of “thinking in public”? Once a blog has been posted, it can’t be truly removed. What if I unintentionally offend someone, or say something that is so stupid it becomes negatively viral?</li>
</ul>
<p>At this point, I am going to make a commitment to write in my daily journal
every day and make a post to my research blog every week. Lets see how this
goes.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/10/Summer+2012+Research+Blog+Experiment">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/10/Summer+2012+Research+Blog+Experiment">ICER Wiki</a> using custom python script. Comment on errors below.</p>Faculty Seminars in Research and Instructional Technology2012-05-08T00:00:00-04:002012-05-08T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-05-08:/faculty-seminars-in-research-and-instructional-technology.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here are the slides for the Intro to HPCC, Intro to MATLAB and Advanced HPCC
Classes.</p>
<p><a href="./images/2012-05-08_Introduction_to_HPCC.pdf">2012-05-08 Introduction to HPCC.pdf</a><br>
<a href="./images/2012-05-08_Introduction_to_Matlab.pdf">2012-05-08 Introduction to Matlab.pdf</a><br>
<a href="./images/2012-05-09_Advanced_HPCC.pdf">2012-05-09 Advanced HPCC.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/08/Faculty+Seminars+in+Research+and+Instructional+Technology">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/08/Faculty+Seminars+in+Research+and+Instructional+Technology">ICER Wiki</a> using custom python …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here are the slides for the Intro to HPCC, Intro to MATLAB and Advanced HPCC
Classes.</p>
<p><a href="./images/2012-05-08_Introduction_to_HPCC.pdf">2012-05-08 Introduction to HPCC.pdf</a><br>
<a href="./images/2012-05-08_Introduction_to_Matlab.pdf">2012-05-08 Introduction to Matlab.pdf</a><br>
<a href="./images/2012-05-09_Advanced_HPCC.pdf">2012-05-09 Advanced HPCC.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/08/Faculty+Seminars+in+Research+and+Instructional+Technology">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/05/08/Faculty+Seminars+in+Research+and+Instructional+Technology">ICER Wiki</a> using custom python script. Comment on errors below.</p>Files as semaphores2012-04-15T00:00:00-04:002012-04-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-04-15:/files-as-semaphores.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>The following is a script that is designed to make it really easy to run a
large number of embarrassingly parallel jobs on our scheduling system. The
trick to getting this to work is …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>The following is a script that is designed to make it really easy to run a
large number of embarrassingly parallel jobs on our scheduling system. The
trick to getting this to work is to use files as <a href="http://en.wikipedia.org/wiki/Semaphore_(programming)">semaphores
</a>. The program
compares a directory of input files with a directory of flag files. If a file
is found in the input directory and not in the flag directory, then this is a
job that still needs to run and the program will create a file in the flag
directory and do the computation. The flag files are used to keep track of
what is running and what is complete.</p>
<p>Strictly speaking, many file systems are not guaranteed to be atomic so this
method is risky but in my experience it works quite well especially if you do
not care if two processes run the same job.</p>
<p><strong>"FilesAsSemaphores.qsub"</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span>
<span class="normal">48</span>
<span class="normal">49</span>
<span class="normal">50</span>
<span class="normal">51</span>
<span class="normal">52</span>
<span class="normal">53</span>
<span class="normal">54</span>
<span class="normal">55</span>
<span class="normal">56</span>
<span class="normal">57</span>
<span class="normal">58</span>
<span class="normal">59</span>
<span class="normal">60</span>
<span class="normal">61</span>
<span class="normal">62</span>
<span class="normal">63</span>
<span class="normal">64</span>
<span class="normal">65</span>
<span class="normal">66</span>
<span class="normal">67</span>
<span class="normal">68</span>
<span class="normal">69</span>
<span class="normal">70</span>
<span class="normal">71</span>
<span class="normal">72</span>
<span class="normal">73</span>
<span class="normal">74</span>
<span class="normal">75</span>
<span class="normal">76</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/sh -login</span>
<span class="c1">#PBS -l nodes=1:ppn=1,walltime=00:30:00,feature=gbe,mem=1gb</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -m a</span>
<span class="c1">#Files as semifors </span>
<span class="c1"># Script written by Dirk Colbry</span>
<span class="c1">#CHANGE the following to filter files that need to be processed</span>
<span class="nv">filefilter</span><span class="o">=</span><span class="s2">"*.do"</span>
<span class="c1">#Create a flags directory if one is not already created</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
mkdir<span class="w"> </span>-p<span class="w"> </span>flags
<span class="c1">#Check to see if job being run from an array. Space jobs so they do not run on</span>
<span class="c1">#top of each other. This is probably unessary</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span>sleep<span class="w"> </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span>
<span class="k">fi</span>
<span class="c1"># Find the most recent file not in the flags directory</span>
<span class="c1"># This indicates that the file has not been proceesed</span>
<span class="k">for</span><span class="w"> </span>file<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$filefilter</span><span class="sb">`</span><span class="w"> </span>
<span class="k">do</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span>./flags/<span class="si">${</span><span class="nv">file</span><span class="si">}</span>.flag<span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="c1">#Create the file so other jobs will not run the same program</span>
<span class="w"> </span>touch<span class="w"> </span>./flags/<span class="si">${</span><span class="nv">file</span><span class="si">}</span>.flag
<span class="w"> </span><span class="nv">dofile</span><span class="o">=</span><span class="nv">$file</span>
<span class="w"> </span><span class="k">break</span>
<span class="w"> </span><span class="k">fi</span>
<span class="k">done</span>
<span class="c1">#Check to see if dofile is empty</span>
<span class="c1">#This indicates all files have been processed</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$dofile</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"All jobs completed!"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
<span class="k">fi</span>
<span class="c1">#CHANGE the following to run the program of interest using the dofile as an input</span>
module<span class="w"> </span>load<span class="w"> </span>stata
stata-se<span class="w"> </span>-b<span class="w"> </span><span class="si">${</span><span class="nv">dofile</span><span class="si">}</span>
<span class="c1">#Save the return value of the last command</span>
<span class="nv">RET</span><span class="o">=</span><span class="nv">$?</span>
<span class="c1">#Put the job statistics on the flag file as a record and to indicate</span>
<span class="c1"># the program execited successfully.</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$RET</span><span class="s2">"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"0"</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span><span class="w"> </span>><span class="w"> </span>./flags/<span class="si">${</span><span class="nv">dofile</span><span class="si">}</span>.flag
<span class="k">else</span>
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"job with dofile=</span><span class="si">${</span><span class="nv">dofile</span><span class="si">}</span><span class="s2"> completed with error </span><span class="si">${</span><span class="nv">RET</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="nv">$RET</span>
<span class="k">fi</span>
<span class="c1">#Check to see if anything is left that needs to be done</span>
<span class="k">for</span><span class="w"> </span>file<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$filefilter</span><span class="sb">`</span><span class="w"> </span>
<span class="k">do</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span>./flags/<span class="si">${</span><span class="nv">file</span><span class="si">}</span>.flag<span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="c1">#flag missing...start another job</span>
<span class="w"> </span>qsub<span class="w"> </span>mainrun.qsub
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="nv">$RET</span><span class="w"> </span>
<span class="w"> </span><span class="k">fi</span>
<span class="k">done</span>
<span class="c1">#Nothing is left to be done... Exit</span>
<span class="c1">#Note, return value (RET) should be zero</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ALL JOBS COMPLETED!"</span>
<span class="nb">exit</span><span class="w"> </span><span class="nv">$RET</span>
</code></pre></div></td></tr></table></div>
<p>Obviously a user needs to modify the script to fit their need (this one was
designed to work with stata). To start the script you can just issue the
following command:</p>
<div class="highlight"><pre><span></span><code>qsub -t 1-149 FilesAsSemaphores.qsub
</code></pre></div>
<p>This will start 149 jobs working. When each job is done it will start another
job until all of the input files have been processed. This script could easily
be modified to work with a list of jobs in a text file or some other method
for listing what needs to be done. One nice feature of this program is that
you can look at the size of the flag files to determine if a job has completed
successfully. If you want, you can delete all of the jobs in your queue and
then delete all files of size zero and restart the entire process.</p>
<p>I do not recommend walltimes of less than 5 minutes as this will unnecessarily
flood the scheduler, please use this script at your own risk.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/04/15/Files+as+semaphores">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/04/15/Files+as+semaphores">ICER Wiki</a> using custom python script. Comment on errors below.</p>Debugging BLCR problem2012-04-10T00:00:00-04:002012-04-10T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-04-10:/debugging-blcr-problem.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>We have isolated the problem. Details of the solution can be found at the
following blog post: <a href="https://wiki.hpcc.msu.edu/x/qKXT">https://wiki.hpcc.msu.edu/x/qKXT</a></p>
<p>We had BLCR working great on our SLES10 HPC system …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>We have isolated the problem. Details of the solution can be found at the
following blog post: <a href="https://wiki.hpcc.msu.edu/x/qKXT">https://wiki.hpcc.msu.edu/x/qKXT</a></p>
<p>We had BLCR working great on our SLES10 HPC system about four months ago (I am
not sure what version of BLCR we where running). We have upgraded our system
to RHEL6.0 and unfortunately BLCR (0.8.3) no longer works. Well, it works
about 80-90% of the time and segfaults another 10-20% when doing the
cr_restart command. I have been trying to come up with a reliable test case to
submit as a bug but the intermediate nature of the problem is making it really
hard to isolate.</p>
<p>I asked the BLCR mailing list and Paul H. Hargrove was quick to reply with
some debug suggestions. One possible problem that was suggested is that having
different /usr/lib/locale/locale-archive files on different nodes could cause
a problem. I did a check using md5sum and we had two different versions of the
file installed on the system. I wrote the following submission script designed
to force a job to only start on a node with the same file. If the md5sum is
different the job just resubmitts itself:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l nodes=1:ppn=1:intel10,walltime=00:10:00,mem=2gb,feature=gbe</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -N short_test</span>
<span class="c1">#PBS -t 1</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
module<span class="w"> </span>load<span class="w"> </span>powertools
<span class="nb">shopt</span><span class="w"> </span>-s<span class="w"> </span>expand_aliases
<span class="nb">echo</span><span class="w"> </span><span class="s2">"### </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span><span class="s2"> running on </span><span class="nv">$HOSTNAME</span><span class="s2"> with </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span><span class="s2">"</span>
<span class="c1"># 4 hours * 60 minutes * 6 seconds - 60 seconds * 5 minutes</span>
<span class="c1">#export BLCR_WAIT_SEC=$(( 4 * 60 * 60 - 60 * 5))</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">PBS_JOBNAME</span><span class="o">=</span><span class="s2">"short_test"</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">BLCR_WAIT_SEC</span><span class="o">=</span><span class="m">30</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">PBS_JOBSCRIPT</span><span class="o">=</span><span class="s2">"</span><span class="nv">$0</span><span class="s2">"</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">BLCR_EMAIL</span><span class="o">=</span><span class="s2">"FALSE"</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Waiting </span><span class="si">${</span><span class="nv">BLCR_WAIT_SEC</span><span class="si">}</span><span class="s2"> seconds to run </span><span class="si">${</span><span class="nv">PBS_JOBSCRIPT</span><span class="si">}</span><span class="s2">"</span>
md5sum<span class="w"> </span>/usr/lib/locale/locale-archive<span class="w"> </span>><span class="w"> </span><span class="nv">$PBS_ARRAYID</span>.md5sum
cat<span class="w"> </span><span class="nv">$PBS_ARRAYID</span>.md5sum
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span>.orig<span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span>diff<span class="w"> </span><span class="nv">$PBS_ARRAYID</span>.md5sum<span class="w"> </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span>.orig
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$?</span><span class="s2">"</span><span class="w"> </span>!<span class="o">=</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Not the same nd3sum restarting"</span>
<span class="w"> </span>qsub<span class="w"> </span>-t<span class="w"> </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span><span class="w"> </span>-N<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBNAME</span><span class="si">}</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBSCRIPT</span><span class="si">}</span>
<span class="w"> </span>sleep<span class="w"> </span><span class="m">120</span>
<span class="w"> </span><span class="nb">exit</span>
<span class="w"> </span><span class="k">fi</span>
<span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Running for the first time"</span>
<span class="w"> </span>cp<span class="w"> </span><span class="nv">$PBS_ARRAYID</span>.md5sum<span class="w"> </span><span class="nv">$PBS_ARRAYID</span>.orig
<span class="k">fi</span>
longjob<span class="w"> </span>./NKprob.sh<span class="w"> </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span>
</code></pre></div></td></tr></table></div>
<p>Unfortunately, all this test showed me was that the difference in the
/usr/lib/locale/locale-archive files was not the problem I am debugging
(although it could be another problem).</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/04/10/Debugging+BLCR+problem">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/04/10/Debugging+BLCR+problem">ICER Wiki</a> using custom python script. Comment on errors below.</p>ASEE North Central Regional Conference Presentation2012-03-29T00:00:00-04:002012-03-29T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-03-29:/asee-north-central-regional-conference-presentation.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Over the weekend I presented a talk on the Cybergreen class that I taught for
2010-2011 school year. The paper was well recieved and even won "Best paper,
second place." Here is a link to my slides.</p>
<p><a href="./images/2012-03-24_colbrydi.pdf">2012-03-24 colbrydi.pdf …</a></p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Over the weekend I presented a talk on the Cybergreen class that I taught for
2010-2011 school year. The paper was well recieved and even won "Best paper,
second place." Here is a link to my slides.</p>
<p><a href="./images/2012-03-24_colbrydi.pdf">2012-03-24 colbrydi.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/29/ASEE+North+Central+Regional+Conference+Presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/29/ASEE+North+Central+Regional+Conference+Presentation">ICER Wiki</a> using custom python script. Comment on errors below.</p>Automatically log into a dev-node from gateway2012-03-27T00:00:00-04:002012-03-27T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-03-27:/automatically-log-into-a-dev-node-from-gateway.html<p>Tags: HPC, example, Migration</p>
<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>You are not allowed to do any computation on the gateway nodes. Mostly they
are there as a firewall and a place to check and submit jobs. Many users find …</p><p>Tags: HPC, example, Migration</p>
<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>You are not allowed to do any computation on the gateway nodes. Mostly they
are there as a firewall and a place to check and submit jobs. Many users find
it annoying to log into gateway and then have to immediately log into one of
the dev nodes. Although I have not tested it thoughouly, you can add the
following script to the beginning of your .bash_profile file and it will
automaticlly log you into a random dev node. Give it a try:</p>
<div class="highlight"><pre><span></span><code><span class="k">if</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"$HOSTNAME"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"gateway-01"</span><span class="w"> </span><span class="p">];</span><span class="w"> </span><span class="n">then</span>
<span class="w"> </span><span class="o">/</span><span class="n">opt</span><span class="o">/</span><span class="n">software</span><span class="o">/</span><span class="n">powertools</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">dev</span>
<span class="n">fi</span>
</code></pre></div>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/27/Automatically+log+into+a+dev-
node+from+gateway">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/27/Automatically+log+into+a+dev-node+from+gateway">ICER Wiki</a> using custom python script. Comment on errors below.</p>PBS quick submission script2012-03-19T00:00:00-04:002012-03-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-03-19:/pbs-quick-submission-script.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>I write a lot of submissions scripts for a lot of users on the HPCC. I find
myself using the same tricks over and over again. Recently I came up with the
following script …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>I write a lot of submissions scripts for a lot of users on the HPCC. I find
myself using the same tricks over and over again. Recently I came up with the
following script as a quick way to cover most of my job submission needs.</p>
<p>What is nice for me is I don't have to rewrite the script for every job. All
of the job options can be overwritten on the command line and the script uses
the PBS_JOBNAME to determine an executable to run in the current directory. It
also replaces any dashes in the name with spaces as a crude method to include
input arguments. Since PBS_ARRAYID is automatically appended to the end of the
PBS_JOBNAME this also allows job arrays to be used as an input variable.</p>
<p>I hope you find this useful,</p>
<ul>
<li>
<p>Dirk</p>
<h1>!/bin/bash</h1>
<h1>PBS -l nodes=1:ppn=1,walltime=08:00:00,mem=2gb,feature=gbe</h1>
<h1>PBS -j oe</h1>
<h1>PBS -m ae</h1>
<h1>PBS -N a.out</h1>
<h1>I personally use openmpi as my default mpi library and not mvapich</h1>
<p>module unload mvapich
module load openmpi</p>
<h1>Change to the original working directory where the qsub command is executed</h1>
<p>cd ${PBS_O_WORKDIR}</p>
<h1>set OMP thread is appropriate for MPI jobs</h1>
<p>export OMP_NUM_THREADS=<code>cat ${PBS_NODEFILE} | wc -l</code></p>
<h1>Run the jobname as a command in the local directory and</h1>
<h1>use the ARRAYID as the first input variable if avaliable</h1>
<h1>Uses '-' in the jobname to designate spaces</h1>
<h1></h1>
<h1>Example:</h1>
<h1></h1>
<h1>Run a simple single thread program</h1>
<h1>qsub -N myprogram quick.qsub</h1>
<h1></h1>
<h1>Run a simple program with three different input numbers</h1>
<h1>qsub -N myprogram -t 100,200,300 quick.qsub</h1>
<h1></h1>
<h1>Run a basic openmp program</h1>
<h1>qsub -N myprogram -l nodes=1:ppn=8 quick.qsub</h1>
<h1></h1>
<h1>Run a basic openmpi program</h1>
<h1>qsub -N mpirun-myprogram -l nodes=64:ppn=1,feature=ib quick.qsub</h1>
<h1>Display the execution command for debugging</h1>
<p>echo ./${PBS_JOBNAME} | sed "s/-/ /g"</p>
<h1>Run the command using the PBS_JOBNAME to determine the executable name and input variables</h1>
<p><code>echo ./${PBS_JOBNAME} | sed "s/-/ /g"</code></p>
<h1>Display the runtime and resources used for the job</h1>
<p>qstat -f ${PBS_JOBID}</p>
</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/03/19/PBS+quick+submission+script">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/03/19/PBS+quick+submission+script">ICER Wiki</a> using custom python script. Comment on errors below.</p>MATLAB on the HPCC - Mid-Morning Break Slides2012-03-06T00:00:00-05:002012-03-06T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-03-06:/matlab-on-the-hpcc-mid-morning-break-slides.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the presentation slides that I put together for the Mid-Morning
break. This week we talked about some of the many different ways to to take
advantage of MATLAB on the HPCC. I hope you find these useful.</p>
<ul>
<li>Dirk …</li></ul><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the presentation slides that I put together for the Mid-Morning
break. This week we talked about some of the many different ways to to take
advantage of MATLAB on the HPCC. I hope you find these useful.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2012-03-02_Matlab.pdf">2012-03-02 Matlab.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/06/MATLAB+on+the+HPCC+-+Mid-
Morning+Break+Slides">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/06/MATLAB+on+the+HPCC+-+Mid-Morning+Break+Slides">ICER Wiki</a> using custom python script. Comment on errors below.</p>Makefile Mystery2012-03-05T00:00:00-05:002012-03-05T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-03-05:/makefile-mystery.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>I was working with a makfile the other day and I came across a "feature" that
I was not aware of. When running my makefile I saw the following lines appear:</p>
<div class="highlight"><pre><span></span><code>cat build.sh …</code></pre></div><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>I was working with a makfile the other day and I came across a "feature" that
I was not aware of. When running my makefile I saw the following lines appear:</p>
<div class="highlight"><pre><span></span><code>cat build.sh >build
chmod a+x build
</code></pre></div>
<p>I could not find any part of the makefile that used either the cat or the
chmod command yet it seemed to work. Upon further testing I discovered that
Make will automatically convert a shell script to an executable. For example,
consider the following makefile</p>
<p><strong>"Makefile"</strong></p>
<div class="highlight"><pre><span></span><code><span class="n">all</span><span class="o">:</span><span class="w"> </span><span class="n">demo</span>
<span class="n">demo</span><span class="o">:</span><span class="w"> </span><span class="n">test</span>
<span class="w"> </span><span class="o">./</span><span class="n">test</span>
<span class="n">clean</span><span class="o">:</span>
<span class="w"> </span><span class="n">rm</span><span class="w"> </span><span class="n">test</span>
</code></pre></div>
<p>This code seems streatforward but there is no rule to build test and my clean
removes test. However, the script works fine if I include the following in the
same directory:</p>
<p><strong>"test.sh"</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"This is a test!!!!, Hello world"</span>
</code></pre></div></td></tr></table></div>
<p>Now when I run make, I get the following output:</p>
<div class="highlight"><pre><span></span><code>>make
cat test.sh >test
chmod a+x test
./test
This is a test!!!!, Hello world
</code></pre></div>
<p>This is a really neat feature but it threw me for a loop.</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/05/Makefile+Mystery">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/03/05/Makefile+Mystery">ICER Wiki</a> using custom python script. Comment on errors below.</p>Tips and Tricks for GPGPU programming on the HPCC2012-02-17T00:00:00-05:002012-02-17T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-02-17:/tips-and-tricks-for-gpgpu-programming-on-the-hpcc.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>This weeks HPCC Mid-Morning Break talk was about developing and running code
on the GPGPU. Attached are my slides.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2011-03-04_-_GPU_on_HPCC.pdf">2011-03-04 - GPU_on_HPCC.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/02/17/Tips+and+Tricks+for+GPGPU+programming+on+the+HPCC">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/02/17/Tips+and+Tricks+for+GPGPU+programming+on+the+HPCC">ICER Wiki</a> using custom python script. Comment on errors below.</p>Tips and Tricks for Shared Network (MPI) Parallel jobs2012-02-17T00:00:00-05:002012-02-17T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-02-17:/tips-and-tricks-for-shared-network-mpi-parallel-jobs.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Attached are slides from last weeks Mid-Morning Break talk. We had a
discussion on running MPI jobs on the HPCC.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2012-02-10_MPI_ON_HPCC.pdf">2012-02-10_MPI_ON_HPCC.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/02/17/Tips+and+Tricks+for+Shared+Network+%28MPI%29+Parallel+jobs">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/02/17/Tips+and+Tricks+for+Shared+Network+%28MPI%29+Parallel+jobs">ICER Wiki</a> using custom python script. Comment on errors below.</p>Tips and Tricks for Shared Memory Parallelization2012-01-28T00:00:00-05:002012-01-28T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-01-28:/tips-and-tricks-for-shared-memory-parallelization.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the slides from my mid-morning break presentation: "Tips and Tricks
for Shared Memory Parallelization: MKL, OpenMP and others".</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2012-01-27_Shared_Memory.pdf">2012-01-27 Shared Memory.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/27/Tips+and+Tricks+for+Shared+Memory+Parallelization">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/27/Tips+and+Tricks+for+Shared+Memory+Parallelization">ICER Wiki</a> using custom python script. Comment on errors below …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Below are the slides from my mid-morning break presentation: "Tips and Tricks
for Shared Memory Parallelization: MKL, OpenMP and others".</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2012-01-27_Shared_Memory.pdf">2012-01-27 Shared Memory.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/27/Tips+and+Tricks+for+Shared+Memory+Parallelization">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/27/Tips+and+Tricks+for+Shared+Memory+Parallelization">ICER Wiki</a> using custom python script. Comment on errors below.</p>Pleasantly Parallel Presentation2012-01-27T00:00:00-05:002012-01-27T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-01-27:/pleasantly-parallel-presentation.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Last week for the mid-morning break I gave a talk on different ways to manage
and run pleaseantly parallel jobs on the HPCC. Here are my slides:</p>
<p><a href="./images/2012-01-20_Plesently_Parallel.pdf">2012-01-20 Plesently Parallel.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/26/Pleasantly+Parallel+Presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/26/Pleasantly+Parallel+Presentation">ICER Wiki</a> using custom …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Last week for the mid-morning break I gave a talk on different ways to manage
and run pleaseantly parallel jobs on the HPCC. Here are my slides:</p>
<p><a href="./images/2012-01-20_Plesently_Parallel.pdf">2012-01-20 Plesently Parallel.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/26/Pleasantly+Parallel+Presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/26/Pleasantly+Parallel+Presentation">ICER Wiki</a> using custom python script. Comment on errors below.</p>Playing around with git2012-01-26T00:00:00-05:002012-01-26T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-01-26:/playing-around-with-git.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<h1>Looking at an older version</h1>
<p>Use "git checkout HASH" where HASH is that long number in the "git log" file.
You can use the command to temporarily revert to an old version in git …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<h1>Looking at an older version</h1>
<p>Use "git checkout HASH" where HASH is that long number in the "git log" file.
You can use the command to temporarily revert to an old version in git. To
come back to the master version and discard any temporary changes just run the
"git checkout master" command.</p>
<h1>pulling in changes made by a collaborator.</h1>
<p>To pull in changes made by another user in a different directory (For example
from a tar.gz file). Use the following command:</p>
<div class="highlight"><pre><span></span><code><span class="n">git</span><span class="w"> </span><span class="n">pull</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">colbrydi</span><span class="o">/</span><span class="n">UserCode</span><span class="o">/</span><span class="n">greerjos</span><span class="o">/</span><span class="n">powertools</span><span class="w"> </span><span class="k">master</span>
</code></pre></div>
<p>Check the changes and then upload them to the external repository:</p>
<div class="highlight"><pre><span></span><code>git log
git push
</code></pre></div>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/26/Playing+around+with+git">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/26/Playing+around+with+git">ICER Wiki</a> using custom python script. Comment on errors below.</p>2012-01-23 Engineering Graduate Students iCER presentation2012-01-23T00:00:00-05:002012-01-23T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-01-23:/2012-01-23-engineering-graduate-students-icer-presentation.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>The following are slides from the presentation I gave to Engineering Graduate
Students for their by-monthly professional development workshop.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2013-01-23_EGR_Grad_talk.pdf">2013-01-23 EGR Grad talk.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/24/2012-01-23+Engineering+Graduate+Students+iCER+presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/24/2012-01-23+Engineering+Graduate+Students+iCER+presentation">ICER Wiki</a> using custom python script. Comment on errors below …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>The following are slides from the presentation I gave to Engineering Graduate
Students for their by-monthly professional development workshop.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="./images/2013-01-23_EGR_Grad_talk.pdf">2013-01-23 EGR Grad talk.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/24/2012-01-23+Engineering+Graduate+Students+iCER+presentation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/24/2012-01-23+Engineering+Graduate+Students+iCER+presentation">ICER Wiki</a> using custom python script. Comment on errors below.</p>2012-01-11 XSEDE - Accessing and Using Advanced Computational Hardware to Make Your Research Go Faster2012-01-13T00:00:00-05:002012-01-13T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-01-13:/2012-01-11-xsede-accessing-and-using-advanced-computational-hardware-to-make-your-research-go-faster.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I gave this talk to <a href="http://beacon-center.org/">BEACON</a>. The talk was
broadcast to all of the BEACON centers across the US and when it is posted to
YouTube I will add a link here. For now, here are a copy of my …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>I gave this talk to <a href="http://beacon-center.org/">BEACON</a>. The talk was
broadcast to all of the BEACON centers across the US and when it is posted to
YouTube I will add a link here. For now, here are a copy of my slides:</p>
<p><a href="./images/2013-01-11_XSEDE_BEACON_Presentation.pdf">2013-01-11 XSEDE BEACON Presentation.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/13/2012-01-11+XSEDE+-+Accessing+and+Using+Advanced+Computational+Hardware+to+Make+Your+Research+Go+Faster">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2013/01/13/2012-01-11+XSEDE+-+Accessing+and+Using+Advanced+Computational+Hardware+to+Make+Your+Research+Go+Faster">ICER Wiki</a> using custom python script. Comment on errors below.</p>Research and Instructional Technology Seminars for Faculty - Introduction to MATLAB, HPCC and Advanced HPCC2012-01-05T00:00:00-05:002012-01-05T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2012-01-05:/research-and-instructional-technology-seminars-for-faculty-introduction-to-matlab-hpcc-and-advanced-hpcc.html<p>Blog post <strong>edited</strong> by Anonymous</p>
<p>I presented a three, three hour classes as part of the MSU Research and
Instructional Technology Seminars for Faculty. Although attendence as low
comparied to the fall we had a great discussion in all of the classes. Below
are copies of my slides:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411697/2012-01-04%20Introduction%20to%20HPCC.pdf?version=1&modificationDate=1325791724000&api=v2">2012-01-04 Introduction …</a></p><p>Blog post <strong>edited</strong> by Anonymous</p>
<p>I presented a three, three hour classes as part of the MSU Research and
Instructional Technology Seminars for Faculty. Although attendence as low
comparied to the fall we had a great discussion in all of the classes. Below
are copies of my slides:</p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411697/2012-01-04%20Introduction%20to%20HPCC.pdf?version=1&modificationDate=1325791724000&api=v2">2012-01-04 Introduction to
HPCC.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411697/2012-01-04%20Introduction%20to%20Matlab.pdf?version=1&modificationDate=1325791724000&api=v2">2012-01-04 Introduction to
Matlab.pdf</a></p>
<p><a href="https://wiki.hpcc.msu.edu/download/attachments/5411697/2012-01-05%20Advanced%20HPCC.pdf?version=1&modificationDate=1325791724000&api=v2">2012-01-05 Advanced
HPCC.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/05/Research+and+Instructional+Technology+Seminars+for+Faculty+-+Introduction+to+MATLAB%2C+HPCC+and+Advanced+HPCC">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2012/01/05/Research+and+Instructional+Technology+Seminars+for+Faculty+-+Introduction+to+MATLAB%2C+HPCC+and+Advanced+HPCC">ICER Wiki</a> using custom python script. Comment on errors below.</p>On Demand MakeFlow PBS script2011-11-11T00:00:00-05:002011-11-11T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-11-11:/on-demand-makeflow-pbs-script.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>We just installed <a href="http://nd.edu/~ccl/software/makeflow/">MakeFlow</a> on our
system as an easy to use workflow manager that uses the familiar "makefile"
syntax. MakeFlow uses a master node and schedules all of the work off to
worker …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>We just installed <a href="http://nd.edu/~ccl/software/makeflow/">MakeFlow</a> on our
system as an easy to use workflow manager that uses the familiar "makefile"
syntax. MakeFlow uses a master node and schedules all of the work off to
worker nodes. To get this working on our system I wanted to have an easy way
for the master node to communicate it's location and port to the worker nodes.
With the help of one of our students we came up with three basic ways for this
to work:</p>
<ul>
<li>Option 1: Schedule a master job and have it schedule worker jobs.</li>
<li>Option 2: Schedule a large job and use pbsdsh to call the worker nodes.</li>
<li>Option 3: Combine Options 1 & 2</li>
</ul>
<p>Which option you use depends a lot on how PBS is set up on your system and
there are different pros and cons to each setup. The following is a
description of how I set it up on our HPCC:</p>
<h1>Option 1: Schedule a master job and have it schedule worker jobs.</h1>
<p>In this option, the system needs to be able to schedule jobs from all of the
compute nodes. If this is the case, then it is easy to pass the host
information to the worker nodes though a system variable. The job array can
have as many single node jobs as it needs and the jobs will get scheduled as
resources become available. The downside to this approach is that the job has
to wait for the workers to be scheduled before any work can get done. Here is
an example script:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/sh -login</span>
<span class="c1">#PBS -l nodes=1:ppn=1,walltime=04:00:00</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#Set up running environment for all workers</span>
module<span class="w"> </span>load<span class="w"> </span>ImageMagick
module<span class="w"> </span>load<span class="w"> </span>cctools
<span class="nb">cd</span><span class="w"> </span><span class="nv">$PBS_O_WORKDIR</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">PORT</span><span class="o">=</span><span class="m">9123</span>
<span class="c1"># Check to see if MAINHOST is set</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">MAINHOST</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="c1">#Run Main process</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">MAINHOST</span><span class="o">=</span><span class="nv">$HOST</span>
<span class="w"> </span><span class="nv">jobid</span><span class="o">=</span><span class="sb">`</span>qsub<span class="w"> </span>-t<span class="w"> </span><span class="m">1</span>-10<span class="w"> </span>-v<span class="w"> </span>MAINHOST<span class="w"> </span>-N<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBNAME</span><span class="si">}</span><span class="w"> </span><span class="nv">$0</span><span class="sb">`</span>
<span class="w"> </span>makeflow<span class="w"> </span>-T<span class="w"> </span>wq<span class="w"> </span>example.makeflow
<span class="w"> </span><span class="c1">#clean up spawned jobs</span>
<span class="w"> </span>qdel<span class="w"> </span><span class="nv">$jobid</span>
<span class="k">else</span>
<span class="w"> </span><span class="c1">#Run Worker process</span>
<span class="w"> </span>work_queue_worker<span class="w"> </span><span class="nv">$MAINHOST</span><span class="w"> </span><span class="nv">$PORT</span>
<span class="k">fi</span>
</code></pre></div></td></tr></table></div>
<h1>Option 2: Schedule a large job and use pbsdsh to call the worker nodes.</h1>
<p>This option can be used if the system is not set up to schedule jobs from the
compute nodes. It has the added benefit of starting immediately. However, the
downside to this approach is that the scheduler needs to be able to schedule a
large block of nodes together which may cause longer queue times. Here is the
example script:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l nodes=4:ppn=1,walltime=04:00:00</span>
<span class="c1">#PBS -j oe</span>
<span class="c1"># Set up running environment for all workers</span>
module<span class="w"> </span>load<span class="w"> </span>ImageMagick
module<span class="w"> </span>load<span class="w"> </span>cctools
<span class="c1"># If no input argument is given then script is running from qsub</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="c1">#Script called from qsub</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$PBS_O_WORKDIR</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PORT</span><span class="o">=</span><span class="m">9123</span>
<span class="k">else</span>
<span class="w"> </span><span class="c1">#Script called manually from pbsdsh</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">MAINHOST</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PORT</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="c1"># Check to see if MAINHOST is set</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">MAINHOST</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">MAINHOST</span><span class="o">=</span><span class="nv">$HOST</span>
<span class="w"> </span><span class="c1">#Run Main process</span>
<span class="w"> </span>makeflow<span class="w"> </span>-T<span class="w"> </span>wq<span class="w"> </span>example.makeflow<span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="nv">PID</span><span class="o">=</span><span class="nv">$!</span>
<span class="w"> </span><span class="nv">tempfile</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>/<span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>.sh
<span class="w"> </span>cp<span class="w"> </span><span class="nv">$0</span><span class="w"> </span><span class="nv">$tempfile</span>
<span class="w"> </span>chmod<span class="w"> </span><span class="m">755</span><span class="w"> </span><span class="nv">$tempfile</span>
<span class="w"> </span>pbsdsh<span class="w"> </span><span class="nv">$tempfile</span><span class="w"> </span><span class="nv">$MAINHOST</span><span class="w"> </span><span class="nv">$PORT</span><span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="c1">#Wait for main makeflow process to finish </span>
<span class="w"> </span><span class="nb">wait</span><span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span><span class="c1">#Clean up jobs </span>
<span class="w"> </span>rm<span class="w"> </span><span class="nv">$tempfile</span>
<span class="k">else</span>
<span class="w"> </span><span class="c1">#Run Worker process</span>
<span class="w"> </span>work_queue_worker<span class="w"> </span><span class="nv">$MAINHOST</span><span class="w"> </span><span class="nv">$PORT</span>
<span class="k">fi</span>
</code></pre></div></td></tr></table></div>
<h1>Option 3: Combine Options 1 & 2</h1>
<p>The third option is to combine Options 1 & 2 together. This way the job can
get started right away and grow as additional workers get added to the queue.</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span>
<span class="normal">48</span>
<span class="normal">49</span>
<span class="normal">50</span>
<span class="normal">51</span>
<span class="normal">52</span>
<span class="normal">53</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l nodes=4:ppn=1,walltime=04:00:00</span>
<span class="c1">#PBS -j oe</span>
<span class="c1"># Set up running environment for all workers</span>
module<span class="w"> </span>load<span class="w"> </span>ImageMagick
module<span class="w"> </span>load<span class="w"> </span>cctools
<span class="c1"># If no input argument is given then script is running from qsub</span>
<span class="c1"># Note: The else statement is only needed for Option 2</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="c1">#Script called from qsub</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$PBS_O_WORKDIR</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PORT</span><span class="o">=</span><span class="m">9123</span>
<span class="k">else</span>
<span class="w"> </span><span class="c1">#Script called manually from pbsdsh</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">MAINHOST</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PORT</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"MAINHOST=</span><span class="si">${</span><span class="nv">MAINHOST</span><span class="si">}</span><span class="s2"> PORT=</span><span class="si">${</span><span class="nv">PORT</span><span class="si">}</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="c1"># Check to see if MAINHOST is set</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">MAINHOST</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">MAINHOST</span><span class="o">=</span><span class="nv">$HOST</span>
<span class="w"> </span><span class="c1">#Run Main process</span>
<span class="w"> </span>makeflow<span class="w"> </span>-T<span class="w"> </span>wq<span class="w"> </span>example.makeflow<span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="nv">PID</span><span class="o">=</span><span class="nv">$!</span>
<span class="w"> </span><span class="c1">#Option 1: Spawn workers as independent PBS jobs</span>
<span class="w"> </span><span class="nv">jobid</span><span class="o">=</span><span class="sb">`</span>qsub<span class="w"> </span>-l<span class="w"> </span><span class="nv">nodes</span><span class="o">=</span><span class="m">1</span>:ppn<span class="o">=</span><span class="m">1</span><span class="w"> </span>-t<span class="w"> </span><span class="m">1</span>-10<span class="w"> </span>-v<span class="w"> </span>MAINHOST<span class="w"> </span>-N<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBNAME</span><span class="si">}</span><span class="w"> </span><span class="nv">$0</span><span class="sb">`</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$jobid</span>
<span class="w"> </span><span class="c1">#Option 2: run workers inside current job</span>
<span class="w"> </span><span class="nv">tempfile</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>/<span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>.sh
<span class="w"> </span>cp<span class="w"> </span><span class="nv">$0</span><span class="w"> </span><span class="nv">$tempfile</span>
<span class="w"> </span>chmod<span class="w"> </span><span class="m">755</span><span class="w"> </span><span class="nv">$tempfile</span>
<span class="w"> </span>pbsdsh<span class="w"> </span><span class="nv">$tempfile</span><span class="w"> </span><span class="nv">$MAINHOST</span><span class="w"> </span><span class="nv">$PORT</span><span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="c1">#Wait for main makeflow process to finish </span>
<span class="w"> </span><span class="nb">wait</span><span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span><span class="c1">#Clean up jobs </span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Cleaning up </span><span class="nv">$jobid</span><span class="s2"> and </span><span class="nv">$tempfile</span><span class="s2">"</span>
<span class="w"> </span>qdel<span class="w"> </span><span class="nv">$jobid</span>
<span class="w"> </span>rm<span class="w"> </span><span class="nv">$tempfile</span>
<span class="k">else</span>
<span class="w"> </span><span class="c1">#Run Worker process</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"RUNNING WORKER PROCESS"</span>
<span class="w"> </span>work_queue_worker<span class="w"> </span><span class="nv">$MAINHOST</span><span class="w"> </span><span class="nv">$PORT</span>
<span class="k">fi</span>
</code></pre></div></td></tr></table></div>
<p>Future Work</p>
<p>I need to be careful with this script because I have not tested the behavior
when two master jobs end up getting scheduled on the same node and use the
same port. I will need to put in a test to make sure this doesn't happen. I
also am planning on wrapping everything into a simple command that will hide
all of the details from the user.</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/11/11/On+Demand+MakeFlow+PBS+script">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/11/11/On+Demand+MakeFlow+PBS+script">ICER Wiki</a> using custom python script. Comment on errors below.</p>New Powertool to help checkpoint jobs2011-10-06T00:00:00-04:002011-10-06T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-10-06:/new-powertool-to-help-checkpoint-jobs.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~wangx147@msu.edu">Xiaoge Wang</a></p>
<p>In a previous blog post I posted my script for automatically checkpointing
jobs using BLCR which enables us to run jobs longer than a week:</p>
<p><a href="http://wiki.hpcc.msu.edu/x/eIHT">http://wiki.hpcc.msu.edu/x/eIHT</a></p>
<p>I didn't like the complexity of the script so I created a …</p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~wangx147@msu.edu">Xiaoge Wang</a></p>
<p>In a previous blog post I posted my script for automatically checkpointing
jobs using BLCR which enables us to run jobs longer than a week:</p>
<p><a href="http://wiki.hpcc.msu.edu/x/eIHT">http://wiki.hpcc.msu.edu/x/eIHT</a></p>
<p>I didn't like the complexity of the script so I created a new Powertool to do
the same thing. I call this tool "longjob" which requires a few modifications
to your submission script. In addition to running jobs longer than a week,
using longjob with a four hour walltime has the following advantages:</p>
<ol>
<li>Run jobs with unknown walltimes</li>
<li>Run jobs on the buy-in nodes (which requires 4 hours or less walltime)</li>
<li>Enables robustness of long jobs due to hardware failure</li>
<li>Run jobs up to a maintenance window without having to wait for that window to complete</li>
</ol>
<p>The following are instructions for trying out longjob on our system. First,
you start with a a basic submission script. For example, consider the
following simple submission script:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l nodes=1:ppn=1,walltime=200:00:00,mem=2gb,feature=gbe</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -m ae</span>
<span class="nb">cd</span><span class="w"> </span><span class="nv">$PBS_O_WORKDIR</span>
<span class="nv">srcdir</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>/bin/
<span class="nv">WORK</span><span class="o">=</span>/mnt/scratch/<span class="si">${</span><span class="nv">USER</span><span class="si">}</span>/KineticSN/<span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
mkdir<span class="w"> </span>-p<span class="w"> </span><span class="si">${</span><span class="nv">WORK</span><span class="si">}</span>
<span class="c1"># Copy files to work directory</span>
cp<span class="w"> </span>-r<span class="w"> </span><span class="nv">$srcdir</span>/*<span class="w"> </span><span class="nv">$WORK</span>/
<span class="c1">#Move to the working directory</span>
<span class="nb">cd</span><span class="w"> </span><span class="nv">$WORK</span>
<span class="c1">#Run my program</span>
./SimulationTest<span class="w"> </span>-scattering_flag<span class="w"> </span><span class="m">0</span><span class="w"> </span>-weak_reaction_flag<span class="w"> </span><span class="m">0</span><span class="w"> </span>-outputVisData<span class="w"> </span><span class="m">100</span>
<span class="nv">ret</span><span class="o">=</span><span class="nv">$?</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="nb">exit</span><span class="w"> </span><span class="nv">$ret</span>
</code></pre></div></td></tr></table></div>
<p>To get longjob to work, the following modificaitons need to be made:</p>
<ol>
<li>Adjust the walltime to be shorter (I suggest 4 hours or less).</li>
<li>Wrap all setup-code that only needs to be run once in an if statement that checks for the checkpoint file (checkfile.blcr). This will ensure that the setup-code only runs the first time the script is run because the first time the script is run there should not be a checkpoint file.</li>
<li>add the "longjob" command before the command in the submission script that you want to checkpoint.</li>
<li>load the powertools module and turn on aliases. i.e. add the following lines of code to the script:<ul>
<li>shopt -s expand_aliases</li>
<li>module load powertools</li>
</ul>
</li>
<li>Set the following enviornment variables as appropriate for your job:<ul>
<li><strong>BLCR_WAIT_SEC</strong> number of seconds the job should wait before checkpointing and restarting. (should be less than your walltime, default is 3 hours and 55 minutes).</li>
<li><strong>PBS_JOBSCRIPT (required)</strong> the path and name of the jobscript to use in the restart. Typically this is the same as your main jobscript and by default you can always add the following line:</li>
<li>export PBS_JOBSCRIPT="$0"</li>
<li><strong>BLCR_OUTPUT</strong> name of the main standardout/standarderr file (Default is output.txt)</li>
<li><strong>BLCR_CHECKFILE</strong> name of the checkpoint file (Default is checkfile.blcr)</li>
</ul>
</li>
</ol>
<p>The following is a modified example script with the changes:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l nodes=1:ppn=1,walltime=04:00:00,mem=2gb,feature=gbe</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -m ae</span>
<span class="nb">shopt</span><span class="w"> </span>-s<span class="w"> </span>expand_aliases
<span class="nb">cd</span><span class="w"> </span><span class="nv">$PBS_O_WORKDIR</span>
module<span class="w"> </span>load<span class="w"> </span>powertools
<span class="c1"># 4 hours * 60 minutes * 6 seconds - 60 seconds * 5 minutes</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">BLCR_WAIT_SEC</span><span class="o">=</span><span class="k">$((</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="m">60</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="m">60</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="m">60</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="m">5</span><span class="k">))</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">PBS_JOBSCRIPT</span><span class="o">=</span><span class="s2">"</span><span class="nv">$0</span><span class="s2">"</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Waiting </span><span class="si">${</span><span class="nv">BLCR_WAIT_SEC</span><span class="si">}</span><span class="s2"> seconds to run </span><span class="si">${</span><span class="nv">PBS_JOBSCRIPT</span><span class="si">}</span><span class="s2">"</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span>checkfile.blcr<span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nv">srcdir</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>/bin/
<span class="w"> </span><span class="nv">WORK</span><span class="o">=</span>/mnt/scratch/<span class="si">${</span><span class="nv">USER</span><span class="si">}</span>/KineticSN/<span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span><span class="si">${</span><span class="nv">WORK</span><span class="si">}</span>
<span class="w"> </span><span class="c1"># Copy files to work directory</span>
<span class="w"> </span>cp<span class="w"> </span>-r<span class="w"> </span><span class="nv">$srcdir</span>/*<span class="w"> </span><span class="nv">$WORK</span>/
<span class="w"> </span><span class="c1">#Run main simulation program</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$WORK</span>
<span class="k">fi</span>
longjob<span class="w"> </span>./SimulationTest<span class="w"> </span>-scattering_flag<span class="w"> </span><span class="m">0</span><span class="w"> </span>-weak_reaction_flag<span class="w"> </span><span class="m">0</span><span class="w"> </span>-outputVisData<span class="w"> </span><span class="m">100</span>
<span class="nv">ret</span><span class="o">=</span><span class="nv">$?</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="nb">exit</span><span class="w"> </span><span class="nv">$ret</span>
</code></pre></div></td></tr></table></div>
<p>If everything works as expected, you should be able to qsub the above file and
it will resubmit itself every four hours until the job completes. Note, this
is a work in progress and I have not tested all cases. For example, one case
that could propose a problem is if the main program gets caught in a loop and
never exits, in this case the code will keep submitting itself indefinitely.</p>
<p>Please email me (colbrydi@msu.edu) if you end up using this code or if you
would like to learn more how longjob is implemented.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/10/06/New+Powertool+to+help+checkpoint+jobs">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/10/06/New+Powertool+to+help+checkpoint+jobs">ICER Wiki</a> using custom python script. Comment on errors below.</p>Presentation to Fulbright faculty Visitors from Iraq2011-08-19T00:00:00-04:002011-08-19T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-08-19:/presentation-to-fulbright-faculty-visitors-from-iraq.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>This week I had a unique oportunity to talk with a group of Fulbright faculty
from Iraq. I gave them a short presentation on how advanced computation is
used in research at MSU and then I gave them a tour …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>This week I had a unique oportunity to talk with a group of Fulbright faculty
from Iraq. I gave them a short presentation on how advanced computation is
used in research at MSU and then I gave them a tour of the HPCC server room.
Here are the slides from my presentation that I compiled from some of my other
presentation:</p>
<p><a href="./images/2011-08-18_Fullbright_Talk.pdf">2011-08-18 Fullbright Talk.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/08/19/Presentation+to+Fulbright+faculty+Visitors+from+Iraq">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/08/19/Presentation+to+Fulbright+faculty+Visitors+from+Iraq">ICER Wiki</a> using custom python script. Comment on errors below.</p>New Faculty Orientation2011-08-17T00:00:00-04:002011-08-17T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-08-17:/new-faculty-orientation.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here are the slides from my "What is ICER" talk that I put together for new
faculty. I gave the talk yesterday and I will be giving it again in a week.</p>
<p><a href="./images/2011-08-16_Faculty_Orientation.pdf">2011-08-16 Faculty Orientation.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/08/17/New+Faculty+Orientation">View
Online</a></p>
<p>Blogpost …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>Here are the slides from my "What is ICER" talk that I put together for new
faculty. I gave the talk yesterday and I will be giving it again in a week.</p>
<p><a href="./images/2011-08-16_Faculty_Orientation.pdf">2011-08-16 Faculty Orientation.pdf</a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/08/17/New+Faculty+Orientation">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/08/17/New+Faculty+Orientation">ICER Wiki</a> using custom python script. Comment on errors below.</p>Simple Multi-Command Job Array2011-06-13T00:00:00-04:002011-06-13T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-06-13:/simple-multi-command-job-array.html<p>Tags: HPC, example, Migration</p>
<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~billspat@msu.edu">Pat Bills</a> - "type"</p>
<p>It is fairly easy to map a single variable to a job array. However it gets
tricky when you have more than one variables because it is not strait forward
to expand the 1D job array. In the past …</p><p>Tags: HPC, example, Migration</p>
<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~billspat@msu.edu">Pat Bills</a> - "type"</p>
<p>It is fairly easy to map a single variable to a job array. However it gets
tricky when you have more than one variables because it is not strait forward
to expand the 1D job array. In the past, I have managed to make a script to
run over all possible combinations of two variables by flattening a 2D array
into a 1D array (see <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/05/19/Massively+Nested+Loops">Massively Nested
Loops</a>
for an example) using syntax such as the following:</p>
<div class="highlight"><pre><span></span><code>v1=`echo<span class="w"> </span>"<span class="cp">${</span><span class="n">PBS_ARRAYID</span><span class="cp">}</span><span class="w"> </span>%<span class="w"> </span><span class="cp">${</span><span class="n">cols</span><span class="cp">}</span>"<span class="w"> </span>|<span class="w"> </span>bc`
v2=`echo<span class="w"> </span>"<span class="cp">${</span><span class="n">PBS_ARRAYID</span><span class="cp">}</span><span class="w"> </span>/<span class="w"> </span><span class="cp">${</span><span class="n">cols</span><span class="cp">}</span>"<span class="w"> </span>|<span class="w"> </span>bc`
</code></pre></div>
<p>However, this code can quickly get out of hand as you get bigger sets. One
simple alternative is to just make a file that contains a list of all of the
variations that you want to run of your command. For example, say you need to
run the following programs with various different inputs:</p>
<p><strong>commands.txt</strong></p>
<div class="highlight"><pre><span></span><code>./myprogram -a 100 -z 3023
./myprogram dosomething different
./myprogram
./myprogram -s 100 -d 1
./myprogram -s 100 -d 2
./myprogram -s 100 -d 3
./myprogram -s 200
./myprogram -s 300
./myprogram -w 400
./myotherporgram
./mythirdprogram
</code></pre></div>
<p>It is fairly easy to make a program to generate your command list or even use
something like excell to generate different input parameters. Then, assuming
all of these commands take the same number of resources, you can use the
folloing following job submission script to run them all in the same job
array:</p>
<p><strong>jobarray.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l walltime=00:05:00</span>
<span class="c1">#PBS -l nodes=1:ppn=1,feature=gbe</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -t 1-100</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="nv">cmd</span><span class="o">=</span><span class="sb">`</span>tail<span class="w"> </span>-n<span class="w"> </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span><span class="w"> </span>commands.txt<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-n<span class="w"> </span><span class="m">1</span><span class="sb">`</span>
<span class="nb">echo</span><span class="w"> </span><span class="si">${</span><span class="nv">cmd</span><span class="si">}</span>
<span class="si">${</span><span class="nv">cmd</span><span class="si">}</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
</code></pre></div></td></tr></table></div>
<p>You should make sure that the array length is the same size as the number of
lines in your commands.txt file. One way to do this is to submit the job as
follows:</p>
<div class="highlight"><pre><span></span><code>qsub -t 1-`cat commands.txt | wc -l` jobarray.qsub
</code></pre></div>
<p>Hope you find this useful,</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/06/30/Simple+Multi-
Command+Job+Array">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/06/30/Simple+Multi-Command+Job+Array">ICER Wiki</a> using custom python script. Comment on errors below.</p>HFSS script2011-06-08T00:00:00-04:002011-06-08T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-06-08:/hfss-script.html<p>Blog post <strong>edited</strong> by [Pat Bills](
https://wiki.hpcc.msu.edu/display/~billspat@msu.edu</p>
<p>) - "Migration of unmigrated content due to installation of a new plugin"</p>
<p>HFSS is available on the MSU HPCC. You can run interactively or in batch mode.</p>
<p>You may run the GUI interactive using the iCER …</p><p>Blog post <strong>edited</strong> by [Pat Bills](
https://wiki.hpcc.msu.edu/display/~billspat@msu.edu</p>
<p>) - "Migration of unmigrated content due to installation of a new plugin"</p>
<p>HFSS is available on the MSU HPCC. You can run interactively or in batch mode.</p>
<p>You may run the GUI interactive using the iCER Remote Desktop Gateway. See
<a href="https://wiki.hpcc.msu.edu/pages/[viewpage.action](./images/viewpage.action)?pageId=1343579">Copy of Connecting with a Remote Desktop
Client</a></p>
<p>you must run HFSS from a development node via a terminal</p>
<p>Connect to the iCER Remote Desktop Gateway</p>
<p>Start a Terminal by clicking The Applications Menu and selecting "Terminal
Emulator"</p>
<p>Once in the terminal, connect to a Development Node, Load the module for HFSS,
and start hfss. When you run for the firs time, it will create a configuration
setup for you. This may take a bit, and appear to be stalled, but please be
patient if the system is busy.</p>
<p><strong>Start HFSS in the Terminal</strong></p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="n">billspat</span><span class="err">@</span><span class="n">rdpgw</span><span class="o">-</span><span class="mi">00</span><span class="w"> </span><span class="o">~</span><span class="p">]</span><span class="o">$</span><span class="w"> </span><span class="n">ssh</span><span class="w"> </span><span class="n">dev</span><span class="o">-</span><span class="n">intel14</span>
<span class="p">[</span><span class="n">billspat</span><span class="err">@</span><span class="n">dev</span><span class="o">-</span><span class="n">intel14</span><span class="w"> </span><span class="o">~</span><span class="p">]</span><span class="o">$</span><span class="w"> </span><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">hfss</span>
<span class="n">Lmod</span><span class="w"> </span><span class="n">Warning</span><span class="p">:</span><span class="w"> </span><span class="n">hfss</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">found</span><span class="p">,</span><span class="w"> </span><span class="n">loading</span><span class="p">:</span><span class="w"> </span><span class="n">HFSS</span><span class="o">/</span><span class="mf">15.0</span>
<span class="p">[</span><span class="n">billspat</span><span class="err">@</span><span class="n">eval</span><span class="o">-</span><span class="n">k40</span><span class="w"> </span><span class="o">~</span><span class="p">]</span><span class="o">$</span><span class="w"> </span><span class="n">hfss</span>
<span class="n">ANSYS</span><span class="w"> </span><span class="n">Electromagnetics</span><span class="w"> </span><span class="mf">15.0</span><span class="w"> </span><span class="n">Configuration</span>
<span class="o">=========================================</span>
<span class="n">Hostname</span><span class="p">:</span><span class="w"> </span><span class="n">eval</span><span class="o">-</span><span class="n">k40</span>
<span class="n">User</span><span class="p">:</span><span class="w"> </span><span class="n">billspat</span>
<span class="o">></span><span class="w"> </span><span class="n">Running</span><span class="w"> </span><span class="n">first</span><span class="o">-</span><span class="n">time</span><span class="w"> </span><span class="n">configuration</span><span class="o">...</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">***</span><span class="w"> </span><span class="n">Skipping</span><span class="w"> </span><span class="n">dependency</span><span class="w"> </span><span class="n">verification</span><span class="w"> </span><span class="n">test</span><span class="w"> </span><span class="o">***</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Retrieving</span><span class="w"> </span><span class="n">user</span><span class="w"> </span><span class="n">settings</span><span class="o">...</span><span class="w"> </span><span class="n">Done</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Applying</span><span class="w"> </span><span class="n">user</span><span class="w"> </span><span class="n">settings</span><span class="o">...</span><span class="w"> </span><span class="n">Done</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Configuring</span><span class="w"> </span><span class="n">OCX</span><span class="w"> </span><span class="n">files</span><span class="o">...</span><span class="w"> </span><span class="n">Done</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Retrieving</span><span class="w"> </span><span class="n">machine</span><span class="w"> </span><span class="n">settings</span><span class="o">...</span><span class="w"> </span><span class="n">Done</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Applying</span><span class="w"> </span><span class="n">machine</span><span class="w"> </span><span class="n">settings</span><span class="o">...</span><span class="w"> </span><span class="n">Done</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Configuring</span><span class="w"> </span><span class="n">binaries</span><span class="o">...</span><span class="w"> </span><span class="n">Done</span>
<span class="n">First</span><span class="o">-</span><span class="n">time</span><span class="w"> </span><span class="n">configuration</span><span class="w"> </span><span class="n">completed</span><span class="w"> </span><span class="n">successfully</span><span class="o">.</span>
</code></pre></div>
<p>At which point the GUI/ Window will start. It will ask you a question about
which directories you'd like to use. Please change the scratch folder:</p>
<p><img alt="" src="https://wiki.hpcc.msu.edu/download/attachments/5411556/hfss-folderchoice.png?version=1&modificationDate=1441138478000&api=v2">The default scratch directory is simply /mnt/scratch. We suggest you createyour own folder on scratch, and enter that folder here. You have to firstcheck the "override" box and then type in your scratch folder.You may also use Batch mode to run HFSS jobs on the cluster. Here is anexample HFSS 15.0 submission script written by Andrew Temme that sets up theHFSS job options. For more detail see his excellent <a href="http://nbviewer.ipython.org/urls/bitbucket.org/temmeand/example-ipython-notebooks/raw/master/MSU-supercomputer-HFSS-and-IPython.ipynb">Python Notebook onHFSS</a></p>
<p><strong>hfssrun.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l walltime=03:59:00,mem=50gb,nodes=19:ppn=1</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -m abe</span>
<span class="c1">#PBS -W x=gres:hfss_solve</span>
module<span class="w"> </span>load<span class="w"> </span>HFSS/15.0
<span class="nb">export</span><span class="w"> </span><span class="nv">OptFile</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>/Options.txt
<span class="nb">export</span><span class="w"> </span><span class="nv">ANSYSEM_JOB_ID</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">ANSYSEM_HOST_FILE</span><span class="o">=</span><span class="nv">$PBS_NODEFILE</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">LINUX_SSH_BINARY_PATH</span><span class="o">=</span>/usr/bin
<span class="nb">export</span><span class="w"> </span><span class="nv">ANSYSEM_LINUX_HPC_UTILS</span><span class="o">=</span>/opt/software/AnsysEM/15.0/AnsysEM15.0/Linux64/schedulers/utils
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="c1"># mkdir -p ${PBS_O_WORKDIR}/scratch</span>
<span class="nb">echo</span><span class="w"> </span>creating<span class="w"> </span>batch<span class="w"> </span>options<span class="w"> </span>list
<span class="nb">echo</span><span class="w"> </span><span class="se">\$</span>begin<span class="w"> </span><span class="se">\'</span>Config<span class="se">\'</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS/NumCoresPerDistributedTask<span class="se">\'</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_NUM_PPN</span><span class="si">}</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS/HPCLicenseType<span class="se">\'</span><span class="o">=</span><span class="se">\'</span>Pool<span class="se">\'</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS/SolveAdaptiveOnly<span class="se">\'</span><span class="o">=</span><span class="m">0</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS/MPIVendor<span class="se">\'</span><span class="o">=</span><span class="w"> </span><span class="se">\'</span>Intel<span class="se">\'</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS-IE/NumCoresPerDistributedTask<span class="se">\'</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_NUM_PPN</span><span class="si">}</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS-IE/HPCLicenseType<span class="se">\'</span><span class="o">=</span><span class="se">\'</span>Pool<span class="se">\'</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS-IE/SolveAdaptiveOnly<span class="se">\'</span><span class="o">=</span><span class="m">0</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\'</span>HFSS-IE/MPIVendor<span class="se">\'</span><span class="o">=</span><span class="se">\'</span>Intel<span class="se">\'</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
<span class="c1"># echo \'tempdirectory\'=\'${PBS_O_WORKDIR}/scratch\' >> ${OptFile}</span>
<span class="nb">echo</span><span class="w"> </span><span class="se">\$</span>end<span class="w"> </span><span class="se">\'</span>Config<span class="se">\'</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
chmod<span class="w"> </span><span class="m">777</span><span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span>
hfss<span class="w"> </span>-ng<span class="w"> </span>-monitor<span class="w"> </span>-distributed<span class="w"> </span>-machinelist<span class="w"> </span><span class="nv">num</span><span class="o">=</span><span class="si">${</span><span class="nv">PBS_NUM_NODES</span><span class="si">}</span><span class="w"> </span>-batchoptions<span class="w"> </span><span class="si">${</span><span class="nv">OptFile</span><span class="si">}</span><span class="w"> </span>-BatchSolve<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBNAME</span><span class="si">}</span>.hfss
</code></pre></div></td></tr></table></div>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/06/08/HFSS+script">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/06/08/HFSS+script">ICER Wiki</a> using custom python script. Comment on errors below.</p>Monitoring Job overutilization2011-04-28T00:00:00-04:002011-04-28T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-04-28:/monitoring-job-overutilization.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>This week I was debugging some user code that was over-utilizing a compute
node. The job was intended to use only 1 cpu but one of the job's libraries
ended up using all the …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>This week I was debugging some user code that was over-utilizing a compute
node. The job was intended to use only 1 cpu but one of the job's libraries
ended up using all the cpus on the node. I needing to run a lot of tests to
see what exactly was causing the problem. Since I didn't want my tests to
over-utilize the nodes (too much), I wrote the following job script that will
run a monitor and kill the job if it goes to high over the cpu utilization:</p>
<h2><strong>overutilize.qsub</strong></h2>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1">#!/bin/bash</span>
<span class="w"> </span><span class="c1">#PBS -l nodes=1:ppn=1,walltime=168:00:00,mem=2gb,feature=gbe</span>
<span class="w"> </span><span class="c1">#PBS -j oe</span>
<span class="w"> </span><span class="c1">#Change to current working directory</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="w"> </span><span class="c1">#Copy the entire testing directory into its own folder</span>
<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="w"> </span>cp<span class="w"> </span>-r<span class="w"> </span>./testdir/*<span class="w"> </span>./<span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="w"> </span><span class="c1">#Make the name of the executable unique so that more than one test can run on the same node</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">name</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"ex+</span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>cut<span class="w"> </span>-d<span class="w"> </span><span class="s2">"."</span><span class="w"> </span>-f<span class="w"> </span><span class="m">1</span><span class="sb">`</span>
<span class="w"> </span>ln<span class="w"> </span>-s<span class="w"> </span>testprogram<span class="w"> </span><span class="si">${</span><span class="nv">name</span><span class="si">}</span>
<span class="w"> </span><span class="c1">#run the testprogram using the new name (including input arguments)</span>
<span class="w"> </span>./<span class="si">${</span><span class="nv">name</span><span class="si">}</span><span class="w"> </span><span class="m">2</span>.5<span class="w"> </span>-15<span class="w"> </span>-7.5<span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PID</span><span class="o">=</span><span class="nv">$!</span>
<span class="w"> </span><span class="c1">#Wait for job to get going</span>
<span class="w"> </span>sleep<span class="w"> </span><span class="m">60</span>
<span class="w"> </span><span class="c1"># Start job monitor</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">per</span><span class="o">=</span><span class="s2">"0"</span>
<span class="w"> </span><span class="o">(</span>
<span class="w"> </span><span class="c1">#Ensure job does not go over the 120% limit</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$per</span><span class="s2">"</span><span class="w"> </span>-lt<span class="w"> </span><span class="s2">"120"</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="c1">#pause between checking</span>
<span class="w"> </span>sleep<span class="w"> </span><span class="m">22</span>
<span class="w"> </span><span class="c1">#Run top in batch mode but with only one iteration</span>
<span class="w"> </span><span class="c1"># Pick out job with the unique executable name and grab the CPU utilization (9th item)</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">per</span><span class="o">=</span><span class="sb">`</span>top<span class="w"> </span>-b<span class="w"> </span>-n<span class="w"> </span><span class="m">1</span><span class="w"> </span>-u<span class="w"> </span><span class="si">${</span><span class="nv">USER</span><span class="si">}</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="si">${</span><span class="nv">name</span><span class="si">}</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="o">{</span><span class="s1">'print $9'</span><span class="o">}</span><span class="sb">`</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"per=</span><span class="nv">$per</span><span class="s2">"</span>
<span class="w"> </span><span class="k">done</span>
<span class="w"> </span><span class="nb">kill</span><span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Killed </span><span class="nv">$PID</span><span class="s2">"</span>
<span class="w"> </span><span class="o">)</span><span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="nv">wpid</span><span class="o">=</span><span class="nv">$!</span>
<span class="w"> </span><span class="c1">#Wait for job to complete</span>
<span class="w"> </span><span class="nb">wait</span><span class="w"> </span><span class="nv">$PID</span>
<span class="w"> </span><span class="c1">#Kill off wait command if it is still running</span>
<span class="w"> </span><span class="nb">kill</span><span class="w"> </span><span class="nv">$wpid</span>
<span class="w"> </span><span class="c1">#Display all the stats for the job</span>
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
</code></pre></div>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/04/28/Monitoring+Job+overutilization">ICER Wiki</a> using custom python script. Comment on errors below.</p>Running jobs longer than one week using BLCR2011-04-21T00:00:00-04:002011-04-21T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-04-21:/running-jobs-longer-than-one-week-using-blcr.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Our submission system is set up with a maximum walltime of one week. This
works fine for most users but sometimes it is nice to be able to run a job
even longer. The …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Our submission system is set up with a maximum walltime of one week. This
works fine for most users but sometimes it is nice to be able to run a job
even longer. The following script uses <a href="https://ftg.lbl.gov/CheckpointRestart/CheckpointRestart.shtml">Berkley Lab Checkpoint Restart
(BLCR)</a> to
automatically save the current state of a job and submit it back to the
scheduler.</p>
<p><strong>long_job.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span>
<span class="normal">48</span>
<span class="normal">49</span>
<span class="normal">50</span>
<span class="normal">51</span>
<span class="normal">52</span>
<span class="normal">53</span>
<span class="normal">54</span>
<span class="normal">55</span>
<span class="normal">56</span>
<span class="normal">57</span>
<span class="normal">58</span>
<span class="normal">59</span>
<span class="normal">60</span>
<span class="normal">61</span>
<span class="normal">62</span>
<span class="normal">63</span>
<span class="normal">64</span>
<span class="normal">65</span>
<span class="normal">66</span>
<span class="normal">67</span>
<span class="normal">68</span>
<span class="normal">69</span>
<span class="normal">70</span>
<span class="normal">71</span>
<span class="normal">72</span>
<span class="normal">73</span>
<span class="normal">74</span>
<span class="normal">75</span>
<span class="normal">76</span>
<span class="normal">77</span>
<span class="normal">78</span>
<span class="normal">79</span>
<span class="normal">80</span>
<span class="normal">81</span>
<span class="normal">82</span>
<span class="normal">83</span>
<span class="normal">84</span>
<span class="normal">85</span>
<span class="normal">86</span>
<span class="normal">87</span>
<span class="normal">88</span>
<span class="normal">89</span>
<span class="normal">90</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/sh -login</span>
<span class="c1">#PBS -j oe</span>
<span class="c1">#PBS -l nodes=1:ppn=1,walltime=24:10:00,mem=2gb</span>
<span class="c1">#PBS -m a</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
<span class="c1">#Berkly Lab Checkpoint Restart script to run a job continuously</span>
<span class="c1">#Written by Dirk Colbry</span>
<span class="c1">#Job restarts itself every 24 hours or 86400 seconds</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">walltime</span><span class="o">=</span><span class="s2">"86400"</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">output</span><span class="o">=</span><span class="s2">"output.txt"</span>
<span class="c1"># Name if main checkpoint file</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">checkpoint</span><span class="o">=</span><span class="s2">"checkfile.blcr"</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Running for the first time"</span>
<span class="w"> </span><span class="c1">#### SET UP JOB,</span>
<span class="w"> </span><span class="c1">#Runs once. Include any job setup commands inside this if block before the cr_run command.</span>
<span class="w"> </span><span class="c1">#Replace the program "supernova 1000" with your program and input arguments</span>
<span class="w"> </span>cr_run<span class="w"> </span>./supernova<span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="m">1</span>><span class="w"> </span><span class="si">${</span><span class="nv">output</span><span class="si">}</span><span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PID</span><span class="o">=</span><span class="nv">$!</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">next</span><span class="o">=</span><span class="m">1</span>
<span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Restarting </span><span class="si">${</span><span class="nv">PBS_ARRAYID</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="c1">#Job running as a restart job</span>
<span class="w"> </span>cr_restart<span class="w"> </span>--no-restore-pid<span class="w"> </span><span class="si">${</span><span class="nv">checkpoint</span><span class="si">}</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">output</span><span class="si">}</span><span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="w"> </span><span class="p">&</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PID</span><span class="o">=</span><span class="nv">$!</span>
<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">next</span><span class="o">=</span><span class="k">$((</span><span class="nv">$PBS_ARRAYID</span><span class="o">+</span><span class="m">1</span><span class="k">))</span>
<span class="k">fi</span>
<span class="c1">#function to run if the program times out</span>
checkpoint_timeout<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Timeout. Checkpointing Job"</span>
<span class="w"> </span><span class="nb">time</span><span class="w"> </span>cr_checkpoint<span class="w"> </span>--term<span class="w"> </span><span class="si">${</span><span class="nv">PID</span><span class="si">}</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"**********"</span>
<span class="w"> </span>tail<span class="w"> </span><span class="si">${</span><span class="nv">output</span><span class="si">}</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">""</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"**********"</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span><span class="s2">"</span><span class="nv">$?</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"0"</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Failed to checkpoint"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">2</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Queueing Next Job"</span>
<span class="w"> </span>chmod<span class="w"> </span><span class="m">644</span><span class="w"> </span>context.<span class="si">${</span><span class="nv">PID</span><span class="si">}</span>
<span class="w"> </span>mv<span class="w"> </span>context.<span class="si">${</span><span class="nv">PID</span><span class="si">}</span><span class="w"> </span><span class="si">${</span><span class="nv">checkpoint</span><span class="si">}</span>
<span class="w"> </span>qsub<span class="w"> </span>-t<span class="w"> </span><span class="si">${</span><span class="nv">next</span><span class="si">}</span><span class="w"> </span>long_job.qsub
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
<span class="o">}</span>
<span class="c1">#set checkpoint timeout</span>
<span class="o">(</span>sleep<span class="w"> </span><span class="si">${</span><span class="nv">walltime</span><span class="si">}</span><span class="p">;</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s1">'Timer Done'</span><span class="p">;</span><span class="w"> </span>checkpoint_timeout<span class="p">;</span><span class="o">)</span><span class="w"> </span><span class="p">&</span>
<span class="nv">timeout</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"starting timer (</span><span class="si">${</span><span class="nv">timeout</span><span class="si">}</span><span class="s2">) for </span><span class="si">${</span><span class="nv">walltime</span><span class="si">}</span><span class="s2"> seconds"</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Waiting on </span><span class="nv">$PID</span><span class="s2">"</span>
<span class="nb">wait</span><span class="w"> </span><span class="si">${</span><span class="nv">PID</span><span class="si">}</span>
<span class="nv">RET</span><span class="o">=</span><span class="nv">$?</span>
<span class="c1">#Check to see if job checkpointed</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">RET</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"143"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="c1">#Job terminated due to cr_checkpoint</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Job seems to have been checkpointed, waiting for checkpoint to complete."</span>
<span class="w"> </span><span class="nb">wait</span><span class="w"> </span><span class="si">${</span><span class="nv">timeout</span><span class="si">}</span>
<span class="w"> </span>qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
<span class="k">fi</span>
<span class="c1">## JOB completed</span>
<span class="c1">#Kill timeout timer</span>
<span class="nb">kill</span><span class="w"> </span><span class="si">${</span><span class="nv">timeout</span><span class="si">}</span>
<span class="c1">#Output the job statistics</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span>
<span class="c1">#Email the user that the job has completed</span>
qstat<span class="w"> </span>-f<span class="w"> </span><span class="si">${</span><span class="nv">PBS_JOBID</span><span class="si">}</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>mail<span class="w"> </span>-s<span class="w"> </span><span class="s2">"JOB COMPLETE"</span><span class="w"> </span><span class="nv">$USER</span>@msu.edu
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Job completed with exit status </span><span class="si">${</span><span class="nv">RET</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">exit</span><span class="w"> </span><span class="m">254</span>
</code></pre></div></td></tr></table></div>
<p>The job will keep submitting itself until the code exits successfully.</p>
<p>Some limitations of this script include:</p>
<ul>
<li>This one uses Job array as an iteration flag so the script will not work in a job array</li>
<li>Currently only been tested on single thread jobs.</li>
</ul>
<p>A simple modification to this script could be made such that each job runs
less than 4 hours. This would allow them to run on the buy-in nodes.</p>
<p>Hope you find this useful,</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/04/21/Running+jobs+longer+than+one+week+using+BLCR">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/04/21/Running+jobs+longer+than+one+week+using+BLCR">ICER Wiki</a> using custom python script. Comment on errors below.</p>Invited Talk at Hope College2011-04-15T00:00:00-04:002011-04-15T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-04-15:/invited-talk-at-hope-college.html<p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>This week I was invited by Dr. Ryan McFall in the <a href="http://www.hope.edu/">Hope College
</a> CSE department to give a seminar talk about using
advanced computational hardware. This time I reordered my slides to be more
example focused and I think it …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migrated to Confluence 4.0"</p>
<p>This week I was invited by Dr. Ryan McFall in the <a href="http://www.hope.edu/">Hope College
</a> CSE department to give a seminar talk about using
advanced computational hardware. This time I reordered my slides to be more
example focused and I think it made for a better presentation. Here is a copy
of the slides if anyone is interested:</p>
<p><a href="./images/2011-04-14_-_Hope_Talk.pdf">2011-04-14 - Hope Talk.pdf</a></p>
<p><a href="http://vimeo.com/22438309">Link to a video of my talk </a></p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/04/15/Invited+Talk+at+Hope+College">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/04/15/Invited+Talk+at+Hope+College">ICER Wiki</a> using custom python script. Comment on errors below.</p>Condor compile of Avida2011-03-11T00:00:00-05:002011-03-11T00:00:00-05:00Dr. Dirk Colbrytag:colbrydi.github.io,2011-03-11:/condor-compile-of-avida.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Today I figured got a condor_compile version of
<a href="http://avida.devosoft.org/">Avida</a> working on
<a href="http://www.teragrid.org/">TeraGrid</a>. First, I downloaded Avida to my Teragrid
account:</p>
<div class="highlight"><pre><span></span><code>> svn checkout https://avida.devosoft.org/svn/stable
</code></pre></div>
<p>Then I modified the build_avida script …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Today I figured got a condor_compile version of
<a href="http://avida.devosoft.org/">Avida</a> working on
<a href="http://www.teragrid.org/">TeraGrid</a>. First, I downloaded Avida to my Teragrid
account:</p>
<div class="highlight"><pre><span></span><code>> svn checkout https://avida.devosoft.org/svn/stable
</code></pre></div>
<p>Then I modified the build_avida script as follows:</p>
<p><strong>build_avida</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/sh</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">CC</span><span class="o">=</span><span class="s2">"condor_compile gcc"</span>
mkdir<span class="w"> </span>-p<span class="w"> </span>cbuild
<span class="nb">cd</span><span class="w"> </span>cbuild
cmake<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="w"> </span>../
make<span class="w"> </span>-j<span class="w"> </span><span class="m">10</span>
make<span class="w"> </span>install
</code></pre></div></td></tr></table></div>
<p>Then I run the build script using the following commands:</p>
<div class="highlight"><pre><span></span><code><span class="o">></span><span class="w"> </span><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">cmake</span>
<span class="o">></span><span class="w"> </span><span class="o">./</span><span class="n">build_avida</span>
<span class="o">></span><span class="w"> </span><span class="o">./</span><span class="n">run_tests</span>
</code></pre></div>
<p>Once Avida builds and tests successfully I write a condor script</p>
<p><strong>avida.condor</strong></p>
<div class="highlight"><pre><span></span><code>#
#<span class="w"> </span><span class="nv">avida</span>.<span class="nv">condor</span>
#<span class="w"> </span><span class="nv">Example</span><span class="w"> </span><span class="nv">avida</span><span class="w"> </span><span class="nv">condor</span><span class="w"> </span><span class="nv">submission</span><span class="w"> </span><span class="nv">script</span>
#
#<span class="w"> </span><span class="nv">Specify</span><span class="w"> </span><span class="nv">your</span><span class="w"> </span><span class="nv">TeraGrid</span><span class="w"> </span><span class="nv">allocation</span><span class="w"> </span><span class="nv">project</span><span class="w"> </span><span class="nv">here</span>.
<span class="o">+</span><span class="nv">TGProject</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"YOUR PROJECT NUMBER HERE"</span>
<span class="nv">universe</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">vanilla</span>
#<span class="nv">PATH</span><span class="w"> </span><span class="nv">TO</span><span class="w"> </span><span class="nv">CONDOR_COMPILE</span><span class="w"> </span><span class="nv">BUILD</span><span class="w"> </span><span class="nv">OF</span><span class="w"> </span><span class="nv">AVIDA</span>
#<span class="w"> </span><span class="nv">The</span><span class="w"> </span><span class="nv">executable</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">run</span>.<span class="w"> </span><span class="nv">Need</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">full</span><span class="w"> </span><span class="nv">path</span>.<span class="w"> </span><span class="o">~/</span><span class="w"> </span><span class="nv">does</span><span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">work</span>.
<span class="nv">executable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="nv">home</span><span class="o">/</span><span class="nv">ba01</span><span class="o">/</span><span class="nv">u124</span><span class="o">/</span><span class="nv">dcolbry</span><span class="o">/</span><span class="nv">avida_condor</span><span class="o">/</span><span class="nv">work</span><span class="o">/</span><span class="nv">avida</span>
#<span class="nv">Sets</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">process</span><span class="w"> </span><span class="nv">id</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">avida</span><span class="w"> </span><span class="nv">seed</span>
<span class="nv">arguments</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">-</span><span class="nv">s</span><span class="w"> </span>$<span class="ss">(</span><span class="nv">Process</span><span class="ss">)</span>
#<span class="w"> </span><span class="nv">false</span>:<span class="w"> </span><span class="nv">The</span><span class="w"> </span><span class="nv">executable</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">already</span><span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">remote</span><span class="w"> </span><span class="nv">machine</span>.
#<span class="w"> </span><span class="nv">true</span>:<span class="w"> </span><span class="nv">Copy</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">executable</span><span class="w"> </span><span class="nv">from</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">local</span><span class="w"> </span><span class="nv">machine</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">remote</span>.
<span class="nv">transfer_executable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">false</span>
#<span class="w"> </span><span class="nv">Filenames</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">standard</span><span class="w"> </span><span class="nv">output</span>,<span class="w"> </span><span class="nv">standard</span><span class="w"> </span><span class="nv">error</span>,<span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">Condor</span><span class="w"> </span><span class="nv">log</span>.
<span class="nv">output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">output</span><span class="o">/</span><span class="nv">out</span>.$<span class="ss">(</span><span class="nv">Process</span><span class="ss">)</span>
<span class="nv">error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">output</span><span class="o">/</span><span class="nv">err</span>.$<span class="ss">(</span><span class="nv">Process</span><span class="ss">)</span>
<span class="nv">log</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">output</span><span class="o">/</span><span class="nv">condor</span>.<span class="nv">log</span>
#<span class="w"> </span><span class="nv">PATH</span><span class="w"> </span><span class="nv">TO</span><span class="w"> </span><span class="nv">WORK</span><span class="w"> </span><span class="nv">DIRECTORY</span>
<span class="nv">initialdir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="nv">home</span><span class="o">/</span><span class="nv">ba01</span><span class="o">/</span><span class="nv">u124</span><span class="o">/</span><span class="nv">dcolbry</span><span class="o">/</span><span class="nv">avida_condor</span><span class="o">/</span><span class="nv">work</span><span class="o">/</span>
#<span class="w"> </span><span class="nv">The</span><span class="w"> </span><span class="nv">following</span><span class="w"> </span><span class="nv">line</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">always</span><span class="w"> </span><span class="nv">required</span>.<span class="w"> </span><span class="nv">It</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">command</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">submit</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">above</span>.
#<span class="w"> </span><span class="nv">The</span><span class="w"> </span><span class="nv">number</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">number</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">jobs</span><span class="w"> </span><span class="nv">that</span><span class="w"> </span><span class="nv">you</span><span class="w"> </span><span class="nv">want</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">run</span>
<span class="nv">queue</span><span class="w"> </span><span class="mi">5</span>
</code></pre></div>
<p>Make an output directory and submit the job to the condor scheduler using the
following commands:</p>
<div class="highlight"><pre><span></span><code>> mkdir /home/ba01/u124/dcolbry/avida_condor/work/output
> condor_submit avida.condor
</code></pre></div>
<p>That did it for me.</p>
<ul>
<li>Dirk</li>
</ul>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/03/11/Condor+compile+of+Avida">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2011/03/11/Condor+compile+of+Avida">ICER Wiki</a> using custom python script. Comment on errors below.</p>Simple Matlab Script Wrapper2010-08-30T00:00:00-04:002010-08-30T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2010-08-30:/simple-matlab-script-wrapper.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Sometimes you have four or five variations on the same MATLAB script. You
could write a different submissions script for each variation, or you can
constantly edit the submission script every time you need …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>Sometimes you have four or five variations on the same MATLAB script. You
could write a different submissions script for each variation, or you can
constantly edit the submission script every time you need to change to a
different variation. However, I find that this can be confusing so I made the
following script to make my life easier:</p>
<p><strong>runMatlab.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash -login</span>
<span class="c1">#PBS -l walltime=01:00:00,nodes=1:ppn=1</span>
<span class="c1">#PBS -l software=MATLAB</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
module<span class="w"> </span>load<span class="w"> </span>matlab
matlab<span class="w"> </span>-nodisplay<span class="w"> </span>-r<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">PBS_JOBNAME</span><span class="si">}</span><span class="s2">"</span>
</code></pre></div></td></tr></table></div>
<p>This is a qsub wrapper script that allows you to specify the name of the
MATLAB script as the name of the job. To run the script just type:</p>
<div class="highlight"><pre><span></span><code>qsub -N myScriptName runMatlab.qsub
</code></pre></div>
<p>Then when you want to run a different script you can type:</p>
<div class="highlight"><pre><span></span><code>qsub -N myDifferentScript runMatlab.qsub
</code></pre></div>
<p>Make sure you do not include the ".m" in the script name. Since command line
resource request override requests in the script, it is possible to also
change the resources from the command line. For example, if you have a really
short test you need to run you can do the following:</p>
<div class="highlight"><pre><span></span><code>qsub -N myquicktest -l walltime=00:05:00 runMatlab.qsub
</code></pre></div>
<p><strong>NOTE:</strong> This above only works for MATLAB scripts (i.e. no input arguments)
and not functions.</p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2010/08/30/Simple+Matlab+Script+Wrapper">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2010/08/30/Simple+Matlab+Script+Wrapper">ICER Wiki</a> using custom python script. Comment on errors below.</p>Using mpi pernode option2010-08-25T00:00:00-04:002010-08-25T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2010-08-25:/using-mpi-pernode-option.html<p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>The mpiexec (mvapich) pernode option will run a single process for on every
machine with the same host name. For example, if you run a job with
nodes=16:ppn=2, the job's PBS_NODEFILE …</p><p>Blog post <strong>edited</strong> by Anonymous - "Migration of unmigrated content due to
installation of a new plugin"</p>
<p>The mpiexec (mvapich) pernode option will run a single process for on every
machine with the same host name. For example, if you run a job with
nodes=16:ppn=2, the job's PBS_NODEFILE could look like the following:</p>
<div class="highlight"><pre><span></span><code>afn-000
afn-000
afn-000
afn-000
afn-002
afn-002
afn-002
afn-002
afn-002
afn-002
afn-002
afn-002
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
scw-049
scw-049
scw-118
scw-118
scw-094
scw-094
scw-006
scw-006
</code></pre></div>
<p>If you run using the -pernode option, the job will run on the following nodes:</p>
<div class="highlight"><pre><span></span><code>afn-000
afn-002
afn-001
scw-049
scw-118
scw-094
scw-006
</code></pre></div>
<p>However, this may be a problem depending on the type of job you are running.
The pernode option may not run the number of nodes asked for by the job script
(in the above example I asked for 16 nodes, however, since some of these 16
are the same node, the pernode option only runs on 7 nodes). Sometimes it is
more desirable to have the mpi job run on the same number of nodes as was
originally requested in the submission script. To do this, I have made the
following script which will generate a config file that can be used to run the
job:</p>
<p><strong>pernodecofig</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/usr/bin/perl</span>
<span class="c1"># Written by Dirk Colbry - iCERi - August 26, 2009</span>
<span class="c1"># (c) 2010 Michigan State University Board of Trustees.</span>
<span class="c1">#</span>
<span class="c1"># Program to generate a mpiexec config file to run an mpi job with one job per node.</span>
<span class="c1"># This fixes the problem with the -pernode mpiexec option which does not work when</span>
<span class="c1"># two of the nodes are allocated on the same machine. </span>
<span class="c1">#</span>
<span class="c1"># This function will run the same number of processes as allocated in the nodes= section of</span>
<span class="c1"># the PBS resource request.</span>
<span class="c1">#</span>
<span class="c1"># pernodeconfig ppn nodefile executable arguments</span>
<span class="c1"># ppn - process per node for the current submission script.</span>
<span class="c1"># nodefile - nodefile for the current submission script (typically PBS_NODEFILE).</span>
<span class="c1"># exectuable - name of the executable to be run on each node.</span>
<span class="c1"># arguments - arguments used for the executable.</span>
<span class="nv">$ppn</span><span class="o">=</span><span class="nv">$ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="nv">$nodefile</span><span class="o">=</span><span class="nv">$ARGV</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="c1">#Parse the nodefile to find the names of each of the allocated nodes.</span>
<span class="k">if</span><span class="p">(</span><span class="o">-</span><span class="n">T</span><span class="w"> </span><span class="nv">$nodefile</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nb">open</span><span class="w"> </span><span class="p">(</span><span class="n">FILE</span><span class="p">,</span><span class="w"> </span><span class="s">"$nodefile"</span><span class="p">)</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="k">print</span><span class="w"> </span><span class="s">"Can't open '$nodefile': $! \n\n"</span><span class="p">;</span>
<span class="w"> </span><span class="nv">@testfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sr"><FILE></span><span class="p">;</span>
<span class="w"> </span><span class="nb">close</span><span class="p">(</span><span class="n">FILE</span><span class="p">);</span>
<span class="w"> </span><span class="nv">$count</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="k">foreach</span><span class="w"> </span><span class="p">(</span><span class="nv">@testfile</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nb">chomp</span><span class="p">;</span>
<span class="w"> </span><span class="nv">$count</span><span class="o">=</span><span class="nv">$count</span><span class="o">+</span><span class="mi">1</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="nv">$count</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">$ppn</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nv">$count</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="k">print</span><span class="w"> </span><span class="s">"$_\t"</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">print</span><span class="w"> </span><span class="bp">STDERR</span><span class="w"> </span><span class="s">"INPUT NOT A TEXT FILE\n"</span><span class="p">;</span>
<span class="w"> </span><span class="nb">die</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1"># Add the executable to the config file</span>
<span class="nv">$numArgs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$#ARGV</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="k">print</span><span class="w"> </span><span class="s">" \: "</span><span class="p">;</span>
<span class="k">foreach</span><span class="w"> </span><span class="nv">$argnum</span><span class="w"> </span><span class="p">(</span><span class="mi">2</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="nv">$#ARGV</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">print</span><span class="w"> </span><span class="s">"$ARGV[$argnum] "</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">print</span><span class="w"> </span><span class="s">"\n"</span>
</code></pre></div></td></tr></table></div>
<p>To use this command you can add the following to your submission script:</p>
<p><strong>pernodetest.qsub</strong></p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/sh</span>
<span class="c1">#PBS -l nodes=16:ppn=2,walltime=00:01:00</span>
<span class="nb">cd</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_O_WORKDIR</span><span class="si">}</span>
./pernodeconfig<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="si">${</span><span class="nv">PBS_NODEFILE</span><span class="si">}</span><span class="w"> </span>hostname<span class="w"> </span>><span class="w"> </span><span class="nv">$PBS_JOBID</span>.cfg
mpiexec<span class="w"> </span>-comm<span class="w"> </span>none<span class="w"> </span>-conf<span class="w"> </span><span class="nv">$PBS_JOBID</span>.cfg
</code></pre></div></td></tr></table></div>
<p>This script will output the following nodes which is may be more correct
depending on your job:</p>
<div class="highlight"><pre><span></span><code>afn-000
afn-000
afn-002
afn-002
afn-002
afn-002
afn-001
afn-001
afn-001
afn-001
afn-001
afn-001
scw-049
scw-118
scw-094
scw-006
</code></pre></div>
<p>Email me if you find this helpful,</p>
<ul>
<li>Dirk</li>
</ul>
<p>-----------------------------------------------<br>
Dr. Dirk Joel Luchini Colbry<br>
Research Specialist<br>
Institute for Cyber-Enabled Research (iCER)<br>
Michigan State University<br>
1445 Biomedical and Physical Sciences Building (BPS),<br>
East Lansing, MI 48824-1226<br>
email: dirk (at) colbry.com<br>
web: <a href="http://www.dirk.colbry.com">http://www.dirk.colbry.com</a><br>
phone: (517) 355-0730</p>
<p>To ask an HPC or iCER related technical question or to schedule a consulting
appointment, please visit: <a href="http://hpcc.msu.edu/contact">http://hpcc.msu.edu/contact</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2010/08/25/Using+mpi+pernode+option">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2010/08/25/Using+mpi+pernode+option">ICER Wiki</a> using custom python script. Comment on errors below.</p>Reordering the Eligible Jobs in Your Queue2010-08-17T00:00:00-04:002010-08-17T00:00:00-04:00Dr. Dirk Colbrytag:colbrydi.github.io,2010-08-17:/reordering-the-eligible-jobs-in-your-queue.html<p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~changc81@msu.edu">Chun-Min Chang</a> - "Migration of unmigrated content due to installation of a new plugin"</p>
<p>I run a lot of different types of jobs with many different resource
requirements. Back when we could have up to 144 eligible jobs in the queue
this was not a problem. However …</p><p>Blog post <strong>edited</strong> by <a href="https://wiki.hpcc.msu.edu/display/~changc81@msu.edu">Chun-Min Chang</a> - "Migration of unmigrated content due to installation of a new plugin"</p>
<p>I run a lot of different types of jobs with many different resource
requirements. Back when we could have up to 144 eligible jobs in the queue
this was not a problem. However, now to make the queuing system more fair,
users can only have 15 eligible jobs in the queue (at any one time) and the
rest of the jobs are set as idle. If all of your jobs require the same types
of resources, this is not really a problem.</p>
<p>However, sometimes I get five jobs in the queue that are waiting on resources
and I want to run a different type of simpler job that should be able to run
immediately but can't because it is in the idle queue instead of the eligible
queue. The problem is that the five, complex resource, jobs that are already
in the eligble queue end up blocking the simpler job that could run.</p>
<p>For example, lets say I have five jobs that need nodes=1:ppn=8 and I want to
add a job that only requires nodes=1:ppn=1 (no memory or significant
walltimes). If there are no 8-core nodes available then the five 8-core jobs
end up blocking my single core job which will not be eligible to run unless
one of the 8-core jobs gets out of the way. To fix this problem I reorder my
queue by putting a temporary hold on the more complex jobs using the "qhold"
command:</p>
<div class="highlight"><pre><span></span><code>qhold JOBID#
</code></pre></div>
<p>Suspending the complex jobs allows my single core job to move to the eligible
queue and be able to run. I can then immediately run the queue resume command
(qrls) and the complex jobs move back into the queue, behind the single core
job. To resume a suspended job in the queue:</p>
<div class="highlight"><pre><span></span><code>qrls JOBID#
</code></pre></div>
<p>Just for fun, I quickly came up with the following scripts that can help move
a job to either the top or the bottom of a users eligible queue:</p>
<p><strong>move2top</strong></p>
<div class="highlight"><pre><span></span><code>#/bin/bash<span class="w"> </span>--login
#DESCRIPTION<span class="w"> </span>Move<span class="w"> </span>a<span class="w"> </span>job<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>top<span class="w"> </span>of<span class="w"> </span>your<span class="w"> </span>eligble<span class="w"> </span>queue
#<span class="w"> </span>This<span class="w"> </span>script<span class="w"> </span>suspends<span class="w"> </span>all<span class="w"> </span>of<span class="w"> </span>your<span class="w"> </span>jobs<span class="w"> </span>and<span class="w"> </span>then<span class="w"> </span>resumes<span class="w"> </span>only
#<span class="w"> </span>the<span class="w"> </span>specified<span class="w"> </span>job<span class="w"> </span>so<span class="w"> </span>it<span class="w"> </span>moves<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>top<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>eligible<span class="w"> </span>queue
#<span class="w"> </span>The<span class="w"> </span>remaining<span class="w"> </span>jobs<span class="w"> </span>are<span class="w"> </span>then<span class="w"> </span>resumed<span class="w"> </span>(after<span class="w"> </span>a<span class="w"> </span>wait<span class="w"> </span>period).
#
#<span class="w"> </span>This<span class="w"> </span>script<span class="w"> </span>allows<span class="w"> </span>users<span class="w"> </span>to<span class="w"> </span>move<span class="w"> </span>a<span class="w"> </span>job<span class="w"> </span>up<span class="w"> </span>if<span class="w"> </span>they<span class="w"> </span>are<span class="w"> </span>being
#<span class="w"> </span>blocked<span class="w"> </span>by<span class="w"> </span>other<span class="w"> </span>jobs.
#
#<span class="w"> </span>Written<span class="w"> </span>by<span class="w"> </span>Dirk<span class="w"> </span>Colbry<span class="w"> </span>August<span class="w"> </span>17,<span class="w"> </span>2010
#<span class="w"> </span>©<span class="w"> </span>2010<span class="w"> </span>Michigan<span class="w"> </span>State<span class="w"> </span>University<span class="w"> </span>Board<span class="w"> </span>of<span class="w"> </span>Trustees.
jobid=$1
if<span class="w"> </span>[<span class="w"> </span>"$1"<span class="w"> </span>==<span class="w"> </span>""<span class="w"> </span>]
then
<span class="w"> </span>echo<span class="w"> </span>"job<span class="w"> </span>id<span class="w"> </span>required"
<span class="w"> </span>exit<span class="w"> </span>1
fi
qhold<span class="w"> </span>`qselect<span class="w"> </span>-s<span class="w"> </span>Q<span class="w"> </span>-u<span class="w"> </span><span class="cp">${</span><span class="n">USER</span><span class="cp">}</span>`
qrls<span class="w"> </span>$1
echo<span class="w"> </span>"Waiting<span class="w"> </span>30<span class="w"> </span>seconds<span class="w"> </span>for<span class="w"> </span>queue<span class="w"> </span>to<span class="w"> </span>settle..."
sleep<span class="w"> </span>30
qrls<span class="w"> </span>`qselect<span class="w"> </span>-s<span class="w"> </span>H<span class="w"> </span>-u<span class="w"> </span><span class="cp">${</span><span class="n">USER</span><span class="cp">}</span>`
showjobs
</code></pre></div>
<p><strong>move2bot</strong></p>
<div class="highlight"><pre><span></span><code>#<span class="o">/</span><span class="nv">bin</span><span class="o">/</span><span class="nv">bash</span><span class="w"> </span><span class="o">--</span><span class="nv">login</span>
#<span class="nv">DESCRIPTION</span><span class="w"> </span><span class="nv">Move</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">job</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">bottom</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">your</span><span class="w"> </span><span class="nv">eligble</span><span class="w"> </span><span class="nv">queue</span>
#<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">script</span><span class="w"> </span><span class="nv">teporarily</span><span class="w"> </span><span class="nv">suspends</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">jobs</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">allow</span><span class="w"> </span><span class="nv">other</span><span class="w"> </span><span class="nv">jobs</span><span class="w"> </span><span class="nv">to</span>
#<span class="w"> </span><span class="nv">move</span><span class="w"> </span><span class="nv">up</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">eligible</span><span class="w"> </span><span class="nv">queue</span>.<span class="w"> </span><span class="nv">The</span><span class="w"> </span><span class="nv">job</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="nv">resumed</span><span class="w"> </span><span class="ss">(</span><span class="nv">after</span>
#<span class="w"> </span><span class="mi">30</span><span class="w"> </span><span class="nv">second</span><span class="w"> </span><span class="k">wait</span><span class="w"> </span><span class="nv">period</span>.
#
#<span class="w"> </span><span class="nv">Written</span><span class="w"> </span><span class="nv">by</span><span class="w"> </span><span class="nv">Dirk</span><span class="w"> </span><span class="nv">Colbry</span><span class="w"> </span><span class="nv">August</span><span class="w"> </span><span class="mi">17</span>,<span class="w"> </span><span class="mi">2010</span>
#<span class="w"> </span>©<span class="w"> </span><span class="mi">2010</span><span class="w"> </span><span class="nv">Michigan</span><span class="w"> </span><span class="nv">State</span><span class="w"> </span><span class="nv">University</span><span class="w"> </span><span class="nv">Board</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">Trustees</span>.
<span class="nv">jobid</span><span class="o">=</span><span class="mh">$1</span>
<span class="k">if</span><span class="w"> </span>[<span class="w"> </span><span class="s2">"$1"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span>]
<span class="k">then</span>
<span class="w"> </span><span class="nv">echo</span><span class="w"> </span><span class="s2">"job id required"</span>
<span class="w"> </span><span class="k">exit</span><span class="w"> </span><span class="mi">1</span>
<span class="nv">fi</span>
<span class="nv">qhold</span><span class="w"> </span><span class="mh">$1</span>
<span class="nv">echo</span><span class="w"> </span><span class="s2">"Waiting 30 seconds for queue to settle..."</span>
<span class="nv">sleep</span><span class="w"> </span><span class="mi">30</span>
<span class="nv">qrls</span><span class="w"> </span><span class="mh">$1</span>
<span class="nv">showjobs</span>
</code></pre></div>
<p>I put copies of these scripts in the powertools module. To use them just type
the following in one of the developer nodes:</p>
<div class="highlight"><pre><span></span><code><span class="n">module</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">powertools</span>
<span class="n">move2top</span><span class="w"> </span><span class="n">JOBID</span>
<span class="n">move2bot</span><span class="w"> </span><span class="n">JOBID</span>
</code></pre></div>
<p>Hope you find this useful,</p>
<ul>
<li>Dirk</li>
</ul>
<hr>
<p>Dr. Dirk Joel Luchini Colbry<br>
Research Specialist<br>
Institute for Cyber-Enabled Research (iCER)<br>
Michigan State University<br>
1445 Biomedical and Physical Sciences Building (BPS),<br>
East Lansing, MI 48824-1226<br>
email: dirk (at) colbry.com<br>
web: <a href="http://www.dirk.colbry.com">http://www.dirk.colbry.com</a><br>
phone: (517) 355-0730</p>
<p>To ask an HPC or iCER related technical question or to schedule a consulting
appointment, please visit: <a href="http://hpcc.msu.edu/contact">http://hpcc.msu.edu/contact</a></p>
<p><a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2010/08/17/Reordering+the+Eligible+Jobs+in+Your+Queue">View
Online</a></p>
<p>Blogpost migrated from <a href="https://wiki.hpcc.msu.edu/display/~colbrydi@msu.edu/2010/08/17/Reordering+the+Eligible+Jobs+in+Your+Queue">ICER Wiki</a> using custom python script. Comment on errors below.</p>