Module System on Aire

Module System on Aire

Managing software environments on HPC systems

What are Modules?

Modules are a way to manage different software environments on HPC systems

Key Benefits: - Dynamic loading/unloading of software packages - Manage different versions and dependencies - Clean separation of environments - Avoid software conflicts

Without Modules: - Software conflicts and version issues - Complex PATH management - Inconsistent environments - Difficult reproducibility

Why Use Modules?

With Modules ✅

  • Clean environments
  • Multiple software versions
  • Easy switching
  • Optimized HPC builds
  • Reproducible research

Without Modules ❌

  • Software conflicts
  • Path management chaos
  • Version inconsistencies
  • Hard to reproduce
  • Performance issues

Basic Module Commands

Listing Available Software

module avail              # List all available modules
module avail python       # List all Python modules
module avail gcc          # List all GCC modules

Try it: Run module avail to see what software is available on Aire

Loading and Managing Modules

# Load a specific version
module load python/3.13.0

# Load default version (not recommended for reproducibility)
module load python

# List currently loaded modules
module list

# Unload a module
module unload python/3.13.0

# Unload all modules
module purge

Exercise: Your First Module

Load the gcc module and verify it’s loaded:

module load gcc
module list

Questions: - What happens if you don’t specify a version? - How does this affect reproducibility?

Answer: Without version specification, you get the default version, which may change over time!

Best Practice: Always Specify Versions

# ❌ Bad - may change over time
module load gcc
module load python

# ✅ Good - reproducible
module load gcc/14.2.0  
module load python/3.13.0

Why? Default versions change when new software is installed, breaking reproducibility

Using Modules in Scripts

Example Python Script

Create hello_world.py:

print("hello world!")

Create job script python_test.sh:

#!/bin/bash
module load python/3.13.0
python hello_world.py

Making Scripts Executable

# Add executable permissions
chmod +x python_test.sh

# Check it's executable (shows *)
ls -F

# Run the script
./python_test.sh

Remember: Always test your scripts interactively before submitting jobs!

Python Best Practice

Recommendation

Instead of basic Python modules, use Miniforge to create conda environments:

module load miniforge/24.3.0
conda create -n myproject python=3.13
conda activate myproject
conda install numpy pandas matplotlib

This provides better dependency management!

Requesting New Software

Centralized Management

Benefits: - Popular software centrally installed - Optimized for HPC hardware - Professional maintenance - Avoids conflicts

Process: 1. Check if software exists: module avail software_name 2. If not available, submit a Research Computing Query 3. Include: software name, version, research justification

How to Request Software

Submit requests through the Research Computing Help Form

Include in your request: - Software name and version - Brief justification for your research - Any specific requirements or dependencies - Timeline if urgent

Navigation: Home > Research IT > Research IT Query

Alternative Software Management

For advanced users who need more control:

Package Managers

  • Spack: Flexible HPC package manager
  • EasyBuild: Automated building framework

Manual Options

  • Self-building: Download and compile yourself
  • Containers: Apptainer/Singularity for complex environments

Spack Example

# Load Spack
module load spack

# Install software
spack install htop

# Load installed software
spack load htop

# Use the software
htop

Use case: When you need software versions not available via modules

Container Example

# Pull a container
apptainer pull docker://ubuntu:20.04

# Run software in container
apptainer exec ubuntu_20.04.sif python my_script.py

Use case: Complex software stacks, ensuring exact reproducibility

Best Practices Summary

Environment Management

  • Use modules to manage software environments
  • Unload modules when no longer needed
  • Use Miniforge for Python/R environments
  • Start clean with module purge

Reproducibility

  • Always specify versions: gcc/14.2.0 not gcc
  • Document everything: Keep track of modules used
  • Use scripts: Automate your module loading
  • Version control: Track your workflow scripts

Module Script Template

#!/bin/bash
# Project setup script
# Author: Your Name
# Date: 2025-01-01

# Start with clean environment
module purge

# Load required modules with versions
module load gcc/14.2.0
module load python/3.13.0  
module load cmake/3.24.2

# Verify modules loaded
echo "Loaded modules:"
module list

# Optional: activate conda environment
# conda activate myproject

Common Workflows

Data Analysis

module load python/3.13.0
module load scipy/1.11.3
module load matplotlib/3.7.2
python analysis.py

Compilation

module load gcc/14.2.0
module load cmake/3.24.2
module load openmpi/4.1.4
cmake . && make -j8

Machine Learning

module load miniforge/24.3.0
conda activate ml-env
python train_model.py

Collaboration Tips

Sharing Environments

Share module commands:

# Document your setup
echo "module load gcc/14.2.0" > setup_modules.sh
echo "module load python/3.13.0" >> setup_modules.sh

Export conda environments:

conda env export > environment.yml
# Share environment.yml file with collaborators

Exercise Time!

Exercise 1: Module Exploration

  1. List all available Python modules: module avail python
  2. Load a specific Python version
  3. Check what’s loaded: module list
  4. Unload and try a different version

Exercise 2: Create Setup Script

Create a script that loads modules for your research:

#!/bin/bash
# My research environment

module purge
module load [your_required_modules]

echo "Environment ready for [your project]"
module list

Troubleshooting Modules

Common Issues

Module not found:

module avail software_name  # Check spelling/availability
module spider software_name # More detailed search

Conflicts between modules:

module purge    # Start fresh
module load module1 module2  # Load in correct order

Version conflicts:

module unload old_version
module load new_version

Summary

Key Points: - Modules provide clean software environments - Always specify versions for reproducibility
- Use scripts to automate and document - Request new software through proper channels - Consider alternatives for special needs

Next: Learn how to run your software on compute nodes with job scheduling!

Next Steps

Now you can manage software environments!

Let’s learn about job scheduling and submission