Embarassingly Parallel Problems

• each task independent
• shared memory
• easy to parallelize

Threading

• easy to set up
  • no extra installs required
• good for embarassingly parallel problems
• shared memory
• can only use a single machine
  • but multiple cores
• Python: joblib

MPI (Message Passing Interface)

• more complex
• each process has its own memory
• can scale across multiple machines
• can adapt to more settings
• Python: mpi4py
  • requires OpenMPI to be installed
  • done for you on OzSTAR

#running scripts with mpi
mpiexec -n <num processes> python3 <path/to/your/file.py>

flowchart LR
    main@{ shape: rounded, label: "Main"}
    worker1@{ shape: rounded, label: "Worker 1"}
    worker2@{ shape: rounded, label: "Worker 2"}
    workerN@{ shape: rounded, label: "Worker N"}
    out@{ shape: rect, label: "Output"}

    main -...->|assign work| worker1
    main -...->|assign work| worker2
    main -...->|assign work| workerN
    worker1 ---->|collect result| main
    worker2 ---->|collect result| main
    workerN ---->|collect result| main
    main ----->|combine results| out

GPU Computing

• GPU: highly specialized hardware
• designed for embarassingly parallel problems
• huge in Deep Learning
  • batch-wise gradient computation
• each block has it's own memory (shared across block)
  • each thread within a block has it's own local memory
• more specialized variants exist: TPU, PPU

software needs to be designed for GPU-computation!