Latest Tweets

Script for NUMA systems

Preamble

On modern NUMA systems, it’s always a good thing to know the total number of physical processors (nodes), the total number of real cores per processor, and if HT is enabled, the total number of virtual cores per processor (which is, of course, twice the number of real cores per socket). At the same time, it would be great to be able to disable HT for a particular physical processor on the fly. This simple BASH script does precisely that.

Using the script

Install numactl before running the script. If numactl is not found, the script will tell you so and exit:

sudo apt-get install numactl

The script can report the real and virtual cores per socket along with the total number of physical processors installed on the system. It works perfectly well for systems that only have 1 socket, of course, so no NUMA in this case.:

cores.sh -h
Usage: /usr/bin/cores.sh [-t] [-c CPU] [-h]
		 -t, reports total number of physical processors (nodes) and quits.
		 -c CPU, reports total and real cores for physical processor CPU.
		 -r CPU, reports real cores for physical processor CPU separated with commas.
		 -d CPU, disables HT on the given CPU (requires root).
		 -h, show this help message.
  Example: /usr/bin/cores.sh -t
           /usr/bin/cores.sh -c 2
           sudo /usr/bin/cores.sh -d 0

To get the total number of cores for, say, physical processor 1 on a 4-socket system, run:

cores.sh -c 1
Total physical processors on this system: 4
Total Cores for processor 1: 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77
Virtual Cores for processor 1:  41 45 49 53 57 61 65 69 73 77 
Real Cores for processor  1:  1 5 9 13 17 21 25 29 33 37

You can disable HT for a particular socket as well. The number of cores that will be disabled are shown in the “Virtual Cores” row. Say, you want to disable HT for CPU0 on a 1-socket system; run:

 sudo cores.sh -d 0
Total physical processors on this system: 1
Total Cores for processor 0: 0 1 2 3 4 5 6 7
Virtual Cores for processor 0:  4 5 6 7 
Real Cores for processor  0:  0 1 2 3
We are about to disable HT on physical cpu: 0
CPU cores to be disabled:  4 5 6 7 
Are you sure? [yn]

If you are completely sure, type “y” and press <ENTER>. Otherwise, type any other key and press <ENTER>.

You can run the script using the “-r” flag in order to output the real cores for a particular physical processor to pass it directly to numactl. Let’s imagine you want to run a 10-core simulation using mpirun on a 4-socket NUMA system. Each physical node has 10 real cores and 10 virtual cores due to HT. You want to run this simulation on node 3 (CPU4):

cores.sh -r 3
Total physical processors on this system: 4
Real Cores for processor  3:  3,7,11,15,19,23,27,31,35,39

Now, you can pass the string of real cores to numactl and run your simulation like this:

numactl --physcpubind=3,7,11,15,19,23,27,31,35,39 mpirun -np 10 \
> --report-bindings binary arguments

Or you can do it using this one-liner instead:

numactl --physcpubind=`cores.sh -r 3|tail -1|cut -d":" -f2|tr -d " "` \
>  mpirun -np 10 --report-bindings binary arguments

To make sure your simulation is running on the chosen cores/sockets, you can run the following command (replace PID with the proper one):

cat /proc/<PID>/status |grep Cpus_allowed_list
Cpus_allowed_list:	4-7

Get the script (cores.sh)

You can download the script from HERE.