Numa Programming

NUMA programming

Contract

Inputs

1. Target system (required): a multi-socket Linux host or VM with NUMA exposed.
2. Workload symptom (optional): poor scaling, memory bandwidth saturation, remote access, or unexpected page migration.
3. Tools available (optional): numactl, numactl-dev, libnuma-dev, perf, lstopo.

Procedure

1. Detect the NUMA topology. Run numactl --hardware, lstopo --of console, and read /sys/devices/system/node/node*/meminfo and node*/cpulist. A distance value of 10 usually means local access; larger values indicate remote access. Done when: the node count, CPU lists, memory sizes, and distance matrix are reported.

   Typical output from numactl --hardware:

   available: 2 nodes (0-1)
   node 0 cpus: 0-15
   node 0 size: 65536 MB
   node 1 cpus: 16-31
   node 1 size: 65536 MB
   node distances:
   node   0   1
     0:  10  21
     1:  21  10
   

2. Bind a process with numactl. Use --cpunodebind and --membind for strict local placement, --interleave=all to spread memory, --preferred for a preferred node with fallback, and --show to inspect the current policy. Done when: the binding command and policy are selected.

   numactl --cpunodebind=0 --membind=0 ./myapp
   numactl --interleave=all ./myapp
   numactl --preferred=0 ./myapp
   numactl --show
   

3. Use the libnuma API. Call numa_available, numa_node_of_cpu, numa_alloc_onnode, numa_alloc_local, mbind, set_mempolicy, and move_pages. Compile with -lnuma. Done when: the C example and API table are shown.

   #include <numa.h>
   #include <numaif.h>
   #include <stdio.h>
   
   int main(void) {
       if (numa_available() < 0) {
           fprintf(stderr, "NUMA not available\n");
           return 1;
       }
       int node = numa_node_of_cpu(0);
       printf("CPU 0 on node %d\n", node);
   
       size_t size = 1024 * 1024 * 1024;
       void *mem = numa_alloc_onnode(size, 0);
       if (!mem) return 1;
   
       unsigned long nodemask = 1UL << 0;
       mbind(mem, size, MPOL_BIND, &nodemask, sizeof(nodemask) * 8, 0);
   
       numa_free(mem, size);
       return 0;
   }
   

   gcc -o numa_test numa_test.c -lnuma
   

   API	Purpose
   numa_alloc_onnode	Allocate on a specific node
   numa_alloc_local	Allocate on the current CPU node
   mbind	Set policy on an existing mapping
   set_mempolicy	Default policy for later allocations
   move_pages	Migrate pages to a target node

4. Build NUMA-aware data structures. Create a per-node pool and allocate from the node of the current CPU. Pin threads to cores on the same node. Done when: the per-node allocation pattern is shown.

   #include <numa.h>
   #include <sched.h>
   
   #define BLOCK_SIZE 64
   #define MAX_NODES 8
   
   struct per_node_pool {
       void *free_list[MAX_NODES];
       int node_count;
   };
   
   void *pool_alloc_numa(struct per_node_pool *p) {
       int node = numa_node_of_cpu(sched_getcpu());
       void *blk = p->free_list[node];
       if (blk) {
           p->free_list[node] = *(void **)blk;
           return blk;
       }
       return numa_alloc_onnode(BLOCK_SIZE, node);
   }
   

5. Align thread affinity. Bind threads to CPUs on a node with pthread_setaffinity_np or numactl --cpunodebind. Match memory allocation to the same node. Done when: the affinity and memory binding match.

   #include <pthread.h>
   #include <sched.h>
   
   cpu_set_t cpuset;
   CPU_ZERO(&cpuset);
   CPU_SET(target_cpu, &cpuset);
   pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset);
   

6. Diagnose remote access. Run perf stat -e cache-misses,cache-references,node-load-misses with CPU and memory bindings on different nodes. Compare against local-local binding. Use node-loads and node-load-misses if the kernel exposes them. Done when: the remote access signature is identified.

   perf stat -e cache-misses,cache-references,node-load-misses \
       numactl --cpunodebind=0 --membind=1 ./myapp
   
   perf stat numactl --cpunodebind=0 --membind=0 ./myapp
   perf stat -e node-loads,node-load-misses,node-stores ./myapp
   

7. Measure the remote access penalty. Touch a large buffer on the local node and then on a remote node. Time the loops. Remote access can be 1.5x to 3x slower than local access on recent multi-socket x86/AMD64 systems with QPI/UPI/Infinity Fabric, depending on the workload and interconnect. Done when: the measurement method is given.

   clock_t start = clock();
   for (size_t i = 0; i < size; i += 4096)
       sum += ((char *)mem)[i];
   

8. Visualize with lstopo. Run lstopo for a graphical view, lstopo --of ascii for text, or lstopo file.png for an image. Done when: the visualization command is selected.

   lstopo
   lstopo --of ascii
   lstopo file.png
   

9. Walk the decision tree. Check topology, verify CPU and memory are on the same node, measure node-load-misses, then choose numactl --membind=local, per-node partitioning, or memory bandwidth reduction. Done when: the next action is named.

   Poor scaling on multi-socket?
   ├── Check numactl --hardware
   ├── Verify thread and memory on the same node
   ├── perf stat node-load-misses
   ├── Remote misses high?
   │   ├── numactl --membind=local
   │   └── Per-node data partitioning
   └── Still slow: memory bandwidth bound; reduce sharing
   

Failure and recovery

Output

1. The NUMA topology and distance matrix.
2. A numactl binding command.
3. libnuma API calls for the use case.
4. A per-node data structure and affinity pattern.
5. A remote access measurement and a decision tree.