Process’s afinity

Using schedutils, one can alter any process’s afinity by running a simple command: taskset. This way, it is feasible to change the GNU/Linux scheduler behaviour when it comes to running processes on a multi-core GNU/Linux box. For example, if we run any command directly, on a dual-core system, GNU/Linux Kernel sets the default afinity for a process this way:

etch32:~/dev/modest.cvs/kmodest-src-2010# taskset -p 2047
pid 2047’s current affinity mask: 3

In this case, taskset is showing us that 2047 process could be executed either on cpu 0 or CPU 1 – affinity mask of 3 -. So, which CPU is running right now this process?

The task_struct * data structure

Thinking back on it, we do know the task_struct * internal data structure, controlled by the GNU/Linux Kernel. We can determine which CPU belongs to a given process at any time thanks to it, no matter if that process is running a system call – Ring 0 – or it is still executing instructions in Ring 3 – user land -. Taking a quick look at this data structure inside the GNU/Linux Kernel sources, it seems very clear how to achieve that:

 767 struct task_struct {
 768     volatile long state;    /* -1 unrunnable, 0 runnable, >0 stopped */
 769     struct thread_info *thread_info;
(...)
 27 struct thread_info {
 28     struct task_struct  *task;      /* main task structure */
 29     struct exec_domain  *exec_domain;   /* execution domain */
 30     unsigned long       flags;      /* low level flags */
 31     unsigned long       status;     /* thread-synchronous flags */
 32     __u32           cpu;        /* current CPU */
 33     int         preempt_count;  /* 0 => preemptable, <0 => BUG */
 34
 35
 36     mm_segment_t        addr_limit; /* thread address space:
 37                            0-0xBFFFFFFF for user-thead
 38                            0-0xFFFFFFFF for kernel-thread
 39                         */
 40     void            *sysenter_return;
 41     struct restart_block    restart_block;
 42
 43     unsigned long           previous_esp;   /* ESP of the previous stack in case
 44                            of nested (IRQ) stacks
 45                         */
 46     __u8            supervisor_stack[0];
 47 };

Clearly, the thread_info pointer has a field called “cpu“, just an unsigned int 32 value storing the CPU number running this process. This data structure is declared in the include/asm-i386/thread_info.h header file.

Using kmodest to determine which CPU is running a given PID process.

Some little changes were made to the original modest.c and umodest.c source files so as to be able to find out which CPU is running our PID process at any moment, just by running a simple command through the umodest user-land utility. Thus, I added this piece of code in order to get “on-the-fly” that very CPU for a given process as soon as a user-land process is trying to read this information after sending the F_GETCPU ioctl command, declared in the common-structures.h header file:

204         /* Read just a byte - 0-255- storing on which cpu this process is being executed : */
205         case F_GETCPU:
206             /* Only one byte to read allowed */
207             if(count<0 || count>sizeof(char))return -EINVAL;
208             /* Okay, get the current CPU and try to copy to user space memory area: */
209             /* So, we need to reference this PID inside task_struct list : */
210             p = find_task_by_pid (pid_cpu);
211             if(p){
212                 k = copy_to_user(bf, &p->thread_info->cpu , sizeof(char));
213                 if(k!=0)return -EFAULT;
214             } else return -EFAULT;
215             /* Update file pointer */
216             *fp_pos+=1; real_read = sizeof(char);
217             break;

Then, I added a few code lines inside umodest utility to allow it to deal with this new functionality:

158     /* Get information about on what CPU this process is being executed */
159     if(strcmp(*(argv+3),"-s")==0){
160         myargs.command = F_GETCPU;
161         ioctl_return = ioctl(drv,F_GETCPU,&myargs);
162         if(ioctl_return!=0){
163             perror("Sending command to device"); return errno;
164         }
165         /* Read from "device" and print it out. It's just a char (0-255) : */
166         syscall_ret = read(drv, &on_which_cpu, sizeof(char));
167         if(syscall_ret==1)printf("Process with PID: %d is running on CPU: %d.\n" , myargs.pid , on_which_cpu);
168         else printf("Error getting CPU\n");
169     }

Using the umodest utility

Now, it is feasible to figure out on which CPU a given process is being executed at any moment this way:

# umodest -p PID -s

For example, running umodest on a dual-core GNU/Linux box to determine syslog’s CPU it gives:

etch32:~/dev/modest.cvs/kmodest-src-2010# taskset -p 2047
pid 2047’s current affinity mask: 3
etch32:~/dev/modest.cvs/kmodest-src-2010# bin/umodest -p 2047 -s
Process with PID: 2047 is running on CPU: 0.

Well, it’s working!. Now, what if we alter that PID’s affinity using taskset? Let’s see what happens:

etch32:~/dev/modest.cvs/kmodest-src-2010# taskset -p 0x2 2047
pid 2047’s current affinity mask: 3
pid 2047’s new affinity mask: 2
etch32:~/dev/modest.cvs/kmodest-src-2010# bin/umodest -p 2047 -s
Process with PID: 2047 is running on CPU: 1.

Now, after using taskset, our syslog process is being executed on CPU number 1, because of its affinity mask, that is, 0x2, i.e 0010, CPU number 1. (Don’t forget the first one is numbered as CPU number zero).

External links and references

Take a look at how to use schedutils and all about process’s affinity by reading this.

The latest kmodest source files with this new functionality can be downloaded from our CVS repository, right here.

There’s a new video showing how this new functionality works right here.