Collecting HW counters for the entire system

[zaddan]

Hi,

Question 1:
Is it possible to collect data for the entire system, i.e, not for a specific executable, but rather let the counters collect data regardless of what's running on the system for some duration of time?

Is it possible to simply do:
perf = Profiler(events_groups=events) # with some events that we've already defined
while (true):
perf.start_counters(0)
time.sleep(20) # sleep for some time
data = perf.read_events()

in order to do system wide data collection (for the duration of 20 sec at a time)?

PS: would prefixing the name of the event with 'SYSTEMWIDE' do the trick?

Question 2:
Is it possible to set a sampling rate for the question above?

Thank you very much for your tool!

[VitorRamos]

Hello,

Question 1:

Unfortunate linux does not provide a straightforward way to collect performance counter systemwide across all CPUs. We can only do it by a specific CPU, as you can see from the perf_event_open syscall specification:

perf_event_open(struct perf_event_attr *attr,
                           pid_t pid, int cpu, int group_fd,
                           unsigned long flags)

       The pid and cpu arguments allow specifying which process and CPU to
       monitor:

       pid == 0 and cpu == -1
              This measures the calling process/thread on any CPU.

       pid == 0 and cpu >= 0
              This measures the calling process/thread only when running on
              the specified CPU.

       pid > 0 and cpu == -1
              This measures the specified process/thread on any CPU.

       pid > 0 and cpu >= 0
              This measures the specified process/thread only when running
              on the specified CPU.

       pid == -1 and cpu >= 0
              This measures all processes/threads on the specified CPU.
              This requires CAP_SYS_ADMIN capability or a
              /proc/sys/kernel/perf_event_paranoid value of less than 1.

       pid == -1 and cpu == -1
              This setting is invalid and will return an error.

However, some counters are not CPU specific which means you can measure the entire system by just creating an event to any CPU. The SYSTEMWIDE at the beginning of the event will measure for all processes/threads only on CPU 0, but in a case of some counters like the energy ones "SYSTEMWIDE:RAPL_ENERGY_PKG" will measure the energy of all CPUs.

Measuring for all CPUs for any counter is something that I want to implement in the future but will require creating multiple events one for each CPU.

Question 2:

No, only when attached to a program.

Thanks !! it's cool to see that are people using :D

[zaddan]

Hi,

Thanks for the quick response.

Regarding question 1: I took a look at your code and if I understand correctly, start_counters only get pid as an input? Is it possible to pass in the cpu id (instead of process id) to collect counters for a cpu (regardless of the threads running on it)?

Regarding question 2: what is the default sampling rate?

Thanks again for your tool! It's truly critical for the paper that I am working on publishing.

[VitorRamos]

1:
Currently, only the process id, but including the cpu id as a parameter is a good idea.

2:
When you use start_counters() there is no sampling, they will count until you manually reset them with reset_events().

What kind of paper are you working on?
If you are interested, there is a paper with more details of this tool that I published back when I developed.
https://ieeexplore.ieee.org/abstract/document/8778199/references#references

[zaddan]

1: How hard is it to modify your code to implement this? I believe you are passing -1 at the moment for cpu id but I should be extending the API to use the user passed argument instead (or it's more difficult than that)?

2: Thanks that makes sense.

3:I am working on a robotics paper but the the main focus is on the system (compute), so I'd like to use your tool to enable the data collection on the system side. Will happily reference your paper, and much appreciate it that you are helping with the tool!

[VitorRamos]

1:
You will only need to modify the perf_event_open adding a cpuid parameter, the -1 is for the process id. Something like this should be enough:

def __create_events(self, pid, cpuid=0):
    ...
    if 'SYSTEMWIDE' in e_name:
        fd= perfmon.perf_event_open(e, -1, cpuid, -1, 0)
    ...

def start_counters(self, pid, cpuid=0):
    ...
    self.__create_events(pid, cpuid)
    ...

3:
Cool, hope this helps.

[zaddan]

Thank you very much for being very responsive! I was able to get the tool collect data for different cpus.