You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 11:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 100 highest latencies:
System rack7slot7s.osadl.org (updated Thu Nov 20, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8589210086,9sleep10-21swapper/119:08:081
872129985,9sleep30-21swapper/319:09:553
852429383,6sleep20-21swapper/219:07:172
744728066,9sleep00-21swapper/019:05:210
8805992813,3cyclictest13-21ksoftirqd/023:56:420
8805992622,1cyclictest13-21ksoftirqd/023:20:500
8805992621,1cyclictest13-21ksoftirqd/000:29:030
8805992621,1cyclictest13-21ksoftirqd/000:29:020
8818992322,1cyclictest0-21swapper/321:26:563
881899223,17cyclictest19092-21diskmemload00:08:253
881899222,19cyclictest17868-21latency_hist19:59:593
881899221,15cyclictest0-21swapper/322:20:193
8818992120,1cyclictest0-21swapper/322:10:203
880599212,14cyclictest13-21ksoftirqd/022:25:240
880599210,5cyclictest161rcuc/022:35:140
880599207,5cyclictest13-21ksoftirqd/021:00:170
8805992017,2cyclictest13-21ksoftirqd/023:50:160
880599200,19cyclictest0-21swapper/000:32:530
208992200,0sleep30-21swapper/321:35:033
881899190,18cyclictest0-21swapper/323:52:593
8814991917,1cyclictest8489-21fschecks_count21:25:202
8814991915,2cyclictest141rcu_preempt00:18:342
880599197,8cyclictest13-21ksoftirqd/023:07:060
8805991914,2cyclictest13-21ksoftirqd/023:45:240
880599190,1cyclictest0-21swapper/021:15:120
880599190,16cyclictest13-21ksoftirqd/023:40:150
881899181,2cyclictest0-21swapper/320:30:183
881899181,14cyclictest0-21swapper/320:25:143
881499187,10cyclictest32-21ksoftirqd/221:42:202
880999188,9cyclictest24987-21apache222:56:491
880599189,1cyclictest141rcu_preempt21:23:130
880599181,5cyclictest13-21ksoftirqd/022:15:170
880599180,1cyclictest0-21swapper/022:33:510
8818991715,1cyclictest0-21swapper/321:30:123
881499172,2cyclictest0-21swapper/222:19:302
8809991716,1cyclictest0-21swapper/100:27:191
8809991716,1cyclictest0-21swapper/100:27:181
880599176,7cyclictest141rcu_preempt00:20:010
880599171,6cyclictest13-21ksoftirqd/021:28:540
8805991713,2cyclictest1937-21ssh00:02:360
880599171,2cyclictest141rcu_preempt23:39:200
8814991614,1cyclictest0-21swapper/223:22:082
881499160,15cyclictest0-21swapper/223:01:082
8809991615,1cyclictest0-21swapper/100:15:091
880599161,1cyclictest141rcu_preempt22:20:250
8818991513,1cyclictest16536-21sh00:12:313
881499153,4cyclictest8803-21cyclictest22:59:022
881499151,2cyclictest141rcu_preempt21:10:222
881499151,2cyclictest141rcu_preempt21:10:212
880599159,5cyclictest13-21ksoftirqd/000:10:170
880599154,9cyclictest14360-21apt-get21:30:120
880599153,3cyclictest141rcu_preempt21:35:140
880599152,6cyclictest11268-21munin-run19:30:010
880599151,7cyclictest13-21ksoftirqd/021:10:230
880599151,7cyclictest13-21ksoftirqd/021:10:230
880599150,0cyclictest0-21swapper/022:02:340
881899145,3cyclictest224-21systemd-journal22:07:203
881899143,10cyclictest0-21swapper/322:29:203
8818991411,2cyclictest987-21cat21:20:093
8818991410,3cyclictest13213-21grep00:35:243
881899140,14cyclictest0-21swapper/321:56:163
8814991411,2cyclictest19091-21turbostat21:10:002
881499140,6cyclictest0-21swapper/223:35:112
881499140,12cyclictest0-21swapper/222:06:232
881499140,10cyclictest0-21swapper/222:38:002
880999141,12cyclictest0-21swapper/122:05:131
880599145,4cyclictest141rcu_preempt23:16:400
880599143,1cyclictest141rcu_preempt23:26:220
8805991411,2cyclictest13-21ksoftirqd/022:48:520
8805991411,2cyclictest13-21ksoftirqd/020:40:010
8805991411,2cyclictest13-21ksoftirqd/000:20:230
880599140,2cyclictest0-21swapper/020:05:110
881899139,3cyclictest8699-21wc20:45:213
881899139,3cyclictest23241-21grep23:55:143
881899139,3cyclictest22871-21tr22:00:173
881899139,3cyclictest13622-21tr23:25:243
8818991311,1cyclictest0-21swapper/323:49:393
8818991310,2cyclictest29067-21head20:20:223
8818991310,2cyclictest22387-21grep22:45:273
8818991310,2cyclictest22049-21wc20:05:253
8818991310,2cyclictest19910-21tr21:10:173
8818991310,2cyclictest19910-21tr21:10:163
8818991310,2cyclictest15158-21turbostat.cron21:55:003
8818991310,2cyclictest1431-21grep23:40:113
8818991310,2cyclictest12564-21grep22:15:273
881899130,13cyclictest0-21swapper/322:52:173
881499136,3cyclictest32-21ksoftirqd/221:48:582
881499133,1cyclictest32-21ksoftirqd/222:25:052
8814991312,1cyclictest17693-21kworker/2:120:25:212
8814991312,1cyclictest0-21swapper/223:15:142
8814991311,1cyclictest37850irq/131-snd_hda23:56:132
881499131,10cyclictest1158-21apt-get23:40:092
8814991310,2cyclictest31918-21grep00:00:262
881499130,1cyclictest0-21swapper/221:50:162
881499130,13cyclictest0-21swapper/222:03:092
881499130,0cyclictest141rcu_preempt20:40:212
880999139,2cyclictest0-21swapper/122:45:141
880999132,5cyclictest224-21systemd-journal23:34:591
8809991310,2cyclictest515-21cat22:30:211
8809991310,2cyclictest32435-21cron20:29:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional