You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 17:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot5s.osadl.org (updated Thu Sep 03, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2602695995240,8cyclictest391ktimers/307:10:123
2602694994936,8cyclictest2609654-21grep07:25:272
2602694994840,6cyclictest2694205-21kworker/u8:1+events_unbound11:35:552
2602693994840,4cyclictest2632464-21munin-plugin-st08:35:011
2602692994843,3cyclictest0-21swapper/008:40:130
2602694994737,6cyclictest2685353-21grep11:05:252
2602693994634,9cyclictest231ktimers/110:25:111
2602694994538,5cyclictest2638899-21kworker/u8:2+events_unbound09:15:352
2602694994526,5cyclictest131rcu_preempt09:51:392
2602693994524,17cyclictest0-21swapper/107:35:261
2602692994538,4cyclictest2696008-21munin-plugin-st11:40:000
2602694994436,6cyclictest2704674-21kworker/u8:2+events_unbound12:10:582
2602693994433,7cyclictest2625211-21grep08:10:251
2602693994429,10cyclictest2640557-21grep08:55:251
260269399442,10cyclictest2690415-21grep11:20:251
2602694994334,7cyclictest2680392-21kworker/u8:0+events_unbound10:55:492
2602693994332,7cyclictest2648018-21sh09:20:011
2602693994330,8cyclictest2633715-21grep08:35:251
260269399431,12cyclictest2699868-21apt-get11:50:151
260269399431,12cyclictest2689181-21sh11:20:001
260269399431,10cyclictest2648271-21apt-get09:20:131
2602694994234,4cyclictest2716671-21cron12:40:002
260269399422,4cyclictest2703621-21/usr/sbin/munin12:00:181
260269399422,4cyclictest2644849-21apt-get09:10:121
260269399421,9cyclictest2691468-21irqstats11:25:171
260269399417,9cyclictest2643762-21sed09:05:271
260269399413,6cyclictest2651036-21grep09:25:251
2602693994133,5cyclictest2620508-21munin-run08:00:001
260269399412,15cyclictest2682139-21apt-get10:59:591
260269399411,8cyclictest2698649-21/usr/sbin/munin11:45:221
260269399411,8cyclictest2669068-21/usr/sbin/munin10:20:171
260269399411,7cyclictest2615636-21/usr/sbin/munin07:45:261
260269399411,12cyclictest2627694-21df_inode08:20:151
2602693994111,25cyclictest2694747-21fschecks_time11:35:161
2602693994110,27cyclictest2716704-21tr12:40:001
2602692994123,6cyclictest131rcu_preempt11:00:260
2602695994034,4cyclictest2698272-21kworker/u8:2+events_unbound11:45:553
2602694994033,4cyclictest2709373-21unixbench_multi12:15:252
260269399403,4cyclictest2679079-21df_inode10:50:131
260269399402,5cyclictest2602690-21cyclictest09:55:121
260269399402,4cyclictest2665324-21/usr/sbin/munin10:10:161
260269399401,8cyclictest2638145-21/usr/sbin/munin08:50:191
260269399401,7cyclictest2706696-21cpu12:10:171
260269399401,7cyclictest2641751-21/usr/sbin/munin09:00:211
260269399401,7cyclictest2641751-21/usr/sbin/munin09:00:211
260269399401,5cyclictest2605178-21/usr/sbin/munin07:15:161
260269399401,4cyclictest2696365-21df_inode11:40:131
260269399401,11cyclictest164-21systemd-journal08:00:111
260269399401,10cyclictest2636831-21/usr/sbin/munin08:45:231
2602693994010,27cyclictest2634565-21fschecks_count08:40:161
2602695993929,7cyclictest2611339-21unixbench_singl07:30:253
2602694993933,4cyclictest2638899-21kworker/u8:2+events_unbound09:10:362
260269399399,26cyclictest1-21systemd07:15:021
260269399394,6cyclictest2653307-21cut09:35:011
260269399392,8cyclictest164-21systemd-journal11:10:501
260269399392,7cyclictest1-21systemd07:25:031
260269399392,4cyclictest2610751-21meminfo07:30:181
2602693993922,14cyclictest0-21swapper/108:15:261
260269399392,11cyclictest2674903-21head10:35:261
260269399391,9cyclictest2675903-21irqstats10:40:171
260269399391,9cyclictest2675903-21irqstats10:40:171
260269399391,8cyclictest2655666-21/usr/sbin/munin09:40:221
260269399391,5cyclictest2629732-21memory08:25:201
260269399391,12cyclictest2708572-21fschecks_time12:15:151
260269399391,12cyclictest2657846-21unixbench_singl09:45:241
260269399391,10cyclictest2672778-21ps10:30:261
260269399391,10cyclictest2672778-21ps10:30:261
2602692993932,4cyclictest0-21swapper/009:54:590
2602692993927,8cyclictest2649317-21grep09:20:240
2602694993831,5cyclictest2628842-21kworker/u8:3+events_unbound08:24:422
2602694993829,7cyclictest311ktimers/212:00:132
2602694993827,7cyclictest0-21swapper/208:05:252
2602694993823,9cyclictest131rcu_preempt11:40:162
2602694993816,5cyclictest131rcu_preempt10:45:262
260269399382,5cyclictest2662255-21irqstats10:00:171
260269399381,9cyclictest2659582-21users09:50:301
260269399381,8cyclictest2618037-21sendmail_mailst07:50:251
260269399381,7cyclictest2685084-21switchtime11:05:241
260269399381,5cyclictest629-21snmpd12:09:521
260269399381,5cyclictest2714548-21grep12:30:291
260269399381,5cyclictest2682139-21apt-get11:00:051
260269399381,5cyclictest2622569-21sshd08:05:141
260269399381,11cyclictest2702011-21memory11:55:161
2602692993810,4cyclictest151rcuc/009:05:170
2602692993810,13cyclictest11-21ksoftirqd/010:40:200
2602692993810,13cyclictest11-21ksoftirqd/010:40:190
260269599371,13cyclictest0-21swapper/312:25:113
2602694993728,5cyclictest0-21swapper/209:45:382
2602694993725,4cyclictest131rcu_preempt10:35:262
2602694993725,4cyclictest131rcu_preempt10:20:142
2602694993718,5cyclictest131rcu_preempt12:25:012
260269399377,3cyclictest0-21swapper/110:50:011
2602693993727,6cyclictest2667182-21fschecks_count10:15:161
260269399372,6cyclictest2693702-21sed11:30:291
260269399372,6cyclictest2653490-21diskstats09:35:141
260269399372,4cyclictest2606900-21/usr/sbin/munin07:20:231
260269399371,7cyclictest2663702-21/usr/sbin/munin10:05:131
260269399371,5cyclictest2712257-21meminfo12:25:191
2602692993728,6cyclictest2706054-21grep12:05:260
2602695993629,4cyclictest2622234-21munin-plugin-st08:05:003
2602695993629,3cyclictest0-21swapper/309:50:193
2602694993630,4cyclictest2668170-21kworker/u8:0+events_unbound10:30:442
2602694993630,4cyclictest2668170-21kworker/u8:0+events_unbound10:30:432
2602694993629,4cyclictest2634161-21cron08:40:012
2602694993629,3cyclictest0-21swapper/208:20:002
2602694993626,6cyclictest0-21swapper/207:45:012
2602693993612,21cyclictest231ktimers/107:41:381
260269399361,10cyclictest2710680-21ntp_states12:20:211
260269299369,5cyclictest11-21ksoftirqd/010:00:150
260269299368,14cyclictest11-21ksoftirqd/011:20:160
2602692993627,6cyclictest2623526-21unixbench_singl08:05:250
2602692993610,5cyclictest11-21ksoftirqd/007:40:010
2602694993525,6cyclictest0-21swapper/210:15:162
2602694993518,4cyclictest131rcu_preempt09:02:332
2602694993518,4cyclictest131rcu_preempt09:02:332
2602694993517,5cyclictest131rcu_preempt09:35:222
2602692993530,3cyclictest0-21swapper/010:20:110
2602692993528,4cyclictest0-21swapper/008:55:250
2602692993526,6cyclictest2660362-21fschecks_count09:55:160
2602692993524,7cyclictest2625198-21grep08:10:250
2602692993522,7cyclictest0-21swapper/007:15:010
2602695993427,4cyclictest2692609-21cut11:30:003
2602695993427,4cyclictest0-21swapper/309:35:253
2602695993418,5cyclictest2694205-21kworker/u8:1+events_unbound11:43:523
2602694993428,4cyclictest2663287-21kworker/u8:2+events_unbound10:12:422
2602694993428,4cyclictest2638899-21kworker/u8:2+events_unbound08:55:292
2602694993427,3cyclictest0-21swapper/210:40:152
2602694993427,3cyclictest0-21swapper/210:40:152
2602694993422,5cyclictest131rcu_preempt08:45:172
2602694993418,5cyclictest131rcu_preempt11:20:172
2602694993415,10cyclictest131rcu_preempt07:55:132
2602694993414,11cyclictest131rcu_preempt09:25:242
2602694993414,10cyclictest131rcu_preempt11:33:032
260269299349,5cyclictest11-21ksoftirqd/012:15:180
260269299349,4cyclictest11-21ksoftirqd/007:20:010
2602692993424,6cyclictest2702561-21unixbench_multi11:55:220
260269299342,10cyclictest11-21ksoftirqd/010:46:260
2602692993418,6cyclictest131rcu_preempt07:23:250
2602695993326,4cyclictest0-21swapper/311:35:243
260269599331,13cyclictest2714511-21grep12:30:293
2602694993325,4cyclictest2643457-21ntp_kernel_err09:05:222
2602694993324,6cyclictest0-21swapper/207:35:262
2602694993322,8cyclictest311ktimers/209:55:252
2602694993319,5cyclictest131rcu_preempt07:30:202
2602694993318,9cyclictest131rcu_preempt09:30:202
2602694993318,4cyclictest131rcu_preempt08:33:292
2602694993317,4cyclictest131rcu_preempt07:22:232
2602694993315,3cyclictest131rcu_preempt07:45:172
2602694993314,4cyclictest131rcu_preempt09:21:342
260269299339,7cyclictest11-21ksoftirqd/012:20:250
260269299338,6cyclictest11-21ksoftirqd/011:10:240
2602692993325,4cyclictest2639288-21munin-plugin-st08:55:000
2602695993225,5cyclictest2659758-21kworker/u8:2+events_unbound09:55:403
2602695993225,4cyclictest0-21swapper/311:55:223
2602695993224,4cyclictest0-21swapper/308:30:013
2602695993216,13cyclictest2602690-21cyclictest10:25:113
2602695993216,11cyclictest391ktimers/312:15:113
260269599321,3cyclictest131rcu_preempt10:50:113
260269499329,14cyclictest32-21ksoftirqd/210:50:222
2602694993222,6cyclictest0-21swapper/210:05:242
2602694993219,5cyclictest131rcu_preempt12:25:152
2602694993218,4cyclictest131rcu_preempt09:40:122
2602694993217,5cyclictest131rcu_preempt11:15:212
2602694993215,4cyclictest131rcu_preempt11:45:222
2602694993215,10cyclictest131rcu_preempt11:25:132
2602694993214,4cyclictest131rcu_preempt08:50:202
2602694993214,10cyclictest131rcu_preempt10:25:152
2602694993212,11cyclictest0-21swapper/207:10:222
2602694993211,13cyclictest32-21ksoftirqd/212:05:172
2602694993211,10cyclictest32-21ksoftirqd/208:00:292
2602694993210,13cyclictest32-21ksoftirqd/212:30:292
260269299329,6cyclictest0-21swapper/012:25:200
260269299329,5cyclictest0-21swapper/009:45:220
260269299329,4cyclictest11-21ksoftirqd/008:30:240
260269299328,5cyclictest11-21ksoftirqd/008:35:240
260269299326,9cyclictest0-21swapper/008:25:200
260269299324,5cyclictest11-21ksoftirqd/010:35:260
2602692993221,7cyclictest2609675-21grep07:25:270
2602692993210,6cyclictest0-21swapper/010:10:000
2602695993123,4cyclictest0-21swapper/310:05:253
2602695993121,6cyclictest2676571-21unixbench_multi10:40:243
2602695993121,6cyclictest2676571-21unixbench_multi10:40:243
2602694993123,4cyclictest0-21swapper/208:25:202
2602694993116,5cyclictest131rcu_preempt08:40:152
2602694993116,4cyclictest131rcu_preempt08:13:182
2602694993111,11cyclictest0-21swapper/207:54:292
2602694993110,11cyclictest32-21ksoftirqd/207:15:162
260269299316,5cyclictest11-21ksoftirqd/011:10:000
260269299315,7cyclictest2655049-21latency_hist09:40:010
2602692993124,4cyclictest11-21ksoftirqd/011:25:150
2602692993124,3cyclictest0-21swapper/012:30:280
2602692993110,7cyclictest11-21ksoftirqd/007:55:160
2602692993110,6cyclictest11-21ksoftirqd/010:35:000
2602692993110,6cyclictest11-21ksoftirqd/010:35:000
2602692993110,5cyclictest11-21ksoftirqd/009:05:010
2602692993110,5cyclictest11-21ksoftirqd/009:05:000
2602695993024,4cyclictest2586284-21kworker/u8:0+events_unbound07:21:413
2602695993024,3cyclictest0-21swapper/309:00:003
2602695993023,4cyclictest0-21swapper/310:10:303
2602695993022,5cyclictest2637584-21idleruntime-cro08:50:003
2602695993020,6cyclictest0-21swapper/307:15:253
2602695993015,11cyclictest391ktimers/310:35:123
2602694993020,6cyclictest0-21swapper/211:55:222
2602694993019,3cyclictest131rcu_preempt11:00:582
2602694993018,5cyclictest2678685-21kworker/u8:3+events_unbound11:10:072
260269299309,6cyclictest11-21ksoftirqd/009:25:140
260269299308,7cyclictest0-21swapper/009:30:240
260269299307,7cyclictest0-21swapper/011:40:140
260269299302,12cyclictest2665722-21cat10:10:230
260269299301,11cyclictest2693270-21date11:30:220
2602695992923,4cyclictest2687226-21kworker/u8:0+events_unbound11:15:503
2602695992922,3cyclictest0-21swapper/309:15:153
2602695992922,3cyclictest0-21swapper/307:30:013
2602695992921,4cyclictest2694293-21cron11:35:003
2602695992921,4cyclictest0-21swapper/309:10:003
2602695992919,6cyclictest0-21swapper/308:30:253
2602695992918,5cyclictest2635863-21kworker/u8:0+events_unbound08:53:173
260269599290,6cyclictest0-21swapper/307:45:133
260269599290,3cyclictest131rcu_preempt09:10:133
260269499298,10cyclictest32-21ksoftirqd/211:50:232
260269499295,12cyclictest32-21ksoftirqd/210:05:012
260269299299,4cyclictest0-21swapper/011:50:190
260269299298,5cyclictest11-21ksoftirqd/010:55:010
260269299298,4cyclictest0-21swapper/010:20:010
260269299297,7cyclictest0-21swapper/009:15:200
260269299297,6cyclictest0-21swapper/011:15:130
2602692992919,7cyclictest2637100-21grep08:45:250
2602695992822,4cyclictest2680392-21kworker/u8:0+events_unbound10:58:253
2602695992821,4cyclictest0-21swapper/309:25:243
2602695992821,3cyclictest0-21swapper/309:25:013
2602695992820,6cyclictest2627570-21kworker/u8:0+events_unbound08:20:193
260269299285,8cyclictest0-21swapper/009:10:160
260269299284,7cyclictest0-21swapper/007:50:160
2602692992820,5cyclictest0-21swapper/012:05:000
2602692992819,4cyclictest0-21swapper/009:40:260
2602692992818,7cyclictest2698174-21fschecks_time11:45:140
260269599277,10cyclictest131rcu_preempt09:05:003
260269599277,10cyclictest131rcu_preempt09:04:593
2602695992721,4cyclictest2685417-21kworker/u8:2+events_unbound11:05:503
2602695992721,4cyclictest2671650-21kworker/u8:1+events_unbound10:30:213
2602695992721,4cyclictest2671650-21kworker/u8:1+events_unbound10:30:213
2602695992720,5cyclictest2632310-21kworker/u8:1+events_unbound08:35:163
2602695992719,6cyclictest2651080-21kworker/u8:0+events_unbound09:30:163
2602695992719,5cyclictest0-21swapper/311:50:283
2602695992719,5cyclictest0-21swapper/308:00:003
2602695992718,5cyclictest2716690-21cut12:40:003
2602695992710,5cyclictest2680392-21kworker/u8:0+events_unbound11:00:053
260269299278,6cyclictest0-21swapper/012:35:180
260269299276,6cyclictest0-21swapper/008:20:150
2602695992620,4cyclictest2678685-21kworker/u8:3+events_unbound11:11:043
2602695992619,4cyclictest0-21swapper/308:10:253
260269299267,6cyclictest0-21swapper/007:45:220
260269299267,5cyclictest11-21ksoftirqd/008:20:010
2602692992615,6cyclictest0-21swapper/011:00:000
260269599259,5cyclictest2647127-21kworker/u8:1+events_unbound09:42:193
260269599254,4cyclictest131rcu_preempt09:45:253
260269599252,4cyclictest0-21swapper/312:00:103
260269599252,4cyclictest0-21swapper/308:05:113
2602695992518,5cyclictest2704674-21kworker/u8:2+events_unbound12:09:023
2602695992518,3cyclictest0-21swapper/310:05:013
2602695992511,11cyclictest381rcuc/307:40:013
260269299253,4cyclictest151rcuc/008:00:230
2602692992518,4cyclictest0-21swapper/007:45:010
260269299251,19cyclictest2708309-21cat12:15:010
260269599249,3cyclictest0-21swapper/310:20:193
260269599246,3cyclictest131rcu_preempt08:40:173
260269599246,10cyclictest40-21ksoftirqd/310:45:273
260269599245,4cyclictest131rcu_preempt12:10:203
2602695992417,4cyclictest2615068-21kworker/u8:2+events_unbound08:17:023
2602695992415,6cyclictest2711153-21grep12:20:253
260269599241,4cyclictest131rcu_preempt07:50:133
260269599240,3cyclictest2689432-21apt-get11:20:113
260269299239,4cyclictest131rcu_preempt10:25:270
2602692992310,7cyclictest131rcu_preempt07:31:240
2602695992216,3cyclictest0-21swapper/307:45:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional