You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 19:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot5s.osadl.org (updated Fri Mar 20, 2026 12:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2523258995144,5cyclictest2582359-21kworker/u8:2+events_unbound10:09:492
2523258995131,9cyclictest131rcu_preempt08:40:152
2523251995040,7cyclictest2637794-21grep12:35:250
2523252994938,7cyclictest2566440-21grep09:10:251
2523252994934,11cyclictest231ktimers/112:10:121
2523265994821,22cyclictest0-21swapper/309:04:343
2523265994821,22cyclictest0-21swapper/309:04:343
2523258994840,5cyclictest2606566-21kworker/u8:3+events_unbound12:05:112
2523258994839,7cyclictest2558549-21kworker/u8:2+events_unbound09:44:412
2523258994836,8cyclictest2554702-21apt08:40:012
2523265994633,8cyclictest2554284-21grep08:35:253
252326599462,11cyclictest2622283-21unixbench_singl11:50:263
2523252994635,7cyclictest2601384-21cron10:55:001
2523265994519,23cyclictest391ktimers/308:05:113
2523258994537,4cyclictest2566871-21munin-plugin-st09:14:592
2523265994418,22cyclictest0-21swapper/310:20:003
2523252994435,5cyclictest2527233-21sendmail07:20:011
252326599433,6cyclictest2607273-21df11:10:133
2523265994322,16cyclictest0-21swapper/307:30:153
252326599430,11cyclictest2631682-21apt-get12:20:123
252326599430,11cyclictest2631682-21apt-get12:20:123
252326599424,32cyclictest2629042-21grep12:10:243
252326599422,5cyclictest2593977-21/usr/sbin/munin10:30:233
252326599422,14cyclictest2528910-21cron07:25:003
252326599421,9cyclictest2609389-21/usr/sbin/munin11:15:183
252326599421,9cyclictest2572016-21cron09:28:013
252326599421,8cyclictest2568083-21tail09:15:233
2523252994231,6cyclictest0-21swapper/112:05:101
252326599411,9cyclictest2549722-21/usr/sbin/munin08:25:133
252326599411,9cyclictest2549722-21/usr/sbin/munin08:25:133
252326599411,9cyclictest2533926-21cron07:39:003
252326599411,8cyclictest2636440-21sshd12:33:453
252326599411,8cyclictest2616562-21meminfo11:35:193
252326599411,7cyclictest2583689-21/usr/sbin/munin10:00:263
252326599411,7cyclictest2570395-21cron09:24:013
252326599411,5cyclictest2629517-21apt-get12:15:063
252326599411,11cyclictest2603665-21df11:00:123
252326599411,11cyclictest2603665-21df11:00:123
252326599411,11cyclictest2580266-21unixbench_multi09:50:243
252326599411,10cyclictest2625229-21ntp_states12:00:223
2523258994133,6cyclictest2624425-21sendmail12:00:002
2523252994128,8cyclictest0-21swapper/108:40:011
2523252994127,8cyclictest2547266-21grep08:15:261
2523251994132,6cyclictest0-21swapper/010:15:150
252326599403,6cyclictest2542171-21users08:00:263
2523265994030,5cyclictest0-21swapper/311:00:003
2523265994030,5cyclictest0-21swapper/311:00:003
252326599401,9cyclictest2525731-21apt-get07:15:123
252326599401,8cyclictest2581180-21/usr/sbin/munin09:55:163
252326599401,5cyclictest2596639-21df_inode10:40:133
2523265994011,25cyclictest0-21swapper/309:40:233
252326599401,11cyclictest2556010-21grep08:40:233
2523265994010,26cyclictest0-21swapper/308:20:193
252326599400,9cyclictest2620804-21sshd11:47:453
2523258994033,4cyclictest2601387-21cron10:55:002
2523252994034,4cyclictest2532552-21kworker/u8:2+events_unbound07:59:251
2523251994027,10cyclictest121ktimers/010:55:110
2523251994027,10cyclictest121ktimers/010:55:110
252326599398,28cyclictest2586752-21/usr/sbin/munin10:10:233
252326599391,8cyclictest2627330-21/usr/sbin/munin12:05:213
252326599391,8cyclictest2585555-21sshd10:06:453
252326599391,8cyclictest2529284-21forks07:25:153
252326599391,7cyclictest2523509-21/usr/sbin/munin07:10:133
252326599391,6cyclictest2559852-21cron08:53:003
252326599391,4cyclictest2595325-21/usr/sbin/munin10:35:203
2523265993914,21cyclictest0-21swapper/307:50:163
252326599391,11cyclictest2566468-21users09:10:263
252326599391,11cyclictest2557312-21memory08:45:183
252326599391,10cyclictest2535975-21/usr/sbin/munin07:45:153
252326599391,10cyclictest2535975-21/usr/sbin/munin07:45:153
2523258993923,6cyclictest2550636-21grep08:25:252
2523258993923,6cyclictest2550636-21grep08:25:252
2523252993929,6cyclictest2560404-21cut08:55:161
2523252993929,6cyclictest2560404-21cut08:55:161
252326599389,26cyclictest2560649-21sort08:55:183
252326599389,26cyclictest2560649-21sort08:55:183
252326599383,30cyclictest2540850-21munin-run08:00:003
2523265993826,7cyclictest0-21swapper/311:55:263
252326599382,5cyclictest2606573-21sshd11:05:463
252326599382,3cyclictest2534432-21/usr/sbin/munin07:40:163
252326599381,9cyclictest2545554-21grep08:10:253
252326599381,6cyclictest2590189-21/usr/sbin/munin10:20:223
252326599381,5cyclictest2574235-21df09:35:143
252326599381,4cyclictest2551982-21meminfo08:30:193
252326599381,3cyclictest2612505-21/usr/sbin/munin11:25:183
252326599381,11cyclictest2598893-21/usr/sbin/munin10:45:223
252326599381,11cyclictest2563891-21grep09:05:173
252325299386,4cyclictest131rcu_preempt11:15:131
2523251993829,4cyclictest2633163-21munin-plugin-st12:25:000
2523251993829,4cyclictest2633163-21munin-plugin-st12:25:000
2523251993824,9cyclictest2623951-21grep11:55:250
252326599372,4cyclictest2591403-21/usr/sbin/munin10:25:183
252326599371,9cyclictest2618910-21users11:40:243
252326599371,9cyclictest2611117-21if_eth111:20:173
252326599371,8cyclictest2546197-21/usr/sbin/munin08:15:173
252326599371,7cyclictest2577774-21fschecks_time09:45:153
252326599371,4cyclictest2633935-21ntpq12:25:203
252326599371,4cyclictest2614202-21/usr/sbin/munin11:30:243
2523258993730,4cyclictest2596277-21cron10:40:002
2523258993729,4cyclictest0-21swapper/209:30:252
2523252993728,5cyclictest0-21swapper/108:15:011
252325199378,10cyclictest11-21ksoftirqd/007:15:220
252326599363,7cyclictest2573435-21grep09:30:253
252326599361,8cyclictest2600607-21sendmail_mailqu10:50:223
252326599361,11cyclictest2637363-21ntp_states12:35:223
2523251993629,4cyclictest0-21swapper/008:10:010
2523258993529,4cyclictest2588933-21kworker/u8:1+events_unbound10:24:522
2523258993522,10cyclictest311ktimers/208:15:112
2523252993527,4cyclictest0-21swapper/111:45:061
2523252993527,4cyclictest0-21swapper/111:15:001
2523252993527,4cyclictest0-21swapper/108:30:001
2523252993527,4cyclictest0-21swapper/108:30:001
2523252993526,5cyclictest2585844-21munin-plugin-st10:10:001
252325199356,11cyclictest2525089-21cat07:15:010
2523251993525,6cyclictest2606925-21cron11:10:000
252325899348,6cyclictest2618817-21unixbench_multi11:40:232
2523258993427,5cyclictest2602996-21kworker/u8:0+events_unbound11:36:412
2523258993427,4cyclictest0-21swapper/210:15:252
252325299343,4cyclictest131rcu_preempt10:55:111
252325299343,4cyclictest131rcu_preempt10:55:111
2523252993426,4cyclictest0-21swapper/111:30:161
2523252993425,4cyclictest0-21swapper/109:30:011
252325299340,4cyclictest131rcu_preempt08:50:131
252325199349,5cyclictest11-21ksoftirqd/009:50:190
252325199346,10cyclictest2604886-21chrt11:03:300
252325199346,10cyclictest2604886-21chrt11:03:300
252325199345,11cyclictest2572423-21/usr/sbin/munin09:30:130
2523251993426,5cyclictest2630981-21unixbench_multi12:15:250
2523251993426,4cyclictest0-21swapper/009:29:380
2523251993410,5cyclictest11-21ksoftirqd/008:40:020
2523258993327,3cyclictest0-21swapper/211:25:012
2523258993325,4cyclictest0-21swapper/210:25:142
2523258993325,4cyclictest0-21swapper/208:05:002
2523252993323,7cyclictest2616273-21fschecks_count11:35:151
2523252993322,5cyclictest2577681-21/usr/sbin/munin09:45:141
252325299330,14cyclictest0-21swapper/111:25:121
252325199339,9cyclictest0-21swapper/010:55:010
252325199336,10cyclictest2549675-21egrep08:25:130
252325199336,10cyclictest2549675-21egrep08:25:130
2523251993322,7cyclictest2596301-21sendmail10:40:010
2523258993225,4cyclictest0-21swapper/209:05:172
2523258993223,6cyclictest0-21swapper/212:40:012
2523252993226,3cyclictest0-21swapper/109:15:191
2523252993225,5cyclictest2530608-21kworker/u8:3+events_unbound07:54:251
2523252993224,4cyclictest0-21swapper/107:45:271
2523252993224,4cyclictest0-21swapper/107:45:271
252325299321,15cyclictest2531896-21grep07:30:251
252325199328,7cyclictest0-21swapper/010:25:190
252325199327,6cyclictest11-21ksoftirqd/011:35:010
252325199325,8cyclictest2531283-21cat07:30:180
2523251993222,6cyclictest0-21swapper/012:05:220
2523251993221,7cyclictest0-21swapper/009:10:250
2523258993125,4cyclictest2507287-21kworker/u8:0+events_unbound08:09:272
2523258993124,4cyclictest0-21swapper/211:05:152
2523258993123,6cyclictest2586303-21kworker/u8:0+events_unbound10:40:422
2523258993123,6cyclictest2558549-21kworker/u8:2+events_unbound09:04:352
2523258993123,6cyclictest2558549-21kworker/u8:2+events_unbound09:04:352
2523258993123,4cyclictest0-21swapper/211:25:162
2523252993120,7cyclictest2580294-21grep09:50:241
2523252993119,8cyclictest2599697-21idleruntime-cro10:50:001
2523252993110,6cyclictest131rcu_preempt08:20:111
252325199318,6cyclictest0-21swapper/009:45:130
2523251993125,4cyclictest2591099-21latency_hist10:25:010
2523251993124,3cyclictest2568583-21apt-get09:20:010
2523251993124,3cyclictest0-21swapper/009:40:010
2523251993122,6cyclictest0-21swapper/012:00:260
2523251993120,7cyclictest2629096-21grep12:10:240
2523251993120,7cyclictest0-21swapper/007:50:230
252325899306,14cyclictest131rcu_preempt07:15:242
252325899305,5cyclictest131rcu_preempt10:00:262
252325299304,4cyclictest131rcu_preempt11:20:251
2523252993023,5cyclictest2582359-21kworker/u8:2+events_unbound10:00:011
2523252993010,5cyclictest131rcu_preempt07:20:141
252325199308,13cyclictest11-21ksoftirqd/012:25:120
252325199305,8cyclictest11-21ksoftirqd/008:55:010
2523251993023,3cyclictest0-21swapper/009:20:250
2523251993023,3cyclictest0-21swapper/008:15:160
2523251993022,4cyclictest2582410-21cron10:00:000
2523251993022,4cyclictest0-21swapper/007:40:000
2523251993020,6cyclictest0-21swapper/010:00:260
2523251993019,6cyclictest0-21swapper/007:55:250
2523251993010,4cyclictest151rcuc/010:40:120
2523258992923,4cyclictest2544501-21kworker/u8:1+events_unbound08:58:062
2523258992923,4cyclictest2544501-21kworker/u8:1+events_unbound08:58:062
2523258992923,4cyclictest2507287-21kworker/u8:0+events_unbound07:40:372
2523258992923,3cyclictest0-21swapper/211:05:002
2523258992923,3cyclictest0-21swapper/211:05:002
2523258992921,4cyclictest0-21swapper/212:20:122
2523258992921,4cyclictest0-21swapper/212:20:122
2523258992919,7cyclictest0-21swapper/211:50:002
252325899291,6cyclictest0-21swapper/207:10:122
252325299299,11cyclictest0-21swapper/107:44:271
252325299298,12cyclictest131rcu_preempt12:35:001
252325299294,4cyclictest131rcu_preempt12:05:001
2523252992922,4cyclictest2587547-21munin-plugin-st10:15:001
2523252992920,5cyclictest0-21swapper/112:00:001
2523252992918,7cyclictest0-21swapper/110:20:001
2523252992917,4cyclictest131rcu_preempt11:55:011
252325199298,13cyclictest11-21ksoftirqd/010:10:210
2523251992919,6cyclictest0-21swapper/008:30:250
2523251992916,9cyclictest2636108-21grep12:30:240
2523258992821,5cyclictest2532552-21kworker/u8:2+events_unbound07:46:412
2523258992821,5cyclictest2532552-21kworker/u8:2+events_unbound07:46:412
2523258992821,4cyclictest0-21swapper/209:50:242
2523258992820,4cyclictest0-21swapper/207:55:252
2523258992816,4cyclictest2632831-21kworker/u8:3+events_unbound12:29:062
2523258992815,9cyclictest311ktimers/209:35:122
252325899281,16cyclictest0-21swapper/211:30:122
252325299288,6cyclictest131rcu_preempt09:30:251
252325299287,12cyclictest24-21ksoftirqd/107:40:001
252325299286,5cyclictest131rcu_preempt09:40:251
2523252992822,4cyclictest2588933-21kworker/u8:1+events_unbound10:39:561
2523252992820,4cyclictest0-21swapper/112:20:191
2523252992820,4cyclictest0-21swapper/112:20:191
2523252992820,4cyclictest0-21swapper/109:05:271
2523252992820,4cyclictest0-21swapper/108:05:001
2523252992813,8cyclictest131rcu_preempt08:30:231
252325199288,6cyclictest0-21swapper/011:15:210
252325199286,4cyclictest0-21swapper/008:55:240
252325199286,4cyclictest0-21swapper/008:55:240
2523251992822,3cyclictest0-21swapper/011:25:250
2523251992819,5cyclictest0-21swapper/011:45:240
2523251992818,6cyclictest0-21swapper/010:50:000
252325899279,3cyclictest131rcu_preempt11:15:172
252325899279,3cyclictest131rcu_preempt07:35:232
252325899272,7cyclictest2548913-21grep08:20:242
252325899272,3cyclictest131rcu_preempt08:30:122
2523258992720,4cyclictest0-21swapper/210:00:002
2523258992719,6cyclictest2570200-21kworker/u8:1+events_unbound09:25:132
252325299278,5cyclictest131rcu_preempt11:07:161
252325299275,4cyclictest131rcu_preempt09:00:251
252325299275,4cyclictest131rcu_preempt09:00:251
2523252992721,4cyclictest2571925-21kworker/u8:0+events_unbound09:35:221
2523252992720,5cyclictest2544501-21kworker/u8:1+events_unbound08:45:141
2523252992720,4cyclictest0-21swapper/112:35:251
2523252992719,4cyclictest0-21swapper/108:44:301
2523252992719,4cyclictest0-21swapper/107:10:251
2523252992717,6cyclictest0-21swapper/108:10:011
252325199277,14cyclictest131rcu_preempt08:14:490
2523251992720,4cyclictest0-21swapper/009:00:250
2523251992720,4cyclictest0-21swapper/009:00:250
2523251992719,5cyclictest0-21swapper/011:40:140
2523251992718,3cyclictest131rcu_preempt08:45:520
2523251992716,6cyclictest0-21swapper/010:05:290
2523251992714,10cyclictest2529152-21apt-get07:25:110
252325899266,4cyclictest131rcu_preempt08:45:242
2523258992621,2cyclictest2507287-21kworker/u8:0+events_unbound07:52:402
2523258992619,3cyclictest0-21swapper/211:10:152
2523258992619,3cyclictest0-21swapper/210:45:252
2523258992618,5cyclictest0-21swapper/210:30:242
2523258992616,7cyclictest2532317-21sh07:35:002
2523252992619,5cyclictest2517445-21kworker/u8:2+events_unbound07:25:151
2523252992618,6cyclictest2570200-21kworker/u8:1+events_unbound10:03:251
2523252992617,7cyclictest2632831-21kworker/u8:3+events_unbound12:29:101
2523252992616,7cyclictest231ktimers/112:15:111
252325299261,13cyclictest0-21swapper/111:41:541
252325199269,12cyclictest0-21swapper/009:05:130
2523251992620,3cyclictest0-21swapper/007:45:000
2523251992617,5cyclictest0-21swapper/011:50:260
2523251992614,9cyclictest2607906-21tr11:10:230
2523251992613,3cyclictest131rcu_preempt09:44:240
2523251992610,4cyclictest0-21swapper/008:40:010
252325899257,6cyclictest32-21ksoftirqd/210:55:242
252325899257,6cyclictest32-21ksoftirqd/210:55:242
252325899254,3cyclictest131rcu_preempt12:15:252
2523258992519,4cyclictest2507287-21kworker/u8:0+events_unbound07:25:372
2523258992518,5cyclictest2507287-21kworker/u8:0+events_unbound08:13:582
2523258992518,4cyclictest2622302-21grep11:50:262
2523258992517,6cyclictest2578693-21kworker/u8:3+events_unbound10:14:342
2523258992517,5cyclictest2578693-21kworker/u8:3+events_unbound09:46:492
252325899251,2cyclictest0-21swapper/212:00:132
2523252992516,5cyclictest0-21swapper/111:05:001
2523252992516,5cyclictest0-21swapper/111:05:001
252325299251,15cyclictest0-21swapper/110:33:451
252325199254,7cyclictest2593337-21perl10:30:150
2523251992518,4cyclictest2617609-21idleruntime-cro11:40:010
2523251992518,3cyclictest0-21swapper/008:00:260
2523251992516,3cyclictest131rcu_preempt07:47:380
2523251992516,3cyclictest131rcu_preempt07:47:380
2523251992515,7cyclictest2548970-21grep08:20:250
2523258992418,4cyclictest2517445-21kworker/u8:2+events_unbound07:20:082
252325899240,13cyclictest0-21swapper/209:15:102
252325899240,11cyclictest0-21swapper/209:23:412
252325299245,9cyclictest131rcu_preempt09:20:241
2523252992418,4cyclictest2586303-21kworker/u8:0+events_unbound10:25:191
252325299240,14cyclictest0-21swapper/110:43:571
2523251992414,7cyclictest2528498-21grep07:20:240
252325199241,19cyclictest0-21swapper/011:20:300
2523258992317,4cyclictest2622611-21kworker/u8:2+events_unbound12:11:432
252325299232,4cyclictest131rcu_preempt10:20:231
252325899223,3cyclictest131rcu_preempt08:50:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional