You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 04:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Apr 10, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
35472235160,22sleep00-21swapper/019:05:480
35472235160,22sleep00-21swapper/019:05:480
22552224162,55sleep20-21swapper/219:05:142
22552224162,55sleep20-21swapper/219:05:132
36472201166,23sleep10-21swapper/119:07:061
36472201166,23sleep10-21swapper/119:07:051
35302201166,23sleep30-21swapper/319:05:343
35302201166,23sleep30-21swapper/319:05:343
3963996158,2cyclictest0-21swapper/223:26:502
3963995955,3cyclictest0-21swapper/221:07:122
258552420,1sleep10-21swapper/121:10:171
3970994015,13cyclictest26929-40dnf22:25:233
396399401,29cyclictest0-21swapper/200:33:152
3963993835,2cyclictest0-21swapper/222:29:292
397099365,21cyclictest16130-21/usr/sbin/munin00:30:193
394999369,13cyclictest25637-21munin-run22:25:010
397099357,22cyclictest0-21swapper/322:40:133
3970993511,10cyclictest808-21snmpd21:25:153
397099349,20cyclictest0-21swapper/322:15:013
397099348,9cyclictest29416-21munin-run21:20:013
3970993410,15cyclictest79592head00:10:163
83652330,4sleep011-21rcuc/000:10:200
397099339,15cyclictest1038-21/usr/sbin/munin20:15:243
397099339,15cyclictest1038-21/usr/sbin/munin20:15:243
397099338,16cyclictest24458-21cstates23:35:133
397099334,14cyclictest9563-21hddtemp_smartct23:00:153
3970993319,5cyclictest19460-21/usr/sbin/munin20:55:193
397099329,5cyclictest87350irq/126-enp1s0-00:20:253
397099329,13cyclictest24624-21/usr/sbin/munin19:55:243
397099328,14cyclictest22584-21grep23:30:153
3970993215,4cyclictest18663-21users22:05:263
3970993212,10cyclictest23443-21perf22:20:003
397099321,11cyclictest0-21swapper/300:39:113
3970993211,12cyclictest5853-21munin-run21:40:023
396399321,19cyclictest0-21swapper/220:21:202
394999321,24cyclictest9908-21kworker/0:021:49:540
397099319,8cyclictest8459-21cstates21:45:143
397099319,6cyclictest8260-21unixbench_singl22:55:243
397099318,12cyclictest15490-21/usr/sbin/munin23:15:243
3970993113,10cyclictest32221-21latency_hist22:40:013
3970993113,10cyclictest32221-21latency_hist22:40:013
3970993111,17cyclictest14395-21tune2fs00:25:153
394999311,23cyclictest10515-21kworker/0:023:12:370
394999311,14cyclictest19659-21munin-run23:25:000
397099309,12cyclictest28673-21/usr/sbin/munin23:45:143
397099304,15cyclictest0-21swapper/321:52:073
3970993015,4cyclictest13629-21df23:10:133
3970993012,15cyclictest3833-21memory00:00:193
394999301,19cyclictest32124-21kworker/0:222:56:530
394999300,11cyclictest27534-21kworker/0:022:32:130
397099299,11cyclictest87350irq/126-enp1s0-00:15:143
3970992921,5cyclictest12358-21smart_sda23:05:233
3970992920,5cyclictest11157-40dnf19:24:243
3970992919,7cyclictest1035-21diskstats23:55:143
3970992913,4cyclictest31179-21unixbench_singl21:20:273
3970992911,8cyclictest28078-21cpu22:30:123
3970992911,6cyclictest4274-21munin-run22:50:013
396399291,11cyclictest0-21swapper/219:51:472
396399290,17cyclictest0-21swapper/223:17:122
395899294,18cyclictest32479-21cut21:25:161
394999292,10cyclictest1365-21kworker/0:221:40:180
394999291,21cyclictest9908-21kworker/0:022:04:460
394999291,10cyclictest10515-21kworker/0:023:30:170
397099289,11cyclictest30986-21latency_hist20:10:013
397099288,6cyclictest87650irq/129-enp1s0-21:30:133
397099287,9cyclictest0-21swapper/321:55:203
397099286,9cyclictest28954-21cat20:05:013
397099285,14cyclictest0-21swapper/300:10:023
397099285,13cyclictest6330-21sshd22:53:223
3970992820,5cyclictest21544-40dnf23:26:143
3970992814,8cyclictest7323-21/usr/sbin/munin21:40:243
3970992810,7cyclictest23658-21taskset22:20:093
396399286,3cyclictest0-21swapper/223:45:112
396399281,12cyclictest0-21swapper/223:20:482
395899284,16cyclictest0-21swapper/123:34:271
394999281,22cyclictest6202-21kworker/0:000:07:020
3949992811,2cyclictest87350irq/126-enp1s0-23:45:110
394999281,11cyclictest25895-21kworker/0:200:00:180
397099277,8cyclictest17908-21sed20:50:253
397099277,12cyclictest11446-21/usr/sbin/munin19:25:153
397099277,12cyclictest0-21swapper/323:40:153
397099276,13cyclictest22428-21/usr/sbin/munin19:50:173
3970992720,4cyclictest6624-21smart_sda19:15:033
3970992716,7cyclictest32290-21ntp_states20:10:193
3970992711,13cyclictest5974-21sshd19:13:073
396399271,17cyclictest0-21swapper/219:23:432
396399271,11cyclictest0-21swapper/222:38:522
396399271,11cyclictest0-21swapper/222:38:512
396399270,9cyclictest0-21swapper/220:08:552
394999271,16cyclictest25000-21kworker/0:021:26:410
394999271,13cyclictest21252-21awk22:15:010
397099269,14cyclictest20941-21cpuspeed_turbos21:00:133
397099268,9cyclictest15838-21/usr/sbin/munin19:35:123
397099266,7cyclictest87350irq/126-enp1s0-21:10:153
3970992619,4cyclictest4527-21smart_sda20:20:233
3970992618,5cyclictest23860-21/usr/sbin/munin21:05:173
3970992618,5cyclictest13119-21ntp_states20:40:203
3970992618,5cyclictest11118-21smart_sda20:35:233
3970992617,6cyclictest20232-21/usr/sbin/munin19:45:243
3970992617,6cyclictest18762-21/usr/sbin/munin23:20:223
3970992617,6cyclictest14868-21/usr/sbin/munin22:00:193
3970992617,6cyclictest14601-21diskstats20:45:143
3970992611,12cyclictest808-21snmpd23:54:203
396399260,12cyclictest28054-21awk23:40:272
396399260,11cyclictest0-21swapper/223:55:512
396399260,10cyclictest0-21swapper/221:45:592
3958992613,7cyclictest0-21swapper/122:48:261
394999262,9cyclictest28136-21kworker/0:023:41:250
394999262,22cyclictest10515-21kworker/0:023:17:370
394999262,17cyclictest25895-21kworker/0:223:58:250
394999262,11cyclictest27534-21kworker/0:022:37:130
394999262,11cyclictest27534-21kworker/0:022:37:130
394999262,10cyclictest9908-21kworker/0:021:50:340
394999262,10cyclictest25000-21kworker/0:021:12:290
394999261,9cyclictest10974-21kworker/0:319:28:380
394999261,11cyclictest9233-21kworker/0:300:23:140
394999261,11cyclictest13254-21kworker/0:019:31:380
3970992518,4cyclictest8337-21hddtemp_smartct20:30:163
3970992518,4cyclictest5815-21/usr/sbin/munin20:25:153
3970992517,5cyclictest18376-21/usr/sbin/munin19:40:123
3970992516,6cyclictest13644-21/usr/sbin/munin19:30:253
3963992522,2cyclictest0-21swapper/222:02:312
396399251,9cyclictest0-21swapper/221:02:222
396399251,9cyclictest0-21swapper/220:34:482
396399251,9cyclictest0-21swapper/200:24:352
396399251,12cyclictest0-21swapper/223:32:392
396399251,11cyclictest0-21swapper/221:38:052
396399251,11cyclictest0-21swapper/220:13:182
396399251,11cyclictest0-21swapper/219:30:552
396399250,9cyclictest0-21swapper/222:57:442
396399250,10cyclictest0-21swapper/221:11:082
395899251,21cyclictest0-21swapper/100:00:241
394999252,10cyclictest25000-21kworker/0:021:20:410
394999251,13cyclictest14502-21kworker/0:221:59:500
394999251,12cyclictest15910-21munin-run00:30:020
394999251,10cyclictest29895-21kworker/0:122:40:250
396399247,3cyclictest0-21swapper/220:03:352
396399241,9cyclictest0-21swapper/223:08:302
396399241,9cyclictest0-21swapper/222:08:012
396399241,8cyclictest0-21swapper/221:26:212
396399241,8cyclictest0-21swapper/219:58:292
396399241,13cyclictest0-21swapper/220:42:562
396399241,11cyclictest0-21swapper/219:16:032
396399241,10cyclictest0-21swapper/220:52:022
396399241,10cyclictest0-21swapper/219:45:092
395899244,12cyclictest0-21swapper/123:15:241
3958992410,4cyclictest28452-21munin-run23:45:001
394999242,8cyclictest13764-21kworker/0:220:47:230
394999242,10cyclictest27534-21kworker/0:022:29:570
394999240,9cyclictest32124-21kworker/0:222:50:290
394999240,11cyclictest32124-21kworker/0:223:00:250
236752240,3sleep09-21rcu_preempt21:05:140
396399239,3cyclictest0-21swapper/222:12:492
396399239,3cyclictest0-21swapper/200:17:092
396399239,2cyclictest0-21swapper/220:26:362
396399233,3cyclictest0-21swapper/200:12:272
396399231,9cyclictest0-21swapper/221:22:372
396399231,9cyclictest0-21swapper/200:26:172
396399231,9cyclictest0-21swapper/200:09:552
396399231,8cyclictest0-21swapper/220:46:382
396399231,6cyclictest0-21swapper/200:36:502
396399231,11cyclictest0-21swapper/219:39:352
396399231,10cyclictest0-21swapper/222:47:502
396399231,10cyclictest0-21swapper/220:36:442
396399231,10cyclictest0-21swapper/219:26:092
396399230,14cyclictest0-21swapper/221:53:152
396399230,11cyclictest0-21swapper/222:52:282
395899233,2cyclictest14357-21kworker/u8:122:31:031
395899231,19cyclictest4507-21sh19:10:131
395899231,16cyclictest0-21swapper/123:59:001
395899231,12cyclictest0-21swapper/121:41:251
3958992310,4cyclictest18025-21df_inode23:20:131
394999233,2cyclictest5851-21kworker/0:019:15:550
394999232,9cyclictest655-21kworker/0:220:18:210
394999232,9cyclictest655-21kworker/0:220:18:210
394999232,9cyclictest4898-21kworker/0:320:26:370
394999232,19cyclictest20960-21kworker/0:222:15:580
394999232,12cyclictest25895-21kworker/0:223:54:130
394999232,10cyclictest17818-21kworker/0:100:32:540
394999232,10cyclictest10515-21kworker/0:023:28:130
394999231,11cyclictest24042-21kworker/0:219:54:080
394999231,10cyclictest32124-21kworker/0:223:05:170
396399228,3cyclictest0-21swapper/222:42:462
396399226,3cyclictest0-21swapper/219:13:542
396399224,3cyclictest0-21swapper/220:57:062
396399223,3cyclictest0-21swapper/223:51:392
396399221,9cyclictest25019-21diskmemload00:04:212
396399221,11cyclictest0-21swapper/223:02:142
396399221,10cyclictest0-21swapper/220:15:242
396399221,10cyclictest0-21swapper/220:15:242
396399220,12cyclictest0-21swapper/219:41:072
395899223,12cyclictest87550irq/128-enp1s0-23:02:421
395899223,11cyclictest87550irq/128-enp1s0-00:15:181
395899222,17cyclictest5745-21grep00:05:151
395899222,10cyclictest20008-21munin-run19:45:011
3958992213,2cyclictest25019-21diskmemload23:38:521
395899221,18cyclictest8878-21processes20:30:241
395899221,18cyclictest24008-21df_inode22:20:151
395899221,12cyclictest22775-21diskstats19:50:141
395899221,11cyclictest0-21swapper/123:53:351
395899221,11cyclictest0-21swapper/123:46:181
395899221,11cyclictest0-21swapper/121:37:541
395899221,11cyclictest0-21swapper/121:23:141
395899220,16cyclictest0-21swapper/100:35:361
394999226,8cyclictest16768-21kworker/0:122:06:060
394999226,3cyclictest13764-21kworker/0:220:44:250
394999221,9cyclictest25000-21kworker/0:021:38:220
394999221,12cyclictest1987-21kworker/0:022:49:290
394999221,12cyclictest13254-21kworker/0:019:39:020
396399216,3cyclictest0-21swapper/221:17:332
396399215,3cyclictest0-21swapper/221:58:092
396399211,9cyclictest0-21swapper/223:39:052
396399211,9cyclictest0-21swapper/222:15:572
396399211,10cyclictest0-21swapper/223:14:462
396399210,7cyclictest0-21swapper/222:30:232
396399210,10cyclictest0-21swapper/221:41:072
395899218,3cyclictest0-21swapper/121:15:401
395899217,11cyclictest15112chrt20:15:161
395899217,11cyclictest15112chrt20:15:161
395899216,9cyclictest0-21swapper/123:05:401
395899212,10cyclictest12439-21latency_hist21:55:011
395899211,4cyclictest31417-21proc_pri22:35:231
395899211,4cyclictest31417-21proc_pri22:35:231
395899211,3cyclictest12683-21basename21:55:131
3958992113,3cyclictest0-21swapper/100:30:041
395899211,18cyclictest0-21swapper/119:49:341
395899211,15cyclictest0-21swapper/122:53:341
395899211,15cyclictest0-21swapper/122:18:151
395899211,12cyclictest13715-21awk00:25:011
395899211,11cyclictest5463-40dnf20:25:131
395899211,11cyclictest27928-21cat22:30:021
395899211,11cyclictest0-21swapper/122:41:431
395899211,10cyclictest0-21swapper/123:28:521
395899211,10cyclictest0-21swapper/122:12:311
395899210,16cyclictest0-21swapper/121:49:051
394999219,3cyclictest25000-21kworker/0:021:16:090
394999217,2cyclictest18870-21kworker/0:219:20:500
394999214,2cyclictest30889-21kworker/0:120:12:450
394999212,9cyclictest24042-21kworker/0:219:55:260
394999212,5cyclictest17401-1kworker/u9:219:45:570
394999212,11cyclictest20840-21/usr/sbin/munin21:00:140
394999212,10cyclictest13764-21kworker/0:220:54:130
394999211,9cyclictest25895-21kworker/0:200:38:540
394999211,12cyclictest22260-21kworker/0:123:37:050
394999211,11cyclictest1365-21kworker/0:221:32:260
394999211,10cyclictest3019-21kworker/0:020:20:210
208382211,2sleep10-21swapper/121:00:111
190642211,2sleep019098-21ntp_states19:40:210
395899209,3cyclictest342-21vmstat20:10:251
395899203,10cyclictest16715-21kworker/u8:222:05:131
395899201,17cyclictest0-21swapper/120:36:281
395899201,17cyclictest0-21swapper/120:20:251
395899201,17cyclictest0-21swapper/119:24:001
395899201,16cyclictest14700-21if_err_enp1s000:25:171
395899201,14cyclictest0-21swapper/122:57:401
395899201,11cyclictest15218-21diskstats22:00:141
395899201,11cyclictest12156-21expr20:40:121
395899201,10cyclictest9308-21sh00:15:011
395899201,10cyclictest0-21swapper/121:30:591
394999204,2cyclictest9321-21kworker/0:120:36:150
394999203,14cyclictest29136-21cut20:05:090
394999202,7cyclictest9233-21kworker/0:300:16:300
394999202,7cyclictest13764-21kworker/0:220:56:130
22642201,15sleep22380-21ls21:30:172
396399193,8cyclictest0-21swapper/222:20:272
395899191,16cyclictest0-21swapper/120:58:201
395899191,16cyclictest0-21swapper/120:50:191
395899191,16cyclictest0-21swapper/120:05:091
395899191,16cyclictest0-21swapper/119:16:171
395899191,14cyclictest0-21swapper/121:07:391
395899191,11cyclictest15614-21awk19:35:011
395899191,10cyclictest16745-21taskset19:35:181
395899191,10cyclictest14469-21/usr/sbin/munin23:10:221
395899191,10cyclictest0-21swapper/120:45:131
395899190,17cyclictest0-21swapper/120:00:211
394999197,2cyclictest4898-21kworker/0:320:31:450
394999191,12cyclictest18870-21kworker/0:219:10:050
3958991815,2cyclictest0-21swapper/119:26:521
395899181,16cyclictest0-21swapper/119:55:121
394999185,3cyclictest26287-21kworker/0:020:02:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional