You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 05:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 08, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2265123760,54sleep20-21swapper/219:05:132
241262238162,25sleep00-21swapper/019:07:410
240572204167,24sleep30-21swapper/319:06:493
240082201164,25sleep10-21swapper/119:06:091
2075121650,4sleep12439799cyclictest22:40:131
2323021440,6sleep02439299cyclictest00:00:010
1628421300,4sleep32440999cyclictest21:15:183
244029910299,2cyclictest0-21swapper/223:01:342
26622580,1sleep00-21swapper/020:45:130
290202550,3sleep20-21swapper/219:15:302
27792550,1sleep00-21swapper/019:30:210
233772510,4sleep07-21ksoftirqd/022:45:180
268352330,2sleep20-21swapper/219:15:012
2439299338,12cyclictest3971-21munin-run00:30:010
24392992810,15cyclictest16521-21munin-run23:45:010
24392992810,15cyclictest14334-21perf23:40:010
2440999275,8cyclictest23828-40dnf22:45:273
2439799270,10cyclictest0-21swapper/100:13:461
24402992622,3cyclictest0-21swapper/223:05:492
24402992610,3cyclictest0-21swapper/223:57:102
2439799261,8cyclictest0-21swapper/121:12:261
2439799261,10cyclictest0-21swapper/100:24:261
2440999259,3cyclictest9908-21needreboot21:00:183
2440999259,3cyclictest9908-21needreboot21:00:183
2440999253,3cyclictest0-21swapper/322:00:373
2439799251,11cyclictest0-21swapper/123:59:141
2439299258,4cyclictest30469-21cstates20:35:140
2439299251,12cyclictest3569-21/usr/sbin/munin23:15:130
2440299243,4cyclictest0-21swapper/200:05:132
2439799243,16cyclictest8075-40dnf19:45:021
2439799241,3cyclictest7495-21grep22:10:111
2439799241,11cyclictest0-21swapper/100:02:061
2439799240,8cyclictest0-21swapper/120:12:541
2439799240,11cyclictest0-21swapper/122:09:381
2439799240,11cyclictest0-21swapper/100:34:101
2439799240,11cyclictest0-21swapper/100:34:101
2439799240,10cyclictest0-21swapper/122:53:591
2439799240,10cyclictest0-21swapper/120:28:151
98482230,2sleep00-21swapper/021:00:180
98482230,2sleep00-21swapper/021:00:180
2440999235,6cyclictest0-21swapper/323:18:323
2440999232,6cyclictest6434-21hddtemp_smartct23:20:143
2440999232,6cyclictest6434-21hddtemp_smartct23:20:143
2440999231,6cyclictest0-21swapper/300:00:463
2440299237,5cyclictest6899-21cstates20:55:142
2440299231,4cyclictest0-21swapper/200:15:242
2440299231,3cyclictest0-21swapper/223:20:212
2440299231,3cyclictest0-21swapper/223:20:212
2440299231,2cyclictest0-21swapper/222:55:152
2439799231,9cyclictest0-21swapper/122:19:001
2439799231,9cyclictest0-21swapper/100:29:501
2439799231,19cyclictest4578-21hddtemp_smartct19:35:161
2439799231,11cyclictest0-21swapper/121:40:321
2439799231,11cyclictest0-21swapper/121:33:061
2439799231,10cyclictest0-21swapper/122:49:051
2439799231,10cyclictest0-21swapper/121:49:341
2439799231,10cyclictest0-21swapper/121:20:441
2439799230,8cyclictest0-21swapper/123:01:391
2439799230,11cyclictest0-21swapper/100:39:541
2439299239,3cyclictest12724-21grep22:20:200
2439299236,5cyclictest0-21swapper/023:26:420
2439299232,8cyclictest26225-21kworker/0:219:12:030
2439299231,4cyclictest0-21swapper/021:49:040
2439299231,11cyclictest420-21munin-run21:55:000
2439299231,11cyclictest31660-21awk19:25:010
13282230,5sleep07-21ksoftirqd/023:10:010
263402220,4sleep339-21rcuc/322:50:253
2440999224,4cyclictest0-21swapper/322:29:103
2440999223,6cyclictest0-21swapper/322:19:033
2440999222,11cyclictest11733-21/usr/sbin/munin22:20:223
2440999221,7cyclictest0-21swapper/323:55:483
2440299227,3cyclictest10605-21cut19:50:112
2440299227,3cyclictest10605-21cut19:50:112
2440299221,7cyclictest29931-21gunzip21:45:242
2440299221,3cyclictest0-21swapper/221:15:492
2440299221,3cyclictest0-21swapper/200:28:212
24402992213,5cyclictest24129-21timerwakeupswit22:45:282
2440299221,2cyclictest0-21swapper/223:49:092
2439799222,17cyclictest15404-21unixbench-2d22:25:251
2439799221,8cyclictest0-21swapper/121:53:421
2439799221,3cyclictest24430-21grep21:35:121
2439799221,18cyclictest8629-21hddtemp_smartct23:25:161
2439799221,16cyclictest0-21swapper/123:34:391
2439799221,10cyclictest0-21swapper/121:56:021
2439799221,10cyclictest0-21swapper/120:30:131
2439799221,10cyclictest0-21swapper/120:20:051
2439799221,10cyclictest0-21swapper/119:11:001
2439799220,9cyclictest0-21swapper/123:10:471
2439799220,7cyclictest0-21swapper/121:28:341
2439299226,6cyclictest0-21swapper/021:27:080
2439299226,10cyclictest26706-21munin-node19:45:030
2439299224,2cyclictest0-21swapper/000:34:180
2439299224,2cyclictest0-21swapper/000:34:180
2439299222,8cyclictest11144-21kworker/0:122:18:330
2439299221,9cyclictest26225-21kworker/0:219:19:450
2439299221,8cyclictest18684-21kworker/0:200:05:350
2439299221,2cyclictest87650irq/129-enp1s0-21:43:520
2439299221,12cyclictest0-21swapper/022:59:130
2439299221,11cyclictest0-21swapper/000:21:050
163332220,2sleep00-21swapper/021:15:200
249912211,15sleep225263-21df22:50:132
2440999218,3cyclictest0-21swapper/321:59:353
2440999211,9cyclictest0-21swapper/300:13:063
2440999211,17cyclictest19912-21users22:35:253
2440299215,8cyclictest10120-21date23:30:022
2440299215,6cyclictest0-21swapper/220:25:582
2440299211,7cyclictest0-21swapper/223:14:512
2440299211,3cyclictest0-21swapper/223:34:522
24402992113,3cyclictest18245-1kworker/u9:200:35:302
2440299211,13cyclictest0-21swapper/200:21:422
2440299211,11cyclictest27540-21grep21:40:212
2439799219,2cyclictest6983-21gunzip23:20:231
2439799219,2cyclictest6983-21gunzip23:20:221
24397992111,8cyclictest3231-21kworker/u8:223:15:261
2439799211,18cyclictest0-21swapper/120:35:121
24397992111,4cyclictest23-21ksoftirqd/123:35:041
2439799211,11cyclictest17192-21ntp_states22:30:221
2439799211,10cyclictest0-21swapper/120:17:131
24397992110,3cyclictest25575-21cpu00:05:111
2439299218,11cyclictest26916-21perf22:55:010
2439299218,10cyclictest8372-21basename00:40:020
2439299213,10cyclictest2402650kworker/u8:121:56:320
2439299212,8cyclictest22879-21kworker/0:100:03:210
2439299211,9cyclictest15104-21kworker/0:221:14:200
2439299211,8cyclictest16498-21kworker/0:020:28:270
2439299211,8cyclictest16498-21kworker/0:020:07:210
24392992112,4cyclictest4202-21sshd22:00:250
2439299211,12cyclictest0-21swapper/000:10:500
2439299211,11cyclictest5612-21fschecks_time22:05:140
2439299211,11cyclictest3907-21/usr/sbin/munin19:35:160
2439299211,11cyclictest20239-21sshd22:39:250
2439299211,11cyclictest18946-21/usr/sbin/munin23:50:150
2439299211,11cyclictest16129-21/usr/sbin/munin22:30:180
2439299211,10cyclictest2445-21kworker/0:022:10:590
2440999206,5cyclictest0-21swapper/321:26:523
2440999205,8cyclictest0-21swapper/323:49:413
2440999203,7cyclictest7018-21expr20:55:133
2440999201,7cyclictest0-21swapper/323:41:453
2440999201,7cyclictest0-21swapper/322:32:043
2440999201,6cyclictest0-21swapper/323:52:403
2440999201,11cyclictest87350irq/126-enp1s0-23:25:223
2440999201,10cyclictest17715-21/usr/sbin/munin21:20:193
2440999200,10cyclictest0-21swapper/323:05:263
2440299206,7cyclictest16147-21mailstats20:00:222
2440299205,8cyclictest21595-21ntp_states22:40:212
2440299201,5cyclictest3286-21diskstats22:00:142
2440299201,3cyclictest0-21swapper/222:35:552
24402992010,3cyclictest87650irq/129-enp1s0-22:08:252
2439799203,7cyclictest755-21rngd23:09:261
2439799202,4cyclictest12475-21munin-run19:55:011
2439799202,4cyclictest12475-21munin-run19:55:011
2439799201,9cyclictest0-21swapper/123:44:181
2439799201,3cyclictest0-21swapper/123:50:281
2439799201,16cyclictest30517-21perl00:15:151
2439799200,6cyclictest0-21swapper/122:04:011
2439799200,4cyclictest0-21swapper/122:35:191
2439299208,2cyclictest26823-21kworker/0:023:04:400
2439299207,2cyclictest9563-21kworker/0:123:45:400
2439299205,5cyclictest0-21swapper/000:16:350
2439299205,2cyclictest29431-21kworker/0:220:50:220
2439299201,9cyclictest21563-21kworker/0:021:31:000
2439299201,6cyclictest0-21swapper/022:27:230
24392992012,5cyclictest1-21systemd19:45:000
2439299201,10cyclictest19296-21/usr/sbin/munin20:10:130
2439299201,10cyclictest14892-21/usr/sbin/munin20:00:140
2439299200,9cyclictest21701-21kworker/u8:021:36:420
2440999197,3cyclictest87350irq/126-enp1s0-19:45:213
2440999192,7cyclictest0-21swapper/321:40:583
2440999191,7cyclictest0-21swapper/300:34:493
2440999191,7cyclictest0-21swapper/300:34:493
2440999191,6cyclictest27978-21grep22:55:173
2440999191,6cyclictest0-21swapper/300:27:143
2440999191,6cyclictest0-21swapper/300:05:143
2440999191,5cyclictest7859-21tune2fs22:10:153
2440999191,5cyclictest0-21swapper/321:48:533
2440999191,16cyclictest0-21swapper/320:52:243
2440999191,11cyclictest0-21swapper/322:41:143
2440999191,10cyclictest8008-21sshd19:44:213
2440999191,10cyclictest347-21sshd21:54:253
2440999191,10cyclictest22111-21/usr/sbin/munin21:30:263
2440999190,9cyclictest0-21swapper/322:05:133
2440299196,3cyclictest87650irq/129-enp1s0-22:18:002
2440299195,7cyclictest13399-21taskset22:21:122
2440299195,4cyclictest16582-21fschecks_count22:30:142
2440299192,6cyclictest3683-21munin-run19:35:012
2440299191,8cyclictest47852expr20:50:122
2440299191,5cyclictest15913-21awk22:30:012
2440299191,3cyclictest0-21swapper/221:35:112
2440299191,3cyclictest0-21swapper/200:10:322
2440299191,10cyclictest8015-21hddtemp_smartct22:10:152
2440299191,10cyclictest0-21swapper/223:40:142
2439799192,14cyclictest8075-40dnf19:45:011
2439799191,8cyclictest0-21swapper/122:20:111
24397991916,2cyclictest0-21swapper/119:15:281
24397991912,2cyclictest18245-1kworker/u9:223:48:411
2439799191,16cyclictest0-21swapper/119:55:101
24397991910,3cyclictest13086-21latency_hist21:10:021
2439799190,12cyclictest16823-21/usr/sbin/munin21:15:271
2439299196,3cyclictest0-21swapper/021:20:080
2439299194,3cyclictest0-21swapper/022:44:240
2439299191,10cyclictest0-21swapper/023:14:510
2439299190,9cyclictest26823-21kworker/0:023:22:280
2439299190,9cyclictest26823-21kworker/0:023:22:280
2439299190,9cyclictest0-21swapper/023:33:020
85212181,2sleep00-21swapper/020:58:020
2440999189,2cyclictest87350irq/126-enp1s0-21:10:223
2440999188,8cyclictest87350irq/126-enp1s0-00:35:213
2440999188,2cyclictest0-21swapper/319:20:243
2440999187,3cyclictest0-21swapper/320:30:163
2440999187,1cyclictest0-21swapper/320:45:213
2440999186,3cyclictest39-21rcuc/300:15:253
2440999185,4cyclictest87350irq/126-enp1s0-19:15:013
2440999184,3cyclictest25909-21chrt20:25:103
2440999184,3cyclictest0-21swapper/321:07:593
2440999182,5cyclictest26284-21munin-run21:40:023
2440999182,13cyclictest4568-21tail19:35:173
2440999181,7cyclictest14662-21latency_hist20:00:023
2440999181,7cyclictest10964-21irqstats23:30:163
2440999181,6cyclictest0-21swapper/323:00:323
2440999181,6cyclictest0-21swapper/320:20:223
2440999181,6cyclictest0-21swapper/320:10:203
2440999181,6cyclictest0-21swapper/300:22:193
2440999181,4cyclictest0-21swapper/323:36:503
2440999181,3cyclictest17098-21/usr/sbin/munin20:05:113
24409991812,3cyclictest1350-21sendmail_mailst20:40:233
2440999181,10cyclictest0-21swapper/319:30:273
2440999180,12cyclictest0-21swapper/320:15:243
2440999180,12cyclictest0-21swapper/320:04:233
2440299186,3cyclictest495-1jbd2/dm-0-820:36:592
2440299185,4cyclictest12699-21/usr/sbin/munin19:55:162
2440299185,3cyclictest3500-21ps20:45:232
2440299185,3cyclictest19833-21sed21:25:022
2440299183,4cyclictest30589-21memory19:20:192
2440299181,9cyclictest31794-21latency21:50:192
2440299181,9cyclictest0-21swapper/221:29:032
2440299181,9cyclictest0-21swapper/219:40:122
2440299181,5cyclictest1609-21latency_hist19:30:022
24402991813,3cyclictest5735-21grep00:30:252
24402991813,3cyclictest5735-21grep00:30:252
2440299181,14cyclictest9783-21grep19:45:272
2440299181,12cyclictest0-21swapper/223:15:152
2439799189,2cyclictest9098-21cstates21:00:131
2439799189,2cyclictest9098-21cstates21:00:131
2439799187,3cyclictest32168-21expr19:25:131
2439799186,3cyclictest27190-21cpuspeed_turbos22:55:121
2439799181,9cyclictest6718-21/usr/sbin/munin20:55:121
24397991814,3cyclictest0-21swapper/120:47:471
24397991814,3cyclictest0-21swapper/120:08:021
2439799181,16cyclictest0-21swapper/120:50:371
2439799181,16cyclictest0-21swapper/119:20:071
2439799181,14cyclictest32577-21sort20:40:131
2439299188,1cyclictest0-21swapper/020:20:260
2439299184,4cyclictest29008-21sshd20:30:200
24392991812,3cyclictest23458-21dump-pmu-power20:20:020
24392991812,3cyclictest12465-21idleruntime-cro19:55:010
24392991812,3cyclictest12465-21idleruntime-cro19:55:000
2439299181,14cyclictest0-21swapper/020:43:270
2439299180,10cyclictest0-21swapper/019:25:250
2440999174,3cyclictest5750irq/9-acpi20:35:173
2440999174,3cyclictest1363-21cpu23:10:123
2440999171,4cyclictest27487-21/usr/sbin/munin19:15:213
2440299176,3cyclictest13718-21/usr/sbin/munin23:35:262
2440299172,4cyclictest17389-21kworker/u8:221:30:142
2440299171,9cyclictest0-21swapper/221:10:462
2440299171,8cyclictest838-21if_enp2s020:40:162
2440299171,7cyclictest3907-21/usr/sbin/munin19:35:202
2440299171,6cyclictest0-21swapper/221:56:312
2440299171,5cyclictest29841-21sshd20:32:202
2440299171,5cyclictest23346-21/usr/sbin/munin00:00:162
2440299171,5cyclictest22172-21hddtemp_smartct20:15:172
2440299171,5cyclictest10177-21smart_sda21:00:242
2440299171,5cyclictest10177-21smart_sda21:00:242
2440299171,5cyclictest0-21swapper/220:20:182
2440299171,10cyclictest0-21swapper/223:50:262
2439799171,5cyclictest909-21sssd_nss20:00:191
24397991712,3cyclictest3469-21seq19:32:251
2440999163,5cyclictest0-21swapper/319:28:563
2440999161,5cyclictest0-21swapper/319:50:233
2440999161,5cyclictest0-21swapper/319:50:233
2440299163,3cyclictest19872-21tune2fs20:10:152
2440299162,3cyclictest17098-21/usr/sbin/munin20:05:172
2440299161,5cyclictest11456-21df21:05:122
24392991610,3cyclictest827-21runrttasks21:07:260
24392991610,3cyclictest12850-21grep19:55:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional