You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-28 - 12:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 28, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
187352238161,24sleep00-21swapper/019:09:020
187432233162,58sleep30-21swapper/319:09:093
1785722250,3sleep10-21swapper/119:05:231
168532203167,24sleep20-21swapper/219:05:102
3142721740,3sleep01889599cyclictest00:30:150
1196221480,3sleep11890299cyclictest20:05:171
1635521380,5sleep21890899cyclictest23:55:182
303822640,1sleep30-21swapper/323:15:023
18222530,3sleep11827-21gltestperf20:55:181
19472460,2sleep20-21swapper/220:55:192
1889599379,13cyclictest6987-21munin-run19:55:010
81522340,6sleep339-21rcuc/319:55:203
1891599331,6cyclictest0-21swapper/300:15:013
18895993221,3cyclictest3164-21switchtime22:10:250
1889599320,11cyclictest0-21swapper/023:35:040
1890299316,8cyclictest0-21swapper/121:35:181
1889599301,14cyclictest23614-21munin-run23:00:010
1891599291,6cyclictest0-21swapper/323:46:243
187642290,4sleep232-21ksoftirqd/200:00:192
1890299281,20cyclictest2581-21latency_hist19:45:011
1891599274,10cyclictest39-21rcuc/300:15:193
18895992719,3cyclictest0-21swapper/000:05:320
18895992718,3cyclictest0-21swapper/022:45:130
1891599265,3cyclictest27059-21fschecks_count00:20:143
1891599261,5cyclictest3189-21switchtime22:10:253
1891599261,4cyclictest0-21swapper/300:05:323
1890899261,4cyclictest0-21swapper/223:20:012
1890899261,4cyclictest0-21swapper/223:20:012
1890299267,16cyclictest28774-21systemd-journal20:42:431
1890299261,3cyclictest24256-21munin-run00:15:011
1890299261,3cyclictest22067-21awk00:10:011
1890299261,23cyclictest989-21awk20:55:011
1890299261,23cyclictest989-21awk20:55:011
1890299261,23cyclictest0-21swapper/123:37:221
1890299261,22cyclictest6309-21hddtemp_smartct21:05:161
1890299261,15cyclictest27390-21munin-run21:55:001
1890299261,12cyclictest13848-21sshd21:20:421
1891599257,4cyclictest19739-21missed_timers21:35:213
1891599254,10cyclictest26044-21/usr/sbin/munin23:05:253
1891599253,9cyclictest39-21rcuc/322:25:213
1891599251,6cyclictest0-21swapper/323:17:513
1891599251,6cyclictest0-21swapper/323:17:513
1891599251,6cyclictest0-21swapper/322:24:083
1891599251,14cyclictest0-21swapper/322:01:113
1891599251,11cyclictest17515-21sed20:15:263
1890299254,17cyclictest24991-40dnf21:47:431
1890299251,6cyclictest10649-21users23:40:271
1890299251,12cyclictest17623-21latency_hist00:00:011
1889599256,16cyclictest30847-21munin-run00:30:010
1889599256,16cyclictest30847-21munin-run00:30:010
1889599253,14cyclictest31251-21cat22:00:290
18895992518,3cyclictest0-21swapper/022:24:080
18895992517,3cyclictest0-21swapper/000:16:380
18895992517,2cyclictest0-21swapper/022:31:180
314112240,2sleep20-21swapper/220:50:132
314112240,2sleep20-21swapper/220:50:132
18915992420,3cyclictest0-21swapper/323:35:153
1891599241,6cyclictest0-21swapper/322:45:123
18915992411,3cyclictest87250irq/129-enp1s0-00:31:063
1891599241,11cyclictest21639-21/usr/sbin/munin22:55:233
1891599241,10cyclictest6568-21sed19:50:263
1890299249,3cyclictest21815-21cstates22:55:131
1890299241,8cyclictest0-21swapper/122:51:021
1890299241,3cyclictest16400-21perf21:30:021
1890299241,21cyclictest19038-40dnf22:47:561
1890299241,21cyclictest0-21swapper/119:55:211
1890299241,20cyclictest3794-21sshd22:14:421
1890299241,11cyclictest3457-21cpu21:00:131
1890299241,11cyclictest32438-21timerandwakeup00:30:261
1890299241,11cyclictest0-21swapper/100:25:171
1890299241,11cyclictest0-21swapper/100:25:171
1890299241,10cyclictest4870-21irqstats22:15:191
18902992410,3cyclictest32104-21cat22:05:131
1890299240,5cyclictest0-21swapper/123:46:421
1891599238,9cyclictest0-21swapper/323:40:173
1891599233,10cyclictest41-21ksoftirqd/320:30:283
1891599232,10cyclictest21991-21/usr/sbin/munin19:15:193
1891599232,10cyclictest14878-21ntp_states20:10:233
1891599231,7cyclictest2383-21head20:55:243
1891599231,6cyclictest0-21swapper/322:30:573
1891599231,5cyclictest0-21swapper/321:27:503
1891599231,4cyclictest0-21swapper/322:50:313
18915992310,3cyclictest87250irq/129-enp1s0-19:20:263
1891599230,11cyclictest0-21swapper/319:39:133
1890299235,7cyclictest0-21swapper/121:14:331
1890299232,19cyclictest0-21swapper/119:10:071
1890299232,14cyclictest87150irq/128-enp1s0-21:55:411
1890299231,8cyclictest0-21swapper/122:35:191
1890299231,5cyclictest0-21swapper/123:05:261
1890299231,3cyclictest30837-21netstat22:00:231
1890299231,3cyclictest11389-21munin-run20:05:011
1890299231,2cyclictest3578-21awk23:25:251
1890299231,2cyclictest0-21swapper/120:15:161
1890299231,20cyclictest17971-21dump-pmu-power20:24:571
1890299231,20cyclictest0-21swapper/123:03:341
1890299231,20cyclictest0-21swapper/119:25:241
1890299231,20cyclictest0-21swapper/100:15:171
1890299231,19cyclictest0-21swapper/120:30:151
1890299231,11cyclictest3573-21if_enp2s019:45:191
1890299230,21cyclictest0-21swapper/119:15:281
1890299230,1cyclictest0-21swapper/122:40:141
18895992316,3cyclictest0-21swapper/022:50:320
18895992316,3cyclictest0-21swapper/021:27:490
18895992316,3cyclictest0-21swapper/000:00:010
18895992313,3cyclictest24821-21needreboot23:00:190
1889599231,15cyclictest0-21swapper/022:27:470
1889599231,15cyclictest0-21swapper/000:25:010
18895992310,3cyclictest87150irq/128-enp1s0-20:31:420
18895992310,3cyclictest21771-21sed20:25:270
281622220,4sleep339-21rcuc/320:40:223
1891599229,3cyclictest87250irq/129-enp1s0-23:59:003
1891599229,3cyclictest1832-21unixbench_singl23:20:263
1891599227,8cyclictest0-21swapper/323:50:183
18915992216,2cyclictest0-21swapper/322:40:283
1891599221,11cyclictest801-21snmpd19:50:013
1891599221,10cyclictest7834-21/usr/sbin/munin21:10:143
1891599221,10cyclictest39-21rcuc/320:05:263
1891599221,10cyclictest26382-21/usr/sbin/munin19:25:223
18915992210,2cyclictest997-21awk20:55:003
18915992210,2cyclictest997-21awk20:55:003
1890899229,4cyclictest0-21swapper/222:53:362
1890899221,6cyclictest0-21swapper/223:47:502
1890899221,4cyclictest4143-21taskset23:26:202
1890899221,4cyclictest0-21swapper/222:35:302
1890899221,4cyclictest0-21swapper/221:16:292
1890899220,15cyclictest86950irq/126-enp1s0-23:54:372
1890299229,3cyclictest14619-21irqstats20:10:191
1890299221,9cyclictest6980-21latency_hist19:55:011
1890299221,9cyclictest2899-21latency_hist00:40:011
1890299221,9cyclictest22274-21chrt20:28:531
1890299221,4cyclictest6605-21diskstats22:20:181
1890299221,20cyclictest0-21swapper/119:37:061
1890299221,1cyclictest0-21swapper/123:25:011
1890299221,19cyclictest0-21swapper/120:49:061
1890299221,10cyclictest24655-21fschecks_time19:20:171
1890299221,10cyclictest0-21swapper/123:53:011
1889599227,7cyclictest0-21swapper/022:37:130
1889599226,8cyclictest519-21/usr/sbin/munin19:40:200
1889599221,14cyclictest0-21swapper/023:08:350
18895992211,2cyclictest9637-21chrt21:12:570
1889599221,12cyclictest31841-21ntp_states19:35:210
1889599220,3cyclictest14790-21kworker/u8:022:43:400
1889599220,16cyclictest0-21swapper/023:16:040
1889599220,16cyclictest0-21swapper/023:16:040
301032210,4sleep339-21rcuc/300:25:253
301032210,4sleep339-21rcuc/300:25:253
1891599217,1cyclictest0-21swapper/322:05:273
1891599215,13cyclictest0-21swapper/321:24:423
1891599214,5cyclictest41-21ksoftirqd/300:00:173
1891599212,11cyclictest0-21swapper/320:35:523
1891599211,7cyclictest28174-21fschecks_time21:55:173
1891599211,7cyclictest21101-21cpuspeed_turbos21:40:123
1891599211,7cyclictest0-21swapper/323:31:293
1891599211,7cyclictest0-21swapper/321:45:313
1891599211,6cyclictest3570-21expr21:00:133
1891599211,4cyclictest0-21swapper/323:00:183
1891599211,12cyclictest0-21swapper/322:38:593
1891599211,10cyclictest0-21swapper/321:50:253
1891599210,7cyclictest41-21ksoftirqd/321:30:203
1890899215,3cyclictest0-21swapper/200:34:552
1890899211,7cyclictest0-21swapper/221:14:472
1890899211,5cyclictest0-21swapper/222:42:572
1890299217,2cyclictest0-21swapper/121:40:111
1890299216,2cyclictest0-21swapper/123:30:171
1890299212,10cyclictest0-21swapper/122:34:501
1890299211,19cyclictest0-21swapper/122:27:461
1890299211,18cyclictest0-21swapper/123:11:521
1890299211,10cyclictest0-21swapper/100:22:461
1890299210,10cyclictest18175-21expr00:00:141
1889599213,13cyclictest0-21swapper/021:30:110
1889599212,12cyclictest2617-21kworker/u8:100:37:320
18895992111,6cyclictest0-21swapper/000:14:420
1889599211,15cyclictest0-21swapper/023:52:190
1889599211,15cyclictest0-21swapper/023:24:190
1889599210,16cyclictest0-21swapper/023:27:540
1889599210,14cyclictest29264-21kworker/u8:222:17:550
1891599204,4cyclictest86950irq/126-enp1s0-21:15:193
1891599202,9cyclictest19098-21/usr/sbin/munin19:10:193
1891599202,9cyclictest12050irq/123-ahci[0020:25:023
1891599201,9cyclictest20407-21/usr/sbin/munin20:25:263
1891599201,8cyclictest801-21snmpd00:35:563
1891599201,3cyclictest29193-21/usr/sbin/munin20:45:123
1891599201,10cyclictest4058-21/usr/sbin/munin22:15:243
1891599200,5cyclictest3466-21needreboot23:25:223
1890899201,5cyclictest0-21swapper/200:38:422
1890899201,4cyclictest0-21swapper/223:02:432
18908992014,3cyclictest29594-21idleruntime-cro22:00:012
1890899200,4cyclictest0-21swapper/222:25:122
1890299201,9cyclictest0-21swapper/123:18:041
1890299201,9cyclictest0-21swapper/123:18:041
1890299201,7cyclictest0-21swapper/121:15:431
1890299201,2cyclictest0-21swapper/121:32:011
1890299200,18cyclictest0-21swapper/119:31:091
1889599209,7cyclictest0-21swapper/000:01:420
1889599208,11cyclictest0-21swapper/023:35:000
1889599206,7cyclictest4688-21smart_sda21:00:240
1889599206,3cyclictest13221-21cut21:20:220
1889599205,8cyclictest11377-21wc21:15:240
1889599203,9cyclictest18200-21/usr/sbin/munin20:20:260
1889599203,6cyclictest29486-21sendmail_mailtr23:10:220
1889599201,5cyclictest3476-21sed19:45:190
1889599201,15cyclictest0-21swapper/021:52:550
1889599201,15cyclictest0-21swapper/021:40:150
1889599201,14cyclictest0-21swapper/023:40:310
1889599201,14cyclictest0-21swapper/021:35:030
1889599201,13cyclictest31707-21kworker/u8:122:07:450
1889599200,15cyclictest0-21swapper/021:57:190
1891599192,4cyclictest2474-21sshd19:43:423
1891599191,6cyclictest29183-21head19:30:163
1890899192,7cyclictest17092-21fschecks_time21:30:162
1890899191,7cyclictest0-21swapper/221:50:162
1890899191,4cyclictest0-21swapper/221:39:372
18908991914,3cyclictest5901-21find23:30:242
1890299191,16cyclictest25470-21hddtemp_smartct20:35:171
1889599197,3cyclictest12490-21gzip23:45:260
1889599197,2cyclictest7600-21turbostat.cron21:10:000
1889599196,4cyclictest32617-21gzip20:50:240
1889599196,4cyclictest32617-21gzip20:50:240
1889599195,7cyclictest13809-21/usr/sbin/munin20:10:290
1889599191,9cyclictest16918-21memory20:15:190
225852181,2sleep20-21swapper/221:40:262
1891599181,5cyclictest0-21swapper/320:00:203
18915991813,3cyclictest6436-21interrupts21:05:183
1890899186,4cyclictest986-1kworker/u9:122:00:032
1890899181,9cyclictest4260-21cat22:15:112
1890899181,7cyclictest0-21swapper/223:07:292
1890899181,7cyclictest0-21swapper/222:06:352
1890899181,7cyclictest0-21swapper/200:26:542
1890899181,7cyclictest0-21swapper/200:26:542
1890899181,6cyclictest0-21swapper/223:21:352
1890899181,3cyclictest0-21swapper/200:12:112
18908991812,3cyclictest21216-21munin-node19:40:142
18908991811,4cyclictest22520-21grep19:15:172
1890899181,14cyclictest1-21systemd00:20:272
1889599185,3cyclictest24207-21strings21:45:210
1889599184,7cyclictest801-21snmpd20:35:100
1889599184,3cyclictest0-21swapper/019:29:000
1889599183,12cyclictest1212-21/usr/sbin/munin20:55:250
1889599181,2cyclictest88750irq/134-enp2s0-19:24:330
1889599180,3cyclictest31179-21perf20:50:010
190862170,2sleep20-21swapper/222:48:332
1890899177,3cyclictest86950irq/126-enp1s0-00:15:132
1890899175,4cyclictest5746-21awk19:50:182
1890899173,4cyclictest28350-21sh19:30:012
1890899172,4cyclictest912-21kworker/u8:021:20:202
1890899172,4cyclictest0-21swapper/221:48:532
1890899171,7cyclictest20683-21sa119:10:272
1890899171,6cyclictest7604-21diskmemload22:55:262
1890899171,6cyclictest2741-21grep22:10:182
1890899171,6cyclictest0-21swapper/223:37:242
1890899171,6cyclictest0-21swapper/222:30:142
1890899171,6cyclictest0-21swapper/222:20:192
1890899171,6cyclictest0-21swapper/221:26:462
1890899171,5cyclictest577-21systemd-journal23:15:012
1890899171,4cyclictest0-21swapper/221:05:232
1890899171,3cyclictest0-21swapper/223:40:062
18908991713,3cyclictest0-21swapper/220:04:272
18908991712,3cyclictest19520-21sort20:20:252
18908991711,4cyclictest29048-21/usr/sbin/munin19:30:162
1890899171,14cyclictest0-21swapper/220:15:062
1889599176,2cyclictest0-21swapper/020:00:030
1889599175,2cyclictest30135-21head19:30:270
1889599171,3cyclictest0-21swapper/019:15:120
18895991713,2cyclictest20087-21strings19:10:210
18895991712,3cyclictest11625-21acpi20:05:150
1889599170,8cyclictest0-21swapper/020:44:420
1889599170,11cyclictest0-21swapper/019:55:150
266062160,2sleep20-21swapper/220:38:062
1890899163,4cyclictest20717-21expr20:25:152
1890899161,6cyclictest0-21swapper/220:05:242
1890899161,14cyclictest0-21swapper/221:02:352
1890899161,14cyclictest0-21swapper/219:59:092
18908991611,3cyclictest30560-21timerandwakeup20:45:262
18908991610,4cyclictest13809-21/usr/sbin/munin20:10:152
18908991610,3cyclictest21120-21ntpq00:05:222
1890899160,14cyclictest0-21swapper/219:20:062
1890899151,2cyclictest0-21swapper/220:30:302
1890899151,12cyclictest28535-21grep20:40:262
18908991510,3cyclictest30777-21/usr/sbin/munin19:35:202
1890899150,14cyclictest0-21swapper/219:45:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional