You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 06:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 16, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
325042237163,25sleep00-21swapper/019:08:020
323512235163,24sleep20-21swapper/219:06:022
324772227153,24sleep30-21swapper/319:07:403
323072200163,24sleep10-21swapper/119:05:301
2744721730,4sleep33275899cyclictest20:10:003
2785921510,3sleep13274799cyclictest23:50:141
2785921510,3sleep13274799cyclictest23:50:141
32754998481,2cyclictest0-21swapper/223:14:472
28092420,1sleep30-21swapper/300:05:223
3274299389,13cyclictest19872-21munin-run22:20:020
3275899373,25cyclictest3058-21cut22:55:013
3275899368,17cyclictest1-21systemd00:10:163
3275899357,17cyclictest19515-21netstat23:30:193
3275899341,25cyclictest0-21swapper/300:27:003
3274299348,12cyclictest4495-21idleruntime-cro21:45:010
3275899337,11cyclictest25225-21crond20:05:003
32758993320,8cyclictest24203-40dnf22:29:243
32758993214,10cyclictest15972-21taskset23:23:403
32758993211,10cyclictest8814-21cat23:05:253
3274799321,20cyclictest12112-21irqstats22:00:171
3275899314,12cyclictest14775-21cpu00:35:123
32747993113,16cyclictest13977-21kworker/u8:121:07:441
32742993113,7cyclictest14075-21grep22:05:150
3275899304,14cyclictest774-21avahi-daemon19:15:023
3275899304,13cyclictest9731-21grep21:55:153
3275899304,13cyclictest30180-21taskset21:29:223
3275899297,7cyclictest1318-21awk00:05:023
3275899295,13cyclictest29855-21/usr/sbin/munin20:15:253
3275899293,19cyclictest15332-21kworker/u8:122:11:453
32758992910,11cyclictest7246-21df_inode21:50:123
3274799291,12cyclictest12068-21kworker/u8:200:29:521
3275899287,7cyclictest18629-21latency_hist19:50:013
3275899287,18cyclictest12432-21sed22:00:213
3275899287,10cyclictest1359-21ntp_states21:35:203
3275899287,10cyclictest1359-21ntp_states21:35:203
3275899286,14cyclictest26083-21users22:30:263
3275899286,13cyclictest18261-21sed21:00:203
3275899285,15cyclictest29622-21latency_hist20:15:003
3275899282,7cyclictest0-21swapper/322:23:303
3275899278,5cyclictest8391-21expr00:20:123
3275899277,5cyclictest13379-21irqstats00:30:193
3275899276,12cyclictest6932-21cat00:15:173
3275899275,6cyclictest87350irq/126-enp1s0-20:50:153
3275899275,4cyclictest31422-21df_inode22:45:133
3275899275,14cyclictest30719-21rt-features23:55:233
3275899274,12cyclictest19806-21cstates21:05:143
3275899274,10cyclictest30104-21kworker/u8:022:07:383
3275899272,21cyclictest21448-21diskmemload23:50:123
3275899272,21cyclictest21448-21diskmemload23:50:123
3275899271,10cyclictest0-21swapper/322:16:493
32758992710,5cyclictest25839-21latency_hist21:20:013
3274799270,19cyclictest23-21ksoftirqd/121:40:111
3274299273,8cyclictest81ktimersoftd/023:10:160
3274299272,13cyclictest2272-21kworker/0:221:45:580
32742992710,4cyclictest13492-21kworker/0:222:29:260
165162270,4sleep19-21rcu_preempt20:55:251
3275899269,6cyclictest25167-21egrep23:45:123
3275899267,6cyclictest6073-21perf20:35:003
3275899267,16cyclictest27758-21ntp_states22:35:203
3275899263,12cyclictest3578-21ntp_states21:40:203
3275899262,14cyclictest1661-21sh20:25:003
3274799269,4cyclictest221ktimersoftd/100:05:231
3274299267,16cyclictest7949-21munin-run00:20:020
3274299267,16cyclictest18288-40dnf23:30:000
3274299261,13cyclictest14538-21munin-run00:35:010
3274299261,13cyclictest12660-21munin-run20:50:010
64892250,4sleep3401ktimersoftd/319:20:173
3275899255,17cyclictest22749-21ntp_states21:10:213
3275899255,12cyclictest0-21swapper/321:45:113
3275899253,8cyclictest9727-21/usr/sbin/munin23:10:253
3275899252,7cyclictest0-21swapper/322:56:003
3275899251,7cyclictest0-21swapper/323:28:053
3275899251,22cyclictest2873-21kworker/u8:023:00:163
32758992510,3cyclictest23972-21ntpq23:40:203
3275499251,5cyclictest0-21swapper/223:50:182
3275499251,5cyclictest0-21swapper/223:50:182
3274799251,18cyclictest23-21ksoftirqd/121:10:111
3274799251,16cyclictest23-21ksoftirqd/122:25:111
3274799251,16cyclictest14930-21/usr/sbin/munin23:20:161
32747992511,12cyclictest9375-21kworker/u8:023:08:461
3274299252,8cyclictest14118-21/usr/sbin/munin23:20:160
3275899247,7cyclictest26094-21cpu21:20:123
3275899245,9cyclictest1-21systemd23:15:153
3275899244,16cyclictest484-21/usr/sbin/munin19:10:123
3275899244,16cyclictest484-21/usr/sbin/munin19:10:113
3275899243,6cyclictest10453-21latency_hist20:45:013
3275899241,7cyclictest14462-21/usr/sbin/munin19:40:243
3275899241,12cyclictest0-21swapper/323:35:133
3274799249,13cyclictest11744-21kworker/u8:219:43:451
3274799245,5cyclictest87550irq/128-enp1s0-22:45:161
3274799244,5cyclictest87550irq/128-enp1s0-00:30:261
3274799241,21cyclictest0-21swapper/122:14:461
3274299249,12cyclictest17062-21idleruntime-cro21:00:010
3274299245,8cyclictest29855-21/usr/sbin/munin20:15:230
3274299242,14cyclictest11988-21kworker/0:000:25:580
3274299241,12cyclictest0-21swapper/021:35:230
3274299241,12cyclictest0-21swapper/021:35:230
32758992314,6cyclictest12618-21diskstats19:35:133
32758992312,6cyclictest827-21runrttasks19:28:153
3275899230,13cyclictest0-21swapper/320:25:153
3275899230,13cyclictest0-21swapper/320:25:153
3274799234,2cyclictest21-21rcuc/123:35:221
3274799231,9cyclictest221ktimersoftd/100:21:101
32747992318,2cyclictest19125-21sshd23:30:141
32747992311,4cyclictest18218-21kworker/u8:023:42:461
3274299235,8cyclictest87550irq/128-enp1s0-22:45:140
3274299231,11cyclictest0-21swapper/023:35:440
32742992310,3cyclictest19078-21kworker/0:121:20:180
3274299230,11cyclictest23040-21awk20:00:010
23892230,2sleep20-21swapper/221:40:012
23892230,2sleep20-21swapper/221:40:012
3275899223,16cyclictest6292-21/usr/sbin/munin20:35:153
32758992213,6cyclictest31058-21/usr/sbin/munin21:30:163
3275899221,13cyclictest0-21swapper/322:40:283
3275899221,11cyclictest10071-21/usr/sbin/munin19:30:133
3274799229,4cyclictest23-21ksoftirqd/121:20:001
3274799224,2cyclictest87550irq/128-enp1s0-23:14:591
3274799221,18cyclictest7575-21hddtemp_smartct21:50:131
3274799221,16cyclictest87550irq/128-enp1s0-22:22:171
3274799221,16cyclictest0-21swapper/121:20:431
3274799221,12cyclictest0-21swapper/121:57:051
3274799220,5cyclictest87550irq/128-enp1s0-00:15:111
3274299228,3cyclictest14462-21/usr/sbin/munin19:40:130
3274299225,3cyclictest0-21swapper/023:05:190
3274299224,7cyclictest30655-21cstates21:30:130
3274299223,8cyclictest81ktimersoftd/021:05:010
3274299223,7cyclictest81ktimersoftd/022:10:150
3274299223,4cyclictest808-21snmpd22:24:190
3274299222,10cyclictest27657-21/usr/sbin/munin20:10:120
3274299221,7cyclictest16660-21/usr/sbin/munin19:45:270
3274299221,5cyclictest827-21runrttasks23:57:590
3274299221,5cyclictest0-21swapper/023:00:190
3274299221,13cyclictest0-21swapper/022:35:000
3274299221,12cyclictest0-21swapper/021:11:210
3274299221,11cyclictest32049-21/usr/sbin/munin20:20:110
32742992210,5cyclictest87550irq/128-enp1s0-19:20:210
3274299220,4cyclictest3202-21kworker/u8:000:10:250
3274299220,14cyclictest1099-21kworker/0:100:02:520
32758992112,6cyclictest18856-21/usr/sbin/munin19:50:223
32758992112,6cyclictest10683-21/usr/sbin/munin20:45:203
32758992111,7cyclictest21056-21/usr/sbin/munin19:55:113
3275499215,4cyclictest909-21sssd_nss23:40:172
3275499210,5cyclictest11744-21kworker/u8:221:27:402
32747992116,3cyclictest0-21swapper/119:55:211
3274799211,18cyclictest0-21swapper/122:36:351
3274799211,13cyclictest0-21swapper/123:28:191
3274799211,11cyclictest2180-21vmstat19:10:261
3274799211,11cyclictest2180-21vmstat19:10:251
3274799211,11cyclictest11753-21ntp_states20:45:201
3274799211,11cyclictest0-21swapper/122:31:361
3274799210,17cyclictest0-21swapper/123:46:331
3274799210,15cyclictest0-21swapper/100:36:181
3274799210,13cyclictest21-21rcuc/121:30:001
3274299219,3cyclictest25687-21sshd23:45:150
3274299215,7cyclictest0-21swapper/019:25:200
3274299215,3cyclictest21448-21diskmemload22:03:530
3274299211,5cyclictest1615-1kworker/u9:121:53:390
3274299211,3cyclictest0-21swapper/023:32:100
3274299211,11cyclictest8713-21expr20:40:110
3274299211,11cyclictest3474-21diskstats22:55:130
3274299211,11cyclictest0-21swapper/000:39:100
3274299211,11cyclictest0-21swapper/000:06:080
3274299211,10cyclictest0-21swapper/021:26:370
3274299210,2cyclictest89150irq/134-enp2s0-21:19:170
3274299210,2cyclictest89150irq/134-enp2s0-20:02:420
115702210,8sleep232-21ksoftirqd/222:00:112
49462201,2sleep10-21swapper/122:58:151
32758992013,4cyclictest15429-21diskstats20:55:153
3275499209,3cyclictest0-21swapper/222:35:162
3275499204,6cyclictest21291-21gltestperf23:35:142
3274799206,5cyclictest755-21rngd21:00:251
3274799205,3cyclictest11744-21kworker/u8:219:17:191
3274799203,4cyclictest87550irq/128-enp1s0-21:37:471
3274799203,4cyclictest87550irq/128-enp1s0-21:37:471
3274799201,9cyclictest0-21swapper/123:16:551
32747992016,3cyclictest0-21swapper/120:21:541
3274799201,2cyclictest17008-21/usr/sbin/munin19:45:141
3274799201,17cyclictest0-21swapper/120:50:121
3274799201,17cyclictest0-21swapper/120:05:581
3274799201,11cyclictest29855-21/usr/sbin/munin20:15:231
3274799201,11cyclictest24326-21ntp_states20:00:211
3274799200,5cyclictest21-21rcuc/100:10:161
3274299209,2cyclictest87350irq/126-enp1s0-23:40:210
3274299206,3cyclictest0-21swapper/022:38:570
3274299204,7cyclictest11861-21kworker/0:219:34:380
3274299203,7cyclictest0-21swapper/023:51:290
3274299203,7cyclictest0-21swapper/023:51:290
3274299202,2cyclictest27422-21crond20:10:010
3274299201,5cyclictest11924-21/usr/sbin/munin23:15:160
32742992011,7cyclictest373-21kworker/u8:122:50:160
3274299201,12cyclictest0-21swapper/021:57:090
3274299201,11cyclictest0-21swapper/000:20:200
3274299200,6cyclictest87350irq/126-enp1s0-19:55:020
66672191,1sleep20-21swapper/221:50:012
64702191,2sleep10-21swapper/121:47:271
3275499192,6cyclictest11072-21munin-run22:00:012
3275499192,3cyclictest0-21swapper/219:42:052
3275499191,6cyclictest0-21swapper/223:03:052
32754991916,2cyclictest0-21swapper/219:48:372
3275499191,3cyclictest16083-21latency_hist23:25:022
32754991913,3cyclictest3130-21find20:25:242
32754991913,3cyclictest3130-21find20:25:242
3275499191,10cyclictest0-21swapper/223:19:002
32754991910,2cyclictest87550irq/128-enp1s0-00:38:522
3275499190,8cyclictest0-21swapper/222:25:102
3275499190,12cyclictest0-21swapper/221:05:202
3274799196,7cyclictest0-21swapper/121:30:151
3274799196,4cyclictest8631-21cut20:40:121
3274799193,2cyclictest23-21ksoftirqd/122:50:121
32747991912,6cyclictest23533-21kworker/u8:019:21:441
3274799191,16cyclictest0-21swapper/120:32:151
3274799191,11cyclictest0-21swapper/123:04:531
3274799191,10cyclictest12268-21/usr/sbin/munin19:35:171
3274799191,10cyclictest0-21swapper/122:41:301
3274799190,18cyclictest0-21swapper/122:18:531
3274799190,17cyclictest0-21swapper/120:12:331
3274299195,3cyclictest87550irq/128-enp1s0-19:17:110
3274299194,8cyclictest5958-21kworker/0:020:37:090
3274299192,4cyclictest3872-21munin-run20:30:000
3274299192,4cyclictest3872-21munin-run20:30:000
3274299191,9cyclictest14845-21idleruntime-cro20:55:010
3274299191,7cyclictest21025-21unixbench_singl21:05:260
146102191,2sleep20-21swapper/222:05:222
94792181,2sleep20-21swapper/219:25:302
3275499189,2cyclictest30-21rcuc/220:55:262
3275499187,3cyclictest87350irq/126-enp1s0-22:50:142
3275499185,3cyclictest87350irq/126-enp1s0-20:35:242
3275499184,4cyclictest11353-21needreboot00:25:182
3275499183,5cyclictest29230-21diskstats22:40:132
3275499183,5cyclictest18763-21latency22:15:182
3275499182,7cyclictest18513-21/usr/sbin/munin23:30:182
3275499182,7cyclictest12268-21/usr/sbin/munin19:35:212
3275499181,8cyclictest4111-21cpu20:30:122
3275499181,7cyclictest31809-21crond20:20:012
3275499181,6cyclictest0-21swapper/219:11:482
3275499181,6cyclictest0-21swapper/219:11:482
3275499181,5cyclictest0-21swapper/223:25:232
3275499181,5cyclictest0-21swapper/222:45:512
32754991812,3cyclictest3174-21ntp_offset19:15:022
32754991812,3cyclictest29499-21/usr/sbin/munin23:55:152
32754991811,4cyclictest3730-21cut21:40:212
3275499180,16cyclictest12662-21perf20:50:022
3274799188,4cyclictest221ktimersoftd/122:05:041
3274799182,3cyclictest31886-21cstates00:00:101
3274799181,9cyclictest21-21rcuc/119:30:151
32747991815,2cyclictest0-21swapper/123:57:371
3274799181,16cyclictest0-21swapper/120:35:051
3274799180,16cyclictest0-21swapper/119:50:171
3274299183,3cyclictest87550irq/128-enp1s0-19:10:130
3274299183,3cyclictest87550irq/128-enp1s0-19:10:130
3274299182,2cyclictest87350irq/126-enp1s0-20:30:130
3274299181,9cyclictest23967-21kworker/u8:022:40:010
3274299180,6cyclictest0-21swapper/019:35:220
100892181,2sleep20-21swapper/220:40:272
49692170,3sleep20-21swapper/222:58:322
3275499175,3cyclictest87350irq/126-enp1s0-00:20:112
3275499174,5cyclictest27690-21cpu20:10:122
3275499173,7cyclictest0-21swapper/200:00:112
3275499171,8cyclictest0-21swapper/221:31:392
3275499171,7cyclictest0-21swapper/220:50:072
3275499171,7cyclictest0-21swapper/200:30:492
3275499171,7cyclictest0-21swapper/200:17:082
3275499171,6cyclictest26138-21cpuspeed_turbos21:20:122
3275499171,6cyclictest2285-21sshd00:05:152
3275499171,6cyclictest0-21swapper/221:50:032
3275499171,6cyclictest0-21swapper/221:15:312
3275499171,6cyclictest0-21swapper/219:55:132
3275499171,4cyclictest0-21swapper/223:48:392
32754991712,3cyclictest5673-21/usr/sbin/munin19:20:162
32754991712,3cyclictest10468-21fschecks_count19:30:142
3275499171,14cyclictest0-21swapper/221:01:482
3275499171,10cyclictest0-21swapper/222:22:582
3275499171,10cyclictest0-21swapper/221:10:142
32754991710,4cyclictest827-21runrttasks20:00:132
3274799174,4cyclictest755-21rngd20:25:581
3274799174,4cyclictest755-21rngd20:25:581
32747991713,3cyclictest0-21swapper/119:27:291
3275499161,7cyclictest20201-21timerandwakeup19:50:232
3275499161,6cyclictest7534-21/usr/sbin/munin23:05:192
3275499161,6cyclictest4836-21ntpq00:10:192
32754991611,3cyclictest32049-21/usr/sbin/munin20:20:192
32754991611,3cyclictest26276-21irqstats20:05:172
32754991611,3cyclictest26087-21vmstat22:30:272
32754991610,3cyclictest16082-21cut22:10:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional