You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 10:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun May 03, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32562240163,26sleep30-21swapper/319:08:133
33672235163,23sleep20-21swapper/219:09:402
33662235162,24sleep10-21swapper/119:09:391
31572235161,23sleep00-21swapper/019:06:560
281602530,4sleep230-21rcuc/223:45:012
200002520,1sleep10-21swapper/119:45:141
156802500,3sleep115679-21gltestperf23:15:141
99192480,1sleep00-21swapper/021:50:010
347799388,14cyclictest10410-21munin-run23:05:020
3477993811,13cyclictest8545-21munin-run19:20:020
3477993810,12cyclictest9-21rcu_preempt20:35:020
3477993610,13cyclictest11980-21munin-run21:55:010
348399354,19cyclictest144192sort00:25:191
347799345,13cyclictest32374-21munin-run23:55:010
3494993310,2cyclictest0-21swapper/322:15:243
347799337,13cyclictest32732-21awk20:15:010
3483993218,9cyclictest1258-40dnf22:41:091
348399313,15cyclictest10051-21cpu21:50:141
3483993119,7cyclictest17267-40dnf19:39:241
348399302,10cyclictest3757-21diskstats21:35:161
349499299,10cyclictest22295-21df_inode19:50:143
349499290,13cyclictest0-21swapper/320:55:073
348399293,10cyclictest19184-21sh23:25:001
348399292,18cyclictest0-21swapper/119:15:041
349499281,14cyclictest0-21swapper/321:22:173
349499281,13cyclictest0-21swapper/320:52:293
349499281,12cyclictest0-21swapper/319:19:453
349499281,11cyclictest0-21swapper/322:41:503
349499280,9cyclictest0-21swapper/321:05:253
348399286,8cyclictest28950-21munin-run21:20:021
348399285,11cyclictest4845-21cstates00:05:131
348399283,13cyclictest28546-21sshd20:05:051
348399282,19cyclictest14075-21/usr/sbin/munin23:10:231
3483992810,15cyclictest18525-21diskstats20:55:141
3477992810,15cyclictest6972-21idleruntime-cro20:30:020
349499279,5cyclictest0-21swapper/321:36:213
349499278,3cyclictest0-21swapper/321:30:253
3494992711,3cyclictest0-21swapper/323:06:493
349499271,13cyclictest0-21swapper/322:30:063
349499271,13cyclictest0-21swapper/320:40:093
349499271,13cyclictest0-21swapper/319:32:343
349499271,13cyclictest0-21swapper/300:38:293
349499271,13cyclictest0-21swapper/300:08:093
349499271,12cyclictest0-21swapper/321:11:053
349499271,12cyclictest0-21swapper/321:11:053
349499271,12cyclictest0-21swapper/320:02:223
349499271,12cyclictest0-21swapper/320:02:223
349499271,12cyclictest0-21swapper/319:57:063
349499271,12cyclictest0-21swapper/300:26:053
349499271,12cyclictest0-21swapper/300:21:173
349499271,12cyclictest0-21swapper/300:01:213
349499271,11cyclictest0-21swapper/323:26:493
349499270,11cyclictest0-21swapper/320:25:063
348399277,17cyclictest18332-21uniq00:35:151
348399275,18cyclictest24150-21/usr/sbin/munin19:55:251
348399271,5cyclictest0-21swapper/123:05:051
3483992715,4cyclictest6940-21/usr/sbin/munin00:10:131
349499269,3cyclictest0-21swapper/321:29:093
349499269,3cyclictest0-21swapper/321:02:253
349499269,2cyclictest0-21swapper/320:35:193
349499268,3cyclictest0-21swapper/319:14:333
349499261,13cyclictest0-21swapper/300:18:093
349499261,12cyclictest0-21swapper/323:49:533
349499261,12cyclictest0-21swapper/322:00:053
349499261,12cyclictest0-21swapper/319:46:343
349499261,11cyclictest0-21swapper/323:18:573
349499261,11cyclictest0-21swapper/322:22:063
349499261,11cyclictest0-21swapper/321:55:533
348399267,16cyclictest28339-21awk20:05:011
348399267,16cyclictest28339-21awk20:05:011
348399266,17cyclictest6953-21latency_hist20:30:011
348399265,17cyclictest20772-21munin-run22:15:001
348399261,5cyclictest0-21swapper/121:40:251
348399260,20cyclictest0-21swapper/122:50:321
347799266,17cyclictest22353-21munin-run21:05:010
349499259,4cyclictest21959-21diskstats23:30:133
349499259,3cyclictest0-21swapper/323:22:173
349499259,3cyclictest0-21swapper/321:49:453
349499259,3cyclictest0-21swapper/320:06:063
349499258,9cyclictest0-21swapper/322:27:503
349499258,3cyclictest0-21swapper/320:21:503
349499258,3cyclictest0-21swapper/319:38:383
349499258,2cyclictest0-21swapper/323:10:193
349499251,13cyclictest0-21swapper/300:13:133
349499251,12cyclictest0-21swapper/323:35:333
349499251,12cyclictest0-21swapper/323:35:333
349499251,12cyclictest0-21swapper/322:12:383
349499251,11cyclictest0-21swapper/320:11:103
3494992510,3cyclictest24547-21diskmemload23:03:013
3494992510,3cyclictest0-21swapper/321:42:493
3494992510,3cyclictest0-21swapper/321:19:173
349099254,11cyclictest13833-21grep00:25:162
348399258,11cyclictest755-21rngd00:01:381
348399256,6cyclictest9383-21/usr/sbin/munin20:35:251
348399254,18cyclictest25772-21turbostat.cron23:40:021
348399254,18cyclictest25772-21turbostat.cron23:40:021
348399252,10cyclictest28206-21/usr/sbin/munin23:45:191
3483992516,6cyclictest30901-21grep20:10:111
3483992516,5cyclictest11271-21/usr/sbin/munin00:20:281
348399251,15cyclictest0-21swapper/123:33:361
348399250,22cyclictest31940-21cut21:25:151
348399250,16cyclictest0-21swapper/122:17:351
347799251,11cyclictest15145-21awk19:35:000
349499249,4cyclictest0-21swapper/322:35:423
349499249,3cyclictest0-21swapper/321:51:133
349499249,3cyclictest0-21swapper/320:30:053
349499249,3cyclictest0-21swapper/319:23:133
349499248,3cyclictest0-21swapper/320:48:533
349499247,3cyclictest0-21swapper/319:26:583
349499241,13cyclictest0-21swapper/322:59:453
349499241,11cyclictest0-21swapper/323:58:133
3494992410,3cyclictest0-21swapper/322:54:493
3494992410,3cyclictest0-21swapper/322:49:013
348399247,14cyclictest4597-21memory19:10:181
348399246,9cyclictest0-21swapper/119:40:241
348399246,7cyclictest14399-21/usr/sbin/munin22:00:201
348399245,16cyclictest30851-21runrttasks21:21:271
348399245,16cyclictest11379-21fschecks_count19:25:141
348399243,7cyclictest87550irq/128-enp1s0-22:45:141
348399243,18cyclictest31719-21sort23:50:251
348399241,17cyclictest0-21swapper/122:05:541
348399240,10cyclictest26688-21smartctl23:40:161
349499239,3cyclictest0-21swapper/322:09:493
349499238,3cyclictest0-21swapper/323:52:173
349499238,3cyclictest0-21swapper/319:41:503
349499237,3cyclictest0-21swapper/323:43:293
349499237,3cyclictest0-21swapper/300:31:093
349099231,4cyclictest25714-21missed_timers21:10:182
349099231,4cyclictest25714-21missed_timers21:10:182
348399235,16cyclictest20448-21ntpq23:25:181
348399234,9cyclictest3026-21taskset21:33:161
348399234,16cyclictest14313-21sort20:45:161
348399233,12cyclictest29605-21sed22:35:011
348399233,11cyclictest15305-21cat19:35:011
348399231,18cyclictest26752-21perf21:15:021
348399231,18cyclictest26752-21perf21:15:021
348399230,15cyclictest0-21swapper/120:15:221
269882230,2sleep20-21swapper/221:15:122
349499227,3cyclictest0-21swapper/320:19:303
348399229,4cyclictest2774-21sshd20:20:061
348399226,13cyclictest1203-21uname23:55:211
348399226,10cyclictest936-21kworker/u8:221:57:501
348399223,16cyclictest27355-21latency_hist22:30:011
348399223,16cyclictest16041-21cpu20:50:121
348399222,8cyclictest29787-21/usr/sbin/munin22:35:191
3483992215,4cyclictest23393-21irqstats21:05:191
3483992215,4cyclictest12653-21ntp_states20:40:191
3483992213,6cyclictest23018-21ntp_states19:50:201
348399221,18cyclictest9684-21sh23:00:221
348399221,18cyclictest579-21systemd-journal19:25:021
348399221,16cyclictest0-21swapper/100:33:461
347799227,6cyclictest0-21swapper/022:28:030
349099218,3cyclictest0-21swapper/223:03:332
349099214,3cyclictest10977-21/usr/sbin/munin23:05:132
349099212,3cyclictest7572-21crond21:45:022
349099211,11cyclictest0-21swapper/200:15:262
348399215,13cyclictest910-21systemd-logind00:18:061
348399214,14cyclictest24130-21sed22:20:191
348399211,8cyclictest808-21snmpd22:55:051
348399211,17cyclictest8934-21pmu-power21:45:221
347799215,6cyclictest0-21swapper/000:07:150
347799215,3cyclictest87350irq/126-enp1s0-21:32:570
347799215,2cyclictest87350irq/126-enp1s0-23:24:460
347799214,7cyclictest22103-21pmu-power22:15:230
347799211,3cyclictest0-21swapper/021:23:070
347799211,11cyclictest15542-21pmu-power22:00:210
347799211,10cyclictest24780-21/usr/sbin/munin21:10:190
347799211,10cyclictest24780-21/usr/sbin/munin21:10:190
3477992110,3cyclictest87550irq/128-enp1s0-20:35:240
347799210,3cyclictest0-21swapper/023:09:130
349099208,4cyclictest1598-21awk22:45:012
349099203,4cyclictest579-21systemd-journal22:05:012
349099201,9cyclictest0-21swapper/220:10:142
349099201,3cyclictest15184-21grep23:15:112
349099201,3cyclictest0-21swapper/222:58:442
349099201,11cyclictest0-21swapper/219:47:562
3483992011,5cyclictest20724-21diskstats21:00:151
348399201,10cyclictest8212-21netstat20:30:201
347799209,3cyclictest17858-21/usr/sbin/munin00:35:210
347799207,3cyclictest0-21swapper/023:15:190
347799206,5cyclictest0-21swapper/021:36:060
347799201,6cyclictest0-21swapper/000:28:170
347799201,4cyclictest0-21swapper/022:53:430
347799201,11cyclictest8605-21sshd00:12:060
347799201,11cyclictest6574-21diskstats22:55:150
347799201,11cyclictest18804-21/usr/sbin/munin22:10:160
347799201,10cyclictest0-21swapper/019:51:040
349099197,4cyclictest32-21ksoftirqd/220:20:012
349099194,3cyclictest87350irq/126-enp1s0-00:20:162
349099193,8cyclictest32-21ksoftirqd/219:50:052
349099192,7cyclictest31716-21diskstats21:25:142
349099191,9cyclictest0-21swapper/222:18:532
349099191,8cyclictest579-21systemd-journal22:50:012
3490991915,2cyclictest30-21rcuc/219:40:012
349099191,15cyclictest21752-21tr23:30:112
349099191,10cyclictest0-21swapper/200:09:482
349099190,14cyclictest5643-21crond19:15:012
347799199,3cyclictest3642-21sshd22:48:070
347799195,3cyclictest14074-21expr20:45:140
347799193,3cyclictest0-21swapper/021:26:230
347799193,3cyclictest0-21swapper/021:16:340
347799191,4cyclictest5809-21cstates21:40:110
347799191,4cyclictest21035-21chrt23:25:420
3477991914,3cyclictest22709-21tr21:05:130
347799191,15cyclictest0-21swapper/000:15:150
347799191,11cyclictest0-21swapper/023:41:110
347799190,5cyclictest0-21swapper/000:31:420
130022190,2sleep00-21swapper/020:40:250
68622181,3sleep20-21swapper/200:10:072
42402181,2sleep20-21swapper/200:02:412
349099186,4cyclictest87350irq/126-enp1s0-22:10:072
349099186,4cyclictest28478-21expr23:45:122
349099185,3cyclictest87350irq/126-enp1s0-23:35:132
349099185,3cyclictest87350irq/126-enp1s0-23:35:132
349099185,3cyclictest13747-21missed_timers23:10:192
349099184,4cyclictest311ktimersoftd/223:30:012
349099181,9cyclictest18515-21df20:55:122
349099181,9cyclictest0-21swapper/222:29:582
349099181,7cyclictest0-21swapper/221:24:162
3490991812,3cyclictest9112-21df_inode19:20:142
3490991812,3cyclictest12543-21df_inode21:55:142
349099181,14cyclictest30456-21runrttasks20:09:182
349099180,8cyclictest0-21swapper/219:27:402
347799189,5cyclictest24547-21diskmemload22:41:090
347799184,3cyclictest24547-21diskmemload22:05:140
347799183,3cyclictest16974-21cut20:50:200
347799183,3cyclictest0-21swapper/023:10:050
347799183,12cyclictest7-21ksoftirqd/021:55:230
347799182,8cyclictest11271-21/usr/sbin/munin00:20:150
347799182,13cyclictest18177-21/usr/sbin/munin20:55:150
3477991812,3cyclictest27591-21/usr/sbin/munin22:30:130
347799181,10cyclictest28556-21/usr/sbin/munin20:05:140
347799181,10cyclictest26348-21/usr/sbin/munin20:00:240
347799181,10cyclictest26348-21/usr/sbin/munin20:00:240
347799181,10cyclictest0-21swapper/020:20:110
347799180,9cyclictest87350irq/126-enp1s0-23:46:420
347799180,3cyclictest0-21swapper/022:20:140
347799180,12cyclictest0-21swapper/022:35:140
283872180,4sleep230-21rcuc/222:30:172
22052181,3sleep00-21swapper/020:15:410
349099177,2cyclictest15138-21awk19:35:012
349099176,3cyclictest87350irq/126-enp1s0-23:20:182
349099174,3cyclictest87350irq/126-enp1s0-00:35:132
349099173,4cyclictest1075-21wc23:55:192
349099173,3cyclictest0-21swapper/220:35:222
349099171,8cyclictest0-21swapper/221:50:282
349099171,6cyclictest3469-21egrep21:35:142
349099171,6cyclictest0-21swapper/222:07:192
349099171,6cyclictest0-21swapper/220:25:132
349099171,6cyclictest0-21swapper/219:55:132
349099171,6cyclictest0-21swapper/219:15:162
349099171,15cyclictest0-21swapper/220:20:242
349099171,10cyclictest32-21ksoftirqd/222:35:152
347799177,3cyclictest808-21snmpd19:29:500
347799175,3cyclictest0-21swapper/019:42:110
347799171,8cyclictest0-21swapper/000:03:580
347799171,7cyclictest0-21swapper/023:30:170
3477991712,3cyclictest8772-21acpi19:20:110
3477991711,4cyclictest808-21snmpd19:55:030
347799171,14cyclictest1810-21grep23:55:260
347799171,13cyclictest81ktimersoftd/019:10:130
3477991711,3cyclictest24547-21diskmemload23:35:200
3477991711,3cyclictest24547-21diskmemload23:35:190
29662171,2sleep20-21swapper/221:32:322
349099162,5cyclictest17895-21df19:40:122
349099161,5cyclictest0-21swapper/223:50:042
349099161,5cyclictest0-21swapper/221:48:072
3490991612,3cyclictest0-21swapper/220:04:002
3490991612,3cyclictest0-21swapper/220:03:592
3490991612,2cyclictest7865-21sh20:30:162
349099161,14cyclictest0-21swapper/220:51:192
349099161,14cyclictest0-21swapper/220:41:332
3490991611,3cyclictest4094-21cpuspeed_turbos22:50:112
3490991611,3cyclictest15664-21/usr/sbin/munin00:30:132
3490991610,3cyclictest23197-21/usr/sbin/munin22:20:232
3490991610,3cyclictest22582-21/usr/sbin/munin21:05:152
349099160,6cyclictest0-21swapper/221:00:222
349099160,2cyclictest87550irq/128-enp1s0-20:45:202
3477991612,3cyclictest0-21swapper/019:35:540
3477991611,3cyclictest20021-21ls19:45:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional