You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 20:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Dec 21, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
197399780,68rtkit-daemon0-21swapper/107:07:271
84379427639,13sleep00-21swapper/007:07:050
8683972750,4sleep10-21swapper/108:15:301
197399740,6rtkit-daemon2069-21gnome-shell08:01:100
84394626444,14sleep20-21swapper/207:09:042
8663242630,13sleep0131ksoftirqd/008:10:210
84388626242,14sleep30-21swapper/307:08:163
844186996111,30cyclictest898707-21sed09:45:023
197399600,9rtkit-daemon1972-21rtkit-daemon08:05:550
844186995917,13cyclictest859857-21users07:50:303
844186995914,19cyclictest867196-21munin-run08:14:593
197399580,9rtkit-daemon1972-21rtkit-daemon07:42:261
844186995715,15cyclictest936960-21users11:30:333
844186995612,27cyclictest931554-21python311:15:293
844186995612,20cyclictest853332-21latency_hist07:35:023
844186995513,10cyclictest899505-21memory09:45:223
844186995513,10cyclictest899505-21memory09:45:223
844186995512,26cyclictest885441-21df09:05:173
197399540,8rtkit-daemon1972-21rtkit-daemon07:46:051
9400712530,21sleep013-21ksoftirqd/011:40:240
844186995313,16cyclictest905596-40lsb_release10:02:013
844186995312,17cyclictest438-20systemd-journal12:29:553
844186995311,26cyclictest938984-21taskset11:39:083
844186995214,16cyclictest849187-21python307:20:273
844186995214,16cyclictest849187-21python307:20:263
844186995214,15cyclictest864881-21python308:05:293
844186995214,15cyclictest858186-21python307:46:103
844186995213,16cyclictest892629-21missed_timers09:25:213
844186995212,17cyclictest846547-21sensors_temp07:15:063
844186995212,15cyclictest939734-21hddtemp_smartct11:40:183
844186995211,17cyclictest903128-21ntp_states09:55:273
844186995210,18cyclictest929850-21sed11:10:273
197399520,8rtkit-daemon1972-21rtkit-daemon07:59:061
844186995117,15cyclictest926361-21python311:00:293
844186995113,16cyclictest889916-21sshd09:19:093
844186995113,15cyclictest959081-21hddtemp_smartct12:35:193
844186995113,15cyclictest901080-21cat09:50:193
844186995112,17cyclictest891862-21cut09:25:013
844186995112,16cyclictest853765-21expr07:35:153
844186995111,15cyclictest861187-21netstat07:55:233
844186995017,13cyclictest1-21systemd11:00:003
844186995017,13cyclictest1-21systemd11:00:003
844186995013,14cyclictest881059-21awk08:55:013
844186995012,15cyclictest934623-21irqstats11:25:203
844186995010,16cyclictest923310-21taskset10:53:003
844186994916,29cyclictest1137-21runrttasks10:20:593
844186994911,15cyclictest882772-21apt-get09:00:053
197399490,35rtkit-daemon918743-21tr10:40:152
844186994817,27cyclictest872473-21ls08:30:023
844186994813,12cyclictest957715-21ps12:30:273
844186994812,15cyclictest907868-21acpi10:10:113
844186994717,14cyclictest893958-21expr09:30:153
844186994711,14cyclictest1-21systemd07:15:013
197399470,38rtkit-daemon1941-21systemd09:30:191
844186994617,24cyclictest863029-21proc_pri08:00:273
844186994615,26cyclictest887886-21uptime09:10:303
844186994519,21cyclictest896329-21python309:35:273
844186994417,23cyclictest950432-21netstat12:10:233
844186994417,23cyclictest950432-21netstat12:10:223
844186994411,30cyclictest2092-21JS10:49:093
844186994410,14cyclictest948872-21smartctl12:05:283
844183994424,16cyclictest861246-21ntp_states07:55:242
9359702430,15sleep1251rcuc/111:30:151
8467612430,8sleep0161rcuc/007:15:110
844186994317,22cyclictest910594-21wc10:15:313
844186994317,22cyclictest850356-21cstates07:25:163
844186994311,28cyclictest1033-21gdbus12:05:013
844181994324,7cyclictest26-21ksoftirqd/108:40:011
844181994324,14cyclictest852156-21df_inode07:30:171
9175192420,5sleep1251rcuc/110:35:211
84418699426,31cyclictest869367-21tr08:20:143
84418699426,14cyclictest0-21swapper/308:40:143
844186994220,18cyclictest907017-21python310:05:313
844186994219,19cyclictest941548-21taskset11:45:203
844186994219,19cyclictest941548-21taskset11:45:203
844186994215,23cyclictest1137-21runrttasks10:44:083
844186994211,28cyclictest3692982-21snapd10:47:553
844186994120,17cyclictest1054-21snmpd08:45:293
844186994120,16cyclictest868417-21find08:15:293
844186994119,18cyclictest953673-21runrttasks12:20:203
844186994116,21cyclictest875476-21users08:35:293
844186994115,12cyclictest945983-21seq12:00:013
844186994112,25cyclictest3693053-21snapd11:20:553
844186994112,25cyclictest3693053-21snapd11:20:553
8766722400,6sleep2876678-21latency08:40:212
844186994015,20cyclictest916830-21apt10:35:113
844186994015,20cyclictest881801-21uname08:55:223
84418399402,33cyclictest857785-21ntp_states07:45:262
844186993920,15cyclictest944286-21chrt11:55:013
844186993919,16cyclictest928406-21seq11:06:403
844186993919,16cyclictest855706-21hddtemp_smartct07:40:183
844186993918,18cyclictest872808-21cpuspeed_turbos08:30:143
844186993918,18cyclictest872808-21cpuspeed_turbos08:30:143
844186993918,17cyclictest915819-21ntpq10:30:253
844186993914,5cyclictest952262-21ps12:15:273
844183993921,13cyclictest900412-21pluginstate09:50:012
844183993921,13cyclictest900412-21pluginstate09:50:002
844181993925,10cyclictest877910-21sort08:45:141
844181993921,6cyclictest26-21ksoftirqd/112:25:301
197399390,26rtkit-daemon4497-21PK-Backend11:38:262
844183993824,10cyclictest954636-21cron12:25:012
844183993815,18cyclictest949883-21tr12:10:132
844183993815,18cyclictest949883-21tr12:10:132
844181993827,7cyclictest899722-21wc09:45:281
844181993827,7cyclictest899722-21wc09:45:271
844181993826,7cyclictest976-21rs:main1
844181993826,7cyclictest438-20systemd-journal07:15:001
844176993820,14cyclictest915469-21df_inode10:30:170
844183993726,7cyclictest850794-21ntp_states07:25:252
844183993721,11cyclictest438-20systemd-journal11:00:072
844183993721,11cyclictest438-20systemd-journal10:59:592
844183993721,11cyclictest438-20systemd-journal10:59:592
844183993721,10cyclictest910816-21cron10:17:012
844183993719,13cyclictest438-20systemd-journal08:40:002
844181993721,12cyclictest908746-21python310:10:281
844181993721,12cyclictest866401-21head08:10:211
844181993720,12cyclictest957783-21python312:30:291
844181993719,13cyclictest26-21ksoftirqd/109:05:001
84418199371,4cyclictest4497-21PK-Backend10:45:461
844176993720,13cyclictest914504-21pkexec10:26:070
844176993719,14cyclictest943540-21ntp_states11:50:260
197399370,28rtkit-daemon1972-21rtkit-daemon08:29:571
9263592360,5chrt0-21swapper/011:00:290
844183993628,5cyclictest0-21swapper/210:08:402
844183993625,7cyclictest1137-21runrttasks10:37:262
844183993621,11cyclictest928687-21/usr/sbin/munin11:10:022
844183993620,12cyclictest932956-21sed11:20:202
844183993620,12cyclictest932956-21sed11:20:202
844183993620,11cyclictest438-20systemd-journal12:31:052
844183993619,13cyclictest944502-21seq11:55:052
844183993619,12cyclictest864298-21ls08:05:152
844181993625,6cyclictest26-21ksoftirqd/112:40:001
844181993621,11cyclictest945007-21interrupts11:55:211
844181993621,11cyclictest919236-21grep10:40:231
844181993618,14cyclictest953560-21df_inode12:20:151
844176993622,10cyclictest881030-21cron08:55:010
84417699362,15cyclictest879316-21munin-plugin-st08:50:010
844176993619,13cyclictest1-21systemd08:35:010
844176993619,13cyclictest1-21systemd08:35:000
197399360,27rtkit-daemon1972-21rtkit-daemon09:20:231
844183993519,12cyclictest952225-21awk12:15:252
844183993519,12cyclictest931905-21sshd11:17:082
844183993519,12cyclictest890805-21sed09:20:212
844183993518,12cyclictest2143-21upowerd08:01:282
844183993517,14cyclictest871674-21uname08:25:222
844183993517,13cyclictest438-20systemd-journal07:40:012
844181993523,8cyclictest863738-21turbostat.cron08:05:001
844181993522,8cyclictest26-21ksoftirqd/110:00:151
844181993521,10cyclictest26-21ksoftirqd/110:10:001
844176993521,10cyclictest912786-21cron10:25:000
844176993520,11cyclictest861080-21load07:55:210
197399350,29rtkit-daemon0-21swapper/008:40:170
197399350,25rtkit-daemon1972-21rtkit-daemon08:55:021
84418399343,27cyclictest847664-21users07:15:282
844183993423,6cyclictest34-21ksoftirqd/211:45:192
844183993423,6cyclictest34-21ksoftirqd/211:45:192
844183993419,11cyclictest939335-21apt11:40:122
844183993419,11cyclictest923-21dbus-daemon09:45:022
844183993419,11cyclictest921219-21sed10:45:292
844183993419,11cyclictest883892-21memory09:00:232
844183993419,11cyclictest868676-21cron08:17:002
844183993419,11cyclictest859090-21expr07:50:142
844183993418,12cyclictest903527-21sshd09:56:072
844183993418,12cyclictest852704-21wc07:30:312
844183993418,11cyclictest959394-21ntp_kernel_err12:35:232
844183993418,11cyclictest438-20systemd-journal07:12:102
844183993416,13cyclictest929317-21df11:10:152
84418399341,13cyclictest924-21NetworkManager10:21:242
84418399341,13cyclictest924-21NetworkManager10:00:442
844181993424,7cyclictest1137-21runrttasks08:24:101
844181993424,5cyclictest26-21ksoftirqd/111:20:011
844181993419,5cyclictest26-21ksoftirqd/111:30:011
844181993417,5cyclictest26-21ksoftirqd/108:40:141
844181993417,13cyclictest892316-21expr09:25:141
84418199341,30cyclictest0-21swapper/111:01:241
84418199341,29cyclictest949445-21cron12:10:011
84418199341,29cyclictest4497-21PK-Backend10:20:091
84418199341,10cyclictest4497-21PK-Backend10:29:501
844176993420,10cyclictest948928-21cat12:05:290
844176993419,11cyclictest857782-21cat07:45:250
844176993417,13cyclictest868081-21sed08:15:220
84417699341,5cyclictest4497-21PK-Backend12:20:190
197399340,5rtkit-daemon0-21swapper/211:35:012
197399340,26rtkit-daemon1972-21rtkit-daemon09:38:081
197399340,26rtkit-daemon1972-21rtkit-daemon09:05:271
844183993319,9cyclictest942997-21expr11:50:142
844183993318,11cyclictest954889-21sshd12:25:082
844183993318,11cyclictest892996-21sed09:25:282
844183993318,11cyclictest885460-21entropy09:05:202
84418399331,7cyclictest0-21swapper/209:19:402
84418399331,7cyclictest0-21swapper/208:22:252
844183993317,11cyclictest901733-21awk09:50:292
844183993316,13cyclictest914453-40apt-check10:25:562
844183993316,13cyclictest895975-21cut09:35:202
84418399331,29cyclictest0-21swapper/208:56:452
844181993323,5cyclictest26-21ksoftirqd/107:15:131
844181993322,6cyclictest26-21ksoftirqd/112:15:261
844181993322,6cyclictest26-21ksoftirqd/110:33:241
844181993322,6cyclictest26-21ksoftirqd/109:55:161
844181993322,6cyclictest26-21ksoftirqd/109:40:131
844181993322,6cyclictest26-21ksoftirqd/109:16:221
844181993321,6cyclictest26-21ksoftirqd/109:50:191
844181993317,12cyclictest909727-21tr10:15:171
844181993317,11cyclictest950708-21sed12:10:301
844181993317,11cyclictest950708-21sed12:10:291
84418199331,28cyclictest4497-21PK-Backend08:07:321
84418199331,27cyclictest4497-21PK-Backend09:11:011
84418199331,13cyclictest4497-21PK-Backend11:42:371
84417699336,12cyclictest13-21ksoftirqd/010:03:340
84417699331,9cyclictest4497-21PK-Backend07:31:270
844176993318,11cyclictest936746-21python311:30:300
844176993318,11cyclictest920266-21seq10:45:100
844176993318,11cyclictest899891-21cat09:45:300
844176993318,11cyclictest899891-21cat09:45:300
844176993315,14cyclictest917191-21df10:35:140
844176993315,14cyclictest888908-21df_inode09:15:160
84417699331,29cyclictest956388-21perf12:34:550
84417699331,29cyclictest4497-21PK-Backend07:40:260
84418399327,6cyclictest9348932ntp_states11:25:262
844183993219,8cyclictest2069-21gnome-shell10:30:092
844183993218,10cyclictest923-21dbus-daemon12:10:012
844183993218,10cyclictest895298-21cat09:35:012
844183993218,10cyclictest34-21ksoftirqd/210:50:202
844183993217,11cyclictest878971-21seq08:45:462
844183993216,12cyclictest879705-21sort08:50:142
844183993215,13cyclictest866378-21cat08:10:222
84418399321,28cyclictest0-21swapper/210:14:182
844181993225,4cyclictest4497-21PK-Backend11:38:561
844181993222,6cyclictest26-21ksoftirqd/111:45:291
844181993222,6cyclictest26-21ksoftirqd/111:45:281
844181993222,6cyclictest26-21ksoftirqd/108:52:271
844181993222,5cyclictest26-21ksoftirqd/112:00:151
844181993221,6cyclictest26-21ksoftirqd/111:05:141
84418199321,8cyclictest4497-21PK-Backend10:58:151
84418199321,8cyclictest4497-21PK-Backend10:58:151
844181993217,11cyclictest873264-21sed08:30:221
844181993217,11cyclictest873264-21sed08:30:221
84418199321,4cyclictest4497-21PK-Backend07:25:191
84418199321,4cyclictest0-21swapper/110:54:141
84418199321,28cyclictest0-21swapper/111:51:451
84418199321,27cyclictest4497-21PK-Backend07:50:211
844181993211,17cyclictest929185-21cstates11:10:121
84417699328,10cyclictest141rcu_preempt10:40:280
844176993225,4cyclictest4497-21PK-Backend11:57:480
844176993225,4cyclictest4497-21PK-Backend11:13:240
844176993225,4cyclictest4497-21PK-Backend09:33:200
844176993224,4cyclictest4497-21PK-Backend12:28:080
844176993219,9cyclictest883146-21cat09:00:010
844176993219,9cyclictest2069-21gnome-shell08:35:090
844176993218,10cyclictest931127-21awk11:15:200
844176993218,10cyclictest924313-21sed10:55:200
844176993218,10cyclictest924313-21sed10:55:190
844176993217,11cyclictest932798-21entropy11:20:180
844176993217,11cyclictest932798-21entropy11:20:170
844176993217,11cyclictest870575-21sshd08:24:080
844176993217,11cyclictest853743-21expr07:35:140
844176993217,11cyclictest850144-21cpuspeed_turbos07:25:150
84417699321,4cyclictest4497-21PK-Backend08:28:360
84417699321,4cyclictest4497-21PK-Backend07:21:410
84417699321,4cyclictest4497-21PK-Backend07:21:400
84417699321,4cyclictest0-21swapper/009:00:100
84417699321,28cyclictest4497-21PK-Backend12:17:300
84417699321,28cyclictest4497-21PK-Backend10:06:280
84417699321,28cyclictest4497-21PK-Backend09:52:250
84417699321,28cyclictest4497-21PK-Backend09:26:270
84417699321,28cyclictest0-21swapper/009:10:310
84417699321,27cyclictest4497-21PK-Backend09:40:120
844183993119,8cyclictest946753-21kernelversion12:00:232
844183993117,10cyclictest855435-21tr07:40:142
844183993115,12cyclictest887325-21wc09:10:202
844183993115,12cyclictest848988-21head07:20:212
844183993115,12cyclictest848988-21head07:20:212
84418399311,27cyclictest0-21swapper/208:32:402
84418399311,27cyclictest0-21swapper/208:32:392
844181993121,6cyclictest26-21ksoftirqd/107:20:531
844181993121,6cyclictest26-21ksoftirqd/107:20:531
844181993120,6cyclictest26-21ksoftirqd/111:20:271
844181993120,6cyclictest26-21ksoftirqd/111:20:271
844176993124,4cyclictest4497-21PK-Backend12:39:430
844176993121,6cyclictest886531-21sh09:10:010
844176993121,5cyclictest13-21ksoftirqd/011:39:110
844176993117,10cyclictest928719-21grep11:10:010
844176993117,10cyclictest903164-21seq09:55:290
844176993116,11cyclictest942004-21sed11:45:290
844176993116,11cyclictest942004-21sed11:45:280
844176993116,11cyclictest891166-21grep09:20:280
844176993115,12cyclictest950662-21sed12:10:290
844176993115,12cyclictest950662-21sed12:10:290
84417699311,4cyclictest4497-21PK-Backend11:27:480
84417699311,4cyclictest4497-21PK-Backend10:50:340
84417699311,4cyclictest4497-21PK-Backend09:37:240
844176993114,14cyclictest4497-21PK-Backend10:15:160
84417699311,27cyclictest4497-21PK-Backend12:00:370
84417699311,13cyclictest907998-21cpuspeed_turbos10:10:140
844176993016,10cyclictest859089-21expr07:50:150
844176993016,10cyclictest846011-21latency_hist07:15:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional