You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 16:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 21, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
162212237163,24sleep20-21swapper/207:06:132
162302203167,24sleep10-21swapper/107:06:201
163782202166,24sleep30-21swapper/307:08:153
162192201164,25sleep00-21swapper/007:06:110
352921530,3sleep31661099cyclictest11:30:253
352921530,3sleep31661099cyclictest11:30:253
166059910299,2cyclictest0-21swapper/208:53:322
16605999791,3cyclictest25156-21ntp_states07:25:232
16605998178,2cyclictest0-21swapper/210:34:102
103382620,1sleep00-21swapper/009:20:150
16262480,5sleep31628-21fschecks_time09:00:163
100852470,1sleep30-21swapper/308:05:183
1661099393,5cyclictest24261-40dnf08:37:313
1659299378,13cyclictest865-21munin-run09:00:010
1661099361,11cyclictest26899-21/usr/sbin/munin08:45:183
1661099361,10cyclictest30948-21/usr/sbin/munin12:35:273
1661099352,22cyclictest0-21swapper/311:38:573
16610993511,3cyclictest31914-21/usr/sbin/munin10:10:173
1661099344,7cyclictest29773-21kworker/u8:112:06:573
16610993411,3cyclictest16515-21/usr/sbin/munin09:35:243
1661099339,3cyclictest15752-21uname10:45:183
1661099331,12cyclictest0-21swapper/312:02:473
1661099331,10cyclictest17137-21/usr/sbin/munin10:50:143
1661099331,10cyclictest0-21swapper/311:05:013
1661099329,3cyclictest8345-21/usr/sbin/munin10:30:283
16610993210,3cyclictest20918-21/usr/sbin/munin09:45:183
1659299326,12cyclictest9106-21perf08:05:010
1661099319,3cyclictest835-21runrttasks09:15:533
1661099319,3cyclictest26551-21/usr/sbin/munin12:25:203
1661099316,5cyclictest5313-21munin-run09:10:013
1661099316,5cyclictest5313-21munin-run09:10:013
1661099314,3cyclictest86950irq/126-enp1s0-11:15:183
1661099313,7cyclictest27244-21kworker/u8:210:00:013
1661099313,5cyclictest10945-21awk11:50:013
1661099311,6cyclictest87050irq/127-enp1s0-07:45:353
16610993111,3cyclictest11473-21expr11:50:133
16610993110,3cyclictest25118-21fschecks_count08:40:153
16610993110,3cyclictest21665-21pluginstate07:20:013
16600993115,13cyclictest26565-21systemd-journal09:55:261
16592993110,11cyclictest3439-21basename07:50:170
1661099309,2cyclictest835-21runrttasks07:41:503
1661099309,2cyclictest3950-21/usr/sbin/munin10:20:233
1661099308,3cyclictest14264-21load11:55:203
1661099307,4cyclictest12592-21fschecks_time09:25:153
1661099306,5cyclictest0-21swapper/308:10:143
1661099301,9cyclictest13691-21sort10:40:193
1661099301,9cyclictest0-21swapper/309:20:123
16610993014,13cyclictest19643-21grep09:40:193
1661099301,12cyclictest13711-21/usr/sbin/munin08:15:173
16610993010,3cyclictest32528-21/usr/sbin/munin11:25:153
16610993010,3cyclictest10810-21expr10:35:123
16610993010,2cyclictest23124-21expr07:20:223
1661099299,3cyclictest30348-21sendmail_mailtr08:50:243
1661099299,3cyclictest21017-21cat08:30:173
1661099299,3cyclictest18653-21cut08:25:133
1661099298,3cyclictest30866-21pmu-power10:05:253
1661099297,10cyclictest87250irq/129-enp1s0-08:04:393
1661099296,12cyclictest0-21swapper/311:06:463
1661099291,5cyclictest4915-21/usr/sbin/munin07:55:193
1661099291,5cyclictest4915-21/usr/sbin/munin07:55:193
1661099291,3cyclictest86950irq/126-enp1s0-09:10:163
1661099291,19cyclictest0-21swapper/307:37:533
16610992910,3cyclictest26522-21expr07:30:143
16610992910,2cyclictest835-21runrttasks10:02:483
1661099290,7cyclictest0-21swapper/310:27:293
1661099289,3cyclictest31455-21cstates08:55:133
1661099288,3cyclictest16401-21tune2fs08:20:163
1661099282,12cyclictest19331-21/usr/sbin/munin10:55:193
1661099281,9cyclictest30999-21hddtemp_smartct11:20:163
1661099281,12cyclictest25932-21/usr/sbin/munin11:10:173
1661099281,10cyclictest0-21swapper/312:30:133
16610992810,3cyclictest20194-21expr12:10:123
1660099288,11cyclictest0-21swapper/110:25:141
99312270,4sleep11660099cyclictest09:20:121
1661099279,3cyclictest3015-21expr07:50:133
1661099274,6cyclictest0-21swapper/309:50:133
1661099273,6cyclictest25414-21ntp_states12:20:213
16610992711,3cyclictest7092-21expr11:40:133
16610992711,3cyclictest14319-21/usr/sbin/munin09:30:173
1661099271,12cyclictest0-21swapper/312:19:573
16592992710,15cyclictest31701-21awk10:10:010
1661099262,12cyclictest17152-21diskstats07:10:153
1661099261,11cyclictest25130-21ntp_kernel_pll_07:25:213
1661099250,4cyclictest0-21swapper/310:18:483
1660099250,8cyclictest0-21swapper/110:38:301
1659299257,16cyclictest28868-21munin-run08:50:000
1660099244,9cyclictest0-21swapper/110:30:131
1660099241,14cyclictest87150irq/128-enp1s0-11:30:241
1660099241,14cyclictest87150irq/128-enp1s0-11:30:241
1659299242,6cyclictest26378-21ntp_states09:55:240
320002231,3sleep232101-21irqstats08:55:182
1660599231,6cyclictest0-21swapper/211:12:522
1660099237,9cyclictest23-21ksoftirqd/109:35:221
1660099231,4cyclictest2144-21munin-run11:30:011
1660099231,2cyclictest22278-21perf08:35:011
1660099231,15cyclictest0-21swapper/107:57:291
1660099231,15cyclictest0-21swapper/107:57:291
16600992310,10cyclictest28465-21systemd-journal07:35:021
142702231,3sleep214380-21nfs4_client11:55:212
1660099221,13cyclictest0-21swapper/111:35:251
1660099221,11cyclictest577-21systemd-journal10:55:011
1660099221,10cyclictest0-21swapper/109:44:171
1659299222,11cyclictest3342-21/usr/sbin/munin09:05:180
1659299222,11cyclictest3342-21/usr/sbin/munin09:05:180
1659299221,11cyclictest0-21swapper/010:50:010
1660599211,8cyclictest19635-21memory09:40:192
1660599211,11cyclictest0-21swapper/211:43:162
1660099219,11cyclictest29773-21kworker/u8:111:54:351
1660099217,12cyclictest23-21ksoftirqd/107:44:591
1660099211,18cyclictest26279-21uptime08:40:261
1660099211,17cyclictest7785-21netstat11:40:201
1660099211,11cyclictest30530-21irqstats10:05:191
1660099211,11cyclictest28526-21perf12:30:011
1660099211,11cyclictest17875-21latency_hist08:25:021
1660099211,10cyclictest0-21swapper/110:59:011
1660099210,5cyclictest0-21swapper/112:15:201
1659299212,7cyclictest0-21swapper/011:22:360
1659299212,11cyclictest20304-21/usr/sbin/munin08:30:140
1659299211,17cyclictest6346-21irqstats09:10:170
1659299211,12cyclictest0-21swapper/010:30:570
16605992013,4cyclictest4715-21cat09:05:272
16605992013,4cyclictest4715-21cat09:05:272
1660099204,14cyclictest31828-21wc12:35:191
16600992015,3cyclictest20797-21ls07:15:201
1660099201,3cyclictest28862-21latency_hist08:50:011
1660099201,11cyclictest508-21cpuspeed_turbos07:45:141
1660099201,11cyclictest31289-21/usr/sbin/munin08:55:191
1660099201,11cyclictest18776-21/usr/sbin/munin07:15:011
1660099201,11cyclictest0-21swapper/111:00:541
1660099201,10cyclictest521-21ntp_states10:10:221
1660099201,10cyclictest24469-21latency_hist08:40:021
1660099201,10cyclictest20071-21latency_hist08:30:021
1660099201,10cyclictest0-21swapper/111:46:361
1660099200,9cyclictest0-21swapper/111:55:141
1660099200,5cyclictest0-21swapper/112:24:271
1660099200,3cyclictest0-21swapper/110:43:571
1660099200,18cyclictest0-21swapper/111:06:591
1660099200,13cyclictest0-21swapper/111:15:271
1659299202,15cyclictest14719-21idleruntime-cro10:45:010
1659299202,10cyclictest19959-21/usr/sbin/munin12:10:240
1659299201,13cyclictest0-21swapper/009:00:120
1659299201,12cyclictest0-21swapper/011:50:060
1659299201,11cyclictest28748-21/usr/sbin/munin12:30:260
1659299201,11cyclictest1737-21/usr/sbin/munin10:15:260
1659299201,11cyclictest14319-21/usr/sbin/munin09:30:200
1659299201,11cyclictest0-21swapper/012:05:270
1660599195,5cyclictest16154-21cstates08:20:142
1660599193,6cyclictest32-21ksoftirqd/209:10:172
1660599192,6cyclictest0-21swapper/210:20:172
1660599192,5cyclictest14354-21sshd10:40:372
1660599192,4cyclictest0-21swapper/209:50:242
1660599191,8cyclictest0-21swapper/212:13:522
1660599191,8cyclictest0-21swapper/209:21:022
1660599191,7cyclictest0-21swapper/212:38:482
1660599191,7cyclictest0-21swapper/210:14:472
1660599191,16cyclictest0-21swapper/211:00:192
1660099198,3cyclictest0-21swapper/112:30:211
1660099197,5cyclictest0-21swapper/109:31:051
1660099191,6cyclictest0-21swapper/109:17:581
16600991916,2cyclictest0-21swapper/108:13:551
1660099191,1cyclictest0-21swapper/110:00:241
1660099191,17cyclictest0-21swapper/109:02:241
1660099191,11cyclictest0-21swapper/111:12:021
1660099191,10cyclictest5873-21df09:10:131
1660099191,10cyclictest5303-21latency_hist09:10:011
1660099191,10cyclictest5303-21latency_hist09:10:011
1660099191,10cyclictest26860-21perl07:30:171
1660099191,10cyclictest10033-21basename08:05:171
1660099191,10cyclictest0-21swapper/112:02:311
1660099191,10cyclictest0-21swapper/109:45:121
1660099190,3cyclictest0-21swapper/109:50:231
1659299199,3cyclictest23310-21irqstats08:35:170
1659299198,10cyclictest0-21swapper/008:55:010
1659299197,3cyclictest0-21swapper/011:10:530
1659299192,4cyclictest9705-21munin-run09:20:010
1659299191,5cyclictest0-21swapper/010:00:250
16592991913,4cyclictest24808-21grep08:40:120
1659299191,12cyclictest0-21swapper/011:57:120
1659299191,10cyclictest3950-21/usr/sbin/munin10:20:260
200382181,2sleep2904-21sssd_nss07:15:132
1660599188,3cyclictest87250irq/129-enp1s0-11:30:262
1660599188,3cyclictest87250irq/129-enp1s0-11:30:262
1660599185,3cyclictest0-21swapper/209:33:062
1660599184,9cyclictest6438-40dnf11:38:572
1660599181,8cyclictest0-21swapper/212:00:022
1660599181,8cyclictest0-21swapper/209:27:562
1660599181,7cyclictest24550-21irqstats11:05:182
1660599181,6cyclictest0-21swapper/211:28:092
1660599181,5cyclictest0-21swapper/209:57:032
1660599181,4cyclictest0-21swapper/211:51:112
16605991811,4cyclictest18318-40dnf09:37:552
1660099189,2cyclictest2746-21strings10:15:221
1660099186,3cyclictest0-21swapper/111:22:211
1660099181,3cyclictest0-21swapper/108:00:281
1660099181,3cyclictest0-21swapper/107:25:311
16600991812,3cyclictest15297-21users08:15:271
1660099181,16cyclictest0-21swapper/112:05:391
1660099181,16cyclictest0-21swapper/110:47:111
1660099181,16cyclictest0-21swapper/109:25:211
1660099181,16cyclictest0-21swapper/108:50:331
1660099181,16cyclictest0-21swapper/107:52:411
1660099181,16cyclictest0-21swapper/107:22:211
1660099181,10cyclictest0-21swapper/110:21:241
1659299189,2cyclictest28530-21awk12:30:010
1659299185,3cyclictest0-21swapper/007:41:270
1659299183,9cyclictest0-21swapper/008:15:350
1659299182,6cyclictest20111-21cat10:55:160
1659299181,4cyclictest0-21swapper/010:50:190
16592991812,3cyclictest13491-21idleruntime-cro08:15:010
1659299181,11cyclictest0-21swapper/011:42:020
1659299180,12cyclictest0-21swapper/008:20:230
215342170,4sleep230-21rcuc/209:45:172
1660599176,4cyclictest1138-21kworker/2:011:20:262
1660599172,6cyclictest8961-21/usr/sbin/munin11:45:152
1660599172,6cyclictest0-21swapper/212:20:142
1660599172,5cyclictest9095-21munin-run08:05:012
1660599171,8cyclictest801-21snmpd08:16:042
1660599171,7cyclictest0-21swapper/212:26:502
1660599171,7cyclictest0-21swapper/210:08:522
1660599171,6cyclictest3722-21idleruntime-cro10:20:012
1660599171,6cyclictest3162-21tune2fs07:50:152
1660599171,6cyclictest23396-21sed07:20:252
1660599171,6cyclictest20399-21ntp_states10:55:212
16605991713,3cyclictest0-21swapper/210:35:372
16605991713,2cyclictest5306-21diskmemload11:17:202
16605991712,3cyclictest30067-21users07:35:252
16605991712,3cyclictest24701-21/usr/sbin/munin08:40:152
1660599171,15cyclictest0-21swapper/208:48:152
1660599171,14cyclictest0-21swapper/212:30:052
1660599170,6cyclictest19251-21pmu-power08:25:202
16600991712,3cyclictest21294-21threads12:10:241
1659299176,3cyclictest32528-21/usr/sbin/munin11:25:190
1659299175,3cyclictest0-21swapper/011:09:210
1659299174,7cyclictest0-21swapper/007:35:130
1659299174,3cyclictest0-21swapper/007:15:370
1659299171,7cyclictest0-21swapper/012:01:020
1659299171,7cyclictest0-21swapper/010:10:160
1659299171,7cyclictest0-21swapper/009:53:360
1659299171,5cyclictest0-21swapper/011:35:410
1659299171,5cyclictest0-21swapper/011:16:520
1659299171,3cyclictest800-21sshd12:37:290
16592991712,3cyclictest8209-21cut10:30:010
16592991712,3cyclictest10453-21pmu-power08:05:230
1659299171,15cyclictest0-21swapper/009:27:590
1659299171,14cyclictest0-21swapper/008:25:080
1659299171,10cyclictest0-21swapper/012:23:030
1660599161,6cyclictest1444-21df09:00:142
1660599161,6cyclictest0-21swapper/212:18:532
1660599161,5cyclictest0-21swapper/210:45:162
1660599161,4cyclictest986-1kworker/u9:108:35:412
16605991612,2cyclictest5306-21diskmemload10:00:262
16605991611,3cyclictest7185-21ntp_kernel_pll_10:25:212
16605991610,3cyclictest9074-21/usr/sbin/munin09:15:252
16605991610,3cyclictest412-21/usr/sbin/munin07:45:182
1660599160,15cyclictest0-21swapper/208:10:592
1659299161,9cyclictest0-21swapper/011:45:180
1659299161,6cyclictest22156-21/usr/sbin/munin12:15:130
1659299161,6cyclictest0-21swapper/009:35:580
1659299161,6cyclictest0-21swapper/007:10:150
1659299161,4cyclictest0-21swapper/009:40:540
1659299161,3cyclictest800-21sshd07:31:290
1659299161,14cyclictest0-21swapper/007:29:320
1659299161,14cyclictest0-21swapper/007:22:440
1659299161,13cyclictest22176-21grep11:00:160
16592991611,3cyclictest1858-21threads07:45:260
1659299161,12cyclictest2368-21/usr/sbin/munin11:30:200
1659299161,12cyclictest2368-21/usr/sbin/munin11:30:200
1659299160,6cyclictest0-21swapper/007:58:170
1659299160,6cyclictest0-21swapper/007:58:170
1660599156,7cyclictest18632-21unixbench_multi10:50:272
1660599151,9cyclictest0-21swapper/207:55:172
1660599151,9cyclictest0-21swapper/207:55:172
1660599151,6cyclictest18188-21fschecks_count12:05:162
1660599151,5cyclictest20531-21expr08:30:132
16605991511,3cyclictest0-21swapper/207:30:102
16605991510,3cyclictest30696-21cpu07:40:122
16605991510,3cyclictest16805-21/usr/sbin/munin07:10:262
1660599150,3cyclictest0-21swapper/208:05:122
1659299151,5cyclictest10540-21/usr/sbin/munin10:35:190
16592991510,3cyclictest21449-21sort09:45:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional