You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-12 - 10:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 12, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
307992237163,24sleep10-21swapper/119:07:431
307202236163,24sleep30-21swapper/319:06:443
290132235163,59sleep20-21swapper/219:05:062
309782204168,24sleep00-21swapper/019:09:580
319321440,4sleep33108899cyclictest20:30:183
319321440,4sleep33108899cyclictest20:30:173
310839910399,3cyclictest0-21swapper/219:11:432
282062990,1sleep20-21swapper/222:40:192
258832700,3sleep03107399cyclictest23:50:140
178462670,3sleep117849-21cpu21:05:161
63252610,2sleep232-21ksoftirqd/221:50:212
31083996158,2cyclictest0-21swapper/200:01:132
113232590,5sleep33108899cyclictest20:50:163
66102540,2sleep019800-21diskmemload23:05:180
81252520,1sleep121-21rcuc/120:40:311
25692520,4sleep230-21rcuc/219:15:172
3107399449,11cyclictest13580-21kworker/0:122:30:160
31073994410,31cyclictest10078-21munin-run23:15:010
31073994111,14cyclictest3774-21munin-run19:20:010
3107399382,10cyclictest6831-21/usr/sbin/munin20:40:230
3107399361,14cyclictest24147-21gltestperf23:45:180
3107399361,14cyclictest24147-21gltestperf23:45:180
31073993611,12cyclictest21997-21munin-run21:15:020
31073993610,3cyclictest6527-21df19:25:130
31073993610,3cyclictest6527-21df19:25:130
3107399351,4cyclictest30091-21kworker/u8:100:05:260
31073993511,3cyclictest1477-21/usr/sbin/munin22:55:240
3107399351,10cyclictest30387-21/usr/sbin/munin20:20:180
3107399351,10cyclictest26625-21/usr/sbin/munin21:25:290
3107399349,2cyclictest13964-21irqstats00:35:200
3107399345,10cyclictest1267-21taskset22:55:010
3107399343,3cyclictest32090-21munin-run00:05:010
31073993411,3cyclictest13642-21grep23:20:260
3107399336,12cyclictest838-21munin-run19:15:010
3107399332,7cyclictest13840-21munin-run22:10:010
31073993312,3cyclictest3962-21cat23:00:140
3107399331,10cyclictest14634-21snmpd21:00:540
3107399329,3cyclictest31528-21fschecks_time21:35:190
3107399329,3cyclictest31528-21fschecks_time21:35:190
3107399329,3cyclictest19800-21diskmemload22:45:170
3107399321,9cyclictest848-21sssd_nss22:03:430
3107399321,8cyclictest27893-21/usr/sbin/munin23:55:160
3107399321,5cyclictest0-21swapper/021:30:230
31073993214,15cyclictest5401-21tr21:50:120
3107399321,3cyclictest25061-21/usr/sbin/munin22:35:250
31073993211,3cyclictest29868-21seq20:16:300
31073993211,3cyclictest17189-21sort22:15:200
3107399321,10cyclictest7457-21/usr/sbin/munin21:55:130
3107399321,10cyclictest14634-21snmpd23:31:230
31073993210,2cyclictest836-21runrttasks23:44:260
3107399319,4cyclictest11999-21ntpq00:30:220
3107399319,3cyclictest836-21runrttasks21:21:030
3107399319,3cyclictest836-21runrttasks20:31:530
3107399319,3cyclictest836-21runrttasks20:31:530
3107399317,2cyclictest4025-21kworker/u8:000:25:190
31073993111,2cyclictest836-21runrttasks21:41:140
31073993110,2cyclictest836-21runrttasks23:19:170
3107399310,14cyclictest0-21swapper/000:15:010
3107399309,3cyclictest11539-21cat20:50:180
3107399306,3cyclictest0-21swapper/023:25:020
3107399302,8cyclictest6389-21chrt20:37:330
3107399302,4cyclictest14634-21snmpd20:00:420
31073993012,3cyclictest21905-21mailstats22:25:240
3107399301,21cyclictest29904-21kworker/0:020:26:520
3107399301,12cyclictest3411-21diskstats21:45:150
31078992910,7cyclictest6383-40dnf20:37:341
3107399299,3cyclictest17957-21cpuspeed_turbos21:05:170
3107399292,24cyclictest5972-21munin-run19:25:010
3107399291,5cyclictest15462-21fschecks_time19:45:190
3107399291,3cyclictest0-21swapper/022:22:330
3107399291,11cyclictest1640-21kworker/0:200:18:180
31073992910,3cyclictest86550irq/126-enp1s0-19:55:230
31073992910,3cyclictest836-21runrttasks19:41:290
3107899280,22cyclictest0-21swapper/121:43:391
3107399289,3cyclictest11630-21ntp_kernel_pll_19:35:250
3107399286,5cyclictest24429-21gltestperf20:05:180
3107399283,3cyclictest0-21swapper/022:40:320
3107399283,10cyclictest0-21swapper/023:36:380
3107399282,10cyclictest13414-21/usr/sbin/munin20:55:270
3108399271,12cyclictest0-21swapper/200:10:422
3107399279,2cyclictest7089-21fschecks_time00:20:170
3107399279,2cyclictest27420-21timerwakeupswit20:10:260
3107399271,6cyclictest0-21swapper/022:13:290
3107399271,4cyclictest10984-21latency_hist20:50:020
3108899261,4cyclictest19295-21kernelversion22:20:193
3107399269,3cyclictest9819-21timerwakeupswit19:30:250
3107399260,2cyclictest86550irq/126-enp1s0-21:16:120
31088992510,12cyclictest10261-21head22:00:153
3107899251,16cyclictest19800-21diskmemload21:50:281
3107899251,10cyclictest0-21swapper/123:44:521
3107899250,18cyclictest0-21swapper/121:00:431
3107399251,12cyclictest19168-21perf19:55:010
31088992416,2cyclictest0-21swapper/323:50:243
3108399246,4cyclictest29858-21ntp_kernel_pll_21:30:222
3107899247,9cyclictest4894-21fschecks_time00:15:151
31078992418,3cyclictest20049-21users22:20:271
3107899241,16cyclictest19800-21diskmemload23:01:531
31078992410,3cyclictest172182expr23:30:141
31088992315,2cyclictest0-21swapper/322:45:173
3108899231,10cyclictest0-21swapper/322:58:193
3108899230,9cyclictest0-21swapper/321:50:213
3107899235,4cyclictest0-21swapper/100:21:111
3107899231,12cyclictest0-21swapper/123:45:251
3107899231,12cyclictest0-21swapper/123:45:251
3107899230,5cyclictest0-21swapper/122:30:131
3107899230,16cyclictest0-21swapper/119:20:161
184592231,3sleep218460-21/usr/sbin/munin22:20:132
3108899228,6cyclictest23299-1kworker/u9:222:53:003
3108899225,8cyclictest21559-21latency_hist20:00:023
3108899222,16cyclictest24163-21sed23:45:183
3108899222,16cyclictest24163-21sed23:45:173
3108899221,9cyclictest0-21swapper/323:40:163
3108899221,6cyclictest0-21swapper/322:32:203
3108899221,10cyclictest3155-21needreboot00:10:203
3108899220,10cyclictest25347-21unixbench_singl20:05:273
3108399225,7cyclictest0-21swapper/222:05:232
3108399223,7cyclictest0-21swapper/222:31:402
3108399222,7cyclictest4601-21grep19:20:162
3107899229,3cyclictest888-21munin-node19:15:001
3107899222,4cyclictest24590-21cat21:20:131
3107899221,8cyclictest0-21swapper/122:02:471
3107899221,5cyclictest23936-21kworker/u8:021:18:001
31078992214,3cyclictest0-21swapper/119:15:011
3107899221,19cyclictest0-21swapper/122:10:141
3107899221,16cyclictest0-21swapper/119:45:491
3107899221,16cyclictest0-21swapper/100:39:411
3107899221,15cyclictest0-21swapper/122:40:131
3107899221,15cyclictest0-21swapper/121:25:101
3107899221,15cyclictest0-21swapper/119:40:341
3107899220,15cyclictest0-21swapper/123:15:191
3108899218,7cyclictest0-21swapper/319:42:153
3108899218,4cyclictest28639-21timerandwakeup22:40:253
3108899218,1cyclictest0-21swapper/322:05:273
3108899217,8cyclictest0-21swapper/321:30:173
3108899214,4cyclictest20932-21unixbench_singl19:55:263
31088992116,3cyclictest31946-21hddtemp_smartct19:10:183
3108899211,18cyclictest0-21swapper/320:25:143
3108899211,14cyclictest0-21swapper/319:25:383
3108899211,14cyclictest0-21swapper/319:25:383
3108899210,9cyclictest0-21swapper/321:00:213
3108399213,2cyclictest86850irq/129-enp1s0-20:05:422
3108399212,6cyclictest0-21swapper/223:20:142
3108399211,9cyclictest0-21swapper/222:57:552
3108399211,7cyclictest17065-21wc23:30:132
3108399211,7cyclictest0-21swapper/223:59:212
3108399211,7cyclictest0-21swapper/222:28:282
3108399211,7cyclictest0-21swapper/221:23:032
3108399211,7cyclictest0-21swapper/221:16:132
3108399211,4cyclictest8388-21unixbench_singl20:40:332
3108399211,12cyclictest0-21swapper/223:17:242
3107899219,3cyclictest25300-21grep20:05:281
3107899219,3cyclictest11885-21egrep22:05:161
3107899212,16cyclictest10731-21grep19:35:171
3107899212,11cyclictest7108-21memory19:25:191
3107899212,11cyclictest7108-21memory19:25:191
3107899212,10cyclictest21929-21diskstats20:00:181
3107899211,8cyclictest0-21swapper/100:05:511
31078992117,3cyclictest0-21swapper/100:26:281
3107899211,6cyclictest8584-21kworker/u8:222:45:521
3107899211,4cyclictest7665-21kworker/u8:221:12:021
3107899211,4cyclictest17946-21kworker/u8:122:56:561
31078992114,2cyclictest0-21swapper/121:58:051
31078992113,3cyclictest0-21swapper/122:52:101
3107899211,17cyclictest26133-21ntp_states22:35:251
3107899211,15cyclictest0-21swapper/123:51:321
3107899211,15cyclictest0-21swapper/123:10:361
3107899211,15cyclictest0-21swapper/123:06:241
3107899211,15cyclictest0-21swapper/122:19:491
3107899211,15cyclictest0-21swapper/121:47:401
3107899211,15cyclictest0-21swapper/120:48:281
3107899211,15cyclictest0-21swapper/119:51:311
3107899211,15cyclictest0-21swapper/100:30:241
3107899211,14cyclictest0-21swapper/120:25:221
3107899211,12cyclictest0-21swapper/123:27:081
3107899211,10cyclictest0-21swapper/122:28:121
3107899210,18cyclictest19135-21cpu23:35:131
3107899210,16cyclictest0-21swapper/100:04:001
3107899210,15cyclictest0-21swapper/120:34:061
3107899210,15cyclictest0-21swapper/120:34:061
269752211,17sleep327202-21perl21:25:173
103642211,16sleep310633-21df23:15:133
3108899206,3cyclictest86850irq/129-enp1s0-00:25:273
3108899201,9cyclictest4964-21grep00:15:163
31088992016,3cyclictest0-21swapper/322:13:243
31088992016,3cyclictest0-21swapper/319:37:153
3108899201,11cyclictest6561-21/usr/sbin/munin00:20:203
3108899201,10cyclictest0-21swapper/321:47:353
3108399204,3cyclictest86850irq/129-enp1s0-21:05:212
3108399202,7cyclictest0-21swapper/223:35:392
3108399202,6cyclictest0-21swapper/223:06:212
3108399202,6cyclictest0-21swapper/200:05:292
3108399202,5cyclictest0-21swapper/222:35:022
3108399201,7cyclictest0-21swapper/223:41:362
3108399201,7cyclictest0-21swapper/223:10:572
3108399201,7cyclictest0-21swapper/222:11:352
3108399201,7cyclictest0-21swapper/222:03:402
3108399201,7cyclictest0-21swapper/221:46:552
3108399201,7cyclictest0-21swapper/219:39:462
3108399201,7cyclictest0-21swapper/200:37:592
3108399201,7cyclictest0-21swapper/200:30:372
3108399201,7cyclictest0-21swapper/200:29:332
3108399201,6cyclictest0-21swapper/222:46:232
3108399201,6cyclictest0-21swapper/221:41:092
3108399201,6cyclictest0-21swapper/221:38:532
3108399201,6cyclictest0-21swapper/221:38:532
3108399201,13cyclictest0-21swapper/220:20:152
3108399200,6cyclictest4759-21ntp_offset23:00:222
3107899209,7cyclictest0-21swapper/121:35:251
3107899209,7cyclictest0-21swapper/121:35:251
3107899209,3cyclictest19735-21entropy19:55:161
3107899208,3cyclictest12522-21grep23:20:151
3107899202,14cyclictest28838-21sed20:15:161
31078992016,3cyclictest0-21swapper/119:30:211
31078992013,3cyclictest0-21swapper/120:57:151
3107899201,15cyclictest0-21swapper/100:13:291
3107899201,10cyclictest29121-21/usr/sbin/munin23:55:231
3107899201,10cyclictest12680-21cat20:50:301
3107899200,15cyclictest0-21swapper/120:21:031
3108899197,7cyclictest0-21swapper/322:37:393
3108899197,3cyclictest0-21swapper/321:10:193
3108899193,13cyclictest28186-21/usr/sbin/munin20:15:153
3108899192,3cyclictest32413-21kworker/3:121:35:223
3108899192,3cyclictest32413-21kworker/3:121:35:223
3108899191,9cyclictest0-21swapper/320:35:173
3108899191,6cyclictest18197-21switchtime23:30:263
3108899191,6cyclictest0-21swapper/323:14:393
3108899191,6cyclictest0-21swapper/320:49:083
3108899191,6cyclictest0-21swapper/300:36:143
3108899191,15cyclictest28533-21gltestperf23:55:153
3108899191,15cyclictest20235-21grep23:35:233
3108899191,15cyclictest1984-21/usr/sbin/munin19:15:143
3108899191,10cyclictest0-21swapper/323:05:433
3108899191,10cyclictest0-21swapper/321:42:243
3108899190,10cyclictest0-21swapper/300:03:383
3108399195,3cyclictest86750irq/128-enp1s0-23:50:242
3108399192,3cyclictest25777-21kworker/2:120:57:222
3108399191,7cyclictest0-21swapper/222:50:122
3108399191,6cyclictest0-21swapper/223:48:402
3108399191,6cyclictest0-21swapper/223:48:402
3108399191,6cyclictest0-21swapper/222:16:322
3108399191,6cyclictest0-21swapper/221:57:142
3108399191,6cyclictest0-21swapper/221:27:002
3108399191,6cyclictest0-21swapper/220:47:302
3108399191,6cyclictest0-21swapper/219:40:212
3108399191,6cyclictest0-21swapper/200:22:152
3108399191,5cyclictest14895-21expr23:25:152
3108399190,7cyclictest0-21swapper/200:15:522
3108399190,6cyclictest0-21swapper/219:55:272
3107899195,3cyclictest26077-21cpuspeed_turbos20:10:131
3107899191,10cyclictest29789-21missed_timers21:30:201
3108899182,3cyclictest86850irq/129-enp1s0-23:25:263
3108899181,9cyclictest0-21swapper/321:55:193
3108899181,7cyclictest0-21swapper/322:17:173
3108899181,6cyclictest0-21swapper/322:28:373
3108899181,6cyclictest0-21swapper/300:30:423
3108899181,5cyclictest0-21swapper/321:17:243
3108899181,5cyclictest0-21swapper/319:30:423
3108899181,3cyclictest836-21runrttasks23:23:273
31088991812,3cyclictest16127-21pmu-power19:45:253
31088991812,3cyclictest1033-21open_inodes00:05:243
3108899181,15cyclictest0-21swapper/323:01:303
3108399183,3cyclictest86750irq/128-enp1s0-20:15:122
3108399181,6cyclictest0-21swapper/221:11:352
3108399181,6cyclictest0-21swapper/221:02:322
3108399181,5cyclictest6194-21/usr/sbin/munin19:25:232
3108399181,5cyclictest6194-21/usr/sbin/munin19:25:232
31083991813,3cyclictest310-21ls20:25:132
31083991812,3cyclictest3530-21open_inodes20:30:232
31083991812,3cyclictest3530-21open_inodes20:30:232
3108899171,6cyclictest0-21swapper/321:05:203
3108899171,5cyclictest25413-21strings21:20:203
31088991713,2cyclictest14154-21if_enp1s020:55:173
31088991712,3cyclictest31855-21grep20:20:263
31088991711,3cyclictest7598-21if_enp2s020:40:233
31088991711,3cyclictest17538-21entropy19:50:133
3108399171,6cyclictest0-21swapper/219:51:292
3108399171,6cyclictest0-21swapper/219:32:152
3108399171,4cyclictest6383-40dnf20:37:312
3108899164,3cyclictest0-21swapper/319:22:243
31088991612,3cyclictest0-21swapper/320:11:423
3108399161,3cyclictest22253-21hddtemp_smartct20:00:202
31083991611,3cyclictest27084-21open_inodes20:10:232
31083991610,4cyclictest11570-21diskstats20:50:192
31083991610,3cyclictest15151-21grep19:45:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional