You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 08:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Oct 20, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29882239162,24sleep30-21swapper/319:08:283
29872238162,24sleep20-21swapper/219:08:282
28892211135,24sleep10-21swapper/119:07:111
30282202165,24sleep00-21swapper/019:08:590
3213997061,8cyclictest0-21swapper/222:17:162
114682660,3sleep011472-21cpuspeed_turbos20:40:140
312972540,3sleep1320699cyclictest23:50:261
266592500,3sleep011-21rcuc/020:00:160
321899443,25cyclictest0-21swapper/321:45:143
321899441,33cyclictest0-21swapper/322:02:103
3213994215,9cyclictest1342-21awk22:45:012
3201994111,14cyclictest30892-21munin-run21:25:010
321399402,19cyclictest0-21swapper/220:16:012
3213993919,9cyclictest5518-21/usr/sbin/munin00:05:242
3213993915,12cyclictest16488-21expr22:05:152
3213993914,15cyclictest23384-21sh21:05:212
296112390,9sleep232-21ksoftirqd/222:35:122
320199381,12cyclictest15806-21kworker/0:222:05:160
3213993715,9cyclictest8070-21tune2fs21:45:142
3213993715,11cyclictest23903-21expr23:35:132
3213993715,11cyclictest23903-21expr23:35:122
3213993615,8cyclictest10624-21cat19:25:012
3213993613,7cyclictest13928-21idleruntime-cro22:00:022
321399361,23cyclictest0-21swapper/221:04:192
3213993611,16cyclictest26076-21/usr/sbin/munin20:00:142
3213993516,8cyclictest2515-21/usr/sbin/munin20:20:272
3213993516,8cyclictest2515-21/usr/sbin/munin20:20:272
3213993514,12cyclictest22849-21/usr/sbin/munin23:30:262
3213993415,7cyclictest25153-21/usr/sbin/munin22:25:142
3213993415,6cyclictest27573-21sshd23:42:462
320199349,12cyclictest640-21munin-run21:30:010
320199349,12cyclictest640-21munin-run21:30:010
320199346,13cyclictest7960-21munin-run23:00:020
320199341,25cyclictest25509-21kworker/0:123:46:000
320199340,26cyclictest24582-21kworker/0:222:27:400
321399338,5cyclictest0-21swapper/223:45:212
321399338,15cyclictest0-21swapper/222:10:272
321399338,15cyclictest0-21swapper/221:15:202
321399338,14cyclictest12888-21/usr/sbin/munin19:30:232
3213993314,5cyclictest22951-21/usr/sbin/munin22:20:282
3213993311,18cyclictest7822-21tail19:15:262
3213993311,18cyclictest7822-21tail19:15:252
320199331,25cyclictest5628-21kworker/0:023:06:180
320199331,25cyclictest13438-21kworker/0:000:28:400
320199331,13cyclictest16351-21kworker/0:322:11:480
320199330,26cyclictest0-21swapper/022:36:300
320199330,26cyclictest0-21swapper/022:18:380
320199330,25cyclictest25509-21kworker/0:123:41:160
320199330,15cyclictest7935-21kworker/0:123:01:040
320199330,15cyclictest5628-21kworker/0:023:22:360
321399327,12cyclictest86850irq/129-enp1s0-23:10:272
3213993212,7cyclictest12151-21taskset23:07:282
3213993211,11cyclictest20888-40dnf23:26:452
3213993210,12cyclictest2310-21cat21:30:252
320199325,13cyclictest5776-21perf22:55:020
320199321,8cyclictest8547-21kworker/0:300:23:380
320199321,14cyclictest4050-21kworker/0:000:10:100
320199321,13cyclictest4768-21kworker/0:021:50:180
320199321,13cyclictest11633-21kworker/0:122:01:080
320199320,18cyclictest27336-21kworker/0:022:34:200
320199320,18cyclictest27336-21kworker/0:022:34:200
320199320,15cyclictest5628-21kworker/0:023:29:120
320199320,15cyclictest24072-21kworker/0:021:17:440
3218993110,9cyclictest1331-21munin-plugin-st22:45:013
321399311,18cyclictest0-21swapper/223:20:582
3213993110,11cyclictest761-21cat21:30:012
3213993110,11cyclictest761-21cat21:30:012
320199311,12cyclictest0-21swapper/022:23:100
320199310,13cyclictest5628-21kworker/0:023:15:220
320199310,12cyclictest1046-21kworker/0:022:45:380
320199310,11cyclictest27739-21kworker/0:023:52:520
321399308,8cyclictest24288-21diskmemload00:26:042
321399308,5cyclictest0-21swapper/219:50:392
3213993017,10cyclictest16602-21/usr/sbin/munin00:30:242
3213993015,12cyclictest5409-21systemd19:15:012
3213993013,13cyclictest31038-21grep20:10:172
320199307,11cyclictest19267-21perf19:45:020
320199300,14cyclictest21043-21kworker/0:223:33:360
320199300,14cyclictest11633-21kworker/0:121:59:140
321899298,14cyclictest86550irq/126-enp1s0-19:45:273
321899297,5cyclictest18335-21perf22:10:023
321399299,8cyclictest7829-21runrttasks22:58:322
3213992920,4cyclictest26011-21/usr/sbin/munin21:10:272
3213992919,6cyclictest4862-21/usr/sbin/munin22:50:222
321399291,11cyclictest11860-21irqstats00:20:192
321399291,11cyclictest0-21swapper/220:55:232
320199291,23cyclictest0-21swapper/000:09:260
320199291,14cyclictest13438-21kworker/0:000:39:200
3201992911,3cyclictest10613-21latency_hist19:25:020
320199291,12cyclictest19091-21kworker/0:120:09:460
320199291,11cyclictest0-21swapper/021:30:460
320199291,11cyclictest0-21swapper/019:16:460
320199291,11cyclictest0-21swapper/019:16:450
320199290,13cyclictest21802-21kworker/0:221:05:220
320199290,13cyclictest19091-21kworker/0:119:59:060
320199290,10cyclictest21043-21kworker/0:223:38:580
320199290,10cyclictest21043-21kworker/0:223:38:580
321399289,11cyclictest24288-21diskmemload22:46:062
321399288,7cyclictest0-21swapper/223:00:182
321399288,7cyclictest0-21swapper/222:30:042
321399288,7cyclictest0-21swapper/222:30:042
3213992820,5cyclictest6421-21ntp_states21:40:242
3213992820,5cyclictest5981-21smart_sda20:25:242
3213992820,5cyclictest30873-21/usr/sbin/munin23:50:212
3213992820,5cyclictest19025-21sshd19:42:142
3213992820,5cyclictest16057-21df_inode20:50:132
3213992820,5cyclictest10456-40dnf19:25:012
3213992819,6cyclictest3694-21/usr/sbin/munin00:00:262
3213992819,6cyclictest28279-21/usr/sbin/munin20:05:162
3213992819,6cyclictest24504-21cut19:55:152
3213992819,6cyclictest15730-21/usr/sbin/munin22:00:282
3213992819,6cyclictest15258-21cat19:35:132
3213992819,6cyclictest14796-21/usr/sbin/munin23:15:182
3213992819,6cyclictest11502-21/usr/sbin/munin20:40:132
3213992818,7cyclictest836-21runrttasks00:15:162
3213992818,7cyclictest17643-21/usr/sbin/munin00:35:142
320199281,3cyclictest16879-21kworker/0:219:50:280
320199281,14cyclictest30845-21kworker/0:221:35:280
320199280,17cyclictest14269-21kworker/0:123:11:320
320199280,13cyclictest13438-21kworker/0:000:34:100
320199280,10cyclictest4768-21kworker/0:021:41:380
321899274,11cyclictest12748-1kworker/u9:123:54:133
3213992720,4cyclictest9741-21/usr/sbin/munin20:35:152
3213992720,4cyclictest3495-21df21:35:142
3213992720,4cyclictest19647-21/usr/sbin/munin19:45:142
3213992720,4cyclictest14551-21/usr/sbin/munin20:45:202
3213992719,5cyclictest7449-21/usr/sbin/munin00:10:192
3213992719,5cyclictest32729-21diskstats23:55:182
3213992719,5cyclictest29084-21/usr/sbin/munin21:20:122
3213992718,6cyclictest9751-21/usr/sbin/munin21:50:202
3213992718,6cyclictest7960-21ntp_kernel_pll_20:30:192
320699272,17cyclictest13095-21sshd20:42:421
320199271,12cyclictest0-21swapper/021:45:420
320199271,10cyclictest7555-21kworker/0:219:25:480
320199270,13cyclictest24072-21kworker/0:021:11:560
320199270,12cyclictest8547-21kworker/0:300:19:080
321899267,16cyclictest86850irq/129-enp1s0-22:51:323
321899261,9cyclictest0-21swapper/320:05:593
321899261,14cyclictest0-21swapper/323:05:023
321899261,11cyclictest0-21swapper/321:04:023
321899261,11cyclictest0-21swapper/319:29:293
320699265,18cyclictest21-21rcuc/100:25:261
320699261,17cyclictest8357-21cstates23:00:131
3206992611,3cyclictest106202latency_hist19:25:021
3201992611,3cyclictest4441-21kworker/0:120:30:200
320199260,9cyclictest4441-21kworker/0:120:47:400
321899257,4cyclictest0-21swapper/323:38:123
321899257,4cyclictest0-21swapper/323:38:123
3206992515,2cyclictest3591-21ntp_states20:20:221
3206992515,2cyclictest3591-21ntp_states20:20:221
320199251,13cyclictest19539-21kworker/0:021:00:240
320199250,9cyclictest1650-21kworker/0:123:59:460
320199250,2cyclictest1650-21kworker/0:100:00:100
320199250,24cyclictest2026-21kworker/0:020:21:200
320199250,24cyclictest2026-21kworker/0:020:21:200
321899241,8cyclictest0-21swapper/323:55:473
321899241,12cyclictest0-21swapper/300:16:243
320699247,3cyclictest29985-21ntp_states21:20:221
320699242,4cyclictest27696-21df_inode22:30:131
320699242,4cyclictest27696-21df_inode22:30:131
320699242,15cyclictest14634-21snmpd22:57:331
320699241,8cyclictest0-21swapper/123:11:591
320699241,6cyclictest0-21swapper/123:08:471
3206992415,2cyclictest0-21swapper/120:50:121
320699241,3cyclictest31696-21gltestperf21:25:171
320699241,3cyclictest31696-21gltestperf21:25:171
320699241,17cyclictest0-21swapper/121:35:181
320199249,3cyclictest3083-21kworker/0:119:12:270
320199245,6cyclictest6327-40dnf20:25:330
320199241,9cyclictest31914-21kworker/0:220:11:480
320199241,14cyclictest24582-21kworker/0:222:40:120
320199241,12cyclictest14647-21kworker/0:119:33:560
320199240,16cyclictest4441-21kworker/0:120:58:340
320199240,10cyclictest14647-21kworker/0:119:35:280
321899239,5cyclictest0-21swapper/320:22:423
321899239,5cyclictest0-21swapper/320:22:413
321899238,3cyclictest0-21swapper/321:54:243
321899237,5cyclictest780-21rngd23:50:023
321899237,3cyclictest86550irq/126-enp1s0-20:50:023
321899236,7cyclictest0-21swapper/319:10:403
321899235,12cyclictest88250irq/135-enp2s0-21:41:323
321899232,11cyclictest780-21rngd20:17:053
321899231,13cyclictest0-21swapper/322:39:383
321899231,12cyclictest0-21swapper/300:10:263
321899230,16cyclictest0-21swapper/300:09:053
320699231,17cyclictest0-21swapper/123:36:101
320699231,17cyclictest0-21swapper/123:36:101
320699231,16cyclictest0-21swapper/123:40:381
320699231,16cyclictest0-21swapper/123:19:571
320699231,16cyclictest0-21swapper/120:27:381
320199231,15cyclictest10928-21kworker/0:220:38:360
321899229,3cyclictest0-21swapper/319:58:253
321899226,6cyclictest22317-21/usr/sbin/munin21:05:273
321899225,2cyclictest41-21ksoftirqd/321:24:063
321899224,15cyclictest2508-21grep21:30:273
321899223,7cyclictest0-21swapper/320:32:073
321899223,6cyclictest41-21ksoftirqd/321:55:433
321899222,9cyclictest88050irq/133-enp2s0-23:08:103
321899221,15cyclictest0-21swapper/322:32:393
321899221,15cyclictest0-21swapper/322:32:393
320699229,5cyclictest0-21swapper/122:35:241
320699227,12cyclictest0-21swapper/121:40:191
320699226,4cyclictest0-21swapper/122:18:561
320699222,9cyclictest0-21swapper/122:05:311
320699222,11cyclictest25829-21hddtemp_smartct22:25:171
320699222,10cyclictest13014-21sh00:25:001
320699221,4cyclictest0-21swapper/123:24:501
320699221,4cyclictest0-21swapper/122:22:471
320699221,3cyclictest0-21swapper/121:18:081
3206992212,3cyclictest930-21/usr/sbin/munin21:30:191
320699221,16cyclictest0-21swapper/122:45:371
320699221,16cyclictest0-21swapper/119:45:281
320699221,16cyclictest0-21swapper/119:34:181
320699221,16cyclictest0-21swapper/100:01:251
320699221,15cyclictest0-21swapper/119:40:401
320699220,16cyclictest0-21swapper/119:28:031
320199221,12cyclictest2026-21kworker/0:020:18:200
320199221,12cyclictest16879-21kworker/0:219:46:100
205222220,4sleep121-21rcuc/122:15:001
321899218,2cyclictest0-21swapper/323:30:263
321899217,3cyclictest0-21swapper/322:47:223
321899216,8cyclictest26700-21load23:40:203
321899216,3cyclictest0-21swapper/319:52:153
321899215,2cyclictest0-21swapper/319:16:203
321899215,2cyclictest0-21swapper/319:16:203
321899213,11cyclictest780-21rngd22:15:563
321899211,3cyclictest0-21swapper/321:27:463
321899211,3cyclictest0-21swapper/321:27:463
321899211,14cyclictest0-21swapper/321:35:203
321899211,13cyclictest780-21rngd20:29:533
321899211,13cyclictest0-21swapper/322:12:123
321899211,12cyclictest11051-21/usr/sbin/munin00:20:173
320699219,3cyclictest15475-21crond20:50:021
3206992117,3cyclictest0-21swapper/123:30:041
320699211,15cyclictest0-21swapper/122:04:341
320699211,15cyclictest0-21swapper/121:56:061
320699211,15cyclictest0-21swapper/121:51:211
320699211,15cyclictest0-21swapper/121:04:401
320699211,15cyclictest0-21swapper/120:11:561
320699211,15cyclictest0-21swapper/119:50:561
320699211,15cyclictest0-21swapper/119:17:331
320699211,15cyclictest0-21swapper/119:17:331
320699211,15cyclictest0-21swapper/100:32:371
320699211,15cyclictest0-21swapper/100:17:021
320699211,14cyclictest0-21swapper/123:26:241
320699211,14cyclictest0-21swapper/121:47:371
320699211,11cyclictest22664-21diskstats21:05:151
320699210,5cyclictest1956-21kworker/u8:021:10:151
320699210,4cyclictest0-21swapper/122:40:251
320699210,16cyclictest0-21swapper/122:52:031
320699210,15cyclictest0-21swapper/119:59:011
320199210,12cyclictest15711-21kworker/0:220:51:200
321899209,2cyclictest18900-21sshd19:40:383
321899208,2cyclictest88050irq/133-enp2s0-20:59:243
321899208,2cyclictest7048-21ntpq22:55:223
321899206,6cyclictest86550irq/126-enp1s0-00:25:283
321899205,3cyclictest86550irq/126-enp1s0-23:15:273
321899203,3cyclictest3075-21latency00:00:193
321899202,7cyclictest88050irq/133-enp2s0-20:42:043
321899202,3cyclictest0-21swapper/320:38:033
321899201,12cyclictest0-21swapper/322:24:043
321899200,15cyclictest0-21swapper/321:13:033
320699202,2cyclictest86750irq/128-enp1s0-23:46:151
3206992016,3cyclictest0-21swapper/100:37:541
320699201,17cyclictest0-21swapper/100:05:131
320699201,15cyclictest0-21swapper/123:58:141
320699201,13cyclictest0-21swapper/119:10:261
320699201,11cyclictest9328-21cstates20:35:131
320699201,10cyclictest0-21swapper/100:10:281
320699200,15cyclictest0-21swapper/120:16:041
320699200,15cyclictest0-21swapper/119:35:501
180932201,2sleep30-21swapper/323:20:233
321899195,3cyclictest0-21swapper/321:15:283
321899195,2cyclictest0-21swapper/323:10:183
321899193,8cyclictest17203-21runrttasks00:32:293
321899191,9cyclictest780-21rngd23:29:063
321899191,13cyclictest0-21swapper/322:27:313
321899191,10cyclictest39-21rcuc/320:00:193
321899190,10cyclictest0-21swapper/300:35:363
320699198,3cyclictest28049-21latency_hist20:05:011
3206991912,2cyclictest0-21swapper/120:59:051
320699191,16cyclictest86550irq/126-enp1s0-20:05:161
320699191,10cyclictest0-21swapper/120:30:261
321899181,9cyclictest12888-21/usr/sbin/munin19:30:123
3218991813,3cyclictest15084-21/usr/sbin/munin19:35:183
3218991813,2cyclictest15713-21/usr/sbin/munin20:50:123
321899181,15cyclictest0-21swapper/319:20:243
321899180,3cyclictest0-21swapper/320:14:183
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional