You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 11:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Nov 15, 2025 00:44:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
310262261183,19sleep30-21swapper/319:09:313
308202235163,58sleep10-21swapper/119:06:531
307082225195,20sleep00-21swapper/019:05:310
291052202165,25sleep20-21swapper/219:05:132
2784021860,3sleep23116699cyclictest22:40:172
31166991241,118cyclictest0-21swapper/223:42:162
109782660,2sleep011-21rcuc/019:35:140
249152650,4sleep33117099cyclictest23:45:263
69632580,3sleep219879-21diskmemload00:20:142
206282570,4sleep13115999cyclictest21:10:161
206282570,4sleep13115999cyclictest21:10:161
134202540,2sleep00-21swapper/000:35:130
133622340,4sleep19-21rcu_preempt19:40:161
3115999324,6cyclictest0-21swapper/100:23:571
116642320,2sleep30-21swapper/320:50:153
3116699314,4cyclictest19892-21perf21:10:012
31159993113,3cyclictest12361-21ntpq20:50:211
253802310,9sleep07-21ksoftirqd/022:35:140
3116699302,11cyclictest28483-21kworker/2:021:20:272
31166992919,2cyclictest28483-21kworker/2:019:15:262
3115999297,2cyclictest0-21swapper/100:05:251
3115999291,2cyclictest19889-21awk21:10:001
121102290,6sleep230-21rcuc/220:50:182
3115999283,23cyclictest0-21swapper/123:10:301
31159992811,3cyclictest0-21swapper/123:55:231
3115499289,4cyclictest836-21runrttasks23:22:340
3115499289,4cyclictest836-21runrttasks23:22:340
3115499285,3cyclictest0-21swapper/019:54:220
31166992724,2cyclictest0-21swapper/223:51:352
3116699271,23cyclictest12119-21kworker/2:123:57:262
31159992712,3cyclictest25027-21latency_hist22:35:021
3115999271,23cyclictest0-21swapper/121:34:491
3115999271,21cyclictest0-21swapper/122:26:481
3115999271,20cyclictest0-21swapper/121:40:171
250192270,4sleep00-21swapper/021:20:190
3116699262,11cyclictest28483-21kworker/2:021:37:412
3116699261,5cyclictest0-21swapper/221:59:122
3116699261,5cyclictest0-21swapper/221:10:162
3116699261,5cyclictest0-21swapper/221:10:162
31166992613,3cyclictest28483-21kworker/2:022:05:572
3116699261,16cyclictest0-21swapper/222:17:442
3116699261,16cyclictest0-21swapper/222:17:442
3116699261,13cyclictest12119-21kworker/2:123:18:532
3116699261,10cyclictest28483-21kworker/2:021:40:192
3115999269,3cyclictest0-21swapper/100:11:411
3115999267,3cyclictest0-21swapper/119:52:301
31159992616,7cyclictest23634-21latency_hist20:05:011
3115499268,7cyclictest26238-21kworker/0:221:44:040
3115499264,3cyclictest0-21swapper/020:41:180
223252260,5sleep13115999cyclictest21:15:131
3117099251,4cyclictest0-21swapper/321:28:383
3116699258,3cyclictest28483-21kworker/2:022:00:212
3116699254,3cyclictest0-21swapper/223:12:142
3116699252,13cyclictest28483-21kworker/2:022:35:572
31166992514,2cyclictest28483-21kworker/2:022:23:012
31166992511,3cyclictest28483-21kworker/2:021:46:532
3116699250,12cyclictest12119-21kworker/2:123:28:372
3116699250,12cyclictest12119-21kworker/2:123:28:372
3115999259,3cyclictest0-21swapper/123:05:181
3115999255,3cyclictest0-21swapper/121:56:061
31159992520,2cyclictest10814-21munin-run00:30:001
31159992520,2cyclictest10814-21munin-run00:30:001
31159992520,2cyclictest0-21swapper/123:25:151
31159992520,2cyclictest0-21swapper/123:25:151
31159992519,3cyclictest9942-21unixbench-2d19:30:271
31159992514,3cyclictest15462-21latency_hist21:00:001
3115999251,19cyclictest0-21swapper/121:29:001
31159992510,3cyclictest10076-21df22:00:141
3115999250,23cyclictest0-21swapper/122:50:021
3115499252,3cyclictest0-21swapper/020:38:540
3115499252,11cyclictest81ktimersoftd/022:50:020
3115499250,23cyclictest0-21swapper/021:12:330
3115499250,23cyclictest0-21swapper/021:12:330
3117099241,7cyclictest0-21swapper/322:47:043
3117099241,5cyclictest0-21swapper/323:59:353
3117099241,4cyclictest0-21swapper/321:16:543
3117099240,15cyclictest0-21swapper/323:10:423
31166992417,4cyclictest27308-21find20:10:222
3116699241,3cyclictest23928-21kworker/u8:221:34:482
3116699241,11cyclictest28483-21kworker/2:022:26:532
31166992410,3cyclictest12119-21kworker/2:123:48:382
3115999246,3cyclictest0-21swapper/122:05:181
3115999241,7cyclictest13170-21tr23:20:171
3115999241,7cyclictest13170-21tr23:20:171
31159992417,4cyclictest11662-21strings19:35:201
3115999241,20cyclictest19355-21irqstats22:20:171
31159992411,3cyclictest0-21swapper/123:52:051
3115999241,11cyclictest6576-21cat00:20:001
3115999241,11cyclictest24219-21diskstats20:05:181
3115999241,11cyclictest24219-21diskstats20:05:181
3115999241,11cyclictest0-21swapper/123:40:181
31159992410,8cyclictest0-21swapper/123:16:291
3115999240,5cyclictest0-21swapper/121:23:021
3115499246,2cyclictest0-21swapper/020:03:180
227332240,4sleep081ktimersoftd/021:15:170
3117099231,6cyclictest23299-1kworker/u9:222:52:433
3117099231,6cyclictest0-21swapper/323:52:573
3117099231,6cyclictest0-21swapper/323:41:353
3116699232,13cyclictest12119-21kworker/2:123:35:332
3116699232,12cyclictest28483-21kworker/2:020:58:382
3116699231,3cyclictest0-21swapper/200:17:492
31166992312,3cyclictest28483-21kworker/2:020:37:022
31166992312,3cyclictest28483-21kworker/2:020:31:502
31166992311,3cyclictest28483-21kworker/2:020:21:412
3116699231,12cyclictest28483-21kworker/2:023:00:052
3116699231,12cyclictest12119-21kworker/2:100:35:092
31166992310,3cyclictest28483-21kworker/2:021:27:412
3116699230,11cyclictest12119-21kworker/2:100:05:262
3115999231,3cyclictest14871-21/usr/sbin/munin22:10:171
31159992312,3cyclictest0-21swapper/100:03:171
31159992312,2cyclictest0-21swapper/122:36:111
3115999231,15cyclictest0-21swapper/123:34:211
3115999230,6cyclictest0-21swapper/122:16:481
3115999230,6cyclictest0-21swapper/122:16:481
3115999230,5cyclictest0-21swapper/121:46:131
3115999230,2cyclictest86850irq/129-enp1s0-23:02:121
3115499239,4cyclictest836-21runrttasks23:49:220
3115499238,4cyclictest12290-21chrt23:19:220
3115499238,3cyclictest15973-21kworker/0:322:14:260
3115499232,3cyclictest86750irq/128-enp1s0-23:38:140
3115499231,4cyclictest0-21swapper/021:55:260
31154992314,3cyclictest0-21swapper/021:32:110
31154992311,5cyclictest0-21swapper/000:15:140
3117099225,4cyclictest19879-21diskmemload21:40:133
3117099221,5cyclictest0-21swapper/321:22:323
3117099221,4cyclictest0-21swapper/323:01:203
3117099221,4cyclictest0-21swapper/322:07:073
3116699229,3cyclictest28483-21kworker/2:019:13:102
3116699224,5cyclictest848-21sssd_nss22:45:232
3116699222,12cyclictest28483-21kworker/2:023:06:212
31166992212,3cyclictest28483-21kworker/2:019:30:052
31166992210,3cyclictest28483-21kworker/2:022:13:012
31166992210,3cyclictest28483-21kworker/2:019:35:572
31166992210,3cyclictest12119-21kworker/2:100:31:252
31166992210,3cyclictest12119-21kworker/2:100:31:252
3115999226,3cyclictest17046-21awk19:50:011
31159992218,3cyclictest0-21swapper/120:26:551
31159992217,4cyclictest0-21swapper/119:10:131
31159992212,2cyclictest0-21swapper/123:35:341
3115999221,10cyclictest6676-21latency_hist20:40:011
31159992210,3cyclictest10171-21ntp_states20:45:251
3115999220,4cyclictest0-21swapper/123:47:201
3115999220,4cyclictest0-21swapper/100:31:081
3115999220,4cyclictest0-21swapper/100:31:081
3115999220,17cyclictest0-21swapper/121:37:091
3115999220,13cyclictest0-21swapper/122:41:231
3115499228,3cyclictest28046-21timerandwakeup21:25:260
3115499227,8cyclictest0-21swapper/000:27:300
3115499227,8cyclictest0-21swapper/000:27:300
3115499225,3cyclictest8102-21kworker/0:219:30:280
3115499225,2cyclictest86650irq/127-enp1s0-23:59:520
31154992215,3cyclictest0-21swapper/023:08:230
31154992210,8cyclictest0-21swapper/022:35:010
3117099216,5cyclictest10946-21fschecks_time23:15:173
3117099213,9cyclictest0-21swapper/322:10:443
3117099212,8cyclictest15468-21munin-run21:00:013
3117099211,5cyclictest23299-1kworker/u9:223:29:133
3117099211,5cyclictest23299-1kworker/u9:223:29:133
3117099211,5cyclictest0-21swapper/323:09:583
3117099211,11cyclictest19935-21if_enp2s023:35:203
3116699219,3cyclictest28483-21kworker/2:021:15:262
3116699219,3cyclictest12119-21kworker/2:100:12:462
3116699216,4cyclictest19879-21diskmemload21:51:022
3116699215,4cyclictest28277-21/usr/sbin/munin20:15:112
3116699214,4cyclictest14634-21snmpd19:20:232
3116699212,13cyclictest28483-21kworker/2:020:04:392
3116699212,12cyclictest28483-21kworker/2:020:47:102
3116699212,12cyclictest12119-21kworker/2:123:23:332
3116699212,12cyclictest12119-21kworker/2:123:23:332
3116699212,11cyclictest28483-21kworker/2:020:41:342
31166992111,3cyclictest12119-21kworker/2:100:00:142
31166992110,3cyclictest28483-21kworker/2:022:51:412
31166992110,3cyclictest28483-21kworker/2:022:34:212
31166992110,3cyclictest12119-21kworker/2:123:30:292
3115999217,3cyclictest0-21swapper/120:41:171
3115999213,3cyclictest1302-21sshd22:54:451
31159992117,3cyclictest0-21swapper/121:00:241
3115999211,18cyclictest0-21swapper/119:21:351
3115999211,17cyclictest30625-21expr20:20:131
3115999210,3cyclictest0-21swapper/121:52:481
3115999210,2cyclictest86850irq/129-enp1s0-00:37:411
3115499219,3cyclictest0-21swapper/022:41:370
3115499218,3cyclictest31440-21sensors_temp00:00:240
3115499215,8cyclictest0-21swapper/021:52:150
3115499212,6cyclictest24887-21kworker/0:023:44:490
3115499212,10cyclictest17260-21/usr/sbin/munin22:15:220
3115499212,10cyclictest17260-21/usr/sbin/munin22:15:220
3115499211,17cyclictest3320-21ntp_states00:10:230
31154992111,3cyclictest0-21swapper/023:54:010
3115499211,10cyclictest17778-21interrupts23:30:190
3115499211,10cyclictest0-21swapper/022:05:200
3115499210,3cyclictest86850irq/129-enp1s0-22:52:050
3117099203,4cyclictest23299-1kworker/u9:200:19:213
3117099201,9cyclictest1169150kworker/u8:222:15:163
3117099201,9cyclictest1169150kworker/u8:222:15:163
3117099201,4cyclictest0-21swapper/323:22:593
3117099201,4cyclictest0-21swapper/323:22:593
3117099201,4cyclictest0-21swapper/322:32:023
3117099201,11cyclictest28270-21/usr/sbin/munin20:15:183
3117099201,11cyclictest14634-21snmpd19:26:453
3117099200,14cyclictest0-21swapper/319:55:183
3116699205,3cyclictest16411-21grep19:45:282
3116699202,6cyclictest19462-21/usr/sbin/munin19:55:152
3116699202,11cyclictest28483-21kworker/2:020:25:102
3116699202,11cyclictest28483-21kworker/2:019:52:292
3116699202,11cyclictest12119-21kworker/2:100:25:422
3116699202,11cyclictest12119-21kworker/2:100:25:422
3116699202,10cyclictest28483-21kworker/2:020:05:172
3116699202,10cyclictest28483-21kworker/2:020:05:172
3116699201,4cyclictest0-21swapper/222:56:402
31166992012,4cyclictest6876-21tail19:25:152
3115999206,8cyclictest0-21swapper/122:58:271
3115999201,7cyclictest2515-21/usr/sbin/munin20:30:241
3115999201,7cyclictest19462-21/usr/sbin/munin19:55:211
31159992017,2cyclictest0-21swapper/119:16:131
3115999201,16cyclictest6375-21grep19:25:121
3115499209,4cyclictest0-21swapper/019:16:360
3115499203,3cyclictest0-21swapper/020:19:340
3115499203,3cyclictest0-21swapper/020:07:180
3115499203,3cyclictest0-21swapper/020:07:180
3115499203,2cyclictest0-21swapper/022:57:160
31154992014,3cyclictest645-21grep20:25:160
31154992014,3cyclictest21272-21grep22:25:160
31154992013,4cyclictest5078-21needreboot19:20:210
3115499201,10cyclictest19623-21sshd21:06:580
31154992010,6cyclictest0-21swapper/022:22:440
31154992010,4cyclictest0-21swapper/021:37:390
173752201,2sleep314634-21snmpd21:01:203
100732200,4sleep011-21rcuc/022:00:140
3117099193,10cyclictest14634-21snmpd21:08:333
3117099192,15cyclictest842-21lldpd21:10:273
3117099192,15cyclictest842-21lldpd21:10:273
3117099191,8cyclictest0-21swapper/321:56:163
3117099191,6cyclictest0-21swapper/322:40:023
3117099191,5cyclictest0-21swapper/321:53:573
3117099191,3cyclictest0-21swapper/300:26:073
3117099191,3cyclictest0-21swapper/300:26:073
3117099191,15cyclictest0-21swapper/322:25:263
3117099191,11cyclictest14634-21snmpd00:31:453
3117099191,11cyclictest14634-21snmpd00:31:453
3116699195,3cyclictest86550irq/126-enp1s0-19:40:222
31159991914,2cyclictest26813-21if_enp1s020:10:171
3115999191,17cyclictest0-21swapper/120:19:271
3115499196,3cyclictest8908-21basename23:10:190
3115499195,9cyclictest0-21swapper/023:02:510
3115499195,7cyclictest0-21swapper/000:34:170
3115499195,7cyclictest0-21swapper/000:34:170
3115499193,3cyclictest0-21swapper/020:23:100
3115499192,3cyclictest0-21swapper/019:48:300
3115499192,10cyclictest1737-21sshd00:06:450
3115499191,5cyclictest15377-21grep23:25:180
3115499191,5cyclictest15377-21grep23:25:180
31154991912,4cyclictest10337-21sendmail_mailqu20:45:280
3115499191,12cyclictest0-21swapper/000:20:410
3115499190,4cyclictest0-21swapper/020:55:260
81672181,2sleep00-21swapper/019:29:080
3117099189,3cyclictest86850irq/129-enp1s0-00:24:073
3117099187,3cyclictest15074-21/usr/sbin/munin19:45:243
3117099186,3cyclictest9738-21/usr/sbin/munin22:00:203
3117099184,5cyclictest4279-21pmu-power21:45:253
3117099184,3cyclictest0-21swapper/322:20:213
3117099183,12cyclictest4678-21grep19:20:183
3117099182,6cyclictest3851-21sh19:20:013
3117099181,9cyclictest14634-21snmpd00:11:253
3117099181,8cyclictest0-21swapper/322:57:053
3117099181,7cyclictest0-21swapper/300:36:053
3117099181,6cyclictest0-21swapper/300:07:213
3117099181,5cyclictest23299-1kworker/u9:223:33:513
31170991812,4cyclictest3145-21grep20:30:163
31170991812,3cyclictest12020-21wc19:35:263
3115499185,7cyclictest0-21swapper/021:45:320
3115499183,3cyclictest86750irq/128-enp1s0-19:58:060
3115499181,4cyclictest0-21swapper/019:40:280
3115499181,11cyclictest0-21swapper/019:10:150
322822171,2sleep30-21swapper/320:23:033
3117099175,4cyclictest86550irq/126-enp1s0-20:40:153
3117099173,11cyclictest31727-21grep21:35:183
3117099172,6cyclictest26073-21/usr/sbin/munin20:10:193
3117099172,6cyclictest12874-21/usr/sbin/munin19:40:183
3117099172,5cyclictest18770-21grep19:50:263
3117099171,7cyclictest25502-21entropy22:35:163
3117099171,7cyclictest0-21swapper/321:30:183
3117099171,4cyclictest784-21NetworkManager19:11:463
31170991712,3cyclictest30175-21/usr/sbin/munin00:00:203
31170991712,3cyclictest21661-21/usr/sbin/munin20:00:233
3117099171,15cyclictest0-21swapper/320:39:593
3117099171,15cyclictest0-21swapper/320:07:273
3117099171,15cyclictest0-21swapper/320:07:273
31170991711,3cyclictest9412-21expr20:45:183
3116699172,3cyclictest15698-21/usr/sbin/munin21:00:272
3115499175,2cyclictest86750irq/128-enp1s0-21:01:560
3115499173,3cyclictest86650irq/127-enp1s0-20:54:340
3115499171,5cyclictest0-21swapper/020:30:200
31154991710,4cyclictest27016-21missed_timers20:10:190
3117099161,9cyclictest0-21swapper/320:25:213
31170991610,3cyclictest9423-21missed_timers19:30:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional