You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 05:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jul 01, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
104582239162,24sleep00-21swapper/019:06:100
107072236163,23sleep30-21swapper/319:09:233
104592233161,58sleep10-21swapper/119:06:111
106462228163,51sleep20-21swapper/219:08:382
1946121620,4sleep3401ktimersoftd/320:40:153
2130521460,4sleep01085399cyclictest20:45:000
10863991147,105cyclictest0-21swapper/221:20:222
10863997370,2cyclictest0-21swapper/220:44:072
190402660,5sleep21086399cyclictest21:50:272
242232630,3sleep19-21rcu_preempt00:35:121
35552610,3sleep03559-21fschecks_time22:30:190
115732570,4sleep01085399cyclictest20:20:290
1085399529,31cyclictest0-21swapper/021:40:010
17862500,1sleep10-21swapper/120:00:141
1085399488,12cyclictest27039-21perf23:25:010
300562460,1sleep10-21swapper/121:05:011
1085399455,13cyclictest23317-21idleruntime-cro20:50:010
1085399442,6cyclictest21868-21munin-run22:00:010
1085399442,6cyclictest21868-21munin-run22:00:010
1085399423,9cyclictest836-21runrttasks21:48:340
1086899413,27cyclictest7872-21ls23:55:003
1086899408,23cyclictest166332cut19:20:203
1086899403,18cyclictest86750irq/128-enp1s0-23:30:243
1086899401,28cyclictest0-21swapper/321:40:153
1086899401,27cyclictest23591-21cpu20:50:123
1086899381,26cyclictest15511-21/usr/sbin/munin21:45:213
1086899381,21cyclictest19714-21/usr/sbin/munin00:25:223
1086899370,30cyclictest0-21swapper/320:55:223
1086899363,24cyclictest4237-21/usr/sbin/munin22:30:273
1086899362,31cyclictest16213-21/usr/sbin/munin23:00:263
1086899362,31cyclictest16213-21/usr/sbin/munin23:00:263
1086899361,6cyclictest0-21swapper/322:00:003
1086899361,6cyclictest0-21swapper/322:00:003
10868993615,18cyclictest29154-21turbostat19:50:013
1086899361,4cyclictest19233-21/usr/sbin/munin23:05:213
1086899361,26cyclictest24371-21sshd19:35:443
1086899353,22cyclictest8049-21hddtemp_smartct22:40:193
1086899351,24cyclictest21835-21wc23:10:253
1086899351,18cyclictest0-21swapper/322:05:173
10853993510,12cyclictest29223-21munin-run23:30:010
1086899341,7cyclictest14634-21snmpd21:11:013
1086899341,5cyclictest0-21swapper/323:29:273
1086899341,4cyclictest0-21swapper/300:11:353
1086899341,3cyclictest5170-21/usr/sbin/munin23:45:273
1086899341,29cyclictest11455-40dnf22:48:473
1086399341,25cyclictest0-21swapper/223:05:462
1086399341,25cyclictest0-21swapper/220:50:162
1086399341,15cyclictest0-21swapper/222:35:362
1086399341,14cyclictest0-21swapper/220:27:242
1086899332,28cyclictest1905-21grep20:00:153
1086899331,3cyclictest25692-21users00:35:263
1086899331,19cyclictest13279-21cat22:50:263
1086399337,10cyclictest12004-21memory21:35:232
10863993322,3cyclictest0-21swapper/220:49:122
1086399331,13cyclictest0-21swapper/222:55:282
1086399331,13cyclictest0-21swapper/220:55:442
1085399331,25cyclictest27963-21kworker/0:121:01:060
1086899324,19cyclictest7429-21cat21:25:193
1086899321,7cyclictest0-21swapper/321:31:083
1086899321,3cyclictest29831-21/usr/sbin/munin22:15:273
1086899321,23cyclictest0-21swapper/322:20:233
1086899321,14cyclictest0-21swapper/320:32:363
1086899321,13cyclictest0-21swapper/320:38:283
1086899320,2cyclictest0-21swapper/300:07:563
1086899320,2cyclictest0-21swapper/300:07:563
1086399329,10cyclictest780-21rngd19:13:362
1086399321,24cyclictest0-21swapper/223:41:282
1086399321,24cyclictest0-21swapper/221:27:362
1086399320,23cyclictest0-21swapper/222:25:302
1085399323,17cyclictest14972-21kworker/0:300:16:200
1085399320,12cyclictest19634-21kworker/0:122:20:240
1086899316,13cyclictest0-21swapper/322:35:323
1086899315,22cyclictest2368-21tr21:15:143
1086899312,15cyclictest11079-21cpu19:10:133
1086899311,4cyclictest0-21swapper/322:26:363
10868993113,15cyclictest12160-21ntp_states21:35:263
1086899311,2cyclictest25806-21if_enp2s023:20:213
1086899311,21cyclictest0-21swapper/300:24:323
1086899311,15cyclictest0-21swapper/300:03:173
1086899311,13cyclictest0-21swapper/319:55:543
1086899311,12cyclictest0-21swapper/319:32:523
1086899310,4cyclictest0-21swapper/322:55:183
1086899310,3cyclictest86750irq/128-enp1s0-21:20:163
1086399314,17cyclictest30-21rcuc/222:19:082
1086399311,11cyclictest0-21swapper/200:19:442
1086399311,10cyclictest0-21swapper/219:59:362
1085399312,22cyclictest19634-21kworker/0:122:35:100
1086899309,18cyclictest22062-21cpuspeed_turbos00:30:143
1086899307,19cyclictest11347-21ntpq20:20:253
1086899304,23cyclictest9575-21/usr/sbin/munin20:15:303
1086899303,24cyclictest27666-21proc_pri22:10:283
1086899301,5cyclictest0-21swapper/323:41:593
1086899301,4cyclictest24321-21unixbench-2d23:15:263
1086899301,24cyclictest29803-21uniq19:50:163
1086899301,24cyclictest29803-21uniq19:50:163
1086899301,15cyclictest26351-21cat19:40:303
1086899301,15cyclictest0-21swapper/323:37:483
1086899301,11cyclictest0-21swapper/321:51:143
1086399301,9cyclictest32196-21kworker/2:123:20:562
1086399301,14cyclictest7340-21unixbench-2d23:50:282
1085399301,13cyclictest26280-21munin-run22:10:010
1086899294,14cyclictest15742-21sshd19:17:443
1086899293,11cyclictest86550irq/126-enp1s0-20:07:443
1086899291,7cyclictest23449-21cat20:50:013
1086899291,5cyclictest0-21swapper/322:01:553
1086899291,25cyclictest29136-21/usr/sbin/munin21:00:253
1086899291,18cyclictest30197-21/usr/sbin/munin21:05:233
10868992911,15cyclictest18357-21/usr/sbin/munin19:25:183
1086899290,27cyclictest0-21swapper/300:15:163
1086399296,9cyclictest11222-21sshd22:45:482
1086399292,12cyclictest780-21rngd20:20:182
1086399291,14cyclictest0-21swapper/200:27:362
1086399291,10cyclictest0-21swapper/221:11:062
1085899291,13cyclictest6934-21ntp_states20:10:241
1085399295,11cyclictest3465-21idleruntime-cro23:45:020
10853992910,16cyclictest9365-21munin-run22:45:010
1086899286,19cyclictest6369-21fschecks_time20:10:173
1086899282,23cyclictest9334-21smart_sda23:55:243
10868992810,15cyclictest13723-21gunzip20:25:253
1086399287,16cyclictest86750irq/128-enp1s0-23:55:282
1086399284,15cyclictest30-21rcuc/223:31:362
1086399284,10cyclictest780-21rngd19:16:362
1086399283,16cyclictest29840-21sshd21:03:482
1086399283,14cyclictest24320-21unixbench-2d23:15:262
10863992811,7cyclictest0-21swapper/222:11:082
1086399281,11cyclictest0-21swapper/222:43:362
1085399282,16cyclictest5444-21kworker/0:019:13:500
1085399281,12cyclictest20209-21kworker/0:020:00:220
1085399281,10cyclictest20209-21kworker/0:019:51:500
1085399281,10cyclictest20209-21kworker/0:019:51:500
1086399274,12cyclictest3410-21latency_hist20:05:012
1086399272,9cyclictest0-21swapper/221:16:162
1086399272,16cyclictest22476-21grep00:30:172
1086399272,10cyclictest780-21rngd19:20:122
1086399272,10cyclictest0-21swapper/220:08:482
1086399271,14cyclictest0-21swapper/223:45:522
1086399271,14cyclictest0-21swapper/221:40:562
1086399271,13cyclictest0-21swapper/219:33:062
1086399271,12cyclictest780-21rngd22:05:362
1086399271,12cyclictest0-21swapper/220:17:042
1086399271,11cyclictest0-21swapper/220:33:522
1086399271,11cyclictest0-21swapper/200:07:542
1086399271,11cyclictest0-21swapper/200:07:542
1086399271,10cyclictest0-21swapper/223:37:122
10863992710,7cyclictest24832-21if_enp1s000:35:172
1085399272,12cyclictest2751-21munin-run22:30:010
1085399272,11cyclictest12740-21kworker/0:221:44:060
1085399271,11cyclictest3269-21kworker/0:220:16:140
1086399265,3cyclictest0-21swapper/219:25:542
1086399265,11cyclictest18305-21latency_hist23:05:012
1086399265,11cyclictest18305-21latency_hist23:05:012
1086399264,12cyclictest88250irq/135-enp2s0-19:48:482
1086399263,12cyclictest18819-21NetworkManager20:39:222
1086399261,8cyclictest0-21swapper/219:37:042
1086399261,11cyclictest0-21swapper/219:41:042
1086399261,10cyclictest0-21swapper/222:01:122
1086399260,9cyclictest0-21swapper/221:30:082
1085899269,11cyclictest87950irq/132-enp2s0-20:27:241
1085399262,13cyclictest23793-21kworker/0:119:45:200
1085399261,19cyclictest5573-21kworker/0:123:51:160
1085399261,11cyclictest31799-21kworker/0:021:30:380
1085399261,11cyclictest24519-21kworker/0:023:33:540
1085399261,10cyclictest5573-21kworker/0:123:49:580
1085399261,10cyclictest24519-21kworker/0:023:38:220
229552251,3sleep122976-21ntp_states00:30:231
1086399258,7cyclictest27252-21/usr/sbin/munin23:25:192
1086399253,3cyclictest0-21swapper/221:05:482
1086399251,16cyclictest0-21swapper/222:20:282
1086399251,11cyclictest32196-21kworker/2:100:12:422
1086399251,10cyclictest32196-21kworker/2:122:34:482
1086399251,10cyclictest0-21swapper/221:47:222
1085899251,14cyclictest0-21swapper/122:50:151
1085399259,2cyclictest5573-21kworker/0:123:55:020
1085399259,2cyclictest11476-21kworker/0:123:00:180
1085399259,2cyclictest11476-21kworker/0:123:00:180
1085399252,10cyclictest14972-21kworker/0:300:13:540
1085399251,9cyclictest20981-21kworker/0:321:06:060
1085399251,12cyclictest12074-21kworker/0:000:03:580
10853992510,12cyclictest18887-21munin-run20:40:010
1086399249,9cyclictest0-21swapper/221:58:402
1086399249,9cyclictest0-21swapper/221:58:402
1086399248,3cyclictest32-21ksoftirqd/220:14:022
1086399246,4cyclictest17879-21forks00:20:182
1086399243,11cyclictest88150irq/134-enp2s0-19:50:242
1086399243,11cyclictest88150irq/134-enp2s0-19:50:242
1086399241,16cyclictest0-21swapper/200:03:282
10863992410,3cyclictest0-21swapper/223:10:422
1085899241,17cyclictest0-21swapper/119:53:491
1085899241,17cyclictest0-21swapper/119:53:491
1085899241,17cyclictest0-21swapper/100:17:051
10858992411,6cyclictest14932-21kworker/u8:200:14:511
1085399249,3cyclictest18003-21kworker/0:219:32:060
1085399248,2cyclictest31799-21kworker/0:021:11:260
1085399246,3cyclictest17171-21kworker/0:000:36:300
1085399242,8cyclictest11476-21kworker/0:123:05:240
1085399242,7cyclictest19634-21kworker/0:122:15:420
122902231,3sleep111051-21/usr/sbin/munin19:10:261
1085899232,18cyclictest17081-21memory19:20:241
1085899232,10cyclictest1318-21perl22:25:191
1085899231,4cyclictest3427-21fschecks_count22:30:181
1085899231,17cyclictest0-21swapper/122:36:461
1085899231,17cyclictest0-21swapper/119:16:521
1085899231,16cyclictest0-21swapper/121:15:141
1085899231,16cyclictest0-21swapper/120:54:541
1085899231,16cyclictest0-21swapper/120:15:221
1085899231,15cyclictest32168-21diskmemload21:48:281
1085899231,14cyclictest15901-21kworker/u8:223:11:131
1085899231,11cyclictest26122-21ntp_states23:20:251
1085899230,6cyclictest0-21swapper/122:44:461
1085899230,15cyclictest0-21swapper/122:19:081
1085899230,11cyclictest0-21swapper/100:07:481
1085899230,11cyclictest0-21swapper/100:07:481
1085399238,2cyclictest28157-21kworker/0:222:11:580
1085399233,10cyclictest23793-21kworker/0:119:36:560
1085399231,9cyclictest9170-21kworker/0:322:47:100
1085399231,9cyclictest21903-21kworker/0:200:30:460
1085399231,9cyclictest19474-21kworker/0:100:27:020
1085399231,9cyclictest11476-21kworker/0:122:50:140
1085399231,8cyclictest12033-21kworker/0:120:32:060
1085399231,6cyclictest0-21swapper/021:21:380
1085399231,10cyclictest5481-21kworker/0:120:13:420
1086399223,14cyclictest88250irq/135-enp2s0-22:53:162
1085899221,5cyclictest0-21swapper/123:57:231
1085899221,4cyclictest0-21swapper/123:26:401
10858992213,3cyclictest0-21swapper/120:30:451
10858992212,3cyclictest0-21swapper/122:49:321
1085899221,18cyclictest2126-21grep23:40:171
1085899221,16cyclictest0-21swapper/123:45:231
1085899221,16cyclictest0-21swapper/123:15:571
1085899221,16cyclictest0-21swapper/123:02:041
1085899221,16cyclictest0-21swapper/123:02:041
1085899221,10cyclictest0-21swapper/121:44:551
1085899220,4cyclictest0-21swapper/121:30:151
1085899220,17cyclictest0-21swapper/121:55:491
1085899220,17cyclictest0-21swapper/121:55:491
1085399229,2cyclictest11476-21kworker/0:122:57:260
1085399228,2cyclictest19634-21kworker/0:122:02:220
1085399227,2cyclictest31799-21kworker/0:021:16:060
1085399222,10cyclictest5481-21kworker/0:120:08:540
1085399221,8cyclictest12033-21kworker/0:120:26:060
1085899213,9cyclictest88050irq/133-enp2s0-20:35:141
1085899211,6cyclictest0-21swapper/121:52:471
10858992115,4cyclictest22117-21grep19:30:281
10858992115,3cyclictest8237-21users21:25:301
10858992115,3cyclictest29737-21expr23:30:141
10858992114,2cyclictest0-21swapper/119:57:051
10858992113,3cyclictest0-21swapper/100:25:151
1085899211,2cyclictest0-21swapper/122:06:481
1085899211,2cyclictest0-21swapper/121:10:041
1085899211,15cyclictest86750irq/128-enp1s0-22:04:251
1085899211,15cyclictest0-21swapper/122:11:451
1085899211,15cyclictest0-21swapper/121:05:091
1085899211,15cyclictest0-21swapper/120:45:501
1085899211,15cyclictest0-21swapper/120:44:441
1085899211,15cyclictest0-21swapper/120:09:121
1085899211,14cyclictest0-21swapper/123:35:131
10858992111,3cyclictest86750irq/128-enp1s0-21:20:101
1085899211,13cyclictest26126-21kworker/u8:122:57:181
1085899211,11cyclictest0-21swapper/121:39:501
1085899210,15cyclictest0-21swapper/119:25:071
1085399212,9cyclictest24519-21kworker/0:023:16:380
1085399212,6cyclictest20791-1kworker/u9:221:29:430
1085399211,8cyclictest22277-21kworker/0:323:12:060
10853992111,2cyclictest12074-21kworker/0:000:09:180
10853992111,2cyclictest12074-21kworker/0:000:09:180
1085899209,6cyclictest0-21swapper/123:05:231
1085899202,2cyclictest23-21ksoftirqd/100:03:501
1085899201,9cyclictest0-21swapper/100:20:171
1085899201,3cyclictest0-21swapper/119:48:391
10858992013,2cyclictest0-21swapper/119:44:461
1085899201,18cyclictest0-21swapper/122:21:201
1085899201,15cyclictest0-21swapper/120:57:361
1085899201,14cyclictest0-21swapper/123:53:051
1085399205,2cyclictest20981-21kworker/0:320:55:090
1085899191,10cyclictest10279-21/usr/sbin/munin20:20:261
1085399196,3cyclictest86750irq/128-enp1s0-19:40:300
1085399194,2cyclictest86650irq/127-enp1s0-19:57:300
10853991914,3cyclictest17867-21cpuspeed_turbos21:50:140
1085399191,10cyclictest5444-21kworker/0:019:21:020
10853991910,2cyclictest15660-21kworker/0:119:25:220
1085399190,9cyclictest5444-21kworker/0:019:15:100
1085899184,5cyclictest88150irq/134-enp2s0-19:36:181
1085399184,3cyclictest88050irq/133-enp2s0-20:52:050
1085399182,8cyclictest17520-21/usr/sbin/munin00:20:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional