You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-19 - 23:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 19, 2024 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
528499491,36cyclictest0-21swapper/008:25:160
528499491,36cyclictest0-21swapper/008:25:160
5314994730,12cyclictest26866irq/58-eth009:55:133
528499472,42cyclictest10192-21apt-get07:15:150
530699461,27cyclictest12130-21ssh09:45:142
529599462,35cyclictest30455-21ssh09:15:151
528499463,39cyclictest18633-21ssh10:10:150
528499461,41cyclictest5441-21apt-get07:10:120
528499450,34cyclictest29527-21/usr/sbin/munin07:35:190
528499443,38cyclictest0-21swapper/007:55:130
528499441,38cyclictest14970-21apt-get07:20:160
5295994320,4cyclictest21-21ksoftirqd/111:40:141
528499432,37cyclictest21659-21apt-get08:35:120
528499432,36cyclictest19698-21apt-get09:50:130
528499431,28cyclictest9696-21/usr/sbin/munin07:45:240
5314994225,12cyclictest26866irq/58-eth011:00:163
5295994220,16cyclictest21-21ksoftirqd/109:50:141
529599420,28cyclictest21-21ksoftirqd/109:25:131
528499420,30cyclictest12019-21cat12:10:230
531499414,21cyclictest26866irq/58-eth009:00:153
531499411,26cyclictest26866irq/58-eth009:35:123
528499412,36cyclictest8404-21apt-get08:55:140
528499411,36cyclictest27786-21apt-get11:20:160
528499411,23cyclictest31251-21ntp_192.168.11409:55:260
528499410,35cyclictest31292-21apt-get08:45:150
528499401,35cyclictest19302-21apt-get11:15:130
528499400,35cyclictest12120-21apt-get09:45:140
531499393,23cyclictest26866irq/58-eth007:35:143
528499392,25cyclictest27885-21sh12:00:160
5284993920,14cyclictest12-21ksoftirqd/008:15:150
528499391,31cyclictest0-21swapper/011:45:160
528499391,26cyclictest16224-21unixbench_singl11:50:330
528499391,23cyclictest0-21swapper/011:13:040
528499390,26cyclictest22504-21diskmemload10:35:560
531499384,22cyclictest26866irq/58-eth012:35:153
528499381,32cyclictest0-21swapper/009:25:140
528499381,24cyclictest10522-21ssh10:26:470
528499380,35cyclictest20494-21apt-get11:35:150
528499380,23cyclictest29093-21ssh10:19:540
530699372,18cyclictest27294-21ssh11:19:562
528499371,31cyclictest1722-21apt-get12:25:130
528499371,31cyclictest0-21swapper/008:10:140
528499371,26cyclictest30443-21apt-get09:15:150
528499371,21cyclictest19751-21diskstats10:55:160
528499370,26cyclictest3504-21if_eth010:00:180
528499370,21cyclictest5017-21ntp_192.168.11407:40:260
528499370,21cyclictest28868-21ntp_192.168.11408:05:270
529599363,22cyclictest19199-21ssh12:15:161
529599361,29cyclictest27213-21ssh11:00:161
528499362,31cyclictest3898-21ssh10:45:120
528499361,32cyclictest32117-21kthreadcore10:20:210
528499361,32cyclictest22921-21ps12:15:310
528499361,32cyclictest21292-21ntp_192.168.11412:35:300
528499361,31cyclictest28589-21ssh10:40:140
528499361,31cyclictest28589-21ssh10:40:140
528499361,26cyclictest7457-21/usr/sbin/munin08:20:160
528499361,20cyclictest0-21swapper/007:25:120
528499360,30cyclictest22892-21ssh09:10:140
531499352,18cyclictest26866irq/58-eth009:15:143
528499351,31cyclictest0-21swapper/009:00:150
528499351,30cyclictest27202-21apt-get11:00:170
528499351,30cyclictest0-21swapper/009:35:130
528499351,30cyclictest0-21swapper/008:40:130
528499351,29cyclictest24493-21apt-get07:30:130
528499351,20cyclictest9305-21ssh12:07:350
528499351,20cyclictest1833-21ssh11:41:550
528499350,20cyclictest26206-21ssh12:20:160
531499345,20cyclictest26866irq/58-eth012:20:173
531499345,15cyclictest26866irq/58-eth010:30:143
5306993422,8cyclictest0-21swapper/209:50:142
530699342,25cyclictest10215-21ssh10:47:562
528499341,29cyclictest2363-21apt-get11:05:110
528499341,29cyclictest0-21swapper/008:00:160
528499341,22cyclictest22504-21diskmemload10:33:280
528499340,30cyclictest16993-21df08:30:160
528499340,22cyclictest24962-21ssh11:55:440
528499340,22cyclictest10586-21ssh09:43:060
528499331,30cyclictest5568-21tr11:25:220
528499331,28cyclictest4136-21irqrtprio08:50:210
528499331,28cyclictest0-21swapper/010:50:160
528499331,23cyclictest28384-21sshd09:34:480
528499331,21cyclictest18540-21ssh11:32:540
528499331,21cyclictest18540-21ssh11:32:540
528499330,30cyclictest11811-21irqstats07:50:230
528499321,19cyclictest11994-21ssh09:22:510
531499311,18cyclictest26866irq/58-eth011:45:163
528499311,27cyclictest16402-21ssh12:33:380
528499311,26cyclictest14750-21find10:05:330
531499307,12cyclictest0-21swapper/312:25:123
530699304,17cyclictest22046irq/40-dwc2_hso11:50:162
529599303,19cyclictest19324-21ssh11:15:141
529599302,22cyclictest17226-21ssh12:34:551
528499300,24cyclictest0-21swapper/009:05:140
531499293,19cyclictest0-21swapper/312:10:183
5314992914,10cyclictest26866irq/58-eth011:15:143
5314992910,12cyclictest0-21swapper/311:25:093
530699292,16cyclictest22046irq/40-dwc2_hso08:55:152
531499283,16cyclictest0-21swapper/310:47:563
530699283,19cyclictest15963-21ssh12:32:552
531499273,17cyclictest0-21swapper/312:04:563
531499272,13cyclictest26866irq/58-eth009:20:143
530699279,11cyclictest27-21ksoftirqd/211:29:082
530699278,11cyclictest27-21ksoftirqd/211:57:592
530699272,10cyclictest27-21ksoftirqd/212:18:382
5306992710,8cyclictest2609-21sshd12:25:202
5306992710,11cyclictest27-21ksoftirqd/211:37:322
529599272,19cyclictest18556-21ssh11:32:551
529599272,19cyclictest18556-21ssh11:32:551
529599272,18cyclictest20516-21ssh11:35:151
529599271,3cyclictest0-21swapper/109:39:101
529599271,17cyclictest2374-21ssh11:05:121
530699266,8cyclictest27-21ksoftirqd/212:38:482
530699266,11cyclictest131rcu_preempt11:20:422
530699264,9cyclictest27-21ksoftirqd/211:40:182
530699263,3cyclictest131rcu_preempt12:13:502
5306992621,3cyclictest27-21ksoftirqd/210:05:142
530699261,17cyclictest22046irq/40-dwc2_hso10:40:152
530699261,17cyclictest22046irq/40-dwc2_hso10:40:152
529599261,5cyclictest15926-21apt-get07:55:141
5295992612,3cyclictest21-21ksoftirqd/110:40:331
5295992612,3cyclictest21-21ksoftirqd/110:40:331
530699256,9cyclictest131rcu_preempt11:46:312
530699254,11cyclictest131rcu_preempt11:30:302
530699254,11cyclictest131rcu_preempt11:30:302
530699253,3cyclictest0-21swapper/212:23:572
530699252,14cyclictest13538-21apt-get09:25:122
530699251,21cyclictest3798-21grep08:50:192
5306992510,4cyclictest131rcu_preempt09:10:262
529599259,7cyclictest0-21swapper/111:46:531
529599259,2cyclictest21-21ksoftirqd/112:02:331
5295992513,9cyclictest21-21ksoftirqd/111:28:021
529599251,19cyclictest12144-21ssh09:45:151
531499242,12cyclictest26866irq/58-eth010:05:143
530699243,14cyclictest22046irq/40-dwc2_hso09:20:132
530699241,5cyclictest27-21ksoftirqd/212:05:172
5306992412,10cyclictest1-21systemd10:04:562
5306992411,5cyclictest25504-21grep08:35:302
530699240,4cyclictest1149-21sshd12:01:552
529599249,8cyclictest0-21swapper/110:39:591
529599248,2cyclictest21-21ksoftirqd/110:20:171
529599243,19cyclictest11484-21ls12:10:201
5295992411,5cyclictest21-21ksoftirqd/110:19:051
528499241,21cyclictest5274-21cyclictest07:10:010
528499241,21cyclictest5274-21cyclictest07:10:010
530699239,11cyclictest13198-21kthreadcore07:50:252
530699237,7cyclictest27-21ksoftirqd/209:32:292
530699233,3cyclictest131rcu_preempt07:40:252
530699233,11cyclictest28893-21ssh09:35:122
530699232,6cyclictest0-21swapper/207:10:132
5306992312,8cyclictest0-21swapper/208:10:272
5306992311,9cyclictest3352-21ssh09:16:252
530699230,6cyclictest25724-21ls08:05:182
529599239,3cyclictest21-21ksoftirqd/110:05:291
529599238,8cyclictest21-21ksoftirqd/108:35:161
529599238,5cyclictest21-21ksoftirqd/108:15:191
529599237,3cyclictest21-21ksoftirqd/111:50:431
529599237,2cyclictest21-21ksoftirqd/110:57:031
529599234,3cyclictest131rcu_preempt12:20:181
529599232,5cyclictest131rcu_preempt11:59:221
5295992315,5cyclictest5101-21ssh10:25:171
5295992311,6cyclictest0-21swapper/108:10:291
5295992311,3cyclictest21-21ksoftirqd/111:20:161
529599231,10cyclictest21-21ksoftirqd/110:10:331
531499221,17cyclictest0-21swapper/310:10:163
530699229,7cyclictest11903-21apt-get10:50:162
530699228,8cyclictest27-21ksoftirqd/208:45:292
530699226,4cyclictest27-21ksoftirqd/207:30:012
530699223,12cyclictest27-21ksoftirqd/211:08:502
5306992210,5cyclictest25151-21sshd10:56:552
530699220,15cyclictest11076-21grep11:10:162
529599228,4cyclictest21-21ksoftirqd/107:50:341
529599227,8cyclictest21-21ksoftirqd/110:02:381
529599225,7cyclictest131rcu_preempt07:45:241
529599225,3cyclictest131rcu_preempt08:25:321
529599225,3cyclictest131rcu_preempt08:25:321
529599224,4cyclictest131rcu_preempt09:40:151
529599221,4cyclictest27559-21kworker/1:109:11:331
5295992212,6cyclictest131rcu_preempt08:45:151
529599221,16cyclictest17685-21rm12:35:171
5295992211,3cyclictest21-21ksoftirqd/112:08:201
5295992211,3cyclictest0-21swapper/112:29:211
5295992210,9cyclictest21-21ksoftirqd/109:22:391
5295992210,3cyclictest0-21swapper/108:20:151
531499219,6cyclictest0-21swapper/311:35:153
531499212,13cyclictest26866irq/58-eth010:20:113
530699219,5cyclictest7101-21sshd08:19:332
530699218,3cyclictest131rcu_preempt09:05:242
530699216,8cyclictest27-21ksoftirqd/209:44:242
530699214,5cyclictest27-21ksoftirqd/207:45:212
530699214,3cyclictest131rcu_preempt08:20:312
530699213,3cyclictest0-21swapper/210:40:012
530699213,2cyclictest0-21swapper/208:25:282
530699213,2cyclictest0-21swapper/208:25:282
530699212,8cyclictest27-21ksoftirqd/207:30:132
530699212,7cyclictest131rcu_preempt09:00:212
530699212,15cyclictest22130-21missed_timers10:10:272
5306992115,4cyclictest261rcuc/207:15:292
530699211,3cyclictest0-21swapper/207:55:262
530699211,11cyclictest30747-21missed_timers11:00:272
529599219,4cyclictest21-21ksoftirqd/109:55:311
529599215,3cyclictest131rcu_preempt11:10:191
529599215,10cyclictest21-21ksoftirqd/110:50:161
529599214,8cyclictest21-21ksoftirqd/109:30:191
529599213,7cyclictest8170-21cron08:55:011
529599210,6cyclictest13538-21irqstats10:30:211
531499204,10cyclictest26866irq/58-eth010:40:163
531499204,10cyclictest26866irq/58-eth010:40:163
5314992011,6cyclictest0-21swapper/310:55:163
530699205,7cyclictest27-21ksoftirqd/210:20:202
530699203,14cyclictest13292-21irqcore10:30:212
530699201,5cyclictest27-21ksoftirqd/208:30:242
530699201,11cyclictest0-21swapper/210:15:322
530699201,10cyclictest20896-21fschecks_time08:00:202
530699200,15cyclictest4577-21apt-get10:25:042
529599209,4cyclictest21-21ksoftirqd/107:15:191
529599209,2cyclictest21-21ksoftirqd/110:45:131
529599209,2cyclictest21-21ksoftirqd/108:05:201
529599207,4cyclictest21-21ksoftirqd/109:05:151
529599206,3cyclictest131rcu_preempt07:35:231
529599205,7cyclictest21-21ksoftirqd/108:00:251
529599203,8cyclictest131rcu_preempt07:10:251
531499193,11cyclictest0-21swapper/312:15:193
531499193,11cyclictest0-21swapper/311:40:123
531499193,11cyclictest0-21swapper/311:32:553
531499193,11cyclictest0-21swapper/311:32:553
531499193,11cyclictest0-21swapper/311:20:083
531499192,12cyclictest0-21swapper/312:34:553
531499192,11cyclictest0-21swapper/308:35:143
531499191,13cyclictest0-21swapper/309:10:373
531499191,12cyclictest0-21swapper/308:20:303
531499191,11cyclictest0-21swapper/307:15:143
5314991910,5cyclictest26866irq/58-eth010:27:483
530699196,10cyclictest0-21swapper/208:40:202
530699192,3cyclictest451-21sed07:35:322
530699191,5cyclictest18388-21ntp_offset07:20:292
530699191,3cyclictest0-21swapper/209:55:132
529599193,12cyclictest0-21swapper/107:30:131
529599191,11cyclictest20129-21ls08:30:251
529599191,10cyclictest21-21ksoftirqd/108:55:271
531499189,6cyclictest26866irq/58-eth009:45:143
531499189,6cyclictest0-21swapper/311:07:123
531499186,7cyclictest0-21swapper/309:50:133
531499183,10cyclictest0-21swapper/311:55:153
531499183,10cyclictest0-21swapper/311:50:143
531499183,10cyclictest0-21swapper/310:50:043
531499182,10cyclictest0-21swapper/308:55:153
531499182,10cyclictest0-21swapper/308:40:053
531499182,10cyclictest0-21swapper/308:05:153
531499182,10cyclictest0-21swapper/307:25:143
531499181,9cyclictest0-21swapper/307:10:113
5314991812,4cyclictest0-21swapper/309:30:263
531499181,14cyclictest0-21swapper/309:40:323
531499181,11cyclictest0-21swapper/312:05:253
5314991810,6cyclictest0-21swapper/309:25:523
529599187,8cyclictest13201-21apt-get09:00:151
529599187,3cyclictest131rcu_preempt07:40:211
529599182,3cyclictest131rcu_preempt07:25:211
531499177,6cyclictest26866irq/58-eth011:10:133
531499174,9cyclictest26866irq/58-eth010:35:173
531499172,9cyclictest0-21swapper/308:10:153
531499172,12cyclictest0-21swapper/310:00:203
531499172,10cyclictest0-21swapper/308:30:153
531499172,10cyclictest0-21swapper/308:00:173
531499172,10cyclictest0-21swapper/307:50:173
531499172,10cyclictest0-21swapper/307:20:173
529599175,4cyclictest21-21ksoftirqd/107:20:151
529599171,8cyclictest21-21ksoftirqd/108:40:211
531499168,6cyclictest0-21swapper/310:15:043
531499162,9cyclictest0-21swapper/308:45:163
531499162,10cyclictest0-21swapper/307:45:143
531499161,9cyclictest0-21swapper/309:10:013
531499151,7cyclictest0-21swapper/307:55:133
531499151,11cyclictest0-21swapper/307:30:243
531499143,6cyclictest0-21swapper/308:25:173
531499143,6cyclictest0-21swapper/308:25:173
531499141,7cyclictest0-21swapper/308:50:203
531499141,7cyclictest0-21swapper/308:20:013
530699141,9cyclictest5388-21latency_hist07:10:012
530699141,9cyclictest5388-21latency_hist07:10:012
529599131,9cyclictest5298-21latency_hist07:10:011
529599131,9cyclictest5298-21latency_hist07:10:011
531499111,7cyclictest0-21swapper/307:40:173
531499101,6cyclictest0-21swapper/307:10:013
531499101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional