You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 04:06
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue May 13, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4806991033,76cyclictest29308-21apt-get22:30:182
47819910084,8cyclictest0-21swapper/023:40:010
4795999881,10cyclictest0-21swapper/119:35:411
4795999881,10cyclictest0-21swapper/119:35:411
4806999768,13cyclictest181rcu_preempt00:25:192
480699969,66cyclictest25439-21apt-get20:10:142
480699953,69cyclictest28668-21apt-get00:15:132
4806999357,12cyclictest181rcu_preempt00:15:022
4806999345,12cyclictest12929-21ssh22:56:112
4806999345,12cyclictest12929-21ssh22:56:112
4806999331,17cyclictest0-21swapper/222:15:232
4795999374,11cyclictest0-21swapper/100:00:361
4781999378,8cyclictest0-21swapper/022:00:330
4806999255,17cyclictest181rcu_preempt00:36:532
4806999255,17cyclictest181rcu_preempt00:36:532
4806999255,14cyclictest181rcu_preempt21:10:232
4806999247,13cyclictest24813-21ssh23:15:242
4806999233,40cyclictest0-21swapper/200:20:442
4806999233,40cyclictest0-21swapper/200:20:442
4806999161,8cyclictest181rcu_preempt21:39:122
4781999170,8cyclictest0-21swapper/021:25:100
38672917,55sleep23908-21processes19:05:422
38672917,55sleep23908-21processes19:05:422
4806999064,16cyclictest3452-21ssh21:47:332
4806999054,15cyclictest181rcu_preempt00:00:392
4806999034,11cyclictest0-21swapper/220:00:032
4806999034,11cyclictest0-21swapper/220:00:032
4806998866,15cyclictest23783-21sh22:20:342
4806998862,9cyclictest181rcu_preempt22:54:592
4806998761,17cyclictest376-21in:imuxsock21:19:582
4806998751,10cyclictest181rcu_preempt21:10:012
4806998742,16cyclictest181rcu_preempt22:06:082
4806998736,26cyclictest4772-21cyclictest21:50:092
4795998768,11cyclictest27317-21ssh23:19:581
4795998757,7cyclictest181rcu_preempt00:28:061
4795998756,7cyclictest181rcu_preempt00:30:201
480699869,67cyclictest13669-21apt-get19:35:142
480699869,67cyclictest13669-21apt-get19:35:142
4806998648,14cyclictest181rcu_preempt21:43:222
4806998629,22cyclictest0-21swapper/223:30:172
4806998629,22cyclictest0-21swapper/223:30:172
4806998562,17cyclictest11256-21latency22:00:242
4806998561,10cyclictest181rcu_preempt23:35:392
4806998557,10cyclictest181rcu_preempt23:25:292
4806998553,13cyclictest181rcu_preempt20:30:402
4806998553,13cyclictest181rcu_preempt20:30:402
4806998551,12cyclictest181rcu_preempt23:24:012
4806998548,13cyclictest181rcu_preempt23:40:362
4806998530,14cyclictest0-21swapper/222:49:372
4806998453,8cyclictest181rcu_preempt22:25:352
4806998453,8cyclictest181rcu_preempt22:25:352
4806998448,8cyclictest181rcu_preempt21:30:172
4806998448,8cyclictest181rcu_preempt21:30:172
4806998443,11cyclictest181rcu_preempt21:25:252
4806998442,13cyclictest181rcu_preempt22:40:032
4806998436,9cyclictest181rcu_preempt00:06:472
478199848,26cyclictest18518-21ssh23:05:170
37802848,47sleep03800-21ntp_kernel_pll_19:05:360
37802848,47sleep03800-21ntp_kernel_pll_19:05:360
4806998355,10cyclictest181rcu_preempt23:08:172
4806998349,12cyclictest181rcu_preempt19:10:212
4806998349,12cyclictest181rcu_preempt19:10:212
4806998331,10cyclictest0-21swapper/223:10:362
4806998327,25cyclictest0-21swapper/223:50:172
4795998369,7cyclictest0-21swapper/119:25:151
4795998368,11cyclictest4772-21cyclictest20:40:291
4795998361,15cyclictest202-21systemd-journal20:19:591
4795998358,9cyclictest181rcu_preempt23:32:231
4795998358,9cyclictest181rcu_preempt23:32:231
4795998353,7cyclictest181rcu_preempt22:52:411
4781998361,14cyclictest23413-21latency_hist20:05:010
4806998253,13cyclictest181rcu_preempt21:20:362
4806998251,13cyclictest181rcu_preempt23:45:292
4806998251,13cyclictest181rcu_preempt23:45:292
4806998250,7cyclictest181rcu_preempt20:45:012
4795998263,15cyclictest12390-21kworker/u17:4+events_unbound23:44:581
4795998254,7cyclictest181rcu_preempt22:01:201
4795998253,7cyclictest181rcu_preempt21:59:081
4781998263,13cyclictest1490-21latency_hist21:45:020
4806998155,9cyclictest181rcu_preempt19:15:382
4806998144,8cyclictest18355-21ssh22:11:332
4806998144,8cyclictest18355-21ssh22:11:332
4806998142,13cyclictest181rcu_preempt21:01:242
4795998161,11cyclictest0-21swapper/100:10:001
4795998160,10cyclictest0-21swapper/120:15:061
4795998160,10cyclictest0-21swapper/119:20:031
4795998154,9cyclictest181rcu_preempt00:23:591
4795998154,9cyclictest181rcu_preempt00:23:591
4795998154,7cyclictest181rcu_preempt00:18:241
4795998153,8cyclictest181rcu_preempt00:11:281
4795998152,7cyclictest181rcu_preempt21:21:321
4795998150,9cyclictest181rcu_preempt23:24:171
4795998149,7cyclictest181rcu_preempt23:55:481
4781998165,10cyclictest31359-21ssh00:20:180
4781998165,10cyclictest31359-21ssh00:20:180
4806998035,17cyclictest0-21swapper/220:00:452
4795998057,7cyclictest181rcu_preempt21:41:471
4795998055,9cyclictest181rcu_preempt22:24:481
4795998054,7cyclictest181rcu_preempt23:37:401
4795998053,7cyclictest181rcu_preempt21:39:221
4795998052,7cyclictest181rcu_preempt21:29:461
4781998067,7cyclictest0-21swapper/020:35:360
4781998067,7cyclictest0-21swapper/020:35:360
4781998060,14cyclictest202-21systemd-journal00:15:000
4781998060,10cyclictest0-21swapper/022:25:300
4781998060,10cyclictest0-21swapper/022:25:300
4806997946,13cyclictest181rcu_preempt21:59:462
4806997941,9cyclictest181rcu_preempt19:20:202
4806997940,8cyclictest181rcu_preempt23:56:582
4795997964,9cyclictest5165-21kworker/u17:3-xprtiod22:45:281
4795997954,8cyclictest181rcu_preempt22:10:581
4795997954,8cyclictest181rcu_preempt22:10:581
4795997953,7cyclictest181rcu_preempt22:39:261
4795997951,7cyclictest181rcu_preempt23:49:351
4795997951,7cyclictest181rcu_preempt23:49:351
4795997951,7cyclictest181rcu_preempt22:06:121
4781997960,9cyclictest0-21swapper/021:50:010
478199793,39cyclictest24378-21seq22:21:210
4806997832,22cyclictest0-21swapper/200:35:002
4806997831,17cyclictest0-21swapper/222:40:172
4806997829,25cyclictest0-21swapper/220:35:462
4806997829,25cyclictest0-21swapper/220:35:462
4795997860,10cyclictest0-21swapper/123:04:011
4795997859,9cyclictest0-21swapper/121:45:361
4795997855,15cyclictest26620-21ntp_kernel_pll_22:25:301
4795997855,15cyclictest26620-21ntp_kernel_pll_22:25:301
4795997853,8cyclictest181rcu_preempt19:51:551
4795997817,31cyclictest31-21ksoftirqd/121:31:131
4795997817,31cyclictest31-21ksoftirqd/121:31:131
4781997817,12cyclictest0-21swapper/000:33:330
4806997744,8cyclictest181rcu_preempt20:15:332
4806997743,10cyclictest181rcu_preempt20:05:362
4806997740,13cyclictest181rcu_preempt20:56:222
4806997740,13cyclictest181rcu_preempt20:56:222
4806997738,13cyclictest181rcu_preempt19:45:212
4806997738,13cyclictest181rcu_preempt19:45:212
4806997734,9cyclictest181rcu_preempt19:27:132
4806997731,10cyclictest0-21swapper/219:30:262
4806997727,31cyclictest0-21swapper/219:40:232
4806997727,31cyclictest0-21swapper/219:40:232
4806997727,27cyclictest0-21swapper/220:27:282
4806997727,14cyclictest0-21swapper/220:50:402
4795997760,11cyclictest29985-21ssh22:30:471
4795997752,8cyclictest181rcu_preempt20:27:431
4795997752,7cyclictest181rcu_preempt21:10:441
479599772,29cyclictest31-21ksoftirqd/123:25:461
4781997762,10cyclictest4772-21cyclictest20:10:460
4781997723,19cyclictest0-21swapper/021:57:450
4806997642,10cyclictest181rcu_preempt19:50:182
4806997626,8cyclictest0-21swapper/220:45:472
4795997660,9cyclictest0-21swapper/119:15:011
4795997660,9cyclictest0-21swapper/119:15:011
4795997657,14cyclictest10481-21latency_hist21:05:031
4795997655,15cyclictest381-21lldpd20:00:001
4795997655,15cyclictest381-21lldpd20:00:001
4795997649,8cyclictest181rcu_preempt00:36:451
4795997649,8cyclictest181rcu_preempt00:36:451
4795997649,10cyclictest181rcu_preempt20:50:501
4795997642,16cyclictest21749-21ssh23:10:231
4781997661,10cyclictest4772-21cyclictest22:56:110
4781997661,10cyclictest4772-21cyclictest22:56:110
4781997660,9cyclictest0-21swapper/023:25:330
4781997658,8cyclictest0-21swapper/022:05:370
4781997657,9cyclictest0-21swapper/020:45:000
4781997657,9cyclictest0-21swapper/019:15:370
4781997657,14cyclictest24777-21sh23:15:200
4781997657,10cyclictest0-21swapper/021:35:000
4781997657,10cyclictest0-21swapper/021:35:000
4781997617,34cyclictest0-21swapper/021:13:100
461427612,39sleep30-21swapper/319:09:593
461427612,39sleep30-21swapper/319:09:593
4806997526,26cyclictest0-21swapper/223:00:262
4795997556,14cyclictest12930-21fschecks_time22:56:111
4795997556,14cyclictest12930-21fschecks_time22:56:111
4795997551,8cyclictest181rcu_preempt19:40:371
4795997551,8cyclictest181rcu_preempt19:40:371
4795997547,7cyclictest181rcu_preempt23:53:401
4781997543,23cyclictest23788-21tune2fs20:05:210
4806997442,12cyclictest181rcu_preempt20:20:362
4795997460,8cyclictest0-21swapper/120:05:021
4795997448,7cyclictest181rcu_preempt22:16:131
4795997445,7cyclictest181rcu_preempt22:42:121
4781997455,8cyclictest0-21swapper/022:45:020
4781997429,37cyclictest20809-21grep19:55:340
4781997429,37cyclictest20809-21grep19:55:340
4795997360,8cyclictest0-21swapper/123:05:231
4795997358,8cyclictest0-21swapper/120:30:231
4795997358,8cyclictest0-21swapper/120:30:231
4795997346,9cyclictest181rcu_preempt21:15:411
4795997346,7cyclictest181rcu_preempt21:00:131
4795997346,7cyclictest181rcu_preempt21:00:131
4781997359,10cyclictest4772-21cyclictest23:02:410
4781997356,9cyclictest0-21swapper/019:25:310
4781997356,8cyclictest0-21swapper/022:15:240
4781997356,7cyclictest0-21swapper/020:15:160
4781997355,9cyclictest0-21swapper/000:25:270
4781997350,17cyclictest32180-21latency_hist22:35:020
4781997342,23cyclictest7235-21cstates00:35:180
4781997342,23cyclictest7235-21cstates00:35:180
4781997339,21cyclictest12390-21kworker/u17:4+nfsiod23:40:070
4781997318,10cyclictest0-21swapper/023:30:150
4781997318,10cyclictest0-21swapper/023:30:150
4795997258,8cyclictest0-21swapper/120:40:051
4795997258,8cyclictest0-21swapper/120:40:051
4795997248,7cyclictest181rcu_preempt21:52:451
4781997256,9cyclictest14195-21ntp_kernel_pll_23:50:280
4781997253,9cyclictest0-21swapper/019:35:140
4781997253,9cyclictest0-21swapper/019:35:140
4781997252,13cyclictest12327-21diskmemload22:35:180
4781997251,15cyclictest907-21latency_hist20:35:140
4781997251,15cyclictest907-21latency_hist20:35:140
4781997243,21cyclictest18240-21sh19:49:580
4781997243,21cyclictest18240-21sh19:49:580
4781997239,23cyclictest21549-21munin-plugin-st21:24:590
478199723,38cyclictest16943-21ssh22:10:150
478199723,38cyclictest16943-21ssh22:10:150
478199719,27cyclictest5402-21kworker/u17:0+xprtiod21:51:130
4781997158,9cyclictest4772-21cyclictest00:01:000
4781997157,9cyclictest4772-21cyclictest22:45:510
4781997156,9cyclictest10658-21ssh23:45:170
4781997156,9cyclictest10658-21ssh23:45:170
4795997051,9cyclictest0-21swapper/120:45:131
4795997050,13cyclictest4772-21cyclictest20:20:231
4795997049,15cyclictest379-21cron19:34:591
4795997041,6cyclictest181rcu_preempt21:06:181
4795997015,24cyclictest31-21ksoftirqd/119:49:281
4795997015,24cyclictest31-21ksoftirqd/119:49:281
478199706,28cyclictest11070-21cpu21:06:140
4781997057,7cyclictest0-21swapper/021:04:580
478199696,30cyclictest15-21ksoftirqd/022:52:430
478199696,30cyclictest15-21ksoftirqd/021:37:310
4781996957,8cyclictest19102-21kworker/u19:2+rpciod00:07:020
4781996953,7cyclictest0-21swapper/020:20:450
4781996952,7cyclictest0-21swapper/023:20:180
4781996944,17cyclictest16580-21kworker/u17:2+nfsiod23:14:500
4781996916,12cyclictest30806-21grep20:25:340
4781996815,16cyclictest29629-21timerandwakeup00:15:490
4781996810,31cyclictest15-21ksoftirqd/019:14:020
4781996810,31cyclictest15-21ksoftirqd/019:14:020
4795996750,12cyclictest0-21swapper/120:10:031
4795996735,9cyclictest58-21kcompactd019:21:191
4781996755,7cyclictest0-21swapper/019:25:010
478199675,27cyclictest17865-21ssh23:55:560
4781996751,10cyclictest1-21systemd19:35:000
4781996650,10cyclictest58-21kcompactd020:45:190
4781996640,20cyclictest5402-21kworker/u17:0+nfsiod21:15:590
478199662,32cyclictest15-21ksoftirqd/019:43:050
478199662,32cyclictest15-21ksoftirqd/019:43:050
4781996548,9cyclictest9135-21cat21:00:170
4814996450,9cyclictest451rcuc/320:16:413
4781996444,14cyclictest0-21swapper/020:59:580
4781996417,14cyclictest6437-21tr20:50:470
4781996416,11cyclictest18492-21/usr/sbin/munin19:50:370
4814996249,9cyclictest451rcuc/323:51:033
4814996148,9cyclictest451rcuc/323:17:083
35622608,32sleep13588-21fw_packets19:05:291
35622608,32sleep13588-21fw_packets19:05:291
4814995946,9cyclictest451rcuc/322:01:203
4814995742,10cyclictest451rcuc/321:11:493
4814995642,9cyclictest451rcuc/322:20:243
481499543,42cyclictest22003-21chrt20:00:243
481499538,32cyclictest12522-21chrt21:10:023
4814995138,8cyclictest451rcuc/323:56:103
4814995138,8cyclictest451rcuc/322:09:383
481499513,38cyclictest23898-21taskset21:28:053
481499513,37cyclictest6062-21chrt22:45:153
481499504,37cyclictest29295-21taskset23:21:573
481499504,33cyclictest4428-21taskset00:30:203
4814995037,8cyclictest451rcuc/319:28:053
481499496,33cyclictest5719-21taskset22:44:583
4814994935,9cyclictest451rcuc/300:13:563
4814994935,10cyclictest451rcuc/321:24:283
481499493,36cyclictest22667-21taskset22:19:593
481499493,36cyclictest21751-21taskset23:10:233
481499493,36cyclictest14661-21taskset22:58:523
481499493,36cyclictest14661-21taskset22:58:523
481499484,34cyclictest31877-21chrt20:30:173
481499484,34cyclictest10179-21taskset22:50:383
481499483,35cyclictest19902-21chrt23:06:303
481499483,34cyclictest3117-21chrt20:40:303
481499483,34cyclictest20389-21taskset19:55:223
481499483,34cyclictest20389-21taskset19:55:223
481499483,23cyclictest28762-21chrt00:15:293
481499474,34cyclictest477-21chrt21:42:473
481499474,33cyclictest6655-21taskset23:38:433
481499474,33cyclictest3909-21chrt21:48:473
481499474,32cyclictest1690-21chrt20:35:443
481499474,32cyclictest1690-21chrt20:35:443
481499474,31cyclictest32179-21chrt22:35:023
4814994734,9cyclictest451rcuc/319:37:013
4814994734,9cyclictest451rcuc/319:37:013
481499473,35cyclictest13365-21taskset23:50:013
481499473,35cyclictest13365-21taskset23:50:013
481499473,34cyclictest3863-21chrt00:29:273
481499473,34cyclictest2665-21chrt23:31:133
481499473,34cyclictest2665-21chrt23:31:133
481499473,34cyclictest19809-21chrt00:00:183
481499473,33cyclictest15611-21chrt19:40:243
481499473,33cyclictest15611-21chrt19:40:243
481499473,32cyclictest29146-21chrt22:30:033
481499473,32cyclictest29146-21chrt22:30:033
481499472,35cyclictest2363-21taskset22:39:153
481499464,33cyclictest7352-21taskset00:35:223
481499464,33cyclictest7352-21taskset00:35:223
481499464,31cyclictest30216-21chrt21:38:463
481499464,31cyclictest23500-21chrt00:05:373
4814994633,7cyclictest0-21swapper/320:10:293
481499463,34cyclictest4859-21chrt21:50:173
481499454,30cyclictest18160-21chrt22:10:593
481499454,30cyclictest18160-21chrt22:10:593
481499453,33cyclictest7250-21taskset19:15:253
481499453,32cyclictest9893-21taskset21:58:273
481499453,32cyclictest8578-21chrt23:40:373
481499453,32cyclictest757-21chrt23:29:193
481499443,31cyclictest13282-21chrt19:35:013
481499442,32cyclictest17596-21chrt19:45:343
481499442,32cyclictest17596-21chrt19:45:343
481499442,31cyclictest18197-21taskset23:05:003
481499433,29cyclictest29708-21taskset20:21:063
481499432,30cyclictest8047-21taskset20:55:403
481499432,30cyclictest8047-21taskset20:55:403
481499423,30cyclictest27072-21chrt21:33:283
481499423,30cyclictest27072-21chrt21:33:283
481499423,29cyclictest19233-21chrt19:50:363
481499423,29cyclictest16100-21taskset21:15:253
481499423,28cyclictest8784-21taskset19:20:233
481499423,26cyclictest4265-21taskset20:45:073
481499422,30cyclictest6321-21taskset19:15:003
481499422,30cyclictest6321-21taskset19:15:003
481499422,30cyclictest23605-21taskset20:05:143
481499414,27cyclictest31979-21taskset00:20:403
481499414,27cyclictest31979-21taskset00:20:403
481499412,30cyclictest7149-21taskset20:54:023
481499412,29cyclictest9147-21chrt21:00:183
481499403,26cyclictest535-21taskset20:31:593
481499403,26cyclictest535-21taskset20:31:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional