You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-14 - 10:07
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sun Sep 14, 2025 00:44:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
723223860,20sleep17230-21cron19:01:381
6055215112,102sleep05888-21/usr/sbin/munin18:57:010
73669912637,52cyclictest0-21swapper/021:37:190
7366991122,23cyclictest16798-21sh21:04:460
7366991112,28cyclictest6914-21rm21:44:050
7366991112,25cyclictest8944-21ssh23:43:440
7366991112,25cyclictest21938-21ssh21:12:390
7366991103,26cyclictest10369-21grep19:07:200
7366991103,26cyclictest10369-21grep19:07:200
7366991102,23cyclictest27858-21ssh21:23:310
7375991093,70cyclictest12847-21latency_hist23:51:401
7375991093,70cyclictest12847-21latency_hist23:51:401
73759910850,31cyclictest181rcu_preempt21:32:261
7366991082,27cyclictest17073-21ssh23:01:120
7366991082,24cyclictest12437-21ssh21:54:100
7366991073,65cyclictest0-21swapper/019:57:180
73669910733,63cyclictest0-21swapper/023:47:020
73669910733,63cyclictest0-21swapper/023:47:020
7366991072,79cyclictest21431-21apt-get00:06:510
73669910713,59cyclictest0-21swapper/019:05:120
73669910713,59cyclictest0-21swapper/019:05:120
7366991062,24cyclictest22360-21taskset19:45:210
7366991062,23cyclictest19691-21ssh23:05:450
7366991052,23cyclictest17632-21seq23:59:230
7366991042,64cyclictest11961-21seq22:51:540
7366991042,24cyclictest8612-21ntp_states21:47:110
7366991042,24cyclictest8612-21ntp_states21:47:110
7366991032,22cyclictest27516-21ssh23:19:070
7366991032,22cyclictest27516-21ssh23:19:070
73669910220,72cyclictest0-21swapper/023:22:120
73669910219,48cyclictest0-21swapper/022:18:110
7366991013,88cyclictest0-21swapper/022:31:540
7366991013,7cyclictest0-21swapper/023:36:550
7366991013,6cyclictest0-21swapper/021:16:530
7366991013,6cyclictest0-21swapper/021:16:530
7366991012,24cyclictest3180-21taskset20:26:310
7366991012,19cyclictest27011-21kworker/u17:0+xprtiod22:26:150
73669910116,55cyclictest12039-21ntpq19:12:550
660521017,65sleep26645-21/usr/sbin/munin18:57:282
7375999946,32cyclictest181rcu_preempt22:37:051
736699993,22cyclictest14845-21runrttasks23:54:260
739099984,34cyclictest9675-21latency_hist20:46:412
736699984,83cyclictest13195-21apt-get20:56:530
7366999847,44cyclictest15258-21ssh21:59:190
7366999843,48cyclictest29814-21ssh00:22:060
7366999843,48cyclictest29814-21ssh00:22:060
736699982,65cyclictest10667-21ssh22:48:440
736699982,65cyclictest10667-21ssh22:48:440
736699982,22cyclictest318-21runrttasks23:28:070
736699982,22cyclictest318-21runrttasks23:28:070
736699982,21cyclictest32260-21runrttasks20:14:370
736699972,78cyclictest24183-21ssh00:11:500
736699972,78cyclictest24183-21ssh00:11:500
736699972,20cyclictest19504-21/usr/sbin/munin19:37:060
7390999640,48cyclictest0-21swapper/222:27:052
739099963,33cyclictest18830-21latency_hist22:06:402
739099963,33cyclictest18830-21latency_hist22:06:402
736699962,61cyclictest32320-21ssh21:32:110
736699962,28cyclictest29277-21cstates21:26:550
736699962,28cyclictest29277-21cstates21:26:550
736699953,21cyclictest25480-21runrttasks23:16:280
736699952,21cyclictest29090-21ssh00:21:120
739099944,33cyclictest545-21/usr/sbin/munin22:31:562
739099943,34cyclictest8290-21switchtime19:02:132
739099943,34cyclictest8290-21switchtime19:02:132
739099942,41cyclictest7718-21vmstat20:37:242
736699943,80cyclictest0-21swapper/020:46:540
736699942,23cyclictest16919-21ntp_kernel_pll_22:02:150
736699942,23cyclictest16919-21ntp_kernel_pll_22:02:150
736699942,21cyclictest19471-21latency_hist19:36:410
739099932,37cyclictest356-21runrttasks22:22:112
736699934,77cyclictest29454-21apt-get20:06:530
736699934,77cyclictest29454-21apt-get20:06:530
7366999332,55cyclictest19186-21sh00:02:070
7366999324,50cyclictest0-21swapper/022:12:300
7366999324,50cyclictest0-21swapper/022:12:300
736699932,23cyclictest1979-21taskset00:29:450
739099924,34cyclictest12198-21wc20:52:162
739099923,37cyclictest26558-21/usr/sbin/munin21:21:592
739099923,35cyclictest3760-21idleruntime20:27:032
739099923,33cyclictest11189-21latency_hist19:11:412
739099923,33cyclictest11189-21latency_hist19:11:412
736699923,78cyclictest22955-21latency_hist23:11:410
736699923,78cyclictest22955-21latency_hist23:11:410
736699923,59cyclictest0-21swapper/022:38:510
7366999235,51cyclictest23148-21latency19:47:050
7366999234,47cyclictest0-21swapper/020:46:410
739099914,38cyclictest32511-21latency_hist20:16:402
739099914,35cyclictest11796-21fschecks_time19:12:492
739099914,34cyclictest23417-21latency_hist21:16:412
739099913,33cyclictest17934-21grep19:31:552
739099913,33cyclictest17934-21grep19:31:552
739099913,33cyclictest10521-21cut23:47:012
739099913,33cyclictest10521-21cut23:47:012
739099913,32cyclictest356-21runrttasks22:43:242
739099912,34cyclictest15343-21cat21:02:042
7366999137,48cyclictest17073-21timerandwakeup19:27:160
7366999137,48cyclictest17073-21timerandwakeup19:27:160
7366999131,52cyclictest0-21swapper/022:07:010
736699912,74cyclictest3477-21apt-get20:26:510
736699912,25cyclictest30216-21apt-get22:26:500
739099904,35cyclictest7995-21/usr/sbin/munin21:47:102
739099904,35cyclictest7995-21/usr/sbin/munin21:47:102
739099904,34cyclictest24851-21fschecks_count22:17:062
739099903,37cyclictest356-21runrttasks20:57:002
739099903,34cyclictest6446-21latency_hist20:36:542
739099903,34cyclictest6446-21latency_hist20:36:542
739099903,34cyclictest19132-21ssh00:02:062
739099903,33cyclictest1597-21latency_hist20:21:422
739099902,34cyclictest11775-21unixbench_multi21:52:172
7375999023,34cyclictest31-21ksoftirqd/123:39:081
736699902,80cyclictest24679-21taskset19:52:360
736699902,74cyclictest27837-21apt-get20:01:520
739099893,35cyclictest24896-21fschecks_count19:52:452
739099893,35cyclictest19868-21switchtime22:07:162
7375998960,22cyclictest18999-21fschecks_time00:02:031
736699893,78cyclictest13070-21/usr/sbin/munin19:16:520
736699893,72cyclictest0-21swapper/019:31:510
7366998925,54cyclictest1155-21sed20:17:170
736699892,25cyclictest14681-21apt-get19:21:530
739099883,34cyclictest12839-21grep22:52:172
739099883,31cyclictest20137-21grep19:37:112
739099883,31cyclictest15604-21systemd-run23:56:382
739099883,29cyclictest3022-21apt-get22:36:512
739099882,33cyclictest4596-21/usr/sbin/munin23:37:092
7390998817,23cyclictest29384-21fschecks_count21:27:002
7390998817,23cyclictest29384-21fschecks_count21:27:002
736699884,73cyclictest0-21swapper/022:41:520
736699883,27cyclictest18540-21/usr/sbin/munin21:07:130
736699883,25cyclictest7291-21/usr/sbin/munin20:37:180
7390998762,18cyclictest21885-21threads19:42:142
739099874,31cyclictest356-21runrttasks20:06:032
7390998735,45cyclictest14548-21diskmemload21:57:172
739099873,35cyclictest26225-21latency_hist21:21:412
739099873,35cyclictest26225-21latency_hist21:21:412
739099873,33cyclictest20286-21/usr/sbin/munin23:07:032
739099873,33cyclictest20286-21/usr/sbin/munin23:07:032
739099873,32cyclictest14409-21apt19:21:382
739099872,38cyclictest21907-21latency00:07:062
739099872,35cyclictest25062-21timerandwakeup00:12:152
739099872,35cyclictest25062-21timerandwakeup00:12:152
7390998723,53cyclictest0-21swapper/221:41:402
739099872,34cyclictest15136-21netstat19:22:092
739099872,32cyclictest7607-21ssh23:42:012
737599874,65cyclictest24612-21apt-get19:51:521
736699873,24cyclictest2832-21unixbench_multi23:32:140
739099863,31cyclictest29436-21/usr/sbin/munin20:07:232
739099863,31cyclictest29436-21/usr/sbin/munin20:07:232
739099862,35cyclictest18466-21sendmail_mailqu21:07:122
7375998660,18cyclictest15068-21fschecks_time22:57:031
737599862,42cyclictest31-21ksoftirqd/119:17:481
7375998611,38cyclictest31-21ksoftirqd/123:58:271
736699864,69cyclictest5113-21apt-get20:31:530
736699864,69cyclictest5113-21apt-get20:31:530
736699864,68cyclictest11584-21apt-get20:51:550
739099854,27cyclictest32077-21cron00:26:372
739099854,27cyclictest32077-21cron00:26:372
7390998533,44cyclictest0-21swapper/220:47:062
739099853,33cyclictest16150-21fschecks_count23:57:032
739099853,31cyclictest711-21ssh00:27:522
739099853,31cyclictest27147-21grep19:57:192
739099853,30cyclictest24276-21latency_hist19:51:412
739099852,34cyclictest31666-21fschecks_count23:27:002
739099852,34cyclictest31666-21fschecks_count23:27:002
737599853,62cyclictest29779-21ssh00:21:501
737599853,62cyclictest29779-21ssh00:21:501
7390998467,11cyclictest331-21dbus-daemon21:36:392
739099844,26cyclictest1843-21/usr/sbin/munin20:22:022
739099843,31cyclictest7780-21latency_hist21:46:402
739099842,28cyclictest16266-21sshd19:26:502
739099842,28cyclictest16266-21sshd19:26:502
7375998456,19cyclictest31791-21apt-key21:31:421
7375998456,19cyclictest31791-21apt-key21:31:421
739099834,27cyclictest356-21runrttasks23:12:452
739099833,30cyclictest15584-21switchtime22:57:162
7390998331,45cyclictest356-21runrttasks23:31:492
7390998327,48cyclictest0-21swapper/200:16:522
739099832,35cyclictest28430-21latency_hist23:21:402
739099832,35cyclictest28430-21latency_hist23:21:402
7390998315,23cyclictest17701-21expr23:01:572
7390998234,40cyclictest0-21swapper/223:22:042
739099822,32cyclictest9690-21ssh22:47:082
739099822,32cyclictest9690-21ssh22:47:082
737599822,56cyclictest31-21ksoftirqd/123:30:571
737599822,56cyclictest31-21ksoftirqd/123:30:571
739099813,26cyclictest22845-21unixbench_multi22:12:172
739099813,26cyclictest22845-21unixbench_multi22:12:172
7375998154,11cyclictest181rcu_preempt22:23:331
7375998115,36cyclictest31-21ksoftirqd/120:51:381
7375998012,36cyclictest31-21ksoftirqd/119:57:251
7375997961,11cyclictest17881-21cstates21:06:561
7375997957,16cyclictest29263-21apt-key00:21:381
7375997949,21cyclictest24406-21apt-key19:51:421
737599791,43cyclictest31-21ksoftirqd/119:30:031
737599791,43cyclictest31-21ksoftirqd/119:30:031
737599788,47cyclictest31-21ksoftirqd/119:43:561
7375997853,14cyclictest0-21swapper/122:26:511
737599784,42cyclictest31-21ksoftirqd/121:56:011
7375997819,30cyclictest31-21ksoftirqd/121:39:181
7375997811,41cyclictest31-21ksoftirqd/123:11:421
7375997748,20cyclictest32431-21sh20:16:381
737599774,43cyclictest31-21ksoftirqd/123:33:451
7375997737,15cyclictest181rcu_preempt22:44:311
7375997720,35cyclictest31-21ksoftirqd/119:24:571
7375997717,29cyclictest31-21ksoftirqd/121:12:061
7375997657,13cyclictest13306-21diskmemload23:52:011
7375997642,13cyclictest181rcu_preempt00:30:281
7375997639,16cyclictest181rcu_preempt19:39:231
737599762,40cyclictest31-21ksoftirqd/123:14:421
737599762,40cyclictest31-21ksoftirqd/122:05:001
737599762,40cyclictest31-21ksoftirqd/122:05:001
737599754,35cyclictest31-21ksoftirqd/123:05:561
7375997538,10cyclictest181rcu_preempt19:08:001
7375997538,10cyclictest181rcu_preempt19:08:001
7375997525,22cyclictest31-21ksoftirqd/122:11:401
7375997510,29cyclictest31-21ksoftirqd/123:23:011
737599747,27cyclictest31-21ksoftirqd/123:19:371
737599747,27cyclictest31-21ksoftirqd/123:19:371
737599738,27cyclictest31-21ksoftirqd/120:21:581
7375997346,19cyclictest14548-21diskmemload22:17:331
737599733,52cyclictest7347-21find20:37:191
7375997326,13cyclictest181rcu_preempt19:31:441
7375997326,13cyclictest181rcu_preempt19:31:441
737599732,32cyclictest31-21ksoftirqd/120:28:521
737599729,37cyclictest31-21ksoftirqd/123:44:481
737599729,26cyclictest31-21ksoftirqd/100:09:311
737599728,38cyclictest31-21ksoftirqd/121:20:131
737599728,38cyclictest31-21ksoftirqd/121:20:131
737599727,26cyclictest31-21ksoftirqd/119:13:561
737599724,29cyclictest31-21ksoftirqd/121:57:191
737599722,46cyclictest26678-21latency_hist00:16:411
737599722,46cyclictest26678-21latency_hist00:16:411
7375997223,25cyclictest31-21ksoftirqd/120:58:451
739699715,46cyclictest3737-21taskset22:37:243
7375997152,9cyclictest0-21swapper/120:52:201
7375997152,12cyclictest0-21swapper/122:32:171
7375997151,13cyclictest365-21systemd-logind20:46:391
7375997140,22cyclictest28728-21grep20:02:121
737599713,35cyclictest31-21ksoftirqd/121:44:371
7375997127,19cyclictest31-21ksoftirqd/120:32:081
7375997127,19cyclictest31-21ksoftirqd/120:32:081
737599712,26cyclictest26895-21load21:22:081
7375997121,25cyclictest0-21swapper/122:14:111
7375997121,25cyclictest0-21swapper/122:14:111
737599705,33cyclictest31-21ksoftirqd/120:07:281
737599705,33cyclictest31-21ksoftirqd/120:07:281
7375997053,10cyclictest0-21swapper/121:06:381
7375997052,6cyclictest181rcu_preempt23:10:021
7375996833,8cyclictest181rcu_preempt22:47:081
7375996833,8cyclictest181rcu_preempt22:47:081
7375996814,32cyclictest31-21ksoftirqd/121:47:111
7375996814,32cyclictest31-21ksoftirqd/121:47:111
7375996734,12cyclictest181rcu_preempt20:19:071
737599663,44cyclictest7982-21latency19:02:051
737599663,44cyclictest7982-21latency19:02:051
737599662,27cyclictest31-21ksoftirqd/122:53:391
7396996550,10cyclictest451rcuc/320:16:573
739699652,49cyclictest32228-21chrt21:31:523
739699642,34cyclictest0-21swapper/323:51:423
739699642,34cyclictest0-21swapper/323:51:423
7396996345,11cyclictest451rcuc/323:09:333
7396996345,11cyclictest451rcuc/323:09:333
739699613,41cyclictest3320-21chrt22:36:413
7396996047,9cyclictest451rcuc/320:26:543
739699597,37cyclictest415-21chrt00:27:363
7396995947,8cyclictest451rcuc/323:39:283
7396995744,9cyclictest451rcuc/321:57:023
739699574,29cyclictest23603-21taskset00:10:573
7396995742,8cyclictest451rcuc/300:04:413
7396995739,14cyclictest451rcuc/321:31:413
7396995739,14cyclictest451rcuc/321:31:413
739699573,39cyclictest20150-21chrt23:06:403
7396995643,9cyclictest451rcuc/300:15:123
7396995643,9cyclictest451rcuc/300:15:123
739699564,36cyclictest24328-21chrt19:51:413
739699563,39cyclictest19758-21chrt22:07:143
739699563,38cyclictest16065-21taskset22:58:013
739699563,36cyclictest18958-21taskset22:06:413
739699563,36cyclictest18958-21taskset22:06:413
739699563,34cyclictest11490-21chrt19:11:553
7396995542,9cyclictest451rcuc/322:53:073
739699552,37cyclictest27039-21chrt00:17:013
739699545,34cyclictest29809-21chrt00:22:053
739699545,34cyclictest29809-21chrt00:22:053
7396995441,8cyclictest451rcuc/321:00:523
7396995441,8cyclictest451rcuc/319:26:063
7396995438,11cyclictest451rcuc/320:39:093
7396995434,14cyclictest451rcuc/319:43:073
664625418,21sleep30-21swapper/318:57:293
7396995341,8cyclictest451rcuc/319:52:353
739699533,41cyclictest5707-21taskset21:42:053
739699532,38cyclictest11763-21chrt20:52:043
739699532,36cyclictest25411-21taskset21:19:363
739699532,36cyclictest25411-21taskset21:19:363
161032533,7sleep30-21swapper/323:57:043
7396995239,9cyclictest451rcuc/321:55:443
739699523,35cyclictest1467-21chrt23:31:373
739699523,35cyclictest1467-21chrt23:31:373
739699522,37cyclictest4859-21chrt21:41:203
739699514,37cyclictest23455-21taskset22:13:593
739699514,37cyclictest23455-21taskset22:13:593
739699514,36cyclictest14457-21taskset23:53:143
739699513,36cyclictest9374-21chrt23:45:033
739699512,35cyclictest28332-21chrt21:25:033
7396995037,9cyclictest451rcuc/320:45:253
739699503,37cyclictest24992-21taskset23:14:553
739699503,36cyclictest32552-21taskset22:31:183
739699502,35cyclictest3591-21chrt23:34:253
739699502,34cyclictest16704-21chrt19:27:073
739699502,34cyclictest16704-21chrt19:27:073
7396994936,9cyclictest451rcuc/320:26:053
7396994936,8cyclictest451rcuc/320:15:403
7396994934,9cyclictest451rcuc/320:10:163
7396994934,9cyclictest451rcuc/320:10:163
739699493,36cyclictest29062-21taskset20:05:183
739699493,36cyclictest10043-21chrt22:47:153
739699493,36cyclictest10043-21chrt22:47:153
739699493,34cyclictest9735-21chrt19:07:033
739699493,34cyclictest9735-21chrt19:07:033
739699493,34cyclictest17274-21chrt21:06:103
739699492,34cyclictest28205-21chrt23:21:203
739699492,34cyclictest28205-21chrt23:21:203
739699492,32cyclictest27761-21chrt20:01:413
7396994835,9cyclictest451rcuc/320:34:113
7396994835,9cyclictest451rcuc/320:34:113
739699483,32cyclictest19281-21chrt19:36:383
739699483,32cyclictest19281-21chrt19:36:383
7396994833,11cyclictest451rcuc/321:48:473
7396994833,11cyclictest451rcuc/321:48:473
739699482,32cyclictest28814-21chrt22:23:153
739699482,32cyclictest19876-21chrt19:37:033
739699482,31cyclictest20825-21chrt21:11:573
739699475,30cyclictest29722-21chrt23:22:173
739699472,31cyclictest17791-21taskset21:06:553
7396994633,9cyclictest451rcuc/322:42:213
739699463,31cyclictest8547-21chrt19:02:173
739699463,31cyclictest8547-21chrt19:02:173
739699452,29cyclictest13969-21chrt19:17:233
7396994432,8cyclictest451rcuc/320:51:083
739699414,27cyclictest25887-21chrt22:17:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional