You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-07 - 16:32
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sat Sep 07, 2024 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11191214025,68sleep211219-21munin-run07:03:392
11266211311,62sleep111275-21latency_hist07:03:421
10742211113,63sleep310839-21grep06:59:243
75802970,5sleep00-21swapper/012:28:390
110862956,15sleep00-21swapper/007:02:140
1135399892,65cyclictest28417-21apt-get12:08:530
11365998041,15cyclictest33-21ksoftirqd/209:33:592
1136599798,33cyclictest16459-21ssh11:49:102
221282780,6sleep20-21swapper/211:04:012
1137099782,53cyclictest27881-21apt-get12:08:513
11370997751,17cyclictest5827-21munin-plugin-st09:38:383
1137099757,43cyclictest16394-21ssh11:48:583
11370997548,17cyclictest14240-21cron09:53:373
11358997527,11cyclictest141rcu_preempt11:07:081
1137099743,48cyclictest13237-21ssh11:44:053
11370997412,37cyclictest13465-21diskstats10:49:203
11365997452,13cyclictest17040-21munin-plugin-st09:58:372
1136599743,27cyclictest141rcu_preempt10:49:152
11353997454,14cyclictest131ktimers/008:38:500
11365997354,12cyclictest0-21swapper/212:08:532
11365997125,25cyclictest33-21ksoftirqd/209:06:432
11358997134,22cyclictest141rcu_preempt09:50:081
11370997048,14cyclictest29456-21unixbench_multi10:19:143
11370997037,9cyclictest141rcu_preempt10:54:053
11370997028,8cyclictest141rcu_preempt08:34:073
11370997028,8cyclictest141rcu_preempt08:34:073
11365997031,8cyclictest33-21ksoftirqd/210:13:372
11365997025,23cyclictest33-21ksoftirqd/210:41:512
11358997050,14cyclictest0-21swapper/108:53:381
11358997026,27cyclictest141rcu_preempt12:04:351
11358997014,37cyclictest15486-21apt-get08:54:131
11358997014,37cyclictest15486-21apt-get08:54:131
11365996925,23cyclictest141rcu_preempt08:13:412
11365996925,23cyclictest141rcu_preempt08:13:412
11358996952,9cyclictest0-21swapper/108:33:381
11358996949,10cyclictest0-21swapper/111:13:391
11353996953,9cyclictest3622-21basename08:18:370
11353996928,20cyclictest13446-21df10:49:180
11370996812,10cyclictest141rcu_preempt09:24:003
11370996812,10cyclictest141rcu_preempt09:24:003
11358996847,14cyclictest355-21systemd-logind09:33:381
11358996846,13cyclictest26328-21unin-run07:48:391
11358996845,7cyclictest141rcu_preempt09:21:141
11358996845,12cyclictest0-21swapper/110:28:381
11353996846,15cyclictest16849-21munin-plugin-st08:58:370
11353996846,15cyclictest16849-21munin-plugin-st08:58:370
11353996813,33cyclictest16333-21apt-get11:48:510
11370996749,12cyclictest11726-21munin-plugin-st08:43:373
11370996733,11cyclictest183-21pr/ttyS009:39:093
11365996737,9cyclictest33-21ksoftirqd/207:13:522
11365996723,13cyclictest141rcu_preempt09:14:132
11365996723,13cyclictest141rcu_preempt09:14:132
11358996738,13cyclictest141rcu_preempt10:21:501
11358996736,14cyclictest141rcu_preempt11:37:261
11358996736,10cyclictest141rcu_preempt09:26:251
11358996736,10cyclictest141rcu_preempt09:26:251
11358996732,6cyclictest0-21swapper/109:55:031
11358996726,25cyclictest141rcu_preempt10:11:101
117092660,14sleep233-21ksoftirqd/211:41:092
1135899669,35cyclictest0-21swapper/109:34:521
1135899664,34cyclictest13156-21apt-get10:48:511
73572650,11chrt231rcuc/110:39:011
1136599659,24cyclictest141rcu_preempt07:53:382
11365996544,15cyclictest5806-21timerwakeupswit12:24:132
11365996544,14cyclictest18575-21idleruntime-cro10:58:382
11365996532,15cyclictest141rcu_preempt11:13:542
11365996532,15cyclictest141rcu_preempt11:13:542
11365996524,33cyclictest33-21ksoftirqd/208:55:022
11365996524,33cyclictest33-21ksoftirqd/208:55:022
11358996547,9cyclictest0-21swapper/111:44:401
11358996517,19cyclictest25-21ksoftirqd/110:16:191
11353996546,12cyclictest1979-21munin-plugin-st08:13:380
11353996546,12cyclictest1979-21munin-plugin-st08:13:380
1135399654,42cyclictest400-21runrttasks11:44:070
11353996532,26cyclictest8650-21modprobe09:43:370
11370996448,9cyclictest0-21swapper/312:14:133
11365996446,11cyclictest11724-21basename08:43:372
11365996435,17cyclictest33-21ksoftirqd/209:22:082
1136599643,30cyclictest25322-21grep11:09:092
11365996420,11cyclictest33-21ksoftirqd/209:49:492
11358996443,8cyclictest141rcu_preempt11:01:521
11358996443,8cyclictest141rcu_preempt11:01:521
11358996443,7cyclictest141rcu_preempt07:18:141
11358996442,15cyclictest29567-21idleruntime-cro07:58:381
11358996441,16cyclictest26033-21systemd-journal12:33:371
11358996441,16cyclictest26033-21systemd-journal12:33:371
11358996430,12cyclictest141rcu_preempt09:09:161
11358996419,25cyclictest141rcu_preempt10:05:061
11358996419,25cyclictest141rcu_preempt10:05:061
11353996443,14cyclictest6902-21idleruntime-cro10:38:370
11370996334,15cyclictest141rcu_preempt11:19:093
11370996322,12cyclictest0-21swapper/311:28:143
11370996314,27cyclictest41-21ksoftirqd/310:07:073
11370996314,27cyclictest41-21ksoftirqd/310:07:073
11365996342,14cyclictest14761-21kworker/u8:109:09:122
11365996326,10cyclictest33-21ksoftirqd/207:23:542
11358996342,11cyclictest0-21swapper/107:24:181
11358996341,14cyclictest0-21swapper/109:08:371
11358996334,8cyclictest141rcu_preempt11:28:351
11353996318,22cyclictest141rcu_preempt09:14:000
11353996318,22cyclictest141rcu_preempt09:14:000
11370996240,7cyclictest41-21ksoftirqd/310:35:273
11370996219,10cyclictest41-21ksoftirqd/310:13:403
11365996246,10cyclictest14968-21sh08:53:382
11365996239,13cyclictest28136-21sh10:18:382
11365996217,13cyclictest141rcu_preempt12:14:132
11358996242,13cyclictest13031-21latency_hist11:43:411
11358996241,8cyclictest141rcu_preempt11:19:521
11358996234,18cyclictest0-21swapper/112:13:381
11358996233,7cyclictest141rcu_preempt12:28:041
11358996229,10cyclictest141rcu_preempt09:59:221
11358996210,11cyclictest11337-21cyclictest09:44:451
11353996232,12cyclictest141rcu_preempt11:13:580
11353996232,12cyclictest141rcu_preempt11:13:580
11370996139,7cyclictest41-21ksoftirqd/311:36:483
11365996140,12cyclictest0-21swapper/208:18:582
11365996129,19cyclictest141rcu_preempt12:32:282
11365996129,19cyclictest141rcu_preempt12:32:282
11358996139,8cyclictest141rcu_preempt10:48:121
11358996138,8cyclictest141rcu_preempt10:34:521
11358996134,14cyclictest141rcu_preempt10:56:411
11353996140,12cyclictest0-21swapper/010:39:050
11353996119,21cyclictest141rcu_preempt12:04:510
11370996045,9cyclictest21410-21kworker/u8:307:53:383
11370996034,16cyclictest41-21ksoftirqd/312:01:173
11365996039,10cyclictest0-21swapper/211:48:402
11365996032,13cyclictest141rcu_preempt11:03:402
11365996032,13cyclictest141rcu_preempt11:03:402
11365996016,19cyclictest141rcu_preempt10:29:202
11365996016,19cyclictest141rcu_preempt10:29:202
1135899605,17cyclictest25-21ksoftirqd/107:49:031
11358996039,8cyclictest141rcu_preempt12:00:341
11358996039,7cyclictest141rcu_preempt11:31:471
11358996037,15cyclictest1-21systemd09:43:391
11358996034,8cyclictest141rcu_preempt11:54:051
11358996021,18cyclictest25-21ksoftirqd/108:04:141
11358996012,11cyclictest25-21ksoftirqd/107:59:001
11358996012,11cyclictest25-21ksoftirqd/107:59:001
11353996042,11cyclictest19814-21modprobe07:28:380
11353996022,29cyclictest0-21swapper/010:09:120
11370995941,11cyclictest26033-21systemd-journal12:33:403
11370995941,11cyclictest26033-21systemd-journal12:33:403
11370995936,7cyclictest141rcu_preempt07:09:153
11370995935,13cyclictest41-21ksoftirqd/311:32:023
11370995923,14cyclictest141rcu_preempt12:04:033
11370995920,16cyclictest41-21ksoftirqd/309:29:043
11370995919,11cyclictest141rcu_preempt08:43:583
11370995910,28cyclictest41-21ksoftirqd/310:28:493
11370995910,28cyclictest41-21ksoftirqd/310:28:493
11365995943,10cyclictest741-21munin-plugin-st11:23:372
11365995939,14cyclictest9331-21if_err_eth009:44:002
11365995924,14cyclictest141rcu_preempt07:34:122
11365995924,14cyclictest141rcu_preempt07:34:122
11365995923,9cyclictest33-21ksoftirqd/207:43:402
1135899596,16cyclictest25-21ksoftirqd/111:13:581
1135899596,16cyclictest25-21ksoftirqd/111:13:581
11358995917,20cyclictest25-21ksoftirqd/108:28:071
11353995945,8cyclictest18901-21ssh10:58:510
11353995945,8cyclictest18901-21ssh10:58:510
11353995926,7cyclictest0-21swapper/008:53:410
11353995924,16cyclictest12-21ksoftirqd/009:03:550
11353995917,15cyclictest141rcu_preempt11:24:010
1137099583,9cyclictest141rcu_preempt11:38:573
11370995834,17cyclictest31165-21kworker/u8:010:03:373
11370995832,16cyclictest18712-21diskmemload09:08:493
11365995841,10cyclictest22833-21runrttasks11:59:052
11365995833,16cyclictest400-21runrttasks08:24:092
11365995832,16cyclictest12465-21ntp_kernel_err08:44:032
1136599583,12cyclictest141rcu_preempt11:24:142
11365995831,15cyclictest0-21swapper/211:29:002
11365995829,11cyclictest33-21ksoftirqd/207:29:132
11365995817,19cyclictest33-21ksoftirqd/212:04:102
11365995816,9cyclictest141rcu_preempt09:59:122
11358995838,7cyclictest141rcu_preempt12:18:121
11358995820,22cyclictest141rcu_preempt10:28:471
11358995820,22cyclictest141rcu_preempt10:28:471
11358995814,23cyclictest0-21swapper/107:36:261
11358995814,23cyclictest0-21swapper/107:36:261
1135399586,34cyclictest0-21swapper/010:58:400
11353995844,8cyclictest7916-21cpuspeed12:28:550
11353995844,8cyclictest7916-21cpuspeed12:28:550
1135399584,17cyclictest141rcu_preempt11:08:390
11353995840,11cyclictest0-21swapper/009:53:370
11353995834,9cyclictest12-21ksoftirqd/007:48:580
1135399582,44cyclictest14761-21kworker/u8:108:58:580
11370995732,7cyclictest141rcu_preempt07:24:173
11370995726,13cyclictest41-21ksoftirqd/311:56:583
11370995724,17cyclictest41-21ksoftirqd/310:39:013
1137099572,13cyclictest141rcu_preempt07:38:583
1137099572,13cyclictest0-21swapper/307:08:403
11370995718,17cyclictest41-21ksoftirqd/307:44:103
11365995738,13cyclictest4598-21unin-run12:23:382
1136599572,21cyclictest29265-21switchtime10:19:132
11365995712,16cyclictest33-21ksoftirqd/209:29:102
1135899577,16cyclictest89372fschecks_count08:34:011
1135899577,16cyclictest89372fschecks_count08:34:011
1135899577,14cyclictest25-21ksoftirqd/108:44:361
11358995729,12cyclictest141rcu_preempt11:53:001
11358995728,13cyclictest141rcu_preempt12:19:471
11358995724,11cyclictest141rcu_preempt07:19:161
11358995717,14cyclictest141rcu_preempt09:15:501
11358995717,14cyclictest141rcu_preempt09:15:501
1135899571,21cyclictest25-21ksoftirqd/108:59:281
11353995734,14cyclictest12906-21latency_hist10:48:400
11353995728,15cyclictest141rcu_preempt08:43:530
11353995727,9cyclictest141rcu_preempt07:39:000
1137099565,22cyclictest17100-21latency_hist09:58:393
11370995641,9cyclictest400-21runrttasks10:15:103
11370995639,9cyclictest0-21swapper/311:04:123
11370995638,8cyclictest0-21swapper/312:24:143
1136599567,25cyclictest141rcu_preempt09:24:112
1136599567,25cyclictest141rcu_preempt09:24:112
11365995639,9cyclictest10110-21basename08:38:382
11365995639,9cyclictest10110-21basename08:38:382
11365995634,11cyclictest0-21swapper/207:48:402
11365995614,13cyclictest141rcu_preempt10:34:142
11365995611,9cyclictest141rcu_preempt10:44:112
11358995636,11cyclictest0-21swapper/107:33:381
11353995640,9cyclictest29456-21unixbench_multi10:19:140
11353995636,12cyclictest1882-21ssh12:18:440
11353995633,12cyclictest0-21swapper/011:19:130
121942552,8sleep112279-21threads07:04:101
11370995534,7cyclictest41-21ksoftirqd/308:59:283
11370995530,13cyclictest0-21swapper/310:23:593
11370995526,13cyclictest141rcu_preempt11:09:053
11370995526,11cyclictest363-21in:imklog08:54:053
11370995526,11cyclictest363-21in:imklog08:54:053
11370995517,12cyclictest141rcu_preempt11:01:223
11370995517,12cyclictest141rcu_preempt11:01:223
1136599557,22cyclictest141rcu_preempt11:53:592
11365995537,12cyclictest17760-21unixbench_multi07:19:152
11365995513,14cyclictest33-21ksoftirqd/210:24:432
11358995539,10cyclictest241ktimers/108:09:051
11358995539,10cyclictest241ktimers/108:09:051
11353995541,8cyclictest4381-21latency_hist10:33:410
11353995541,8cyclictest4381-21latency_hist10:33:410
11353995540,9cyclictest8510-21systemd-run08:33:380
11353995532,7cyclictest12-21ksoftirqd/009:37:160
11353995531,14cyclictest0-21swapper/008:08:370
11353995519,16cyclictest0-21swapper/011:11:190
11353995518,8cyclictest12-21ksoftirqd/009:24:010
11353995518,8cyclictest12-21ksoftirqd/009:24:010
11353995512,8cyclictest141rcu_preempt11:41:170
11370995431,7cyclictest41-21ksoftirqd/308:23:393
11370995431,12cyclictest0-21swapper/308:13:593
11370995429,13cyclictest41-21ksoftirqd/308:09:023
11370995429,13cyclictest41-21ksoftirqd/308:09:023
11370995427,14cyclictest41-21ksoftirqd/311:16:143
11370995427,14cyclictest41-21ksoftirqd/311:16:143
11370995422,17cyclictest141rcu_preempt10:48:403
11370995412,11cyclictest0-21swapper/307:13:523
1136599545,16cyclictest0-21swapper/207:09:142
11365995438,10cyclictest8692-21systemd-run09:43:382
11365995417,10cyclictest33-21ksoftirqd/208:18:392
1135899542,21cyclictest25-21ksoftirqd/108:18:541
11353995441,8cyclictest23330-21kworker/u8:011:34:110
11353995434,9cyclictest0-21swapper/009:33:400
11353995434,7cyclictest0-21swapper/007:53:410
1135399543,21cyclictest22743-21timerandwakeup09:09:120
1135399542,11cyclictest12-21ksoftirqd/009:44:090
11353995419,12cyclictest12882-21munin-run07:08:370
11353995411,33cyclictest23046-21cron07:38:380
11353995411,33cyclictest23046-21cron07:38:380
11370995326,8cyclictest41-21ksoftirqd/308:04:023
11370995326,8cyclictest141rcu_preempt09:21:453
11370995326,14cyclictest41-21ksoftirqd/308:28:393
11370995320,9cyclictest41-21ksoftirqd/307:54:103
11370995313,8cyclictest17319-21ntpq07:19:053
1136599533,17cyclictest141rcu_preempt11:34:012
1135899537,12cyclictest25-21ksoftirqd/107:08:441
11353995330,7cyclictest12-21ksoftirqd/010:04:520
11353995330,7cyclictest12-21ksoftirqd/010:04:520
11353995330,17cyclictest31248-21/usr/sbin/munin10:24:030
1137099522,15cyclictest21713-21apt-get07:33:493
1137099522,15cyclictest21713-21apt-get07:33:493
11370995216,8cyclictest141rcu_preempt07:29:003
11370995216,16cyclictest141rcu_preempt09:13:563
11370995216,16cyclictest141rcu_preempt09:13:563
11370995212,7cyclictest141rcu_preempt09:48:193
1136599524,16cyclictest17648-21perl08:59:322
11365995231,11cyclictest0-21swapper/207:04:122
11365995225,13cyclictest0-21swapper/208:03:582
11365995219,13cyclictest0-21swapper/207:59:032
11365995219,13cyclictest0-21swapper/207:59:032
1135399527,23cyclictest141rcu_preempt11:59:040
11353995233,12cyclictest26033-21systemd-journal11:54:440
11353995218,8cyclictest0-21swapper/012:14:000
11353995216,10cyclictest28710-21kworker/0:008:03:390
11353995216,10cyclictest28710-21kworker/0:008:03:390
11370995135,11cyclictest26033-21systemd-journal09:07:503
11370995126,13cyclictest41-21ksoftirqd/308:33:413
11370995117,14cyclictest41-21ksoftirqd/312:23:323
11370995112,17cyclictest141rcu_preempt07:59:013
11370995112,17cyclictest141rcu_preempt07:59:013
11365995132,12cyclictest0-21swapper/208:33:382
11365995111,21cyclictest0-21swapper/210:03:442
11365995111,21cyclictest0-21swapper/210:03:442
1135899519,13cyclictest25-21ksoftirqd/107:39:251
11358995119,13cyclictest141rcu_preempt08:41:201
1135399517,12cyclictest18033-21awk09:59:090
1135399515,12cyclictest0-21swapper/007:44:010
1135399513,10cyclictest0-21swapper/009:18:490
11353995125,16cyclictest8250irq/69-2188000.07:33:400
11353995123,7cyclictest0-21swapper/011:33:370
1137099507,8cyclictest141rcu_preempt08:49:073
11365995026,15cyclictest29119-21unixbench_multi07:54:122
11353995034,9cyclictest0-21swapper/010:14:110
11353995024,8cyclictest15932-21ssh09:55:010
11353995010,23cyclictest8734-21/usr/sbin/munin08:34:150
11353995010,23cyclictest8734-21/usr/sbin/munin08:34:150
11358994914,9cyclictest241ktimers/108:13:551
11353994937,8cyclictest183-21pr/ttyS007:09:120
11353994928,7cyclictest12-21ksoftirqd/008:28:410
11353994926,15cyclictest0-21swapper/008:23:380
1135399482,10cyclictest0-21swapper/007:14:130
11353994713,7cyclictest12-21ksoftirqd/007:19:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional