You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 19:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Nov 14, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11936991214,100cyclictest14905-21apt-get10:00:150
577921160,4sleep05780-21grep11:39:540
11936991139,90cyclictest29174-21apt-get12:20:140
11936991139,90cyclictest29174-21apt-get12:20:140
119469910990,14cyclictest14570-21latency_hist09:59:552
119469910663,37cyclictest23581-21idleruntime-cro07:44:522
119369910312,82cyclictest30256-21seq11:25:080
119469910251,43cyclictest0-21swapper/211:35:092
119469910152,35cyclictest0-21swapper/208:30:152
119419910172,21cyclictest14482-21apt-key09:59:551
1194699992,27cyclictest355-21runrttasks09:02:362
1118729910,62sleep211241-21expr07:05:402
11946999756,35cyclictest31548-21threads08:05:282
11946999756,35cyclictest31548-21threads08:05:282
1194199976,9cyclictest0-21swapper/108:20:091
11946999652,38cyclictest0-21swapper/209:20:102
11946999448,41cyclictest0-21swapper/210:00:362
11941999479,9cyclictest21765-21apt11:10:061
11941999474,14cyclictest21963-21sendmail07:39:531
11941999438,49cyclictest0-21swapper/110:10:061
11946999359,9cyclictest181rcu_preempt08:15:042
11946999278,10cyclictest11923-21cyclictest11:00:202
11946999245,40cyclictest31989-21fschecks_time10:30:152
11946999243,42cyclictest0-21swapper/211:54:562
1194199924,51cyclictest871-21sendmail-mta09:40:061
1194199924,51cyclictest871-21sendmail-mta09:40:061
11936999255,29cyclictest11923-21cyclictest11:34:550
11936999255,29cyclictest11923-21cyclictest11:34:550
11946999170,14cyclictest23831-21/usr/sbin/munin07:45:042
11946999169,10cyclictest0-21swapper/211:29:562
11941999151,32cyclictest0-21swapper/109:20:301
11941999148,36cyclictest0-21swapper/112:15:291
11941999148,36cyclictest0-21swapper/112:15:291
1194199914,44cyclictest1092-21grep11:30:291
1194199914,44cyclictest1092-21grep11:30:291
11936999155,15cyclictest28727-21latency_hist10:24:540
11936999155,15cyclictest28727-21latency_hist10:24:540
11936999132,29cyclictest15-21ksoftirqd/008:14:180
11936999132,29cyclictest15-21ksoftirqd/008:14:180
1194699902,42cyclictest13124-21fschecks_time08:50:152
1194199903,42cyclictest25393-21latency_hist07:49:561
11936998920,58cyclictest21770-21apt-get11:10:120
11946998849,33cyclictest0-21swapper/209:46:402
11946998849,33cyclictest0-21swapper/209:46:402
1194699883,8cyclictest0-21swapper/207:35:092
1194699883,47cyclictest28629-21munin-run12:19:532
1194699883,47cyclictest28629-21munin-run12:19:532
11941998845,36cyclictest0-21swapper/111:00:331
11936998835,25cyclictest181rcu_preempt12:10:110
11936998812,53cyclictest29183-21runrttasks08:00:080
11936998812,53cyclictest29183-21runrttasks08:00:080
11946998769,12cyclictest18040-21fschecks_time10:05:132
11946998750,31cyclictest0-21swapper/210:45:192
11946998750,31cyclictest0-21swapper/210:45:192
11946998746,34cyclictest0-21swapper/212:25:292
11946998745,32cyclictest0-21swapper/208:20:212
11946998742,38cyclictest0-21swapper/209:14:542
11946998734,44cyclictest17559-21acpi12:00:042
1194199874,39cyclictest13365-21ssh10:55:181
1194199874,39cyclictest13365-21ssh10:55:181
11946998646,33cyclictest0-21swapper/210:55:212
11946998646,33cyclictest0-21swapper/210:55:212
1194699863,35cyclictest2322-21fschecks_count10:35:182
11941998643,37cyclictest0-21swapper/111:40:281
11941998639,40cyclictest0-21swapper/110:25:161
11941998639,40cyclictest0-21swapper/110:25:161
11941998637,14cyclictest181rcu_preempt07:50:161
1194199863,49cyclictest32395-21apt-get08:10:051
1194199863,49cyclictest32395-21apt-get08:10:051
1194199862,38cyclictest17240-21munin-run11:59:521
111412867,55sleep111186-21switchtime07:05:391
11946998566,13cyclictest355-21runrttasks12:35:212
1194699853,36cyclictest19555-21/usr/sbin/munin11:05:282
1194699853,36cyclictest19555-21/usr/sbin/munin11:05:282
1194199853,49cyclictest18691-21apt-get11:05:041
1194199853,49cyclictest18691-21apt-get11:05:041
1194199853,41cyclictest14904-21/usr/sbin/munin10:00:051
1194199853,39cyclictest32026-21basename10:30:151
1194199853,38cyclictest23328-21sort12:10:091
11941998533,43cyclictest19630-21tune2fs09:10:141
11946998467,11cyclictest2485-21cstates12:30:102
11946998443,35cyclictest23166-21unixbench_singl09:16:102
1194699843,8cyclictest0-21swapper/210:10:052
1194699843,8cyclictest0-21swapper/207:30:092
11941998461,16cyclictest5235-21/usr/sbin/munin12:35:261
1194199844,38cyclictest5857-21latency_hist11:39:551
1194199843,39cyclictest1818-21runrttasks09:36:061
1194199843,36cyclictest28329-21grep09:25:301
1194199843,36cyclictest13566-21sort08:50:271
11946998346,30cyclictest0-21swapper/207:24:562
11946998345,30cyclictest0-21swapper/208:04:562
1194699833,38cyclictest11421-21/usr/sbin/munin08:45:142
1194699833,37cyclictest15516-21/usr/sbin/munin11:55:462
1194699833,34cyclictest21204-21/usr/sbin/munin12:05:282
1194699832,8cyclictest0-21swapper/208:10:082
1194699832,8cyclictest0-21swapper/208:10:082
11941998366,10cyclictest0-21swapper/111:45:161
11941998363,10cyclictest0-21swapper/110:50:291
11941998363,10cyclictest0-21swapper/110:50:291
1194199834,34cyclictest19727-21/usr/sbin/munin07:30:281
1194199834,24cyclictest28023-21/usr/sbin/munin11:20:271
1194199833,39cyclictest19110-21ssh09:09:551
1194199833,39cyclictest19110-21ssh09:09:551
1194199833,36cyclictest23291-21/usr/sbin/munin10:15:091
1194199833,34cyclictest22519-21cut07:40:151
1194199833,10cyclictest0-21swapper/108:00:331
1194199833,10cyclictest0-21swapper/108:00:331
11946998244,32cyclictest0-21swapper/207:55:422
11946998241,33cyclictest0-21swapper/210:15:142
1194699823,8cyclictest0-21swapper/208:00:292
1194699823,40cyclictest9386-21basename08:39:532
1194699823,37cyclictest8906-21/usr/sbin/munin11:45:102
1194699823,34cyclictest4013-21/usr/sbin/munin09:40:222
1194699823,34cyclictest4013-21/usr/sbin/munin09:40:222
1194699823,32cyclictest24448-21/usr/sbin/munin11:15:212
1194699823,32cyclictest10463-21fschecks_time10:50:172
1194699823,32cyclictest10463-21fschecks_time10:50:172
1194699822,35cyclictest27612-21/usr/sbin/munin11:20:162
1194699822,34cyclictest22732-21/usr/sbin/munin11:11:062
1194199825,32cyclictest355-21runrttasks08:39:571
1194199824,9cyclictest0-21swapper/109:00:051
1194199824,36cyclictest1963-21latency08:15:191
1194199824,35cyclictest2103-21sshd10:35:101
1194199824,34cyclictest618-21ssh12:25:301
11941998236,38cyclictest0-21swapper/111:15:141
11941998235,40cyclictest0-21swapper/107:55:301
1194199823,41cyclictest355-21runrttasks08:07:111
1194199823,41cyclictest355-21runrttasks08:07:111
1194199823,38cyclictest6297-21ssh09:45:101
1194199823,38cyclictest6297-21ssh09:45:101
1194199823,37cyclictest9674-21/usr/sbin/munin09:50:241
1194199823,37cyclictest30696-21grep09:31:261
1194199823,36cyclictest17967-21ssh10:05:111
1194199823,34cyclictest5007-21fschecks_count08:25:191
1194199823,30cyclictest12172-21cpuspeed07:10:091
11936998259,12cyclictest0-21swapper/010:34:560
11936998254,20cyclictest1-21systemd08:55:100
11946998165,9cyclictest0-21swapper/207:54:542
11946998154,19cyclictest615-21cpu09:35:102
1194699813,37cyclictest29311-21/usr/sbin/munin10:25:182
1194699813,37cyclictest29311-21/usr/sbin/munin10:25:182
1194699813,35cyclictest29602-21/usr/sbin/munin12:21:072
1194699813,35cyclictest29602-21/usr/sbin/munin12:21:072
1194699813,32cyclictest15509-21munin-run07:19:532
1194699813,31cyclictest2556-21ssh11:33:592
1194699813,31cyclictest2556-21ssh11:33:592
11946998125,50cyclictest0-21swapper/208:25:232
1194699812,34cyclictest12399-21/usr/sbin/munin07:10:152
1194199814,36cyclictest355-21runrttasks11:53:281
1194199814,36cyclictest29667-21sh12:21:081
1194199814,36cyclictest29667-21sh12:21:081
1194199814,36cyclictest16180-21grep07:20:191
1194199814,34cyclictest2333-21ssh12:30:051
1194199814,10cyclictest0-21swapper/110:20:081
1194199814,10cyclictest0-21swapper/110:20:081
1194199813,37cyclictest10047-21latency_hist10:49:561
1194199813,37cyclictest10047-21latency_hist10:49:561
1194199813,34cyclictest5148-21fschecks_count10:40:161
11941998129,42cyclictest0-21swapper/111:25:001
1193699818,43cyclictest5981-21/usr/sbin/munin11:40:200
11936998159,15cyclictest207-21systemd-journal09:24:110
11936998157,17cyclictest12895-21ssh10:54:560
11936998157,17cyclictest12895-21ssh10:54:560
11936998156,18cyclictest309-21systemd-logind08:39:540
1193699814,48cyclictest15-21ksoftirqd/008:52:200
1194699803,8cyclictest0-21swapper/209:05:082
1194699803,8cyclictest0-21swapper/209:05:082
1194699803,34cyclictest6153-21cat11:40:112
1194699803,34cyclictest15800-21modprobe08:59:522
1194699802,40cyclictest23599-21taskset12:10:162
1194699802,34cyclictest30818-21/usr/sbin/munin09:31:302
1194699802,34cyclictest27404-21sshd09:25:102
1194699802,33cyclictest27574-21taskset10:21:282
1194699802,33cyclictest27574-21taskset10:21:282
1194199804,36cyclictest22495-21/usr/sbin/munin09:15:171
1194199804,35cyclictest12826-21latency_hist08:49:561
1194199804,33cyclictest6662-21fschecks_time08:30:151
1194199804,32cyclictest18221-21timerwakeupswit07:25:291
1194199803,34cyclictest14714-21fschecks_time08:55:161
11936998053,20cyclictest19061-21diskmemload10:59:540
11936998053,20cyclictest19061-21diskmemload10:59:540
1193699803,38cyclictest15-21ksoftirqd/008:39:570
1193699803,34cyclictest15-21ksoftirqd/010:09:350
1193699803,34cyclictest15-21ksoftirqd/008:21:430
1193699802,47cyclictest15-21ksoftirqd/008:29:320
1194699793,36cyclictest18739-21sh07:29:532
1194699793,34cyclictest7172-21munin-run10:44:532
1194199794,30cyclictest20672-21fschecks_time12:05:151
1194199793,36cyclictest17552-21acpi12:00:041
11941997913,30cyclictest8166-21df08:35:131
11936997960,10cyclictest19061-21diskmemload09:10:100
1193699793,41cyclictest18672-21/usr/sbin/munin11:05:210
1193699793,41cyclictest18672-21/usr/sbin/munin11:05:210
1193699792,45cyclictest15-21ksoftirqd/010:28:080
1193699792,45cyclictest15-21ksoftirqd/010:28:080
1177927911,37sleep011803-21idleruntime-cro07:09:530
1194699783,35cyclictest9631-21ssh09:50:222
1194699783,35cyclictest10122-21/usr/sbin/munin08:40:222
11941997829,41cyclictest0-21swapper/107:15:311
1193699787,58cyclictest355-21runrttasks09:50:550
1193699783,33cyclictest15-21ksoftirqd/012:17:220
1193699783,33cyclictest15-21ksoftirqd/012:17:220
1193699782,40cyclictest18730-21modprobe07:29:530
11936997814,29cyclictest15-21ksoftirqd/007:17:410
11936997759,12cyclictest19082-21ssh09:09:530
11936997759,12cyclictest19082-21ssh09:09:530
11936997728,34cyclictest16244-21needreboot07:20:210
11936997727,37cyclictest18616-21unixbench_singl12:00:300
11936997653,16cyclictest27078-21latency_hist11:19:560
11936997643,9cyclictest11923-21cyclictest09:31:370
1193699763,60cyclictest4094-21ssh12:32:350
1195199753,39cyclictest14270-21taskset10:55:383
1195199753,39cyclictest14270-21taskset10:55:383
1193699755,35cyclictest4310-21timerandwakeup09:40:290
1193699755,35cyclictest4310-21timerandwakeup09:40:290
1193699755,32cyclictest15-21ksoftirqd/009:17:140
11936997536,23cyclictest0-21swapper/011:55:070
1193699753,37cyclictest15-21ksoftirqd/007:32:560
1193699752,38cyclictest15-21ksoftirqd/007:38:230
1193699743,48cyclictest15-21ksoftirqd/007:11:140
11936997345,21cyclictest26795-21latency_hist07:54:560
1193699732,42cyclictest15-21ksoftirqd/007:58:020
11936997315,29cyclictest161ktimers/011:22:090
11936997312,42cyclictest24117-21fschecks_time07:45:150
1195199723,40cyclictest27640-21chrt11:20:173
1193699725,21cyclictest15-21ksoftirqd/011:03:410
11936997245,19cyclictest1-21systemd09:01:110
1193699724,22cyclictest15-21ksoftirqd/008:07:540
1193699724,22cyclictest15-21ksoftirqd/008:07:540
1193699723,53cyclictest19061-21diskmemload09:59:560
11936997229,30cyclictest355-21runrttasks10:48:190
11936997229,30cyclictest355-21runrttasks10:48:190
11936997156,10cyclictest19061-21diskmemload12:35:510
1193699715,31cyclictest15-21ksoftirqd/008:18:350
11936997150,14cyclictest25036-21ssh10:17:240
1193699714,57cyclictest3138-21grep10:35:330
1193699714,34cyclictest11662-21/usr/sbin/munin11:50:140
1193699706,29cyclictest15-21ksoftirqd/012:27:270
1193699703,36cyclictest355-21runrttasks12:05:200
1193699702,40cyclictest15-21ksoftirqd/010:11:510
1195199692,36cyclictest30547-21chrt09:31:023
1193699693,34cyclictest6922-21proc_pri09:45:250
1193699693,34cyclictest6922-21proc_pri09:45:250
1195199683,35cyclictest24479-21chrt11:15:063
1195199683,22cyclictest27301-21chrt07:55:063
1193699689,21cyclictest15-21ksoftirqd/007:40:530
11936996848,14cyclictest10239-21sh11:46:200
1193699683,43cyclictest15-21ksoftirqd/008:30:420
11936996824,34cyclictest12813-21latency_hist08:49:560
11936996823,12cyclictest301132cat09:29:570
1195199673,34cyclictest479-21taskset08:10:243
1195199673,34cyclictest479-21taskset08:10:243
1195199672,37cyclictest22258-21taskset09:15:103
1195199665,31cyclictest21838-21chrt11:10:163
1195199663,34cyclictest3019-21taskset12:30:253
1195199662,33cyclictest7781-21taskset08:34:543
1193699664,27cyclictest15-21ksoftirqd/009:38:000
1195199653,31cyclictest17106-21chrt10:03:523
1195199653,22cyclictest2109-21chrt10:35:123
11936996535,21cyclictest5344-21wc10:40:200
1195199644,30cyclictest20195-21chrt07:33:073
1195199643,28cyclictest21697-21taskset09:14:253
1195199643,27cyclictest27696-21chrt10:21:513
1195199643,27cyclictest27696-21chrt10:21:513
1195199642,36cyclictest19832-21taskset10:08:303
1195199642,33cyclictest13116-21chrt11:51:323
1195199642,28cyclictest9384-21chrt11:45:203
1195199633,28cyclictest16553-21chrt07:20:283
1195199633,26cyclictest9412-21chrt10:48:243
1195199633,26cyclictest9412-21chrt10:48:243
1195199633,23cyclictest18580-21chrt07:28:013
1195199632,21cyclictest31436-21chrt08:05:273
1195199632,21cyclictest31436-21chrt08:05:273
1195199623,25cyclictest13123-21chrt07:10:453
1195199623,25cyclictest11256-21chrt08:45:063
1195199623,22cyclictest5539-21chrt12:35:163
1195199623,22cyclictest32165-21chrt12:25:113
1195199622,27cyclictest6812-21chrt09:45:213
1195199622,27cyclictest6812-21chrt09:45:213
1195199622,22cyclictest4085-21chrt08:20:293
1195199622,22cyclictest10210-21chrt09:50:443
1195199616,45cyclictest0-21swapper/312:05:143
1195199614,24cyclictest1895-21chrt11:31:473
1195199614,24cyclictest1895-21chrt11:31:473
1195199613,31cyclictest26479-21chrt09:23:243
1195199613,27cyclictest31973-21chrt11:27:583
1195199613,25cyclictest1559-21chrt08:15:063
1195199613,23cyclictest28179-21chrt12:18:343
1195199613,23cyclictest28179-21chrt12:18:343
1195199613,22cyclictest4037-21taskset09:40:223
1195199613,22cyclictest4037-21taskset09:40:223
1195199613,22cyclictest13712-21chrt08:50:283
1195199613,21cyclictest19404-21chrt12:02:473
1195199612,26cyclictest7886-21chrt11:42:533
1195199612,22cyclictest14754-21chrt11:55:013
1195199612,20cyclictest18190-21chrt11:04:203
11951996110,40cyclictest0-21swapper/309:35:163
1195199604,24cyclictest30176-21chrt10:25:563
1195199604,24cyclictest30176-21chrt10:25:563
1195199603,41cyclictest12255-21taskset09:55:173
1195199603,29cyclictest6263-21chrt10:41:243
1195199603,28cyclictest4947-21chrt11:37:393
1195199603,26cyclictest27543-21chrt09:25:153
1195199603,24cyclictest30199-21taskset08:00:573
1195199603,24cyclictest30199-21taskset08:00:573
1195199603,24cyclictest21191-21chrt10:10:253
1195199603,23cyclictest10950-21chrt08:44:373
1195199602,28cyclictest20382-21taskset11:07:243
1195199602,28cyclictest20382-21taskset11:07:243
1195199602,24cyclictest22743-21chrt07:40:203
1195199602,24cyclictest13919-21chrt07:15:023
1195199602,23cyclictest23246-21chrt12:10:043
1195199602,22cyclictest407-21chrt10:30:463
1195199602,21cyclictest23290-21chrt10:15:023
1195199602,20cyclictest26855-21chrt07:54:533
1195199593,24cyclictest8019-21chrt08:35:063
1195199593,21cyclictest12136-21chrt10:53:273
1195199593,21cyclictest12136-21chrt10:53:273
1195199592,27cyclictest17301-21chrt09:03:083
1195199592,25cyclictest15482-21chrt08:55:473
1195199592,24cyclictest31484-21chrt12:24:183
1195199592,24cyclictest31484-21chrt12:24:183
1195199592,23cyclictest4775-21chrt08:24:583
1195199582,24cyclictest24925-21chrt07:46:363
1195199582,23cyclictest17861-21chrt09:05:103
1195199582,23cyclictest17861-21chrt09:05:103
1195199572,21cyclictest21592-21chrt07:35:303
11951995110,33cyclictest0-21swapper/307:09:583
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional