You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 23:42
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Sep 17, 2025 12:44:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218389913437,60cyclictest0-21swapper/012:29:420
21838991163,34cyclictest18100-21acpi09:41:350
21697211312,67sleep221744-21idleruntime-cro07:01:252
21838991113,34cyclictest29643-21fschecks_time09:01:480
21838991113,34cyclictest29643-21fschecks_time09:01:480
21838991093,36cyclictest11687-21switchtime11:27:010
21838991093,35cyclictest11365-21timerandwakeup10:27:030
21838991092,27cyclictest27234-21switchtime09:57:070
21838991092,27cyclictest27234-21switchtime09:57:070
21838991074,40cyclictest29037-21diskmemload10:56:390
21838991074,30cyclictest21829-21cyclictest07:01:280
218389910733,7cyclictest0-21swapper/011:36:570
21838991073,36cyclictest10756-21mailstats09:26:570
21838991073,34cyclictest3291-21cut11:11:580
21838991073,34cyclictest3291-21cut11:11:580
21838991073,34cyclictest27503-21taskset11:56:370
21838991073,34cyclictest27503-21taskset11:56:370
21838991073,29cyclictest356-21runrttasks08:52:590
218389910729,70cyclictest1590-21unixbench_multi12:07:010
21838991063,41cyclictest9595-21cut08:01:470
21838991054,45cyclictest21978-21cstates11:46:410
21838991054,45cyclictest21978-21cstates11:46:410
218389910543,54cyclictest0-21swapper/011:16:490
21838991054,26cyclictest356-21runrttasks10:31:400
218389910535,7cyclictest0-21swapper/010:47:580
21838991053,40cyclictest4970-21/usr/sbin/munin10:16:420
21838991052,44cyclictest10202-21munin-run10:26:240
218389910439,59cyclictest31814-21/usr/sbin/munin10:07:160
21838991043,33cyclictest26406-21apt10:56:230
21838991043,33cyclictest26406-21apt10:56:230
21838991043,29cyclictest25659-21sed11:52:020
218389910430,13cyclictest0-21swapper/011:21:430
21838991034,34cyclictest32162-21/usr/sbin/munin07:31:480
21838991034,34cyclictest32162-21/usr/sbin/munin07:31:480
21838991034,34cyclictest11610-21latency_hist12:26:270
21838991034,34cyclictest11610-21latency_hist12:26:270
218389910337,59cyclictest0-21swapper/008:21:430
21838991033,40cyclictest31870-21cron07:31:240
21838991033,40cyclictest31870-21cron07:31:240
21838991033,33cyclictest8433-21sendmail_mailqu07:57:020
21838991033,33cyclictest6991-21idleruntime-cro09:21:240
21838991032,31cyclictest21544-21latency_hist11:46:280
21838991024,35cyclictest356-21runrttasks09:51:040
21838991024,34cyclictest18810-21/usr/sbin/munin10:41:510
21838991024,34cyclictest12652-21/usr/sbin/munin08:11:580
218389910239,53cyclictest0-21swapper/009:16:270
218389910234,8cyclictest0-21swapper/009:33:350
21838991023,41cyclictest356-21runrttasks11:32:430
21838991023,41cyclictest356-21runrttasks11:32:430
21838991023,32cyclictest15363-21/usr/sbin/munin09:36:510
218389910227,12cyclictest0-21swapper/011:06:260
218389910227,11cyclictest0-21swapper/009:21:480
21838991014,35cyclictest21664-21grep08:37:010
218389910127,11cyclictest0-21swapper/007:07:010
21838991012,34cyclictest22571-21/usr/sbin/munin08:42:260
21838991012,34cyclictest22571-21/usr/sbin/munin08:42:260
218389910115,30cyclictest28765-21latency_hist07:21:270
218389910034,58cyclictest0-21swapper/009:06:530
21838991003,38cyclictest18861-21tr08:31:240
21838991003,37cyclictest12528-21sort08:11:270
21838991003,36cyclictest14260-21/usr/sbin/munin08:16:460
21838991003,34cyclictest25703-21/usr/sbin/munin07:11:500
21838991003,33cyclictest31628-21grep10:06:260
21838991003,33cyclictest31628-21grep10:06:260
21838991003,32cyclictest6113-21/usr/sbin/munin07:51:530
21838991003,32cyclictest6113-21/usr/sbin/munin07:51:530
2183899994,30cyclictest356-21runrttasks09:53:240
2183899994,30cyclictest356-21runrttasks09:53:240
2183899994,30cyclictest30564-21fschecks_time12:01:460
2183899993,36cyclictest2354-21latency_hist11:11:260
2183899984,33cyclictest16230-21/usr/sbin/munin10:38:050
2183899983,37cyclictest3196-21fschecks_time07:41:490
2183899983,35cyclictest19560-21latency08:31:540
2183899983,35cyclictest19560-21latency08:31:540
2183899983,33cyclictest27661-21/usr/sbin/munin08:57:400
2183899983,30cyclictest3100-21grep10:12:030
2183899983,30cyclictest3100-21grep10:12:030
2183899982,33cyclictest6106-21latency_hist12:16:270
2183899982,33cyclictest6106-21latency_hist12:16:270
2183899982,30cyclictest27966-21systemd-journal12:21:230
21838999741,49cyclictest14750irq/292-2188000.ethernet07:26:240
2183899973,32cyclictest22758-21switchtime07:01:590
2111529710,60sleep121179-21timerandwakeup06:57:121
21861999673,17cyclictest12921-21ssh10:31:242
2183899964,34cyclictest4542-21/usr/sbin/munin07:47:080
2183899964,34cyclictest4542-21/usr/sbin/munin07:47:080
2183899963,32cyclictest24165-21/usr/sbin/munin08:46:520
2183899962,33cyclictest1324-21/usr/sbin/munin07:36:480
21861999460,24cyclictest16199-21latency_hist11:36:272
21861999460,24cyclictest16199-21latency_hist11:36:272
21861999066,16cyclictest26618-21timerandwakeup08:52:022
21861998836,39cyclictest2953-21apt-get07:41:372
21861998822,27cyclictest1046-21seq11:07:382
21861998723,26cyclictest17576-21runrttasks09:40:432
275412860,10sleep10-21swapper/108:56:281
21861998622,24cyclictest12207-21seq09:30:412
21861998463,13cyclictest17266-21latency_hist08:26:262
21861998352,18cyclictest23823-21apt-key10:51:292
21861998321,24cyclictest29003-21seq11:58:212
21861998321,24cyclictest29003-21seq11:58:212
21851998368,8cyclictest0-21swapper/107:56:461
21851998361,14cyclictest29037-21diskmemload12:01:371
2185199825,53cyclictest7394-21apt-get10:21:321
21851998255,19cyclictest32346-21sh09:06:521
21861998122,20cyclictest1123-21seq09:09:052
21851998151,23cyclictest24221-21sh09:51:581
21851998151,23cyclictest24221-21sh09:51:581
47532800,5sleep20-21swapper/211:15:342
47532800,5sleep20-21swapper/211:15:342
21861998022,27cyclictest18804-21seq08:31:132
21861998022,26cyclictest5471-21seq12:15:032
21861998022,26cyclictest5471-21seq12:15:032
21851998053,21cyclictest30438-21unixbench_multi11:02:021
21861997961,12cyclictest8246-21ssh10:22:382
21861997960,13cyclictest457-21switchtime07:32:002
21861997960,13cyclictest457-21switchtime07:32:002
21861997960,13cyclictest23261-21sendmail09:51:242
21861997951,20cyclictest21533-21grep08:37:002
21851997952,21cyclictest7648-21sh09:21:501
2127327924,36sleep30-21swapper/306:57:143
21861997817,27cyclictest0-21swapper/209:26:072
21861997753,15cyclictest18426-21fschecks_time09:41:472
21861997724,20cyclictest356-21runrttasks10:00:572
21861997724,20cyclictest356-21runrttasks10:00:572
21861997723,24cyclictest25078-21seq07:07:132
21851997758,10cyclictest0-21swapper/110:47:551
21861997622,24cyclictest13874-21seq08:14:492
21861997618,22cyclictest0-21swapper/211:50:552
21861997618,22cyclictest0-21swapper/211:50:552
21851997656,11cyclictest1880-21systemctl10:11:231
21851997654,14cyclictest27966-21systemd-journal07:51:251
21851997654,14cyclictest27966-21systemd-journal07:51:251
2185199764,53cyclictest24080-21apt-get10:51:391
2185199764,53cyclictest24080-21apt-get10:51:391
21861997553,11cyclictest0-21swapper/210:11:442
21861997553,11cyclictest0-21swapper/210:11:442
21861997529,35cyclictest0-21swapper/212:06:352
21861997529,34cyclictest22589-21apt-get08:41:372
21861997529,34cyclictest22589-21apt-get08:41:372
21851997551,14cyclictest12913-21cron10:31:231
21851997549,20cyclictest984-21sh12:06:491
21861997452,15cyclictest29040-21idleruntime-cro09:01:242
21861997452,12cyclictest0-21swapper/207:47:062
21861997452,12cyclictest0-21swapper/207:47:062
21861997447,19cyclictest6413-21sh10:18:122
2186199744,53cyclictest16369-21chrt10:37:102
21861997421,28cyclictest13789-21seq12:30:032
21861997418,27cyclictest356-21runrttasks11:22:462
21861997418,23cyclictest5768-21grep09:17:062
21861997418,19cyclictest20255-21seq10:43:162
21861997416,29cyclictest32002-21latency_hist11:06:272
21851997456,12cyclictest3878-21runrttasks11:12:481
21851997456,12cyclictest3878-21runrttasks11:12:481
21861997353,9cyclictest0-21swapper/207:26:272
21861997352,11cyclictest0-21swapper/210:07:042
2186199733,50cyclictest23515-21apt-get10:51:402
2186199733,50cyclictest23515-21apt-get10:51:402
21861997320,19cyclictest7380-21seq11:20:062
21851997352,15cyclictest29037-21diskmemload10:17:011
21851997337,26cyclictest0-21swapper/110:01:451
21851997337,26cyclictest0-21swapper/110:01:451
21851997329,34cyclictest0-21swapper/111:26:591
21861997254,13cyclictest21829-21cyclictest11:51:352
21861997252,10cyclictest0-21swapper/210:02:012
21861997252,10cyclictest0-21swapper/210:02:012
21861997154,11cyclictest27966-21systemd-journal08:11:242
21861997144,14cyclictest0-21swapper/212:06:272
21861997121,20cyclictest26798-21seq07:13:122
21861997113,25cyclictest0-21swapper/211:28:542
21851997155,11cyclictest356-21runrttasks10:14:331
21851997155,11cyclictest356-21runrttasks10:14:331
21851997148,14cyclictest31777-21kworker/u17:4-rpciod11:41:371
21851997139,24cyclictest6000-21date12:16:261
21851997139,24cyclictest6000-21date12:16:261
2185199713,50cyclictest16373-21ssh10:37:101
2185199712,28cyclictest18029-21seq08:26:561
21861997052,12cyclictest19790-21mailstats08:32:022
21861997052,12cyclictest19790-21mailstats08:32:022
2186199703,44cyclictest8822-21cat12:21:262
2185199705,52cyclictest27171-21apt-get08:56:421
21851997054,10cyclictest27966-21systemd-journal12:26:231
21851997054,10cyclictest27966-21systemd-journal12:26:231
21851997046,18cyclictest331-21dbus-daemon11:11:261
21851997045,17cyclictest356-21runrttasks11:56:261
21851997045,16cyclictest7218-21latency_hist09:21:281
21851997039,26cyclictest21829-21cyclictest10:01:271
21851997039,26cyclictest21829-21cyclictest10:01:271
21868996923,38cyclictest0-21swapper/310:21:573
21861996929,35cyclictest0-21swapper/207:26:392
21861996929,35cyclictest0-21swapper/207:26:392
21861996915,19cyclictest8353-21ntp_states07:57:002
21851996940,24cyclictest0-21swapper/110:46:261
21861996850,12cyclictest365-21systemd-logind09:36:252
21861996818,38cyclictest10364-21ssh12:22:582
21861996818,38cyclictest10364-21ssh12:22:582
21851996849,15cyclictest21829-21cyclictest09:43:441
2185199682,27cyclictest12793-21seq09:31:431
21861996735,7cyclictest181rcu_preempt07:16:592
21861996732,25cyclictest14278-21apt-get08:16:412
21861996714,23cyclictest0-21swapper/207:39:002
21851996752,10cyclictest13192-21ntp_offset08:11:551
2185199672,50cyclictest22085-21apt-get08:41:371
2185199672,50cyclictest22085-21apt-get08:41:371
2186199663,43cyclictest356-21runrttasks09:53:452
2186199663,43cyclictest356-21runrttasks09:53:452
2186199663,41cyclictest18916-21latency_hist11:41:262
2185199665,25cyclictest15746-21latency_hist08:21:271
21851996650,9cyclictest25739-21latency_hist08:51:271
21851996634,25cyclictest0-21swapper/112:26:591
2185199663,41cyclictest11636-21processes08:06:581
2185199662,26cyclictest18503-21seq11:40:331
21851996546,14cyclictest10682-21latency_hist11:26:271
2185199654,47cyclictest12506-21grep09:31:261
21851996543,12cyclictest0-21swapper/107:31:191
21851996543,12cyclictest0-21swapper/107:31:191
2185199653,43cyclictest22236-21latency_hist08:41:261
21851996525,14cyclictest181rcu_preempt09:04:441
21851996525,14cyclictest181rcu_preempt09:04:441
21861996446,14cyclictest17141-21kworker/u17:3+events_unbound08:51:242
21861996443,15cyclictest29380-21cpu09:01:422
21861996443,15cyclictest29380-21cpu09:01:422
21861996433,21cyclictest19099-21apt-get11:41:422
21861996423,28cyclictest6879-21cut07:52:052
21861996423,28cyclictest6879-21cut07:52:052
21851996446,13cyclictest1-21systemd09:16:251
21851996437,19cyclictest30263-21modprobe07:26:241
2185199643,46cyclictest14210-21grep10:32:021
2185199643,31cyclictest619050/usr/sbin/munin12:16:401
2186199633,38cyclictest27156-21latency10:56:532
2185199633,44cyclictest16773-21unixbench_multi08:21:581
21851996334,22cyclictest0-21swapper/111:57:021
21851996334,22cyclictest0-21swapper/111:57:021
2185199633,41cyclictest29160-21latency_hist11:01:261
21851996332,24cyclictest0-21swapper/111:46:411
21851996332,24cyclictest0-21swapper/111:46:411
2185199632,19cyclictest17366-21seq09:40:021
2186899624,39cyclictest27249-21taskset08:56:263
2186899623,44cyclictest29210-21taskset11:01:253
2186899623,40cyclictest483-21chrt12:06:273
21868996218,26cyclictest0-21swapper/310:52:053
21868996218,26cyclictest0-21swapper/310:52:053
21861996245,12cyclictest9401-21/usr/sbin/munin08:01:482
21861996222,29cyclictest2236-21sort09:11:442
2185199624,23cyclictest3171-21tune2fs07:41:481
21851996241,13cyclictest0-21swapper/111:36:241
21851996241,13cyclictest0-21swapper/111:36:241
2185199623,42cyclictest356-21runrttasks08:31:401
2185199623,42cyclictest356-21runrttasks08:31:401
2185199623,37cyclictest28824-21latency_hist07:21:271
21851996232,22cyclictest0-21swapper/108:01:371
21861996124,30cyclictest13831-21/usr/sbin/munin10:31:572
21851996146,9cyclictest1814-21netstat07:36:521
21851996142,13cyclictest25475-21sh07:11:251
2186899604,38cyclictest3102-21chrt10:12:033
2186899604,38cyclictest3102-21chrt10:12:033
21851996045,9cyclictest7802-21sh11:21:231
2186899594,40cyclictest14397-21chrt10:32:203
2186899594,38cyclictest16122-21chrt08:21:443
2186899594,36cyclictest21144-21chrt10:46:273
21868995941,12cyclictest0-21swapper/311:16:233
21868995941,12cyclictest0-21swapper/311:16:233
2186899593,38cyclictest26672-21taskset09:56:513
2186899593,38cyclictest26672-21taskset09:56:513
21861995945,8cyclictest0-21swapper/207:02:012
21851995943,11cyclictest356-21runrttasks09:46:531
2185199592,41cyclictest32374-21fschecks_count07:31:471
2185199592,41cyclictest32374-21fschecks_count07:31:471
2186899584,41cyclictest15613-21chrt08:21:243
2186899583,40cyclictest7403-21chrt10:21:233
2186899583,39cyclictest18854-21chrt08:31:243
2186899583,37cyclictest23297-21chrt11:48:063
2186899583,37cyclictest23297-21chrt11:48:063
21868995826,18cyclictest0-21swapper/311:31:233
21851995841,12cyclictest356-21runrttasks07:12:011
2185199583,32cyclictest22311-21fschecks_time07:01:481
2186899573,40cyclictest23598-21chrt10:51:263
21851995741,9cyclictest0-21swapper/107:51:531
21851995741,9cyclictest0-21swapper/107:51:531
2186899565,35cyclictest25846-21chrt07:11:413
2186899564,40cyclictest10090-21chrt12:22:033
2186899564,40cyclictest10090-21chrt12:22:033
2186899564,37cyclictest31485-21taskset07:27:003
2186899564,37cyclictest31485-21taskset07:27:003
2186899564,36cyclictest1900-21chrt07:36:573
2186899564,33cyclictest18403-21chrt09:41:473
2186899563,39cyclictest765-21chrt09:07:583
2186899563,34cyclictest13855-21chrt09:32:063
2186899562,37cyclictest29733-21chrt10:02:023
2186899562,37cyclictest29733-21chrt10:02:023
2186899554,34cyclictest16896-21chrt11:36:523
2186899554,33cyclictest5056-21taskset07:46:573
2186899554,33cyclictest5056-21taskset07:46:573
2186899553,38cyclictest5131-21taskset09:16:503
2186899553,38cyclictest14262-21chrt12:31:243
2186899553,37cyclictest21463-21taskset11:46:123
2186899552,36cyclictest15629-21chrt11:34:573
2186899552,36cyclictest15629-21chrt11:34:573
2186899546,26cyclictest0-21swapper/308:41:503
2186899546,26cyclictest0-21swapper/308:41:503
2186899544,36cyclictest7305-21taskset11:19:533
2186899544,35cyclictest346-21taskset10:07:173
2186899543,37cyclictest9829-21chrt08:01:523
2186899543,37cyclictest17486-21chrt10:38:083
2186899543,35cyclictest27702-21chrt07:16:523
2186899543,33cyclictest900-21kworker/3:1+events08:01:243
2186899542,39cyclictest31106-21chrt11:03:523
2186899542,39cyclictest29835-21chrt07:22:023
2186899534,37cyclictest6809-21chrt07:52:043
2186899534,37cyclictest6809-21chrt07:52:043
2186899534,35cyclictest15997-21taskset09:36:573
2186899534,32cyclictest26572-21chrt11:54:313
2186899533,36cyclictest8855-21chrt11:22:003
2186899532,38cyclictest11646-21chrt10:27:143
2186899532,36cyclictest32702-21taskset07:31:553
2186899532,36cyclictest32702-21taskset07:31:553
2186899523,33cyclictest32395-21taskset11:06:443
2186899523,31cyclictest6413-21chrt12:16:443
2186899522,34cyclictest7300-21chrt09:21:363
2186899522,33cyclictest3554-21chrt12:11:423
2186899522,33cyclictest3554-21chrt12:11:423
2186899515,35cyclictest22364-21chrt09:48:283
2186899514,33cyclictest28896-21chrt08:59:563
2186899513,34cyclictest946-21taskset12:06:473
2186899513,34cyclictest3524-21chrt09:13:013
2186899513,34cyclictest22047-21taskset07:01:413
2186899504,31cyclictest20716-21chrt08:36:353
2186899504,30cyclictest11370-21taskset08:06:503
2186899484,27cyclictest24723-21chrt07:07:003
2186899483,34cyclictest24600-21chrt08:46:503
2186899473,31cyclictest28486-21taskset11:57:023
2186899473,31cyclictest28486-21taskset11:57:023
2186899473,30cyclictest11331-21chrt09:27:453
2186899463,30cyclictest3453-21chrt07:41:553
2186899444,28cyclictest23592-21chrt09:51:363
2186899444,28cyclictest23592-21chrt09:51:363
2186899443,30cyclictest12679-21taskset08:11:403
2186899443,28cyclictest30456-21chrt09:02:273
2186899443,28cyclictest30456-21chrt09:02:273
2186899413,28cyclictest20115-21chrt08:32:053
2186899413,28cyclictest20115-21chrt08:32:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional