You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 18:30
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Nov 21, 2025 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2435321018,58sleep124438-21unixbench_multi07:05:131
2490821009,63sleep024910-21munin-run07:09:280
25010999760,12cyclictest181rcu_preempt10:54:532
2500299963,74cyclictest28359-21ssh09:59:481
25002999480,10cyclictest24980-21cyclictest11:40:001
25010998961,8cyclictest181rcu_preempt08:52:052
24990998957,23cyclictest21636-21threads08:35:000
25002998873,10cyclictest25756-21ssh10:52:341
25002998869,11cyclictest32372-21kworker/u19:3-ext4-rsv-conversion08:19:441
25010998756,10cyclictest181rcu_preempt09:28:322
25002998726,26cyclictest1878-21sh09:12:431
2500299872,59cyclictest16840-21ssh09:40:031
25002998724,26cyclictest0-21swapper/110:15:021
25002998722,31cyclictest355-21runrttasks11:52:221
24990998659,19cyclictest16128-21sh09:39:250
241722867,54sleep224217-21sensors_temp07:05:102
25010998556,21cyclictest24980-21cyclictest11:04:472
25002998557,20cyclictest28195-21latency_hist09:59:291
25002998557,20cyclictest28195-21latency_hist09:59:291
25010998448,8cyclictest181rcu_preempt09:54:522
25010998448,8cyclictest181rcu_preempt09:54:522
25002998464,14cyclictest31190-21kernelversion12:00:141
24990998454,11cyclictest181rcu_preempt09:58:510
24990998454,11cyclictest181rcu_preempt09:58:510
24990998412,53cyclictest7045-21kworker/u19:1+ext4-rsv-conversion09:59:430
25010998364,12cyclictest26202-21kworker/u17:2-rpciod12:12:342
25010998354,8cyclictest181rcu_preempt09:09:282
25002998360,16cyclictest25831-21cstates08:49:421
25002998325,23cyclictest0-21swapper/110:22:151
2499099836,61cyclictest30821-21kworker/u19:0+ext4-rsv-conversion09:04:450
25002998260,15cyclictest347-21lldpd11:22:151
2500299823,65cyclictest30844-21cpu09:04:451
25002998225,26cyclictest28468-21taskset07:18:251
25002998225,26cyclictest28468-21taskset07:18:251
24990998248,11cyclictest181rcu_preempt12:00:030
24990998246,26cyclictest0-21swapper/011:49:270
24990998246,11cyclictest181rcu_preempt10:39:380
24990998241,10cyclictest181rcu_preempt07:14:440
24990998241,10cyclictest181rcu_preempt07:14:440
2499099823,33cyclictest4714-21fschecks_time07:44:450
25010998157,9cyclictest181rcu_preempt10:49:442
25002998127,25cyclictest0-21swapper/109:22:451
24990998152,22cyclictest2577-21latency_hist09:14:270
25010998055,7cyclictest181rcu_preempt11:01:592
25010998055,7cyclictest181rcu_preempt11:01:592
25010998039,17cyclictest181rcu_preempt10:09:282
25010998025,12cyclictest0-21swapper/211:49:402
25002998024,24cyclictest355-21runrttasks09:19:021
25002998024,24cyclictest355-21runrttasks09:19:021
25002998024,21cyclictest0-21swapper/111:55:461
25010997953,8cyclictest181rcu_preempt07:09:582
25010997949,11cyclictest181rcu_preempt09:34:582
25002997935,29cyclictest28973-21apt-get08:59:361
25002997929,38cyclictest14067-21latency_hist08:14:281
25002997926,23cyclictest32621-21taskset10:06:121
25002997919,27cyclictest0-21swapper/110:59:541
25002997919,27cyclictest0-21swapper/110:59:541
24990997940,25cyclictest20648-21ssh11:41:350
25010997862,10cyclictest9564-21ssh10:24:202
25010997855,9cyclictest181rcu_preempt10:42:002
25010997851,10cyclictest181rcu_preempt08:56:452
2501099783,56cyclictest17038-21cpu09:40:392
25002997851,22cyclictest24980-21cyclictest12:04:371
25002997827,24cyclictest11994-21taskset09:30:091
25002997827,24cyclictest11994-21taskset09:30:091
25002997821,27cyclictest10604-21/usr/sbin/munin11:24:571
25002997821,22cyclictest0-21swapper/111:37:341
2499099785,58cyclictest17055-21cpu09:40:390
24990997837,16cyclictest181rcu_preempt08:44:390
2499099782,28cyclictest5918-21ssh09:19:470
25010997755,9cyclictest181rcu_preempt08:34:252
25010997750,12cyclictest181rcu_preempt11:17:232
25010997750,12cyclictest181rcu_preempt11:17:232
25010997736,8cyclictest181rcu_preempt10:15:032
25010997724,33cyclictest0-21swapper/210:38:292
25010997722,12cyclictest0-21swapper/207:14:402
25010997722,12cyclictest0-21swapper/207:14:402
25002997726,22cyclictest20993-21sh12:38:121
25002997725,23cyclictest10173-21taskset09:27:581
24990997756,15cyclictest10136-21fschecks_count10:24:460
24990997756,15cyclictest10136-21fschecks_count10:24:460
24990997756,12cyclictest0-21swapper/007:19:490
24990997755,14cyclictest6779-21sendmail10:19:240
25010997642,17cyclictest23072-21ssh10:48:092
25010997637,17cyclictest181rcu_preempt11:27:172
25010997633,14cyclictest181rcu_preempt08:39:562
25010997612,36cyclictest30633-21apt-get12:00:002
24990997650,20cyclictest7383-21fschecks_time10:19:470
24990997645,9cyclictest181rcu_preempt12:04:440
24990997644,11cyclictest181rcu_preempt10:54:360
25010997556,12cyclictest309-21systemd-logind08:29:262
25010997554,8cyclictest181rcu_preempt12:29:552
25010997553,8cyclictest181rcu_preempt09:52:072
25010997551,9cyclictest28524-21ssh11:55:022
25010997551,8cyclictest181rcu_preempt08:09:392
25010997547,8cyclictest181rcu_preempt08:34:472
25010997543,10cyclictest181rcu_preempt12:29:282
25010997542,8cyclictest181rcu_preempt10:29:282
25010997542,8cyclictest181rcu_preempt10:29:282
25010997540,14cyclictest181rcu_preempt11:12:202
25002997542,23cyclictest24123-21apt-get08:44:421
25002997516,30cyclictest10477-21ntp_kernel_err12:19:541
25002997513,28cyclictest0-21swapper/109:49:481
25010997451,8cyclictest181rcu_preempt08:24:192
25010997450,8cyclictest181rcu_preempt09:59:522
25010997450,10cyclictest181rcu_preempt08:46:352
25010997433,13cyclictest181rcu_preempt11:48:402
25010997429,23cyclictest0-21swapper/209:09:562
25010997427,26cyclictest0-21swapper/211:30:012
2501099742,51cyclictest32173-21diskmemload09:17:392
2501099742,51cyclictest32173-21diskmemload09:17:392
25010997421,24cyclictest0-21swapper/210:29:582
25002997438,12cyclictest181rcu_preempt08:30:231
25002997425,44cyclictest0-21swapper/108:04:371
25002997422,23cyclictest0-21swapper/112:30:481
24990997442,23cyclictest17450-21ssh11:35:390
24990997437,28cyclictest0-21swapper/011:19:490
25010997352,8cyclictest181rcu_preempt11:42:222
25010997351,8cyclictest181rcu_preempt07:49:362
25010997346,8cyclictest181rcu_preempt07:34:432
25010997343,8cyclictest181rcu_preempt09:29:572
25010997343,8cyclictest181rcu_preempt09:29:572
25010997332,13cyclictest181rcu_preempt12:18:112
25002997354,13cyclictest12408-21latency_hist10:29:281
25002997354,13cyclictest12408-21latency_hist10:29:281
25002997349,17cyclictest13309-21/usr/sbin/munin11:29:521
25002997337,22cyclictest28852-21apt-get07:19:401
25002997328,32cyclictest20990-21/usr/sbin/munin10:44:351
25002997323,22cyclictest15811-21ssh09:38:241
25002997316,25cyclictest0-21swapper/112:14:461
2499099739,29cyclictest292-21dbus-daemon10:49:250
24990997335,30cyclictest0-21swapper/011:59:290
24990997334,25cyclictest13360-21ssh11:29:390
25010997246,9cyclictest181rcu_preempt08:03:472
25010997246,9cyclictest181rcu_preempt08:03:472
25010997243,11cyclictest181rcu_preempt12:37:192
25010997237,13cyclictest181rcu_preempt09:19:532
25010997227,14cyclictest181rcu_preempt10:13:552
25010997224,36cyclictest29186-21if_eth007:19:472
25002997238,20cyclictest4459-21apt-get07:44:351
25002997231,34cyclictest0-21swapper/110:34:361
25002997224,18cyclictest2507-21taskset07:38:541
2499099729,9cyclictest0-21swapper/007:24:380
24990997250,13cyclictest4785-21rm11:14:280
24990997238,14cyclictest181rcu_preempt12:27:220
24990997234,9cyclictest181rcu_preempt08:15:040
24990997226,25cyclictest31647-21latency_hist11:04:270
24990997226,25cyclictest31647-21latency_hist11:04:270
2499099722,32cyclictest3426-21grep09:14:570
2499099722,32cyclictest3426-21grep09:14:570
24990997213,15cyclictest15-21ksoftirqd/008:59:590
254192710,7sleep0211rcuc/011:50:000
25010997151,14cyclictest12437-21latency_hist08:09:272
25010997150,8cyclictest181rcu_preempt07:30:372
25010997148,9cyclictest181rcu_preempt12:05:012
25010997143,8cyclictest181rcu_preempt07:54:502
25010997134,14cyclictest181rcu_preempt07:40:032
25010997132,16cyclictest181rcu_preempt07:45:452
2500299714,57cyclictest21697-21ssh11:44:391
25002997126,34cyclictest27324-21/usr/sbin/munin08:54:371
25002997123,18cyclictest1456-21ssh11:08:081
24990997147,14cyclictest0-21swapper/010:39:240
24990997128,12cyclictest181rcu_preempt08:34:280
25017997011,36cyclictest0-21swapper/308:30:113
25010997044,9cyclictest181rcu_preempt08:59:592
25010997031,20cyclictest181rcu_preempt09:46:242
25002997052,11cyclictest0-21swapper/111:10:001
25002997049,15cyclictest316-21rs:main1
25002997047,15cyclictest19097-21latency_hist08:29:291
25002997033,22cyclictest2861-21apt-get07:39:441
25002997032,27cyclictest26720-21latency_hist07:14:291
25002997022,39cyclictest22279-21sh08:39:261
25002997017,24cyclictest0-21swapper/110:43:201
24990997052,12cyclictest32742-21ssh10:06:340
24990997038,20cyclictest181rcu_preempt07:10:010
24990997036,13cyclictest181rcu_preempt12:18:100
2499099703,45cyclictest11007-21head12:20:030
24990997033,7cyclictest181rcu_preempt08:19:380
2499099703,32cyclictest1516-21/usr/sbin/munin10:09:560
24990997029,22cyclictest181rcu_preempt08:09:570
2499099702,33cyclictest7873-21grep07:54:460
25010996948,8cyclictest181rcu_preempt11:20:272
25010996945,9cyclictest181rcu_preempt12:20:032
25010996932,14cyclictest181rcu_preempt08:19:282
25010996929,14cyclictest181rcu_preempt07:24:562
25002996952,11cyclictest24980-21cyclictest09:49:141
2500299693,32cyclictest6973-21grep07:50:031
25002996929,29cyclictest9535-21fschecks_time07:59:471
25002996929,29cyclictest9535-21fschecks_time07:59:471
25002996929,29cyclictest14797-21/usr/sbin/munin08:14:591
25002996913,28cyclictest0-21swapper/112:09:421
24990996932,29cyclictest0-21swapper/012:34:520
24990996932,22cyclictest20111-21ssh09:44:550
2499099692,53cyclictest11139-21fschecks_count09:29:460
2499099692,53cyclictest11139-21fschecks_count09:29:460
24990996923,9cyclictest181rcu_preempt10:29:590
25010996839,9cyclictest181rcu_preempt11:35:012
2500299683,40cyclictest31802-21awk07:29:251
25002996831,28cyclictest14513-21ssh10:32:071
25002996830,28cyclictest1825-21ssh10:09:471
25002996823,17cyclictest32173-21diskmemload10:55:181
2499099684,43cyclictest15-21ksoftirqd/009:50:000
2499099684,30cyclictest17633-21sort08:24:470
2499099682,43cyclictest2118-21tail07:35:020
2500299674,37cyclictest300-21switchtime07:29:581
2500299674,33cyclictest12824-21ssh12:24:331
2500299673,44cyclictest23894-21latency_hist08:44:271
2500299673,41cyclictest5587-21processes11:14:571
2500299673,41cyclictest5587-21processes11:14:571
24990996736,22cyclictest4316-21latency_hist07:44:280
24990996735,9cyclictest181rcu_preempt11:17:410
24990996735,9cyclictest181rcu_preempt11:17:410
2402526713,35sleep30-21swapper/307:05:023
25017996617,32cyclictest15045-21chrt08:15:033
24990996647,9cyclictest0-21swapper/008:04:390
24990996645,15cyclictest24980-21cyclictest08:54:540
24990996641,18cyclictest8855-21sh09:24:550
24990996632,7cyclictest181rcu_preempt10:49:570
24990996629,14cyclictest181rcu_preempt12:09:360
2499099655,9cyclictest181rcu_preempt07:29:590
2499099653,35cyclictest15-21ksoftirqd/011:24:480
2499099652,38cyclictest27075-21idleruntime-cro08:54:250
2499099652,31cyclictest18669-21cut12:34:280
2501799646,47cyclictest30836-21chrt09:04:403
24990996431,13cyclictest181rcu_preempt11:06:330
2501799635,47cyclictest28307-21chrt09:59:403
25017996326,24cyclictest792-21taskset11:06:043
25017996323,27cyclictest27882-21taskset08:54:563
25017996323,25cyclictest3221-21taskset10:11:553
24990996333,8cyclictest181rcu_preempt07:49:390
2501799624,42cyclictest22825-21taskset09:49:513
25017996232,19cyclictest8689-21chrt09:24:503
25017996230,21cyclictest6814-21chrt10:19:253
25017996226,22cyclictest29958-21chrt09:00:023
25017996225,26cyclictest30910-21taskset11:02:193
25017996225,26cyclictest30910-21taskset11:02:193
2501799622,41cyclictest0-21swapper/311:59:403
25017996220,29cyclictest17497-21taskset12:31:533
24990996248,10cyclictest24980-21cyclictest08:04:240
24990996248,10cyclictest24980-21cyclictest08:04:240
2501799613,40cyclictest25255-21taskset09:54:413
2501799613,40cyclictest25255-21taskset09:54:413
25017996127,21cyclictest8204-21chrt07:54:543
25017996126,22cyclictest3124-21chrt12:07:463
25017996125,24cyclictest8559-21taskset10:21:153
25017996125,23cyclictest21420-21chrt08:34:553
25017996122,25cyclictest1511-21taskset07:34:493
24990996138,14cyclictest0-21swapper/008:40:020
25017996028,20cyclictest10159-21chrt11:23:583
25017996027,21cyclictest10579-21chrt11:24:313
25017996026,22cyclictest16566-21chrt09:39:293
25017996024,22cyclictest655-21chrt09:09:553
25017996023,25cyclictest16365-21taskset08:19:573
25017996023,24cyclictest13625-21taskset08:10:493
25017996022,24cyclictest9898-21taskset12:19:333
25017996022,24cyclictest21586-21taskset09:48:043
25017996021,26cyclictest27966-21taskset10:55:593
25017996021,26cyclictest23594-21taskset10:49:273
25017995926,21cyclictest27083-21chrt11:54:213
25017995926,20cyclictest20797-21chrt12:37:403
25017995925,23cyclictest20480-21chrt11:41:043
25017995925,22cyclictest12745-21chrt10:29:423
25017995924,21cyclictest5920-21taskset09:19:473
25017995924,20cyclictest6472-21chrt07:49:513
25017995923,24cyclictest2767-21taskset09:14:373
25017995923,24cyclictest2767-21taskset09:14:373
25017995923,23cyclictest10276-21taskset10:24:483
25017995923,23cyclictest10276-21taskset10:24:483
25017995923,22cyclictest23838-21chrt10:49:393
25017995922,25cyclictest9058-21taskset12:17:553
25017995912,32cyclictest0-21swapper/309:40:033
2501799583,40cyclictest14424-21chrt11:30:023
25017995825,22cyclictest27585-21chrt11:54:413
25017995825,21cyclictest23185-21chrt11:46:183
25017995825,20cyclictest4904-21chrt07:44:493
25017995824,23cyclictest15505-21chrt12:29:263
25017995824,21cyclictest25738-21chrt08:49:373
25017995726,20cyclictest31098-21chrt10:04:303
25017995725,21cyclictest2167-21chrt11:09:323
25017995725,20cyclictest20082-21chrt10:42:383
25017995724,21cyclictest26609-21chrt07:14:283
25017995724,21cyclictest25113-21chrt08:45:043
25017995724,21cyclictest17463-21taskset11:35:413
25017995724,21cyclictest16036-21chrt10:34:523
25017995724,21cyclictest13308-21taskset09:34:223
25017995724,21cyclictest13308-21taskset09:34:223
25017995724,20cyclictest3905-21chrt07:40:483
25017995724,20cyclictest28478-21chrt07:18:333
25017995724,20cyclictest28478-21chrt07:18:333
25017995723,23cyclictest22519-21chrt08:39:373
25017995624,21cyclictest5091-21taskset11:14:423
25017995624,21cyclictest5091-21taskset11:14:423
25017995624,20cyclictest29393-21chrt07:19:553
25017995623,21cyclictest31608-21chrt07:27:083
25017995622,22cyclictest10467-21chrt08:02:113
25017995622,22cyclictest10467-21chrt08:02:113
25017995523,20cyclictest32214-21chrt07:29:433
25017995523,19cyclictest4547-21chrt12:09:533
25017995521,21cyclictest18458-21chrt08:25:323
25017995423,20cyclictest10942-21chrt08:04:293
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional