You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 02:04
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun Jul 06, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2002021969,70sleep020067-21processes07:02:030
2002021969,70sleep020067-21processes07:02:030
19615216112,113sleep119654-21fschecks_count07:01:481
19615216112,113sleep119654-21fschecks_count07:01:481
209279911746,42cyclictest31478-21/usr/sbin/munin10:11:201
209259911743,23cyclictest378-21rs:main0
20925991086,47cyclictest7080-21apt-get08:01:530
209259910651,14cyclictest20912-21cyclictest09:49:250
20927991053,81cyclictest20824-21sh09:51:421
2318021010,6sleep10-21swapper/110:55:081
209279910050,43cyclictest20328-21apt-key09:51:211
20939999521,49cyclictest0-21swapper/309:51:433
2092799958,77cyclictest23197-21apt-get09:56:341
20927999447,38cyclictest0-21swapper/112:16:591
20927999345,40cyclictest0-21swapper/109:16:451
20927999133,46cyclictest0-21swapper/110:36:351
20927999133,46cyclictest0-21swapper/110:36:351
20925999068,9cyclictest181rcu_preempt09:41:420
20925999068,9cyclictest181rcu_preempt09:41:420
2093999897,45cyclictest0-21swapper/312:01:593
20927998940,42cyclictest0-21swapper/107:51:581
20927998934,44cyclictest0-21swapper/112:02:251
2092799884,79cyclictest0-21swapper/110:41:351
2092799883,72cyclictest0-21swapper/108:21:301
207632886,59sleep20-21swapper/207:06:172
207632886,59sleep20-21swapper/207:06:172
20927998740,40cyclictest0-21swapper/112:21:521
20927998655,24cyclictest4768-21kworker/u19:2+events_unbound10:36:201
20927998655,24cyclictest4768-21kworker/u19:2+events_unbound10:36:201
2092799863,76cyclictest0-21swapper/111:46:381
2092799863,76cyclictest0-21swapper/111:46:381
2092799863,68cyclictest7080-21apt-get08:01:351
20939998539,28cyclictest0-21swapper/309:56:493
20934998540,35cyclictest21587-21ps09:52:292
2092799853,68cyclictest3168-21apt-get12:11:311
2092799853,54cyclictest0-21swapper/111:06:591
20927998534,41cyclictest0-21swapper/109:01:461
2092799843,65cyclictest0-21swapper/111:26:311
20927998362,14cyclictest23946-21cpu10:56:351
20927998362,14cyclictest23946-21cpu10:56:351
2092799834,68cyclictest0-21swapper/107:46:341
2092799833,73cyclictest0-21swapper/107:31:311
2092799833,73cyclictest0-21swapper/107:31:311
2092799833,68cyclictest0-21swapper/111:41:331
2092799833,68cyclictest0-21swapper/108:16:331
20925998352,10cyclictest181rcu_preempt10:11:370
20927998261,15cyclictest3134-21unixbench_singl10:17:001
2092799823,65cyclictest0-21swapper/108:31:341
2092599823,63cyclictest10280-21/usr/sbin/munin08:11:580
2092599823,63cyclictest10280-21/usr/sbin/munin08:11:580
20934998164,11cyclictest4357-21kworker/u18:2-nfsiod09:26:202
20934998157,16cyclictest25453-21ssh09:59:212
20934998139,8cyclictest181rcu_preempt11:47:002
20934998139,8cyclictest181rcu_preempt11:47:002
2092799813,73cyclictest0-21swapper/110:26:311
2092799813,65cyclictest0-21swapper/109:26:301
2092799813,65cyclictest0-21swapper/109:26:301
20925998153,9cyclictest181rcu_preempt11:16:550
2092799803,61cyclictest4153-21apt-get09:21:311
20927998034,38cyclictest0-21swapper/111:56:371
20927998034,38cyclictest0-21swapper/111:56:371
20927998032,40cyclictest0-21swapper/107:56:551
2092799802,71cyclictest0-21swapper/109:11:321
2092799802,71cyclictest0-21swapper/109:11:321
20934997964,11cyclictest20912-21cyclictest10:34:222
20934997964,11cyclictest20912-21cyclictest10:34:222
2092799793,65cyclictest0-21swapper/107:21:321
2092799792,71cyclictest0-21swapper/110:01:301
20925997958,15cyclictest9398-21latency_hist09:31:220
20925997958,15cyclictest9398-21latency_hist09:31:220
20925997958,14cyclictest2785-21latency_hist11:16:210
20925997956,17cyclictest1775-21latency_hist07:46:200
20925997951,20cyclictest8677-21latency_hist11:26:220
20925997951,20cyclictest8677-21latency_hist11:26:220
20934997859,12cyclictest0-21swapper/212:11:412
20934997857,14cyclictest20912-21cyclictest07:21:222
20934997844,11cyclictest559-21sendmail-mta11:35:512
20927997853,15cyclictest0-21swapper/108:11:591
20927997853,15cyclictest0-21swapper/108:11:591
20934997756,14cyclictest20532-21kworker/u17:1+xprtiod09:16:012
20934997756,14cyclictest20532-21kworker/u17:1+xprtiod09:16:012
20934997729,13cyclictest14740-21ssh11:36:362
2092799773,68cyclictest0-21swapper/109:36:331
20927997734,36cyclictest0-21swapper/111:31:471
20927997733,37cyclictest0-21swapper/108:51:191
2092799772,62cyclictest0-21swapper/107:26:371
20925997758,12cyclictest27429-21ssh10:01:570
20925997755,8cyclictest181rcu_preempt10:16:290
20925997742,25cyclictest15125-21sh10:41:180
20925997742,25cyclictest15125-21sh10:41:180
2093999762,58cyclictest7097-21chrt08:01:423
2093499765,50cyclictest29722-21apt-get12:01:342
20934997650,7cyclictest181rcu_preempt08:41:332
20934997650,7cyclictest181rcu_preempt08:41:332
20934997646,9cyclictest181rcu_preempt07:56:302
20934997639,10cyclictest181rcu_preempt09:16:472
2092799764,57cyclictest313-21latency_hist07:41:221
2092799764,57cyclictest313-21latency_hist07:41:221
2092799763,61cyclictest0-21swapper/108:56:351
20927997635,33cyclictest0-21swapper/108:11:211
20927997634,35cyclictest0-21swapper/112:26:481
20927997634,33cyclictest0-21swapper/111:56:211
20927997633,36cyclictest0-21swapper/107:41:361
20925997661,9cyclictest26074-21ssh11:56:030
20925997654,15cyclictest11732-21latency_hist12:26:200
2093999753,38cyclictest23652-21taskset08:52:133
20934997558,11cyclictest356-21runrttasks11:06:462
20934997553,13cyclictest0-21swapper/208:26:552
20934997548,18cyclictest9320-21fschecks_count12:21:402
20934997537,15cyclictest19393-21ssh09:48:462
2092799753,65cyclictest0-21swapper/111:11:351
2092799753,62cyclictest5852-21apt-get11:21:341
2092799753,62cyclictest5852-21apt-get11:21:341
2092799753,56cyclictest26930-21ssh11:01:401
2092799752,68cyclictest0-21swapper/107:16:321
20925997553,16cyclictest18244-21latency_hist08:36:220
2092599754,54cyclictest24993-21latency_hist08:56:220
20925997537,29cyclictest10902-21ssh10:31:520
20925997537,29cyclictest10902-21ssh10:31:520
20925997533,12cyclictest181rcu_preempt11:56:370
20925997533,12cyclictest181rcu_preempt11:56:370
20925997531,35cyclictest0-21swapper/011:00:420
20925997531,35cyclictest0-21swapper/011:00:420
20934997458,11cyclictest20912-21cyclictest12:08:372
20934997432,12cyclictest181rcu_preempt12:31:212
20927997448,18cyclictest20912-21cyclictest09:41:551
20927997448,18cyclictest20912-21cyclictest09:41:551
2092799744,65cyclictest0-21swapper/110:46:361
2092799744,65cyclictest0-21swapper/110:46:361
2092799743,66cyclictest0-21swapper/110:21:331
2092799743,65cyclictest0-21swapper/108:52:051
2092799743,64cyclictest0-21swapper/112:06:341
2092799743,61cyclictest2932-21apt-get11:16:321
2092799743,46cyclictest356-21runrttasks08:36:471
20925997459,9cyclictest23153-21rm09:56:130
20925997458,10cyclictest14325-21ssh09:40:150
20925997457,11cyclictest6417-21ssh09:25:520
20925997455,13cyclictest30078-21ssh10:06:570
20925997443,11cyclictest181rcu_preempt09:01:370
20925997437,27cyclictest8889-21grep08:06:420
20925997436,10cyclictest181rcu_preempt11:31:410
20925997410,12cyclictest181rcu_preempt11:31:200
20934997354,12cyclictest11915-21/usr/sbin/munin08:16:502
2093499734,46cyclictest23453-21apt-get08:51:552
2093499732,51cyclictest19853-21latency_hist08:41:222
2092799734,62cyclictest0-21swapper/111:36:311
2092799733,65cyclictest0-21swapper/109:31:341
2092799733,65cyclictest0-21swapper/107:06:351
2092799733,65cyclictest0-21swapper/107:06:351
20934997254,11cyclictest5970-21cstates11:21:402
20934997254,11cyclictest5970-21cstates11:21:402
20934997252,10cyclictest18068-21kernelversion11:41:452
20934997225,14cyclictest181rcu_preempt11:56:482
20934997225,14cyclictest181rcu_preempt11:56:482
2092799723,59cyclictest29366-21grep09:06:561
2092799722,5cyclictest0-21swapper/107:11:321
2092799722,48cyclictest16571-21latency_hist08:31:211
2092799722,31cyclictest16257-21runrttasks12:32:251
20925997252,13cyclictest365-21systemd-logind07:26:200
20925997251,8cyclictest181rcu_preempt07:36:210
20925997251,8cyclictest181rcu_preempt07:36:210
20925997251,14cyclictest2826-21cron12:11:180
20925997249,17cyclictest4886-21cpu10:21:350
20925997248,8cyclictest181rcu_preempt09:14:320
20925997248,8cyclictest181rcu_preempt09:14:320
20925997248,10cyclictest181rcu_preempt09:57:270
2092599723,30cyclictest18371-21/usr/sbin/munin08:36:470
2093999717,34cyclictest8571-21taskset10:26:583
20934997152,13cyclictest2342-21kernelversion07:46:442
20934997148,13cyclictest0-21swapper/210:51:542
20934997148,12cyclictest0-21swapper/207:32:042
20934997148,12cyclictest0-21swapper/207:32:042
20934997142,19cyclictest6633-21cat12:16:462
20934997132,30cyclictest9825-21cron10:31:182
2092799713,62cyclictest0-21swapper/110:11:321
20925997158,8cyclictest15-21ksoftirqd/011:08:090
20925997157,10cyclictest20912-21cyclictest11:45:010
20925997155,9cyclictest0-21swapper/012:02:230
20925997148,17cyclictest8864-21ssh12:21:210
20925997144,11cyclictest181rcu_preempt07:06:370
20925997144,11cyclictest181rcu_preempt07:06:370
20925997140,11cyclictest181rcu_preempt08:16:390
2093999703,29cyclictest28224-21chrt09:06:223
20934997051,13cyclictest31501-21systemd-run10:11:192
20934997050,13cyclictest3475-21kworker/u17:2-rpciod11:16:492
2093499703,50cyclictest14936-21latency_hist08:26:212
2093499702,49cyclictest26733-21cpu09:01:362
2092799702,57cyclictest19998-21apt-get08:41:321
2092799702,57cyclictest19998-21apt-get08:41:321
20925997054,10cyclictest22319-21ssh11:48:430
20925997054,10cyclictest22319-21ssh11:48:430
2092599703,35cyclictest24780-21apt-get07:16:340
20925997017,12cyclictest181rcu_preempt09:06:410
2093499694,43cyclictest8474-21latency_hist08:06:212
20934996936,8cyclictest7175-21modprobe10:26:182
20925996954,8cyclictest15-21ksoftirqd/008:56:220
20925996953,10cyclictest7848-21ssh10:26:430
20925996952,11cyclictest15397-21/usr/sbin/munin10:41:470
20925996944,10cyclictest181rcu_preempt08:41:310
20925996944,10cyclictest181rcu_preempt08:41:310
2093999682,28cyclictest27273-21chrt07:21:573
2093499685,44cyclictest24182-21/usr/sbin/munin11:51:512
2093499684,44cyclictest811-21sendmail_mailtr11:11:562
20934996841,18cyclictest26431-21modprobe09:01:192
20934996812,22cyclictest26559-21fschecks_count07:21:422
20925996853,9cyclictest20558-21kworker/u17:4+rpciod10:51:430
20925996826,26cyclictest19097-21timerandwakeup10:46:560
20925996826,26cyclictest19097-21timerandwakeup10:46:560
20925996824,34cyclictest15154-21grep11:36:450
2093999673,23cyclictest20968-21chrt10:51:303
2093999673,23cyclictest20968-21chrt10:51:303
2093999672,27cyclictest6567-21chrt11:21:573
2093999672,27cyclictest6567-21chrt11:21:573
20934996753,10cyclictest28960-21kworker/u17:0-rpciod09:11:132
2093499674,44cyclictest356-21runrttasks12:32:252
20934996742,13cyclictest0-21swapper/207:41:202
20934996742,13cyclictest0-21swapper/207:41:202
2093499673,43cyclictest21084-21/usr/sbin/munin07:06:392
2093499673,43cyclictest21084-21/usr/sbin/munin07:06:392
20934996731,12cyclictest181rcu_preempt10:44:252
20925996738,11cyclictest181rcu_preempt09:16:450
2092599673,56cyclictest31435-21fschecks_count07:36:440
2092599673,56cyclictest31435-21fschecks_count07:36:440
20925996731,26cyclictest5509-21grep07:56:410
20925996718,24cyclictest8220-21kworker/u17:1-rpciod12:29:340
2093499665,32cyclictest15127-21latency_hist10:41:202
2093499665,32cyclictest15127-21latency_hist10:41:202
20934996644,14cyclictest58-21kcompactd008:35:332
20934996642,12cyclictest0-21swapper/209:41:422
20934996642,12cyclictest0-21swapper/209:41:422
20934996635,8cyclictest10593-21sort08:11:452
20934996635,8cyclictest10593-21sort08:11:452
2093499663,47cyclictest29039-21latency_hist10:06:242
2093499663,41cyclictest356-21runrttasks07:55:392
20925996651,9cyclictest3751-21diskmemload12:11:490
20925996644,7cyclictest181rcu_preempt08:26:550
20925996641,11cyclictest181rcu_preempt12:36:110
2093999654,27cyclictest22105-21taskset08:46:503
2093999652,26cyclictest26918-21taskset11:01:393
2093499654,36cyclictest356-21runrttasks11:28:052
2093499654,35cyclictest25525-21sh10:58:342
2093499654,35cyclictest25525-21sh10:58:342
20934996539,16cyclictest0-21swapper/208:06:422
20925996548,9cyclictest0-21swapper/008:21:550
20925996540,16cyclictest26715-21/usr/sbin/munin11:01:520
2092599652,53cyclictest21595-21/usr/sbin/munin08:46:430
2092599652,40cyclictest1973-21/usr/sbin/munin07:46:470
2093999644,25cyclictest14135-21chrt12:29:493
2093999643,36cyclictest29260-21taskset09:06:553
2093999643,26cyclictest9677-21taskset09:31:393
2093999643,24cyclictest1590-21chrt12:07:443
2093999643,24cyclictest14771-21chrt10:40:033
2093999643,24cyclictest14771-21chrt10:40:033
2093999643,23cyclictest30295-21chrt11:06:533
2093999643,23cyclictest28070-21chrt11:58:123
2093999643,23cyclictest28070-21chrt11:58:123
2093999643,23cyclictest22476-21chrt07:11:193
2093999643,23cyclictest22476-21chrt07:11:193
2093999643,23cyclictest18508-21chrt09:46:553
2093999642,28cyclictest26474-21chrt09:01:203
2093999642,23cyclictest31884-21chrt07:36:553
2093999642,23cyclictest31884-21chrt07:36:553
2093499644,42cyclictest26912-21df11:01:382
20934996444,10cyclictest0-21swapper/210:48:462
20934996444,10cyclictest0-21swapper/210:48:462
2093499643,46cyclictest12509-21fschecks_time09:36:432
2093499643,32cyclictest28863-21taskset07:26:572
2093499643,32cyclictest24629-21latency_hist07:16:212
20934996432,20cyclictest32352-21ntpq10:11:482
20934996427,13cyclictest3732-21ssh10:18:472
20925996442,8cyclictest181rcu_preempt07:51:530
201502647,39sleep30-21swapper/307:02:063
201502647,39sleep30-21swapper/307:02:063
2093999633,26cyclictest4761-21taskset09:21:513
2093999633,23cyclictest7790-21chrt09:26:533
2093999633,23cyclictest7790-21chrt09:26:533
2093999633,23cyclictest28401-21chrt07:26:473
2093999633,23cyclictest17633-21chrt11:41:303
2093999633,22cyclictest7483-21taskset12:17:373
2093999633,22cyclictest15616-21chrt09:41:523
2093999633,22cyclictest15616-21chrt09:41:523
2093999633,20cyclictest6757-21chrt10:24:563
2093999632,29cyclictest23044-21taskset11:50:503
2093999632,29cyclictest23044-21taskset11:50:503
2093999632,25cyclictest24947-21chrt11:52:493
2093999632,23cyclictest15156-21chrt08:26:333
2093999632,22cyclictest8785-21chrt11:26:313
2093999632,22cyclictest21215-21chrt10:51:383
2093999632,20cyclictest4649-21chrt12:13:033
2093499633,42cyclictest23004-21munin-run08:51:192
2093499633,33cyclictest10494-21unixbench_multi09:31:562
2093499633,32cyclictest1926-21latency_hist07:46:222
20925996346,11cyclictest9475-21/usr/sbin/munin09:31:410
20925996317,37cyclictest29315-21cron07:31:190
2093999624,22cyclictest17097-21chrt08:31:433
2093999623,24cyclictest16826-21chrt12:34:023
2093999623,23cyclictest3612-21chrt10:18:233
2093999623,23cyclictest18670-21chrt08:36:413
2093999623,23cyclictest10316-21chrt08:11:383
2093999623,23cyclictest10316-21chrt08:11:383
2093999623,22cyclictest10615-21chrt12:23:083
2093999622,26cyclictest670-21chrt11:11:523
2093999622,25cyclictest32441-21taskset09:12:433
2093999622,25cyclictest32441-21taskset09:12:433
2093999622,25cyclictest10468-21taskset10:31:403
2093999622,25cyclictest10468-21taskset10:31:403
2093999622,24cyclictest13471-21taskset11:33:483
2093999622,22cyclictest29760-21chrt07:31:413
2093999622,22cyclictest29760-21chrt07:31:413
2093999622,22cyclictest16628-21chrt11:39:193
2093999622,21cyclictest32641-21taskset10:11:553
2093999622,21cyclictest24428-21chrt07:15:453
20934996237,16cyclictest0-21swapper/209:26:542
20934996237,16cyclictest0-21swapper/209:26:542
2093999614,20cyclictest4057-21chrt11:17:033
2093999613,23cyclictest31437-21chrt10:11:143
2093999613,23cyclictest21090-21chrt08:43:093
2093999613,23cyclictest21090-21chrt08:43:093
2093999613,22cyclictest13276-21chrt09:36:583
2093999612,25cyclictest390-21taskset07:41:333
2093999612,23cyclictest25213-21chrt10:57:333
2093999612,23cyclictest25213-21chrt10:57:333
2093999612,22cyclictest11935-21chrt08:16:283
2093999612,21cyclictest27659-21chrt10:02:393
2093999612,21cyclictest2180-21chrt09:16:563
2093999609,37cyclictest0-21swapper/310:41:403
2093999603,25cyclictest25801-21chrt07:17:233
2093999603,21cyclictest3162-21chrt07:49:193
2093999603,21cyclictest14634-21chrt08:23:133
2093999603,19cyclictest4747-21chrt07:54:003
2093999602,21cyclictest8848-21chrt08:06:403
2093999592,21cyclictest6505-21chrt08:00:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional