You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 07:22
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Nov 20, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
112179912353,12cyclictest0-21swapper/223:55:082
112179912353,12cyclictest0-21swapper/223:55:082
11042212115,71sleep111065-21munin-plugin-st19:09:311
11042212115,71sleep111065-21munin-plugin-st19:09:311
11210991122,48cyclictest9177-21ssh22:55:091
11210991122,48cyclictest9177-21ssh22:55:091
112109911147,19cyclictest19880-21kworker/u17:1+events_unbound20:19:311
112179911044,15cyclictest0-21swapper/220:59:342
11222991053,84cyclictest9074-21taskset22:54:523
11222991053,84cyclictest9074-21taskset22:54:523
112179910542,18cyclictest23592-21fschecks_count22:24:512
112179910253,10cyclictest7225-21idleruntime23:44:592
11217991023,64cyclictest9584-21apt-get21:04:552
1121099973,63cyclictest24470-21apt-get00:14:451
1121099973,63cyclictest24470-21apt-get00:14:451
11210999478,9cyclictest11300-21kworker/u17:3-rpciod00:35:121
1121799923,66cyclictest25011-21apt-get00:14:462
1121799923,66cyclictest25011-21apt-get00:14:462
1121799922,66cyclictest9054-21ssh22:54:482
1121799922,66cyclictest9054-21ssh22:54:482
1121099923,64cyclictest20614-21apt-get23:14:451
1121099923,64cyclictest20614-21apt-get23:14:451
11217999059,13cyclictest181rcu_preempt23:22:362
11202999047,26cyclictest2370-21modprobe20:44:310
11217998971,11cyclictest24775-21kworker/u17:2-xprtiod22:34:552
11210998761,18cyclictest1-21systemd23:12:041
11210998761,18cyclictest1-21systemd23:12:041
989828610,52sleep29910-21fschecks_count19:05:002
989828610,52sleep29910-21fschecks_count19:05:002
11217998623,30cyclictest0-21swapper/200:30:172
11210998545,20cyclictest0-21swapper/119:59:511
11210998461,17cyclictest3992-21ssh00:34:331
11217998360,16cyclictest11192-21cyclictest20:59:552
11217998355,20cyclictest355-21runrttasks21:49:492
11202998350,11cyclictest181rcu_preempt21:25:560
11202998350,11cyclictest181rcu_preempt21:25:560
11210998263,12cyclictest0-21swapper/121:05:281
1121799812,57cyclictest1036-21apt-get20:39:572
11210998144,15cyclictest181rcu_preempt21:19:561
11210998127,30cyclictest2640-21sh23:37:011
1121099812,60cyclictest27370-21apt-get23:24:461
11217998057,13cyclictest0-21swapper/220:09:312
11217998055,14cyclictest0-21swapper/220:45:242
11210998059,15cyclictest14153-21idleruntime-cro22:09:311
11210998059,15cyclictest14153-21idleruntime-cro22:09:311
11202998058,15cyclictest30658-21sh23:29:540
11202998047,11cyclictest11192-21cyclictest21:34:330
974527922,34sleep30-21swapper/319:04:523
974527922,34sleep30-21swapper/319:04:523
11217997957,12cyclictest0-21swapper/223:53:052
11217997951,9cyclictest181rcu_preempt20:09:562
11217997942,9cyclictest181rcu_preempt21:33:192
11210997964,8cyclictest0-21swapper/100:04:571
11210997959,10cyclictest0-21swapper/123:30:041
11210997956,13cyclictest0-21swapper/120:40:481
11202997952,11cyclictest355-21runrttasks23:39:500
11202997942,11cyclictest181rcu_preempt21:15:040
976327811,36sleep00-21swapper/019:04:540
976327811,36sleep00-21swapper/019:04:540
11217997859,7cyclictest181rcu_preempt22:49:472
11217997858,10cyclictest0-21swapper/200:20:042
11217997854,17cyclictest30548-21fschecks_count23:29:532
11217997829,29cyclictest21149-21/usr/sbin/munin23:15:312
11217997829,29cyclictest21149-21/usr/sbin/munin23:15:312
11210997848,21cyclictest3085-21uniq20:45:101
11210997847,9cyclictest4567-21ssh22:47:031
1122299772,53cyclictest1330-21taskset00:29:503
11217997753,18cyclictest10946-21diskmemload23:26:082
11217997733,18cyclictest58-21kcompactd023:04:582
11217997721,26cyclictest0-21swapper/221:37:332
11217997721,26cyclictest0-21swapper/221:37:332
11202997761,10cyclictest27192-21unixbench_singl21:35:080
11202997761,10cyclictest27192-21unixbench_singl21:35:080
11202997756,15cyclictest691-21cron20:29:300
11202997756,15cyclictest691-21cron20:29:300
11202997751,8cyclictest181rcu_preempt23:50:000
1120299774,46cyclictest15-21ksoftirqd/000:19:450
1120299773,34cyclictest15-21ksoftirqd/022:19:460
11217997660,10cyclictest17143-21cpu19:49:552
11217997657,13cyclictest645-21gpgv20:39:342
11217997657,13cyclictest645-21gpgv20:39:342
11217997657,10cyclictest0-21swapper/220:50:092
11217997655,8cyclictest181rcu_preempt21:25:592
11217997655,8cyclictest181rcu_preempt21:25:592
11217997655,10cyclictest0-21swapper/223:14:342
11217997655,10cyclictest0-21swapper/223:14:342
11217997653,13cyclictest0-21swapper/219:44:332
11217997644,9cyclictest181rcu_preempt21:24:262
11217997626,19cyclictest13762-21kworker/u17:0-ext4-rsv-conversion20:34:332
11217997626,19cyclictest13762-21kworker/u17:0-ext4-rsv-conversion20:34:332
11210997660,11cyclictest0-21swapper/119:51:301
11210997658,11cyclictest0-21swapper/120:35:001
11210997658,11cyclictest0-21swapper/120:35:001
11210997656,10cyclictest0-21swapper/123:19:521
11210997656,10cyclictest0-21swapper/121:09:431
1121099762,48cyclictest355-21runrttasks19:29:541
11202997654,15cyclictest8459-21latency_hist22:54:340
11217997551,13cyclictest0-21swapper/219:54:482
11217997551,13cyclictest0-21swapper/219:54:482
11217997540,7cyclictest181rcu_preempt22:03:272
11217997534,11cyclictest181rcu_preempt19:35:582
11210997555,9cyclictest0-21swapper/122:14:531
11210997555,9cyclictest0-21swapper/122:14:531
11210997554,15cyclictest8819-21sh21:59:551
11210997551,14cyclictest0-21swapper/122:29:551
11202997545,24cyclictest4295-21latency_hist00:34:350
11217997453,14cyclictest0-21swapper/221:09:572
11217997450,13cyclictest0-21swapper/222:59:482
11217997445,16cyclictest181rcu_preempt22:14:592
11217997445,16cyclictest181rcu_preempt22:14:592
11210997460,8cyclictest0-21swapper/121:49:491
11210997459,10cyclictest316-21rs:main1
11210997454,11cyclictest0-21swapper/121:30:071
11210997451,10cyclictest0-21swapper/119:14:441
11210997448,14cyclictest0-21swapper/122:14:341
11210997447,12cyclictest0-21swapper/100:24:331
11202997460,9cyclictest58-21kcompactd019:56:580
11202997460,9cyclictest58-21kcompactd019:56:580
11202997458,10cyclictest32065-21ssh21:44:440
11202997456,11cyclictest15570-21systemctl19:39:310
11202997444,23cyclictest0-21swapper/023:45:040
11217997356,10cyclictest0-21swapper/222:40:072
11217997354,13cyclictest19757-21timerandwakeup00:05:082
11217997353,10cyclictest0-21swapper/221:39:522
11217997352,12cyclictest0-21swapper/220:24:572
11217997352,12cyclictest0-21swapper/220:24:572
11217997352,10cyclictest0-21swapper/200:25:092
11210997324,33cyclictest20456-21ssh21:24:551
11210997324,33cyclictest20456-21ssh21:24:551
11202997356,10cyclictest677-21latency_hist20:39:330
11202997356,10cyclictest677-21latency_hist20:39:330
11202997354,13cyclictest5835-21idleruntime-cro20:54:310
11202997354,11cyclictest10946-21diskmemload23:56:410
11202997354,11cyclictest10946-21diskmemload23:56:410
11202997344,23cyclictest26340-21/usr/sbin/munin22:30:000
11217997252,9cyclictest0-21swapper/221:15:072
11217997246,8cyclictest181rcu_preempt00:35:142
11210997254,10cyclictest0-21swapper/122:49:521
11210997246,9cyclictest181rcu_preempt21:56:101
11210997222,9cyclictest181rcu_preempt23:04:311
11202997252,13cyclictest10946-21diskmemload23:17:460
11202997252,13cyclictest10946-21diskmemload23:17:460
11202997250,16cyclictest316-21rs:main0
190552710,3sleep0161ktimers/000:04:510
11217997144,10cyclictest181rcu_preempt22:06:272
11217997144,10cyclictest181rcu_preempt22:06:272
11217997123,24cyclictest0-21swapper/223:40:552
11210997154,12cyclictest0-21swapper/123:40:031
11210997153,13cyclictest0-21swapper/119:29:351
11210997153,13cyclictest0-21swapper/119:29:351
11210997152,9cyclictest0-21swapper/119:54:491
11210997152,9cyclictest0-21swapper/119:54:491
11210997151,11cyclictest0-21swapper/121:04:321
11210997150,12cyclictest0-21swapper/123:49:591
11210997150,11cyclictest0-21swapper/123:54:591
11210997150,11cyclictest0-21swapper/123:54:591
11210997150,10cyclictest0-21swapper/119:44:331
11210997147,13cyclictest0-21swapper/120:24:351
11210997147,13cyclictest0-21swapper/120:24:351
11210997131,14cyclictest2678-21kworker/u17:1-xprtiod00:00:091
11210997130,8cyclictest181rcu_preempt20:54:461
11217997055,9cyclictest0-21swapper/221:45:032
11217997052,13cyclictest11192-21cyclictest22:33:062
11217997051,8cyclictest0-21swapper/222:24:332
11217997050,11cyclictest0-21swapper/222:47:532
11217997046,14cyclictest0-21swapper/200:14:332
11210997055,8cyclictest0-21swapper/121:35:001
11210997055,8cyclictest0-21swapper/121:35:001
11210997054,8cyclictest0-21swapper/120:34:331
11210997054,8cyclictest0-21swapper/120:34:331
11210997053,11cyclictest0-21swapper/123:04:551
11210997047,17cyclictest0-21swapper/122:34:571
11210997027,29cyclictest23447-21wc20:09:341
11202997055,9cyclictest8399-21rm21:59:340
11202997051,13cyclictest14064-21latency_hist19:29:340
11202997051,13cyclictest14064-21latency_hist19:29:340
11202997045,18cyclictest11396-21ssh21:09:500
317762690,7sleep10-21swapper/121:44:311
11217996946,15cyclictest0-21swapper/221:55:062
11217996945,8cyclictest181rcu_preempt22:13:002
11210996955,8cyclictest0-21swapper/100:14:331
11210996954,10cyclictest23057-21cron22:24:301
11202996955,10cyclictest11192-21cyclictest21:40:020
11202996953,10cyclictest14722-21ssh23:04:310
11202996943,11cyclictest181rcu_preempt19:11:300
11202996943,11cyclictest181rcu_preempt19:11:300
11202996928,30cyclictest31219-21ntpq00:24:590
11217996853,8cyclictest0-21swapper/223:35:042
11217996850,12cyclictest20396-21cpuspeed19:59:512
11217996849,9cyclictest0-21swapper/220:15:062
1121799682,44cyclictest14063-21apt-get19:29:452
11210996851,11cyclictest11192-21cyclictest23:45:271
11210996844,18cyclictest0-21swapper/121:14:461
11202996854,10cyclictest2-21kthreadd19:54:320
11202996853,10cyclictest11192-21cyclictest22:46:040
11202996852,10cyclictest18702-21ssh22:15:390
11202996852,10cyclictest18702-21ssh22:15:390
11202996848,14cyclictest21038-21latency_hist23:14:350
11202996848,14cyclictest21038-21latency_hist23:14:350
11202996839,22cyclictest11300-21kworker/u17:3+nfsiod00:16:320
11202996839,22cyclictest11300-21kworker/u17:3+nfsiod00:16:320
11202996830,26cyclictest8455-21kworker/u17:1+nfsiod22:56:380
11202996830,26cyclictest8455-21kworker/u17:1+nfsiod22:56:380
11210996751,7cyclictest0-21swapper/100:25:051
11210996750,11cyclictest0-21swapper/120:09:551
11210996749,7cyclictest0-21swapper/122:40:101
11210996746,10cyclictest0-21swapper/120:54:331
11210996743,10cyclictest181rcu_preempt20:24:541
11210996743,10cyclictest181rcu_preempt20:24:541
11202996750,11cyclictest9229-21kworker/0:121:04:330
11202996750,11cyclictest5288-21ssh21:54:330
11202996750,11cyclictest20315-21/usr/sbin/munin20:00:070
11202996736,22cyclictest16189-21modprobe19:44:310
11202996713,33cyclictest15-21ksoftirqd/023:22:360
11217996648,9cyclictest0-21swapper/220:19:572
11217996648,9cyclictest0-21swapper/220:19:572
11217996645,12cyclictest0-21swapper/223:59:532
11217996631,7cyclictest181rcu_preempt19:45:142
11210996652,8cyclictest0-21swapper/119:39:331
11202996649,11cyclictest12801-21cron19:19:310
11202996647,13cyclictest10946-21diskmemload00:36:380
11202996639,20cyclictest28752-21kworker/u17:4+nfsiod22:13:440
11202996620,33cyclictest999-21ls23:34:470
1122299658,20cyclictest0-21swapper/323:24:493
11217996547,11cyclictest0-21swapper/219:14:332
11217996547,11cyclictest0-21swapper/219:14:332
1121799653,38cyclictest12981-21latency_hist19:19:342
11202996549,10cyclictest26520-21cron20:19:310
11202996548,10cyclictest23409-21ssh22:24:460
11202996547,11cyclictest369-21ntpd23:26:090
11202996544,15cyclictest29296-21/usr/sbin/munin22:34:540
11202996537,21cyclictest28752-21kworker/u17:4+nfsiod22:08:220
11202996537,21cyclictest28752-21kworker/u17:4+nfsiod22:08:220
11202996535,23cyclictest0-21swapper/000:14:340
1122299648,26cyclictest0-21swapper/300:14:463
1122299648,26cyclictest0-21swapper/300:14:463
11217996430,7cyclictest181rcu_preempt19:24:342
11210996449,8cyclictest0-21swapper/119:49:351
11202996449,11cyclictest11192-21cyclictest21:23:000
1120299643,46cyclictest9002-21latency21:59:590
11210996332,11cyclictest23615-21ssh22:24:511
11202996346,11cyclictest783-21timerwakeupswit22:40:100
11222996211,21cyclictest0-21swapper/321:04:553
11210996245,9cyclictest0-21swapper/119:14:321
11210996245,9cyclictest0-21swapper/119:14:321
11210996231,23cyclictest13439-21sh19:24:311
1120299627,31cyclictest15-21ksoftirqd/020:04:460
1120299625,41cyclictest6442-21runrttasks20:54:530
11202996226,27cyclictest14985-21/usr/sbin/munin23:05:000
11202996141,13cyclictest355-21runrttasks20:24:340
11202996141,13cyclictest355-21runrttasks20:24:340
1120299613,28cyclictest3826-21unixbench_multi20:45:270
1120299604,27cyclictest9874-21fschecks_time21:05:130
11217995923,7cyclictest181rcu_preempt19:25:402
11217995923,7cyclictest181rcu_preempt19:25:402
11202995942,10cyclictest181rcu_preempt19:44:470
11202995941,13cyclictest11192-21cyclictest20:09:500
1122299583,37cyclictest7486-21taskset21:57:383
11202995844,10cyclictest0-21swapper/019:20:540
1120299583,39cyclictest31354-21latency_hist20:34:300
1120299583,39cyclictest31354-21latency_hist20:34:300
1120299583,39cyclictest14538-21runrttasks19:29:360
1122299563,29cyclictest0-21swapper/319:29:453
1122299543,30cyclictest0-21swapper/323:14:453
1122299543,30cyclictest0-21swapper/323:14:453
1122299542,41cyclictest19195-21taskset22:17:033
1122299542,41cyclictest19195-21taskset22:17:033
1122299542,35cyclictest16424-21taskset21:18:083
1122299542,34cyclictest25002-21taskset23:20:043
1122299532,36cyclictest621-21taskset00:29:133
1122299532,33cyclictest15547-21chrt23:59:313
1122299532,33cyclictest15547-21chrt23:59:313
1122299523,34cyclictest24460-21taskset00:14:303
11222995211,22cyclictest0-21swapper/320:39:573
1122299513,33cyclictest30188-21taskset22:35:063
1122299503,34cyclictest25880-21chrt22:29:073
1122299503,34cyclictest22218-21taskset22:22:103
1122299503,32cyclictest7480-21taskset23:45:043
1122299503,31cyclictest9544-21chrt22:00:103
1122299503,31cyclictest9476-21chrt21:04:343
1122299503,31cyclictest356-21chrt23:34:023
1122299503,31cyclictest14612-21chrt22:09:463
1122299503,28cyclictest5253-21taskset20:50:093
1122299493,30cyclictest23901-21chrt20:09:593
1122299493,29cyclictest4606-21taskset00:35:013
1122299492,32cyclictest21378-21chrt21:25:213
1122299492,32cyclictest21378-21chrt21:25:213
1122299492,31cyclictest20208-21chrt23:13:233
1122299492,31cyclictest20208-21chrt23:13:233
1122299483,34cyclictest2440-21taskset22:44:263
1122299483,32cyclictest6432-21chrt23:44:153
1122299483,29cyclictest2826-21chrt22:44:473
1122299473,32cyclictest18557-21taskset21:20:433
1122299472,31cyclictest12965-21chrt21:11:483
1122299472,30cyclictest13529-21taskset22:07:473
1122299472,30cyclictest13529-21taskset22:07:473
1122299463,32cyclictest12014-21chrt22:59:423
1122299463,27cyclictest8884-21chrt22:54:353
1122299463,26cyclictest26981-21chrt20:19:553
1122299463,26cyclictest26981-21chrt20:19:553
11222994612,22cyclictest26092-21chrt21:34:413
11222994612,22cyclictest26092-21chrt21:34:413
1122299453,31cyclictest11350-21taskset19:09:363
1122299453,31cyclictest11350-21taskset19:09:363
1122299453,30cyclictest9972-21chrt23:49:503
1122299453,27cyclictest17728-21taskset19:52:333
1122299452,31cyclictest21710-21chrt20:04:323
1122299452,30cyclictest28483-21chrt22:33:043
1122299452,27cyclictest895-21kworker/3:1+events21:44:343
1122299444,28cyclictest2976-21taskset21:49:563
1122299444,27cyclictest18740-21taskset00:04:333
1122299443,30cyclictest30822-21chrt21:41:543
1122299443,29cyclictest2836-21chrt23:37:353
1122299433,29cyclictest29288-21taskset00:21:193
1122299433,24cyclictest26698-21taskset20:19:343
1122299432,27cyclictest20073-21chrt00:05:273
1122299432,23cyclictest895-21kworker/3:1+events20:34:343
1122299432,23cyclictest895-21kworker/3:1+events20:34:343
1122299423,28cyclictest23350-21taskset21:29:513
1122299423,26cyclictest6202-21chrt20:54:463
1122299423,26cyclictest32318-21chrt20:35:063
1122299423,26cyclictest32318-21chrt20:35:063
1122299423,26cyclictest21922-21taskset20:04:403
1122299413,28cyclictest16030-21taskset19:42:283
1122299413,27cyclictest28819-21taskset20:24:573
1122299413,27cyclictest28819-21taskset20:24:573
1122299413,26cyclictest18669-21taskset19:54:353
1122299413,26cyclictest18669-21taskset19:54:353
1122299412,29cyclictest13656-21taskset19:24:353
1122299412,27cyclictest13716-21taskset19:25:033
1122299412,27cyclictest13716-21taskset19:25:033
1122299402,27cyclictest14984-21taskset23:04:403
1122299393,27cyclictest16464-21taskset19:44:553
1122299392,27cyclictest15201-21taskset19:35:033
1122299392,25cyclictest12592-21taskset19:17:013
1122299382,27cyclictest2881-21chrt20:44:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional