You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 12:30
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Nov 27, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
543821100,8sleep25398-21ssh23:44:402
2231521100,7sleep222299-21apt-get21:29:162
4465210411,66sleep14551-21/usr/sbin/munin19:04:491
5085991003,71cyclictest25207-21apt-get22:29:192
5069999959,8cyclictest181rcu_preempt22:44:390
493429812,47sleep04969-21munin-run19:09:040
48342947,66sleep20-21swapper/219:07:432
5085999161,10cyclictest181rcu_preempt21:49:152
5085999161,10cyclictest181rcu_preempt21:49:152
5069999167,15cyclictest0-21swapper/021:29:400
5085998869,12cyclictest5060-21cyclictest23:24:062
5069998852,15cyclictest19247-21/usr/sbin/munin23:15:260
5085998743,16cyclictest181rcu_preempt21:44:202
5069998766,14cyclictest24246-21kworker/u19:0-rpciod21:15:580
5069998766,14cyclictest24246-21kworker/u19:0-rpciod21:15:580
508599863,67cyclictest10602-21apt-get21:09:182
5069998653,8cyclictest181rcu_preempt23:56:070
5085998458,11cyclictest181rcu_preempt19:24:062
5069998447,8cyclictest181rcu_preempt20:29:150
5069998447,8cyclictest181rcu_preempt20:29:150
5069998447,13cyclictest12240-21latency_hist19:29:060
5085998313,10cyclictest0-21swapper/222:44:192
5069998345,8cyclictest0-21swapper/020:04:360
5085998257,17cyclictest13861-21latency23:59:302
5085998257,17cyclictest13861-21latency23:59:302
5085998238,13cyclictest181rcu_preempt20:54:042
5085998230,27cyclictest0-21swapper/219:14:212
5085998224,25cyclictest0-21swapper/219:24:312
5069998254,8cyclictest181rcu_preempt21:49:010
5069998249,9cyclictest181rcu_preempt00:31:490
5069998248,11cyclictest181rcu_preempt21:49:250
5069998248,11cyclictest181rcu_preempt21:49:250
5085998167,10cyclictest5060-21cyclictest20:59:372
5085998167,10cyclictest5060-21cyclictest20:59:372
5085998152,21cyclictest316-21rs:main2
5085998135,13cyclictest181rcu_preempt22:22:212
5077998167,8cyclictest0-21swapper/123:44:331
5069998159,15cyclictest316-21rs:main0
5069998151,7cyclictest181rcu_preempt22:06:270
5085998065,11cyclictest5060-21cyclictest00:14:142
5085998050,10cyclictest181rcu_preempt00:04:182
5085998046,11cyclictest181rcu_preempt23:54:442
5085998039,13cyclictest181rcu_preempt21:19:252
5069998056,18cyclictest6731-21latency_hist19:14:060
5085997956,15cyclictest32152-21sh20:34:032
5085997956,15cyclictest32152-21sh20:34:032
5085997954,14cyclictest25608-21sh00:19:402
5085997911,33cyclictest39-21ksoftirqd/223:51:592
5069997959,13cyclictest25759-21netstat21:34:300
5085997861,10cyclictest13112-21latency_hist22:09:062
5085997850,19cyclictest21912-21latency_hist19:59:052
5085997843,9cyclictest22351-21ssh22:24:262
5077997863,11cyclictest5060-21cyclictest19:38:301
5069997855,16cyclictest24944-21apt-key22:29:080
5069997849,8cyclictest181rcu_preempt20:14:400
5085997760,12cyclictest17599-21kernelversion19:44:282
5085997760,12cyclictest17599-21kernelversion19:44:282
5085997755,16cyclictest5263-21sh21:54:382
5085997755,14cyclictest26037-21switchtime21:34:382
5085997753,17cyclictest7630-21fschecks_count21:59:262
5085997753,17cyclictest7630-21fschecks_count21:59:262
5085997752,9cyclictest19430-21ssh21:24:182
5077997759,11cyclictest1206-21systemd-run23:39:041
5077997757,14cyclictest1931-21sh22:44:291
5069997756,7cyclictest181rcu_preempt19:44:140
5069997756,7cyclictest181rcu_preempt19:44:140
5069997751,10cyclictest181rcu_preempt22:19:110
5085997661,11cyclictest5060-21cyclictest20:02:182
5085997642,18cyclictest181rcu_preempt23:04:062
5077997660,10cyclictest513-21kworker/u19:1-events_unbound21:50:071
5077997660,10cyclictest513-21kworker/u19:1-events_unbound21:50:071
5069997657,14cyclictest29965-21latency_hist00:29:060
5069997653,9cyclictest181rcu_preempt20:59:300
5069997653,9cyclictest181rcu_preempt20:59:300
5069997648,21cyclictest13904-21latency_hist19:34:070
5069997636,16cyclictest181rcu_preempt19:38:270
5085997556,13cyclictest26027-21switchtime20:09:392
5085997533,8cyclictest181rcu_preempt21:14:392
5085997533,8cyclictest181rcu_preempt21:14:392
5077997560,11cyclictest5060-21cyclictest23:25:311
5077997560,11cyclictest5060-21cyclictest23:25:311
5077997555,14cyclictest21399-21sh00:13:511
506999756,42cyclictest15-21ksoftirqd/021:59:350
506999756,42cyclictest15-21ksoftirqd/021:59:350
5069997555,14cyclictest1207-21idleruntime-cro22:44:020
506999752,43cyclictest15-21ksoftirqd/019:57:110
5085997456,10cyclictest10419-21apt-config21:09:072
5085997426,9cyclictest181rcu_preempt23:19:062
5077997457,11cyclictest207-21systemd-journal19:44:291
5077997457,11cyclictest207-21systemd-journal19:44:291
5077997455,13cyclictest355-21runrttasks21:44:171
5077997442,23cyclictest3566-21modprobe20:49:031
5069997455,14cyclictest20468-21unixbench_singl21:24:400
5069997445,12cyclictest181rcu_preempt23:09:070
5069997437,10cyclictest0-21swapper/020:39:320
508599736,42cyclictest0-21swapper/200:38:002
508599734,48cyclictest5479-21fschecks_count19:09:252
5085997340,11cyclictest31402-21ssh00:30:442
508599733,45cyclictest28872-21find21:39:362
5085997334,16cyclictest19024-21entropy19:49:252
5085997334,16cyclictest19024-21entropy19:49:252
508599733,40cyclictest31934-21timerandwakeup22:39:372
5085997331,14cyclictest0-21swapper/222:11:532
5085997331,13cyclictest181rcu_preempt23:43:422
507799739,28cyclictest27068-21cron00:24:031
5077997351,16cyclictest4088-21idleruntime-cro23:44:031
5077997351,15cyclictest7061-21latency_hist21:59:061
5069997330,9cyclictest181rcu_preempt23:12:310
5069997325,7cyclictest0-21swapper/022:09:190
5085997246,11cyclictest181rcu_preempt22:59:062
5085997245,18cyclictest7146-21sh22:54:052
5085997237,13cyclictest181rcu_preempt20:19:062
5085997233,18cyclictest181rcu_preempt19:34:222
5077997255,12cyclictest24008-21kworker/u17:2+rpciod22:29:031
5077997255,10cyclictest28759-21diskmemload21:39:321
506999728,35cyclictest15-21ksoftirqd/019:20:240
5069997258,10cyclictest5060-21cyclictest21:14:060
5069997212,31cyclictest15-21ksoftirqd/023:20:510
5085997150,8cyclictest39-21ksoftirqd/223:34:082
5085997148,13cyclictest0-21swapper/223:09:402
5077997156,9cyclictest27779-21kworker/u17:1-nfsiod23:32:211
5077997153,13cyclictest6947-21kworker/u18:1+nfsiod23:50:011
507799715,30cyclictest31-21ksoftirqd/119:40:181
5077997139,23cyclictest16500-21sh22:14:261
507799713,38cyclictest31-21ksoftirqd/121:00:001
507799713,38cyclictest31-21ksoftirqd/121:00:001
5069997156,8cyclictest0-21swapper/000:34:420
5069997155,10cyclictest29051-21ntp_kernel_err20:19:300
5069997155,10cyclictest29051-21ntp_kernel_err20:19:300
5069997155,10cyclictest13969-21ssh23:59:330
5069997155,10cyclictest13969-21ssh23:59:330
5069997154,11cyclictest292-21dbus-daemon23:44:040
5069997132,11cyclictest181rcu_preempt20:34:200
5069997122,7cyclictest0-21swapper/020:09:200
5069997116,37cyclictest0-21swapper/000:11:290
5085997052,11cyclictest0-21swapper/200:24:442
5085997038,14cyclictest0-21swapper/223:24:322
5085997038,14cyclictest0-21swapper/223:24:322
507799705,56cyclictest13240-21ssh23:04:181
5077997055,10cyclictest5060-21cyclictest20:29:301
5077997055,10cyclictest5060-21cyclictest20:29:301
5077997053,10cyclictest32405-21kworker/u18:000:34:031
507799704,28cyclictest31-21ksoftirqd/123:11:111
5077997040,21cyclictest7066-21cron22:54:031
5077997040,21cyclictest28271-21modprobe20:19:031
5077997029,9cyclictest301ktimers/123:14:181
507799702,27cyclictest31-21ksoftirqd/122:04:041
507799702,27cyclictest31-21ksoftirqd/122:04:041
5077997012,17cyclictest29149-21grep20:19:341
5077997012,17cyclictest29149-21grep20:19:341
5069997055,10cyclictest25869-21ntp_offset22:29:500
5069997051,13cyclictest28940-21sensors_temp21:39:370
5069997050,14cyclictest8659-21sh19:19:040
5069997048,8cyclictest181rcu_preempt23:53:020
5069997041,9cyclictest181rcu_preempt21:54:130
5069997038,10cyclictest181rcu_preempt23:32:590
5069997037,7cyclictest181rcu_preempt22:34:320
5085996947,12cyclictest0-21swapper/219:44:072
5085996939,13cyclictest181rcu_preempt20:54:292
5085996935,8cyclictest181rcu_preempt22:14:252
5085996930,13cyclictest181rcu_preempt20:39:232
5077996953,9cyclictest0-21swapper/121:39:041
507799694,27cyclictest31-21ksoftirqd/121:24:331
5077996922,9cyclictest301ktimers/100:24:411
5077996920,10cyclictest9950-21apt-get23:54:191
5069996950,13cyclictest26458-21sh00:22:020
5069996946,8cyclictest181rcu_preempt20:54:360
5069996934,12cyclictest0-21swapper/022:55:290
5069996931,8cyclictest181rcu_preempt00:04:530
506999692,23cyclictest16881-21irqstats21:19:270
5085996836,8cyclictest181rcu_preempt20:46:482
5085996833,15cyclictest181rcu_preempt20:07:212
5077996851,12cyclictest0-21swapper/119:19:211
5077996846,7cyclictest181rcu_preempt00:37:221
5077996834,25cyclictest8102-21grep19:14:411
5077996829,29cyclictest8919-21grep21:04:251
5077996822,19cyclictest31-21ksoftirqd/100:05:201
506999686,33cyclictest15-21ksoftirqd/022:14:210
36252688,35sleep30-21swapper/319:04:223
5085996736,10cyclictest181rcu_preempt23:30:032
508599672,37cyclictest19845-21/usr/sbin/munin00:09:382
5077996750,11cyclictest23179-21sh23:19:391
5077996749,8cyclictest181rcu_preempt22:35:421
5077996749,11cyclictest13233-21systemd-run21:14:041
507799674,57cyclictest16730-21fschecks_count21:19:241
5077996745,15cyclictest0-21swapper/119:59:411
507799674,40cyclictest19805-21grep19:49:401
507799674,40cyclictest19805-21grep19:49:401
507799673,35cyclictest31-21ksoftirqd/119:57:151
5077996726,9cyclictest301ktimers/120:49:241
5069996746,12cyclictest31648-21proc_pri23:34:350
5069996737,7cyclictest181rcu_preempt23:01:300
5069996732,9cyclictest181rcu_preempt19:59:390
5085996633,10cyclictest181rcu_preempt22:34:322
508599663,23cyclictest30578-21munin-run20:29:042
508599663,23cyclictest30578-21munin-run20:29:042
5077996647,13cyclictest22281-21/usr/sbin/munin21:29:311
5077996642,8cyclictest181rcu_preempt00:18:021
5077996634,23cyclictest6837-21lockfile-create20:59:041
507799662,32cyclictest31-21ksoftirqd/122:09:271
5077996619,8cyclictest301ktimers/100:03:401
5077996619,8cyclictest301ktimers/100:03:401
5077996615,10cyclictest21729-21latency_hist22:24:051
507799661,26cyclictest31-21ksoftirqd/119:09:231
5069996639,18cyclictest0-21swapper/021:04:380
5069996631,9cyclictest181rcu_preempt23:29:060
5069996631,9cyclictest181rcu_preempt23:29:060
5069996630,20cyclictest4702-21fschecks_count22:49:260
5077996550,10cyclictest316-21rs:main1
507799654,37cyclictest31028-21/usr/sbin/munin22:39:301
507799654,32cyclictest10294-21/usr/sbin/munin22:59:381
5077996542,15cyclictest0-21swapper/120:09:071
507799653,40cyclictest536-21cat20:39:061
5077996520,21cyclictest31-21ksoftirqd/121:15:471
5077996520,21cyclictest31-21ksoftirqd/121:15:471
506999654,52cyclictest15904-21grep19:39:260
5069996529,11cyclictest181rcu_preempt20:49:310
5069996512,35cyclictest30588-21wc20:29:050
5069996512,35cyclictest30588-21wc20:29:050
5085996439,16cyclictest0-21swapper/219:29:392
507799644,25cyclictest31-21ksoftirqd/120:44:041
5077996418,8cyclictest301ktimers/122:29:521
5077996417,7cyclictest301ktimers/120:09:291
5077996415,11cyclictest301ktimers/122:56:351
5069996427,10cyclictest181rcu_preempt23:46:470
5085996340,14cyclictest0-21swapper/220:39:062
5085996326,13cyclictest181rcu_preempt20:19:382
5085996326,13cyclictest181rcu_preempt20:19:382
507799635,39cyclictest12272-21latency_hist19:29:061
5077996317,9cyclictest301ktimers/119:29:421
5069996247,8cyclictest0-21swapper/019:49:400
5069996247,8cyclictest0-21swapper/019:49:400
506999619,27cyclictest15-21ksoftirqd/020:44:090
24672613,8sleep30-21swapper/321:49:403
24672613,8sleep30-21swapper/321:49:403
5093996028,22cyclictest13286-21chrt22:09:203
5077996043,11cyclictest355-21runrttasks20:29:041
5077996043,11cyclictest355-21runrttasks20:29:041
5093995536,14cyclictest451rcuc/300:31:163
5093995235,10cyclictest451rcuc/320:04:393
509399523,38cyclictest7390-21taskset22:54:153
509399514,37cyclictest19599-21chrt21:24:223
509399514,36cyclictest14355-21taskset21:14:383
509399514,36cyclictest14355-21taskset21:14:383
5093995137,9cyclictest451rcuc/321:09:493
509399512,39cyclictest17542-21taskset00:05:253
509399503,36cyclictest2681-21taskset22:44:503
509399493,37cyclictest23036-21chrt22:24:403
509399493,37cyclictest10084-21chrt19:20:243
36712492,6sleep30-21swapper/320:49:073
509399484,34cyclictest22578-21taskset21:29:253
509399484,34cyclictest19616-21taskset22:19:293
509399483,36cyclictest18241-21taskset22:17:313
509399483,35cyclictest14055-21chrt23:59:373
509399483,35cyclictest14055-21chrt23:59:373
509399483,34cyclictest21938-21taskset23:19:043
509399483,32cyclictest12342-21chrt19:29:073
509399482,35cyclictest28115-21chrt21:39:093
509399474,33cyclictest28503-21taskset22:34:283
509399474,32cyclictest15951-21chrt23:09:063
509399473,34cyclictest8790-21taskset23:50:453
509399473,34cyclictest7939-21taskset20:59:393
509399473,34cyclictest7939-21taskset20:59:393
509399473,34cyclictest18075-21chrt19:44:383
509399473,34cyclictest18075-21chrt19:44:383
509399473,33cyclictest24342-21chrt00:19:033
509399473,32cyclictest19513-21taskset00:09:303
509399472,36cyclictest9145-21chrt21:04:293
509399472,35cyclictest3967-21taskset23:43:443
509399472,35cyclictest16153-21taskset23:09:193
509399472,35cyclictest1088-21chrt00:34:393
5093994710,26cyclictest26258-21chrt21:34:403
509399468,28cyclictest4422-21chrt23:44:173
509399464,32cyclictest29423-21chrt20:19:383
509399464,32cyclictest29423-21chrt20:19:383
5093994634,8cyclictest451rcuc/319:57:353
509399463,33cyclictest13108-21taskset23:59:043
509399463,32cyclictest16923-21chrt21:19:283
509399463,31cyclictest28015-21taskset23:29:093
509399462,35cyclictest25860-21taskset23:24:393
509399462,35cyclictest25860-21taskset23:24:393
509399462,34cyclictest31258-21chrt22:39:233
509399462,34cyclictest30614-21taskset20:29:053
509399462,34cyclictest30614-21taskset20:29:053
509399454,29cyclictest7264-21chrt22:54:063
509399453,33cyclictest6106-21chrt19:09:393
509399453,32cyclictest25179-21chrt00:19:313
509399452,34cyclictest628-21chrt23:37:363
509399452,33cyclictest19484-21taskset19:49:363
509399452,33cyclictest19484-21taskset19:49:363
509399444,30cyclictest27694-21taskset00:24:303
509399444,28cyclictest4476-21taskset21:54:173
509399443,31cyclictest4717-21chrt20:49:393
509399443,30cyclictest7341-21taskset21:59:073
509399443,30cyclictest7341-21taskset21:59:073
509399443,29cyclictest25170-21taskset22:29:143
5093994431,9cyclictest451rcuc/323:24:033
509399433,28cyclictest15659-21taskset19:39:203
509399432,25cyclictest7857-21taskset19:14:373
509399423,30cyclictest27180-21chrt20:14:273
509399423,27cyclictest11414-21taskset22:59:423
509399422,28cyclictest31279-21chrt21:44:243
509399414,28cyclictest12717-21taskset22:08:273
509399413,28cyclictest5710-21chrt20:54:243
509399403,27cyclictest32547-21taskset20:36:103
509399403,27cyclictest31799-21taskset20:29:383
509399403,27cyclictest31799-21taskset20:29:383
509399403,27cyclictest22255-21chrt19:59:243
509399403,26cyclictest12560-21chrt19:29:233
509399402,27cyclictest15597-21taskset19:39:073
509399402,26cyclictest25303-21taskset20:09:133
509399393,26cyclictest879-21taskset20:39:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional