You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 08:45
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Mon Dec 01, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
176939910867,34cyclictest17679-21cyclictest23:33:501
17688991074,49cyclictest26661-21timerandwakeup22:59:260
17688991074,49cyclictest26661-21timerandwakeup22:59:260
176939910364,31cyclictest3599-21diskmemload23:15:141
17688991033,31cyclictest30158-21unixbench_multi23:05:100
1683121037,66sleep116877-21mailstats19:04:331
17688991004,41cyclictest355-21runrttasks21:53:540
17688991003,32cyclictest11119-21cron23:28:470
17688991003,32cyclictest11119-21cron23:28:470
17697999979,12cyclictest16750-21latency_hist21:48:502
1768899993,33cyclictest17277-21/usr/sbin/munin23:39:050
1768899993,33cyclictest17277-21/usr/sbin/munin23:39:050
17693999769,18cyclictest19357-21apt-key20:48:511
1769399967,65cyclictest29171-21ssh23:04:461
17688999638,19cyclictest4147-21kworker/u17:0+nfsiod22:47:490
17688999538,19cyclictest28948-21kworker/u17:3+nfsiod23:36:230
1752629412,51sleep017558-21awk19:08:480
17688999337,19cyclictest28948-21kworker/u17:3+nfsiod23:59:410
17693999251,34cyclictest31574-21apt-get22:14:051
1769799913,33cyclictest32271-21/usr/sbin/munin22:14:232
1768899914,36cyclictest25414-21latency_hist22:03:510
1768899914,36cyclictest25414-21latency_hist22:03:510
1768899913,38cyclictest355-21runrttasks23:33:050
17697998955,27cyclictest24617-21kworker/u17:2-ext4-rsv-conversion22:23:482
17697998941,41cyclictest0-21swapper/222:08:502
17697998920,29cyclictest25661-21cat22:03:522
1769799883,32cyclictest1073-21/usr/sbin/munin19:54:192
17693998854,27cyclictest0-21swapper/122:59:231
17693998854,27cyclictest0-21swapper/122:59:231
17697998766,16cyclictest17679-21cyclictest23:15:032
1769799873,30cyclictest2429-21/usr/sbin/munin21:24:212
17693998749,32cyclictest25801-21diskmemload22:49:011
17693998743,38cyclictest8052-21ntp_kernel_pll_20:14:171
17693998738,39cyclictest0-21swapper/121:03:511
17697998635,44cyclictest0-21swapper/222:30:002
17697998635,44cyclictest0-21swapper/222:30:002
1769799863,29cyclictest32554-21ntpq21:19:152
17697998623,9cyclictest0-21swapper/221:03:492
17697998622,8cyclictest0-21swapper/221:18:062
1769399866,61cyclictest23125-21ssh22:54:521
1769399866,61cyclictest23125-21ssh22:54:521
17693998644,35cyclictest0-21swapper/120:09:181
17693998626,42cyclictest0-21swapper/121:08:491
17697998534,43cyclictest30022-21unixbench_multi23:59:252
1769799853,31cyclictest22697-21ls20:58:502
1769399853,27cyclictest10686-21idleruntime-cro22:33:481
1769399853,27cyclictest10686-21idleruntime-cro22:33:481
1769799849,12cyclictest0-21swapper/223:32:112
1769799843,31cyclictest20052-21sshd19:14:082
1769799843,31cyclictest20052-21sshd19:14:082
1769799843,29cyclictest24655-21grep21:04:122
17693998456,20cyclictest31543-21apt-key19:48:501
17693998437,39cyclictest0-21swapper/123:19:251
1769399843,27cyclictest1653-21/usr/sbin/munin19:54:191
1720128411,44sleep30-21swapper/319:04:393
17697998361,14cyclictest207-21systemd-journal23:50:092
1769799834,27cyclictest6875-21grep20:09:272
1769799833,18cyclictest21783-21apt-get19:19:082
17697998327,47cyclictest0-21swapper/200:22:182
17697998327,47cyclictest0-21swapper/200:22:182
17697998325,51cyclictest691-21cron22:58:482
17697998325,51cyclictest691-21cron22:58:482
1769799832,29cyclictest17232-21latency_hist20:43:522
17697998322,9cyclictest0-21swapper/222:13:192
1769799824,29cyclictest1-21systemd00:13:482
1769799824,29cyclictest1-21systemd00:13:482
17697998236,39cyclictest0-21swapper/221:39:122
17697998234,41cyclictest0-21swapper/223:58:482
1769799823,31cyclictest13401-21runrttasks22:38:212
17697998232,44cyclictest0-21swapper/221:59:252
17697998232,43cyclictest0-21swapper/219:44:052
17697998229,44cyclictest0-21swapper/219:13:522
17697998229,44cyclictest0-21swapper/219:13:522
17697998228,47cyclictest0-21swapper/200:08:502
17697998216,20cyclictest16792-21timerwakeupswit00:34:262
17697998216,20cyclictest16792-21timerwakeupswit00:34:262
17697998214,21cyclictest9946-21sh23:25:202
17697998214,21cyclictest9946-21sh23:25:202
1769399823,37cyclictest22649-21latency_hist21:58:501
17693998231,43cyclictest0-21swapper/122:13:511
1769399823,11cyclictest26123-21apt-get21:09:011
1769399822,23cyclictest11944-21mailstats21:39:221
17693998215,30cyclictest17257-21http20:43:481
17697998136,39cyclictest11567-21xargs20:24:242
1769799813,59cyclictest28932-21ssh23:04:042
17697998122,11cyclictest0-21swapper/223:44:222
17697998113,22cyclictest8696-21wc21:34:152
17697998113,22cyclictest8696-21wc21:34:152
17697998113,22cyclictest25738-21cat19:29:232
17693998137,37cyclictest0-21swapper/119:14:161
17693998137,37cyclictest0-21swapper/119:14:161
1769399813,23cyclictest11507-21/usr/sbin/munin22:34:161
17693998131,41cyclictest0-21swapper/120:50:131
17693998127,43cyclictest0-21swapper/119:38:511
17693998127,43cyclictest0-21swapper/119:38:511
17693998127,40cyclictest0-21swapper/100:29:151
1769399812,39cyclictest9919-21idleruntime-cro00:23:471
1769399812,39cyclictest9919-21idleruntime-cro00:23:471
1769399812,23cyclictest15143-21/usr/sbin/munin20:34:261
1769399812,23cyclictest15143-21/usr/sbin/munin20:34:261
1769399812,22cyclictest9297-21awk21:34:261
1769399812,22cyclictest9297-21awk21:34:261
17693998112,29cyclictest23102-21fschecks_time21:59:111
17693998112,29cyclictest23102-21fschecks_time21:59:111
1769799803,31cyclictest20947-21grep20:53:472
1769799803,26cyclictest18711-21grep20:44:432
17697998032,41cyclictest0-21swapper/222:49:262
17697998028,45cyclictest0-21swapper/221:54:072
17697998028,45cyclictest0-21swapper/200:24:102
17697998025,48cyclictest0-21swapper/223:19:102
17697998019,21cyclictest3464-21cat19:59:232
17697998019,21cyclictest3464-21cat19:59:232
17697998010,23cyclictest2294-21apt-config23:13:502
1769399803,28cyclictest1406-21ssh00:09:001
1769399803,28cyclictest1406-21ssh00:09:001
17693998031,41cyclictest0-21swapper/121:29:111
17693998030,40cyclictest0-21swapper/122:04:061
17693998027,47cyclictest0-21swapper/122:44:101
1769399802,38cyclictest207-21systemd-journal23:37:091
1769399802,25cyclictest20198-21basename23:43:581
17688998063,10cyclictest22337-21systemd-run22:53:490
17697997954,14cyclictest0-21swapper/223:38:512
17697997945,30cyclictest17679-21cyclictest22:41:582
17697997945,30cyclictest17679-21cyclictest22:41:582
1769799793,29cyclictest7375-21sshd21:32:102
1769799793,25cyclictest17679-21cyclictest19:08:522
17697997929,43cyclictest19612-21sh21:53:482
1769799792,31cyclictest10689-21latency_hist20:23:512
1769799792,28cyclictest4927-21runrttasks22:23:552
1769799792,24cyclictest17706-21/usr/sbin/munin23:39:122
1769799792,24cyclictest17706-21/usr/sbin/munin23:39:122
17693997961,12cyclictest18365-21systemd-run00:38:481
17693997961,12cyclictest18365-21systemd-run00:38:481
17693997938,34cyclictest0-21swapper/120:30:141
17693997938,34cyclictest0-21swapper/120:30:141
1769399793,39cyclictest26062-21ssh23:54:021
1769399793,25cyclictest7906-21latency_hist22:28:511
1769399793,23cyclictest9561-21/usr/sbin/munin20:19:151
1769399792,38cyclictest23746-21ntp_states23:49:191
1769399792,20cyclictest23391-21apt-get19:23:581
17697997861,11cyclictest207-21systemd-journal20:08:502
17697997861,11cyclictest207-21systemd-journal20:08:502
1769799783,32cyclictest14035-21latency_hist20:33:502
1769799783,32cyclictest14035-21latency_hist20:33:502
1769799783,27cyclictest4276-21/usr/sbin/munin00:14:122
1769799783,22cyclictest28780-21apt-key23:03:512
17697997824,46cyclictest0-21swapper/220:14:052
17697997818,21cyclictest29396-21grep19:39:262
17697997818,21cyclictest29396-21grep19:39:262
17697997814,23cyclictest13704-21processes00:29:232
17693997858,13cyclictest207-21systemd-journal21:20:101
1769399783,35cyclictest21783-21apt-get19:19:121
1769399783,25cyclictest17508-21/usr/sbin/munin23:39:091
1769399783,25cyclictest17508-21/usr/sbin/munin23:39:091
1769399783,23cyclictest17273-21fschecks_time21:49:111
1769399783,23cyclictest10418-21/usr/sbin/munin00:24:081
17693997832,37cyclictest0-21swapper/123:24:031
17693997832,37cyclictest0-21swapper/123:24:031
17693997826,44cyclictest0-21swapper/123:10:241
1769399782,25cyclictest4630-21fschecks_time00:14:071
1769399782,24cyclictest18013-21/usr/sbin/munin19:09:081
1769399782,24cyclictest18013-21/usr/sbin/munin19:09:081
1769399782,23cyclictest31567-21cron00:03:461
17693997814,29cyclictest29844-21cat21:14:241
17688997860,11cyclictest1-21systemd23:53:480
17697997756,11cyclictest0-21swapper/219:38:482
17697997756,11cyclictest0-21swapper/219:38:482
1769799773,25cyclictest26236-21awk21:09:052
1769799772,25cyclictest32141-21fschecks_count19:49:202
1769399772,24cyclictest32142-21fschecks_count19:49:201
1769799763,22cyclictest26479-21ssh22:59:232
17697997622,8cyclictest0-21swapper/222:44:172
1769799762,20cyclictest14746-21sed20:34:182
1769799762,20cyclictest14746-21sed20:34:182
1769399763,23cyclictest16688-21modprobe21:48:471
1769399763,23cyclictest11328-21tr20:24:181
17693997632,39cyclictest0-21swapper/122:23:091
1769399763,22cyclictest3125-21/usr/sbin/munin21:24:221
1769399762,23cyclictest29093-21/usr/sbin/munin19:39:221
1769399762,23cyclictest29093-21/usr/sbin/munin19:39:221
1769399762,23cyclictest25253-21tune2fs19:29:111
1769399762,23cyclictest1185-21latency_hist00:08:491
1769399762,21cyclictest22185-21/usr/sbin/munin20:54:271
17693997615,8cyclictest0-21swapper/122:40:211
17693997615,8cyclictest0-21swapper/122:40:211
17688997659,11cyclictest292-21dbus-daemon19:33:490
1770299754,44cyclictest0-21swapper/319:19:013
1769799752,22cyclictest23806-21cat19:24:122
1769399752,23cyclictest2877-21/usr/sbin/munin19:59:071
1769399752,23cyclictest2877-21/usr/sbin/munin19:59:071
17688997559,10cyclictest25801-21diskmemload21:48:400
1769399743,23cyclictest4324-21/usr/sbin/munin20:03:581
1769399743,23cyclictest4324-21/usr/sbin/munin20:03:581
17688997355,12cyclictest292-21dbus-daemon19:18:490
17688997355,12cyclictest292-21dbus-daemon19:18:490
17688997257,9cyclictest25801-21diskmemload22:09:240
17688997257,9cyclictest19358-21kworker/u19:0+flush-179:022:54:020
17688997257,9cyclictest19358-21kworker/u19:0+flush-179:022:54:020
17688997254,12cyclictest25801-21diskmemload21:29:110
17688997253,13cyclictest2537-21latency_hist23:13:520
17688997252,14cyclictest7767-21latency_hist22:28:510
17688997252,14cyclictest316-21rs:main0
17688997247,10cyclictest7784-21ssh23:22:340
17688997154,11cyclictest13146-21ssh22:37:370
17688997057,9cyclictest17679-21cyclictest00:29:310
17688997047,17cyclictest14004-21cut20:33:500
17688997047,17cyclictest14004-21cut20:33:500
17688997043,20cyclictest8194-21kworker/u17:4+nfsiod22:42:170
17688997043,20cyclictest8194-21kworker/u17:4+nfsiod22:42:170
17688997039,20cyclictest7739-21grep00:19:120
17688997039,20cyclictest7739-21grep00:19:120
17702996924,13cyclictest0-21swapper/319:34:023
17702996924,13cyclictest0-21swapper/319:34:023
17688996951,12cyclictest2675-21acpi23:13:590
1770299689,32cyclictest0-21swapper/323:09:043
17702996710,50cyclictest0-21swapper/322:29:503
17702996710,50cyclictest0-21swapper/322:29:503
17688996753,9cyclictest9276-21timerandwakeup22:30:110
17688996753,9cyclictest9276-21timerandwakeup22:30:110
17688996752,8cyclictest28948-21kworker/u17:3+xprtiod23:58:510
17688996750,11cyclictest26519-21ssh22:04:230
17688996740,20cyclictest32683-21kworker/u17:1+nfsiod00:34:590
17688996740,20cyclictest32683-21kworker/u17:1+nfsiod00:34:590
17688996739,20cyclictest10584-21grep20:23:500
17688996732,20cyclictest19914-21kworker/u17:0+nfsiod00:11:290
17688996732,20cyclictest19914-21kworker/u17:0+nfsiod00:11:290
17688996653,8cyclictest26156-21ssh21:09:050
17688996647,13cyclictest2585-21netstat22:19:150
1768899664,35cyclictest14244-21/usr/sbin/munin20:34:070
1768899664,35cyclictest14244-21/usr/sbin/munin20:34:070
1768899663,34cyclictest29030-21acpi21:14:000
17688996552,9cyclictest21735-21kworker/u19:2-rpciod00:14:110
17688996551,10cyclictest17679-21cyclictest21:34:130
17688996551,10cyclictest17679-21cyclictest21:34:130
17688996539,19cyclictest28948-21kworker/u17:3+nfsiod00:06:070
17688996535,21cyclictest20935-21sh20:53:470
17688996534,22cyclictest7283-21modprobe20:13:470
17688996439,19cyclictest32012-21kworker/u17:3+nfsiod21:23:460
17688996435,20cyclictest32280-21find22:14:230
17688996351,8cyclictest4154-21kworker/u17:2+xprtiod21:49:110
17688996346,12cyclictest355-21runrttasks20:54:180
17688996338,19cyclictest28948-21kworker/u17:3+nfsiod00:28:390
1768899633,45cyclictest10993-21fschecks_count20:24:110
17688996247,8cyclictest0-21swapper/023:44:160
17688996247,10cyclictest0-21swapper/020:43:500
17688996246,10cyclictest17679-21cyclictest19:13:490
17688996246,10cyclictest17679-21cyclictest19:13:490
17688996241,14cyclictest5305-21grep20:04:240
17688996241,14cyclictest5305-21grep20:04:240
17688996233,21cyclictest19083-21modprobe20:48:470
17688996232,21cyclictest23696-21ls20:59:240
1770299613,35cyclictest0-21swapper/320:49:013
17702996118,31cyclictest0-21swapper/323:14:283
17688996145,10cyclictest12643-21runrttasks21:40:290
1768899613,31cyclictest355-21runrttasks21:24:010
17688996133,19cyclictest2971-21grep19:59:110
17688996133,19cyclictest2971-21grep19:59:110
1768899613,25cyclictest26345-21apt-get19:34:020
1768899613,25cyclictest26345-21apt-get19:34:020
17688996043,12cyclictest0-21swapper/019:58:480
17688996040,13cyclictest31426-21latency_hist19:48:500
1770299594,41cyclictest14022-21chrt21:43:513
17702995942,11cyclictest0-21swapper/322:54:543
17702995942,11cyclictest0-21swapper/322:54:543
1770299584,39cyclictest17575-21taskset20:43:513
1770299583,36cyclictest28572-21taskset22:08:513
17702995812,39cyclictest0-21swapper/323:04:023
17688995845,9cyclictest12940-21kworker/u18:1-flush-179:019:25:140
1768899583,40cyclictest8892-21cron20:18:470
1770299575,37cyclictest28650-21taskset23:03:503
1770299575,37cyclictest28650-21taskset23:03:503
1770299574,33cyclictest0-21swapper/319:49:013
1768899573,34cyclictest31911-21/usr/sbin/munin19:49:340
1770299565,38cyclictest14144-21taskset23:33:493
17702995611,28cyclictest0-21swapper/320:29:423
17702995611,28cyclictest0-21swapper/320:29:423
1768899563,43cyclictest21764-21/usr/sbin/munin19:19:230
1770299558,31cyclictest0-21swapper/320:44:163
1770299555,36cyclictest30322-21chrt19:44:093
1770299555,36cyclictest12736-21chrt00:28:473
1770299554,36cyclictest2773-21chrt22:19:203
1770299553,37cyclictest23333-21chrt23:49:093
17702995530,16cyclictest316182sleep300:03:493
1770299552,40cyclictest13052-21chrt22:37:203
17688995543,7cyclictest0-21swapper/019:39:040
17688995543,7cyclictest0-21swapper/019:39:040
1770299543,37cyclictest24174-21chrt21:03:483
17702995431,13cyclictest0-21swapper/319:14:123
17702995431,13cyclictest0-21swapper/319:14:123
1770299534,37cyclictest14446-21chrt22:39:193
1770299534,37cyclictest14446-21chrt22:39:193
1770299533,39cyclictest26379-21chrt19:33:493
1770299533,35cyclictest379-21chrt21:19:243
1770299533,35cyclictest1399-21chrt00:08:583
1770299533,35cyclictest1399-21chrt00:08:583
1770299533,33cyclictest14267-21taskset20:34:033
1770299533,33cyclictest14267-21taskset20:34:033
1770299532,35cyclictest1229-21chrt00:08:503
1770299524,36cyclictest19449-21chrt19:13:523
1770299524,34cyclictest6918-21taskset00:18:423
1770299523,36cyclictest6344-21chrt22:25:163
1770299523,36cyclictest22648-21chrt21:58:503
1770299513,39cyclictest12176-21taskset20:28:483
1770299513,37cyclictest7398-21taskset00:19:033
1770299513,37cyclictest7398-21taskset00:19:033
1770299513,36cyclictest29756-21chrt22:09:283
1770299513,35cyclictest9005-21taskset21:34:233
1770299513,35cyclictest9005-21taskset21:34:233
1770299513,34cyclictest18415-21taskset22:45:593
1770299513,34cyclictest16846-21chrt00:34:273
1770299513,34cyclictest16846-21chrt00:34:273
1770299504,35cyclictest1847-21chrt19:54:223
1770299503,38cyclictest3538-21chrt21:24:263
1770299503,35cyclictest10718-21chrt23:27:373
1770299503,35cyclictest10718-21chrt23:27:373
1770299503,34cyclictest26375-21chrt21:09:103
1770299503,34cyclictest17264-21chrt23:38:563
1770299503,34cyclictest17264-21chrt23:38:563
1770299503,34cyclictest14213-21chrt00:29:453
1770299502,37cyclictest18785-21chrt19:09:273
1770299502,35cyclictest7509-21taskset23:21:463
1770299502,33cyclictest20196-21chrt23:43:583
1770299494,33cyclictest1034-21chrt22:16:493
1770299493,35cyclictest16803-21chrt23:38:233
1770299492,36cyclictest9969-21chrt20:19:253
1770299492,36cyclictest6180-21chrt20:09:123
17702994913,28cyclictest0-21swapper/323:58:503
1770299484,33cyclictest7774-21chrt20:14:103
1770299483,36cyclictest31671-21chrt21:18:473
1770299483,35cyclictest14895-21taskset21:44:243
1770299483,32cyclictest19834-21chrt22:49:073
17702994832,9cyclictest451rcuc/322:03:043
17702994832,9cyclictest451rcuc/322:03:043
1770299474,34cyclictest21953-21chrt20:54:243
1770299474,28cyclictest25177-21taskset21:04:243
1770299472,37cyclictest6671-21taskset21:30:063
1770299463,35cyclictest17751-21chrt21:49:223
1770299463,34cyclictest3037-21taskset19:59:123
1770299463,34cyclictest3037-21taskset19:59:123
1770299443,28cyclictest4621-21chrt20:04:103
1770299443,28cyclictest4621-21chrt20:04:103
1770299433,26cyclictest28925-21chrt19:39:153
1770299433,26cyclictest28925-21chrt19:39:153
1770299423,29cyclictest23649-21taskset19:24:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional