You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-24 - 17:39
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Oct 24, 2025 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
871421610,18sleep2371rcuc/207:05:442
9710991243,9cyclictest0-21swapper/007:09:370
97109911865,47cyclictest21111-21runrttasks09:16:080
9710991153,25cyclictest20097-21/usr/sbin/munin12:09:440
97109911135,60cyclictest5997-21ssh10:47:330
9710991113,18cyclictest352-21runrttasks09:39:470
9710991092,27cyclictest13807-21seq10:02:060
8994210910,67sleep19036-21/usr/sbin/munin07:05:561
2643521090,5sleep20-21swapper/211:24:332
9710991082,27cyclictest2640-21ssh10:40:250
9710991074,23cyclictest12052-21ssh10:58:280
9710991073,26cyclictest925-21ssh10:37:390
9710991072,25cyclictest329-21rm09:38:170
97169910649,47cyclictest0-21swapper/208:14:572
9710991062,25cyclictest5733-21rm09:47:390
9710991054,27cyclictest8588-21apt-get08:44:460
9710991054,27cyclictest8588-21apt-get08:44:460
9710991053,25cyclictest21652-21apt-get07:44:450
9710991052,26cyclictest9891-21taskset11:51:320
9710991052,26cyclictest9891-21taskset11:51:320
9710991052,24cyclictest24914-21ssh10:22:050
9710991052,24cyclictest11040-21ssh09:57:190
9710991052,24cyclictest11040-21ssh09:57:190
9710991052,23cyclictest28167-21ssh11:26:140
9716991042,6cyclictest0-21swapper/211:44:482
9710991043,22cyclictest29672-21ssh09:32:030
9710991043,22cyclictest25787-21cron12:19:330
9710991043,22cyclictest25787-21cron12:19:330
9710991042,25cyclictest30483-21ssh12:25:490
9716991033,30cyclictest22774-21seq07:46:552
9716991033,28cyclictest14794-21seq09:01:382
9716991032,33cyclictest22536-21taskset10:18:212
97149910338,57cyclictest0-21swapper/110:15:111
97109910347,49cyclictest0-21swapper/008:14:340
9710991033,7cyclictest0-21swapper/012:34:460
9710991033,7cyclictest0-21swapper/012:34:460
9710991033,23cyclictest8397-21ssh10:51:260
9716991023,30cyclictest440-21seq09:38:372
9710991027,20cyclictest18674-21kworker/u17:2+rpciod10:09:070
9710991023,24cyclictest6458-21taskset08:36:470
9710991022,27cyclictest21077-21seq07:40:430
9716991014,31cyclictest26567-21seq09:26:012
97169910115,50cyclictest12042-21apt-get08:54:502
9710991013,6cyclictest0-21swapper/009:09:500
9710991013,6cyclictest0-21swapper/009:09:500
9710991013,23cyclictest18848-21ssh10:10:370
9710991012,24cyclictest7302-21ssh11:47:360
9710991012,23cyclictest17311-21sh11:07:170
9710991012,22cyclictest31215-21ssh11:32:030
9710991012,21cyclictest8576-21ssh09:53:180
9710991003,25cyclictest16422-21ssh12:03:360
9710991002,29cyclictest28414-21ssh12:23:240
9710991002,23cyclictest12634-21ssh11:56:090
971699992,29cyclictest16217-21seq07:25:332
9710999949,42cyclictest0-21swapper/009:04:590
971099992,6cyclictest0-21swapper/008:14:490
971099992,23cyclictest19624-21ssh12:09:080
971099992,23cyclictest19624-21ssh12:09:080
971699983,40cyclictest9936-21apt-get10:54:492
971699983,27cyclictest21042-21seq07:40:152
9710999838,50cyclictest0-21swapper/008:29:560
971099982,33cyclictest20459-21sh10:14:450
971099982,23cyclictest27437-21unixbench_singl08:00:130
971099982,23cyclictest27437-21unixbench_singl08:00:130
971099982,23cyclictest2249-21idleruntime-cro11:39:330
971099982,23cyclictest2249-21idleruntime-cro11:39:330
971099982,22cyclictest8089-21taskset08:41:440
971099982,22cyclictest8089-21taskset08:41:440
971099974,23cyclictest1343-21ssh12:31:280
971099972,23cyclictest17314-21ntpq07:30:010
971699962,31cyclictest6356-21runrttasks08:35:262
971699962,28cyclictest19574-21seq07:37:232
971099963,6cyclictest0-21swapper/007:14:490
971099963,22cyclictest27489-21ssh10:26:380
971099962,26cyclictest19364-21vmstat11:10:130
971099954,25cyclictest28116-21apt-get08:04:460
971099954,25cyclictest28116-21apt-get08:04:460
971099954,25cyclictest13686-21apt-get08:59:460
971099952,21cyclictest16252-21taskset07:25:590
971699942,6cyclictest0-21swapper/208:49:482
9714999435,8cyclictest0-21swapper/109:04:561
9714999432,8cyclictest0-21swapper/110:25:081
9714999430,20cyclictest1854-21grep09:40:091
971099942,25cyclictest23839-21chrt09:20:580
971099942,21cyclictest14871-21ssh11:03:300
971099942,21cyclictest14871-21ssh11:03:300
971499932,36cyclictest14525-21taskset12:00:011
971099934,26cyclictest27712-21modprobe09:29:340
971099933,22cyclictest21574-21kernelversion11:15:030
971099933,21cyclictest11359-21taskset08:52:220
9710999331,56cyclictest0-21swapper/011:20:010
9716999248,33cyclictest0-21swapper/207:15:022
971699922,31cyclictest19470-21ssh10:12:312
971699922,30cyclictest352-21runrttasks09:51:472
9714999236,8cyclictest0-21swapper/111:24:551
971099925,12cyclictest18417-21apt-get07:34:480
971099923,6cyclictest0-21swapper/008:24:460
971099923,6cyclictest0-21swapper/008:19:520
971099923,27cyclictest11246-21modprobe07:14:340
971099923,26cyclictest2527-21apt-get11:39:460
971099922,9cyclictest14793-21seq07:22:580
971699912,32cyclictest30927-21runrttasks08:12:592
9714999136,8cyclictest0-21swapper/109:20:001
971499912,27cyclictest25490-21grep07:55:081
971099913,21cyclictest24280-21taskset07:50:330
9716999051,32cyclictest0-21swapper/210:00:022
971699902,25cyclictest25663-21seq12:19:122
971699902,25cyclictest25663-21seq12:19:122
9714999033,8cyclictest0-21swapper/112:14:541
9714999033,8cyclictest0-21swapper/112:14:541
971499903,31cyclictest2457-21modprobe12:34:331
9714999020,21cyclictest9538-21wc10:54:361
9714999010,27cyclictest0-21swapper/107:39:561
971099903,22cyclictest352-21runrttasks10:30:020
9710999022,57cyclictest24859-21find07:54:460
971099902,24cyclictest12283-21grep08:55:210
9716998955,28cyclictest0-21swapper/211:29:372
9716998952,31cyclictest0-21swapper/209:19:522
9716998945,37cyclictest0-21swapper/209:14:332
9716998945,37cyclictest0-21swapper/209:14:332
971499894,33cyclictest25768-21cat10:24:361
9714998939,7cyclictest0-21swapper/109:15:041
9714998936,9cyclictest0-21swapper/111:00:051
9714998936,9cyclictest0-21swapper/111:00:051
9714998936,8cyclictest0-21swapper/109:00:091
9714998935,8cyclictest0-21swapper/110:35:051
971499893,34cyclictest6321-21grep08:35:131
9714998932,8cyclictest0-21swapper/112:10:111
971699883,27cyclictest28210-21grep09:29:552
9714998833,9cyclictest0-21swapper/108:10:031
971499883,38cyclictest11781-21sh11:55:031
9714998833,8cyclictest0-21swapper/112:05:091
9714998833,8cyclictest0-21swapper/112:05:091
971499883,32cyclictest13189-21grep10:00:141
971499883,31cyclictest4695-21switchtime09:45:091
9714998826,13cyclictest0-21swapper/110:09:591
9716998751,29cyclictest0-21swapper/207:09:512
971699873,7cyclictest0-21swapper/207:19:502
971699872,7cyclictest0-21swapper/211:34:492
971699872,7cyclictest0-21swapper/211:34:492
9714998766,15cyclictest302-21rs:main1
9714998737,8cyclictest0-21swapper/108:19:581
9714998734,8cyclictest0-21swapper/110:40:101
971499873,37cyclictest31904-21latency_hist12:29:361
971499873,33cyclictest18709-21sort11:10:001
971499873,32cyclictest13654-21sort07:19:511
9716998650,30cyclictest0-21swapper/208:04:562
9716998650,30cyclictest0-21swapper/208:04:562
971699864,36cyclictest32237-21ssh12:29:462
971699862,35cyclictest2450-21timerwakeupswit10:40:152
971699862,30cyclictest4607-21grep09:45:082
971699862,24cyclictest27811-21sh12:21:442
971499864,30cyclictest352-21runrttasks09:24:501
9714998635,8cyclictest0-21swapper/108:05:041
9714998635,8cyclictest0-21swapper/108:05:041
9714998634,9cyclictest0-21swapper/108:27:241
971499863,34cyclictest29395-21ssh10:30:171
971499863,33cyclictest10101-21latency_hist08:49:361
971499863,33cyclictest10101-21latency_hist08:49:361
971499862,34cyclictest32766-21latency_hist08:19:361
9716998539,38cyclictest0-21swapper/207:29:542
971699853,30cyclictest27163-21cat08:00:102
971699853,30cyclictest27163-21cat08:00:102
9714998537,42cyclictest17623-21proc_pri09:10:071
9714998537,42cyclictest17623-21proc_pri09:10:071
971499853,37cyclictest23499-21fschecks_time07:49:571
971499853,34cyclictest7874-21unixbench_multi08:40:131
971499853,34cyclictest7874-21unixbench_multi08:40:131
971499853,31cyclictest825-21unixbench_singl11:35:141
971499853,31cyclictest825-21unixbench_singl11:35:141
971499853,31cyclictest352-21runrttasks11:09:081
971499853,30cyclictest18904-21ntp_kernel_err07:35:041
9714998531,8cyclictest0-21swapper/111:14:481
971699843,69cyclictest29534-21apt-get11:29:492
971699842,6cyclictest0-21swapper/207:54:512
971699842,37cyclictest30256-21/usr/sbin/munin12:25:122
971699842,36cyclictest4788-21ps10:45:062
971499844,30cyclictest352-21runrttasks11:40:031
971499844,29cyclictest352-21runrttasks08:00:191
971499844,29cyclictest352-21runrttasks08:00:191
971499843,35cyclictest30460-21ssh09:34:351
971499843,32cyclictest29512-21/usr/sbin/munin11:29:541
971499843,31cyclictest12131-21fschecks_count07:14:571
971499843,29cyclictest10920-21find10:56:011
971499843,28cyclictest17494-21latency_hist10:09:371
971499842,34cyclictest3964-21fschecks_time08:29:561
9716998347,29cyclictest0-21swapper/210:33:402
971699833,39cyclictest15746-21sshd11:04:552
971699833,35cyclictest7751-21xargs10:50:122
971699833,35cyclictest12131-21/usr/sbin/munin11:55:122
971699833,33cyclictest9428-21timerwakeupswit08:45:102
971699833,33cyclictest9428-21timerwakeupswit08:45:102
971699833,32cyclictest18708-21ls11:10:002
971699833,28cyclictest12422-21latency_hist10:59:372
971699833,28cyclictest12422-21latency_hist10:59:372
971699833,26cyclictest7209-21cut08:39:582
971699833,26cyclictest7209-21cut08:39:582
971699832,37cyclictest2988-21sshd12:34:552
971699832,37cyclictest2988-21sshd12:34:552
971499834,32cyclictest9221-21timerandwakeup11:50:131
971499834,32cyclictest9221-21timerandwakeup11:50:131
971499833,35cyclictest15676-21netstat07:25:011
971499833,33cyclictest31552-21ssh09:35:101
971499833,31cyclictest24648-21proc_pri11:20:081
971499833,30cyclictest4680-21ntp_kernel_err10:45:011
971499832,31cyclictest9165-21latency_hist09:54:371
9716998257,17cyclictest0-21swapper/209:09:362
971699825,64cyclictest20098-21apt-get12:09:462
9716998240,32cyclictest0-21swapper/211:39:522
971699823,7cyclictest0-21swapper/211:49:502
971699823,7cyclictest0-21swapper/211:49:502
971699823,32cyclictest9718-21/usr/sbin/munin09:55:002
971699823,32cyclictest9718-21/usr/sbin/munin09:55:002
971699823,29cyclictest15921-21seq10:05:102
971499823,33cyclictest26813-21sh12:20:081
971499823,32cyclictest11903-21latency_hist09:59:371
971499823,32cyclictest11903-21latency_hist09:59:371
971499823,31cyclictest9893-21/usr/sbin/munin07:09:581
971499823,31cyclictest22981-21modprobe07:49:341
971699814,31cyclictest1414-21/usr/sbin/munin08:20:122
971699814,24cyclictest25683-21munin-plugin-st10:24:332
971699813,31cyclictest1449-21latency_hist10:39:362
971699812,7cyclictest0-21swapper/208:29:452
971699812,6cyclictest0-21swapper/212:04:482
971699812,6cyclictest0-21swapper/212:04:482
971499813,28cyclictest18281-21latency_hist07:34:371
971699804,31cyclictest21813-21/usr/sbin/munin11:15:102
971699803,33cyclictest27239-21ssh10:25:482
971699803,32cyclictest14117-21/usr/sbin/munin11:59:472
971699803,31cyclictest20402-21/usr/sbin/munin09:15:072
971499804,28cyclictest5407-21/usr/sbin/munin11:44:521
971499803,34cyclictest2763-21/usr/sbin/munin12:35:021
971499803,34cyclictest2763-21/usr/sbin/munin12:35:021
971699793,30cyclictest23259-21/usr/sbin/munin07:49:462
971699792,35cyclictest1123-21ssh09:39:492
971499793,34cyclictest12757-21mailstats08:55:351
964627911,41sleep30-21swapper/307:09:373
971699784,12cyclictest2131-21apt-get08:24:462
972399613,46cyclictest16978-21taskset12:04:373
972399593,42cyclictest11251-21chrt07:14:343
972399583,44cyclictest5740-21taskset12:39:343
972399583,44cyclictest5740-21taskset12:39:343
972399574,41cyclictest31152-21chrt10:34:343
972399574,40cyclictest22384-21chrt12:13:283
972399573,40cyclictest23153-21chrt07:49:373
972399573,40cyclictest19280-21chrt11:10:123
972399573,36cyclictest3969-21chrt09:44:513
972399573,34cyclictest912-21kworker/3:1+events11:04:503
972399564,37cyclictest24099-21chrt11:19:523
972399563,41cyclictest29373-21chrt11:29:373
972399563,40cyclictest996-21taskset09:39:373
972399563,40cyclictest7112-21taskset09:50:023
972399563,38cyclictest23814-21chrt12:15:063
972399563,38cyclictest23814-21chrt12:15:063
972399563,38cyclictest20295-21taskset10:14:373
972399554,39cyclictest778-21chrt08:19:583
972399554,39cyclictest1909-21taskset09:40:103
972399554,38cyclictest8773-21taskset11:50:033
972399554,38cyclictest8773-21taskset11:50:033
972399554,38cyclictest25487-21chrt07:55:083
972399554,38cyclictest18963-21taskset07:35:073
972399554,37cyclictest32696-21chrt08:19:343
972399553,40cyclictest11590-21chrt09:59:013
972399553,40cyclictest11590-21chrt09:59:013
972399553,39cyclictest11831-21chrt11:55:053
972399553,37cyclictest26843-21chrt12:20:093
972399553,37cyclictest14717-21chrt10:04:363
972399552,41cyclictest398-21chrt10:36:083
972399552,39cyclictest27541-21taskset09:29:083
972399552,38cyclictest20718-21chrt07:40:063
972399544,38cyclictest15077-21taskset10:04:533
972399544,32cyclictest2272-21chrt08:24:523
972399543,40cyclictest5513-21taskset10:46:043
972399543,39cyclictest3057-21chrt11:40:023
972399543,38cyclictest23268-21chrt09:20:083
972399543,37cyclictest28314-21chrt08:04:563
972399543,37cyclictest28314-21chrt08:04:563
972399542,38cyclictest5739-21taskset11:44:563
972399542,37cyclictest7248-21chrt08:39:593
972399542,37cyclictest7248-21chrt08:39:593
972399542,37cyclictest15270-21taskset07:24:513
972399542,37cyclictest14216-21chrt09:00:043
972399542,36cyclictest30197-21chrt12:25:113
972399534,38cyclictest14369-21taskset07:20:093
972399534,36cyclictest17410-21chrt07:30:053
972399533,39cyclictest2385-21chrt12:34:233
972399533,38cyclictest28355-21chrt09:29:573
972399533,38cyclictest17362-21chrt09:10:003
972399533,38cyclictest17362-21chrt09:10:003
972399533,37cyclictest10430-21chrt10:55:453
972399533,36cyclictest24114-21chrt10:20:083
972399533,36cyclictest21724-21chrt11:15:063
972399533,36cyclictest19792-21chrt09:14:503
972399533,35cyclictest7956-21chrt10:50:143
972399533,32cyclictest4392-21chrt08:30:073
972399532,37cyclictest32740-21chrt11:35:043
972399532,37cyclictest32740-21chrt11:35:043
972399524,35cyclictest14968-21chrt11:03:473
972399524,35cyclictest14968-21chrt11:03:473
972399524,35cyclictest12342-21chrt08:55:223
972399523,37cyclictest30225-21chrt08:10:023
972399523,36cyclictest2272-21chrt10:40:133
972399522,35cyclictest5434-21chrt08:34:533
972399514,35cyclictest24067-21chrt07:50:103
972399513,38cyclictest28167-21chrt10:28:473
972399512,37cyclictest22152-21chrt10:17:043
972399504,34cyclictest11118-21chrt08:50:113
972399503,37cyclictest30452-21chrt11:30:143
972399495,31cyclictest26935-21chrt08:00:023
972399495,31cyclictest26935-21chrt08:00:023
972399493,33cyclictest9713-21chrt08:46:553
972399493,33cyclictest9713-21chrt08:46:553
972399483,35cyclictest18224-21chrt12:05:143
972399483,35cyclictest18224-21chrt12:05:143
972399473,29cyclictest12770-21chrt07:15:113
972399434,27cyclictest16440-21chrt09:07:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional