You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 20:42
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Thu Dec 04, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25363214610,93sleep125419-21timerandwakeup07:05:251
25130212113,69sleep024536-21/usr/sbin/munin07:05:160
26052991193,96cyclictest11514-21apt-get10:29:211
260409911121,52cyclictest0-21swapper/007:44:420
260609910952,50cyclictest723-21diskmemload10:04:032
260609910952,50cyclictest723-21diskmemload10:04:032
26052991093,97cyclictest1348-21apt-get11:08:481
26052991093,97cyclictest1348-21apt-get11:08:481
26052991093,76cyclictest28649-21apt-get10:58:501
26052991093,76cyclictest28649-21apt-get10:58:501
26040991073,27cyclictest11687-21ssh10:29:190
26040991072,41cyclictest11364-21seq09:28:350
26052991066,79cyclictest9240-21cstates12:19:271
26052991066,79cyclictest9240-21cstates12:19:271
26040991062,25cyclictest23518-21runrttasks10:50:150
26040991062,25cyclictest23518-21runrttasks10:50:150
26040991053,29cyclictest691-21cron10:43:340
260409910512,23cyclictest18318-21cat12:34:100
260409910512,23cyclictest18318-21cat12:34:100
260409910235,59cyclictest0-21swapper/011:44:100
26040991022,21cyclictest0-21swapper/011:08:520
26040991022,21cyclictest0-21swapper/011:08:520
26040991012,25cyclictest10942-21sh12:22:040
26040991012,25cyclictest10942-21sh12:22:040
2606099994,35cyclictest25873-21fschecks_time10:53:592
2604099993,25cyclictest4419-21ps09:14:070
2604099992,18cyclictest19558-21apt-get09:43:460
26060999828,12cyclictest0-21swapper/207:33:372
26060999828,12cyclictest0-21swapper/207:33:372
2604099983,27cyclictest29596-21modprobe07:18:340
2604099983,12cyclictest3090-21apt-get10:13:500
2604099983,12cyclictest3090-21apt-get10:13:500
2604099982,29cyclictest7137-21rm12:14:510
26060999735,9cyclictest0-21swapper/211:04:582
26060999735,9cyclictest0-21swapper/211:04:582
26060999732,8cyclictest0-21swapper/212:38:362
26060999732,8cyclictest0-21swapper/212:38:362
2604099973,29cyclictest2241-21/usr/sbin/munin07:34:080
2604099973,29cyclictest2241-21/usr/sbin/munin07:34:080
2604099973,28cyclictest27849-21grep11:54:070
2604099973,28cyclictest26691-21modprobe08:48:340
26060999637,9cyclictest0-21swapper/209:09:502
26060999637,9cyclictest0-21swapper/209:09:502
2606099963,45cyclictest19581-21ssh09:43:502
2604099964,10cyclictest31416-21apt-get11:03:500
2604099964,10cyclictest31416-21apt-get11:03:500
2604099963,29cyclictest16948-21latency_hist08:18:360
2604099963,29cyclictest16948-21latency_hist08:18:360
2604099963,24cyclictest9321-21/usr/sbin/munin10:24:070
2604099963,24cyclictest9321-21/usr/sbin/munin10:24:070
2604099963,24cyclictest14516-21fschecks_count10:33:570
2604099963,21cyclictest15882-21ssh12:29:540
2604099962,21cyclictest7519-21unixbench_multi09:19:110
2604099962,21cyclictest14864-21processes09:34:100
2606099953,40cyclictest32062-21fschecks_time09:03:572
2606099953,30cyclictest14024-21fschecks_time08:08:562
2606099952,41cyclictest27528-21sort09:58:492
2604099954,27cyclictest9422-1kworker/0:0H+kblockd08:58:500
2604099954,21cyclictest14124-21ssh12:28:140
26040999539,48cyclictest0-21swapper/010:02:510
2604099953,31cyclictest3580-21ntp_kernel_pll_12:09:030
2604099953,30cyclictest7947-21grep11:19:130
2604099953,30cyclictest7947-21grep11:19:130
2604099953,27cyclictest24511-21modprobe09:53:330
2604099953,27cyclictest24511-21modprobe09:53:330
2604099953,27cyclictest10966-21wc07:58:580
2604099953,25cyclictest5588-21/usr/sbin/munin11:14:120
2604099953,23cyclictest30162-21apt-get10:03:460
2604099953,23cyclictest30162-21apt-get10:03:460
2604099953,21cyclictest24814-21latency11:48:590
26040999516,25cyclictest1874-21wc09:09:110
26040999516,25cyclictest1874-21wc09:09:110
26060999442,46cyclictest16259-21fschecks_time11:33:572
2606099943,36cyclictest6433-21grep07:44:092
2606099943,34cyclictest12309-21taskset10:29:582
2606099943,33cyclictest8562-21latency_hist10:23:382
2606099943,33cyclictest8497-21apt-config12:18:362
2604099944,28cyclictest16319-21/usr/sbin/munin11:33:580
2604099943,27cyclictest18792-21latency_hist08:23:370
2604099942,28cyclictest30280-21cat07:19:010
2606099933,36cyclictest28560-21/usr/sbin/munin08:54:082
2606099933,36cyclictest28560-21/usr/sbin/munin08:54:082
2606099933,33cyclictest26487-21fschecks_time07:08:562
26060999332,14cyclictest0-21swapper/211:59:592
26060999332,12cyclictest0-21swapper/208:33:362
2606099932,37cyclictest17100-21cpu10:38:492
2606099932,34cyclictest13885-21idleruntime-cro09:33:332
26060999323,33cyclictest24426-21grep11:48:502
26040999334,52cyclictest0-21swapper/007:39:050
2604099933,30cyclictest7427-21ssh10:20:450
2604099933,30cyclictest7427-21ssh10:20:450
2604099933,30cyclictest1447-21cat07:29:090
2604099933,30cyclictest1447-21cat07:29:090
2604099933,28cyclictest20674-21grep08:28:550
2604099933,27cyclictest11772-21cstates09:28:530
2604099933,21cyclictest26198-21runrttasks10:54:050
2604099932,26cyclictest10720-21perl11:23:570
2604099932,23cyclictest23563-21latency_hist08:38:360
2604099932,22cyclictest29055-21fschecks_time10:59:170
2604099932,22cyclictest29055-21fschecks_time10:59:170
2606099924,30cyclictest355-21runrttasks11:47:352
26060999236,8cyclictest0-21swapper/212:19:372
26060999236,8cyclictest0-21swapper/212:19:372
2606099923,36cyclictest4138-21fschecks_time07:38:582
2606099923,36cyclictest18395-21munin-plugin-st08:23:332
2606099923,34cyclictest19788-21/usr/sbin/munin10:43:522
2606099923,33cyclictest26946-21/usr/sbin/munin08:48:522
2606099923,32cyclictest7628-21/usr/sbin/munin11:19:472
2606099923,32cyclictest7628-21/usr/sbin/munin11:19:472
2605299928,70cyclictest22868-21sh10:48:501
2605299928,70cyclictest22868-21sh10:48:501
2604099923,31cyclictest16968-21/usr/sbin/munin09:38:500
2604099923,28cyclictest19346-21/usr/sbin/munin11:39:020
2604099923,28cyclictest19346-21/usr/sbin/munin11:39:020
2604099923,26cyclictest861-21/usr/sbin/munin12:04:060
2604099923,24cyclictest28303-21sh08:53:340
2604099923,22cyclictest18889-21apt-get08:23:440
26040999230,54cyclictest0-21swapper/010:13:330
26040999228,57cyclictest0-21swapper/011:28:570
2604099922,21cyclictest24828-21apt-get09:53:460
2606099914,35cyclictest11970-21latency_hist08:03:362
2606099914,33cyclictest17745-21timerandwakeup09:39:102
26060999136,7cyclictest0-21swapper/209:53:582
2606099912,36cyclictest2184-21cpuspeed11:09:382
2606099912,36cyclictest2184-21cpuspeed11:09:382
2606099912,35cyclictest15612-21fschecks_count08:13:562
2606099912,35cyclictest15612-21fschecks_count08:13:562
2606099912,32cyclictest12180-21/usr/sbin/munin08:04:072
2604099912,30cyclictest30128-21fschecks_count11:58:550
2604099912,24cyclictest28587-21cpu08:53:530
2604099912,24cyclictest28587-21cpu08:53:530
2606099905,30cyclictest26028-21cyclictest07:08:382
2606099904,36cyclictest29715-21timerwakeupswit10:59:322
2606099904,36cyclictest29715-21timerwakeupswit10:59:322
26060999036,8cyclictest0-21swapper/209:49:072
26060999036,8cyclictest0-21swapper/209:49:072
2606099903,37cyclictest31728-21fschecks_time07:23:572
2606099903,36cyclictest23732-21latency10:50:202
2606099903,36cyclictest23732-21latency10:50:202
2606099903,34cyclictest1329-21ssh12:04:132
2606099903,32cyclictest376-21sshd09:15:582
2606099902,40cyclictest27227-21cstates11:53:502
2606099902,32cyclictest22308-21cut08:33:552
26060999021,13cyclictest0-21swapper/209:34:102
26040999037,46cyclictest0-21swapper/007:23:580
26040999035,51cyclictest0-21swapper/009:04:110
26040999033,50cyclictest0-21swapper/008:09:000
2604099903,22cyclictest7322-21/usr/sbin/munin07:48:450
2604099902,25cyclictest20044-21runrttasks10:43:550
2606099894,32cyclictest26002-21grep08:44:082
2606099893,36cyclictest14503-21fschecks_count10:33:572
2606099893,35cyclictest3606-21modprobe07:38:332
2606099893,35cyclictest3606-21modprobe07:38:332
2606099893,34cyclictest5732-21ssh10:18:372
2606099893,34cyclictest5732-21ssh10:18:372
2606099893,33cyclictest6503-21/usr/sbin/munin09:19:122
2606099893,33cyclictest29905-21cpu07:18:522
2604099894,27cyclictest26476-21cut07:08:560
2604099893,29cyclictest12180-21/usr/sbin/munin08:04:020
2606099883,38cyclictest355-21runrttasks11:15:122
2606099883,36cyclictest8324-21latency_hist10:23:352
2606099883,32cyclictest7646-21fschecks_time07:49:332
26052998821,41cyclictest30217-21cpu08:58:501
2604099883,28cyclictest8931-21/usr/sbin/munin07:54:070
2604099883,28cyclictest23951-21fschecks_count08:38:570
2606099874,27cyclictest13149-21/usr/sbin/munin11:28:562
2606099873,33cyclictest25078-21sh08:43:342
2606099873,32cyclictest10716-21fschecks_time11:23:572
2606099872,36cyclictest11311-21modprobe09:28:332
2606099872,31cyclictest9424-21awk07:54:012
26060998721,26cyclictest28033-21cpu07:13:432
26052998726,24cyclictest30143-21fschecks_time07:18:581
2606099864,34cyclictest8618-21/usr/sbin/munin10:23:552
2606099864,34cyclictest8618-21/usr/sbin/munin10:23:552
2606099864,32cyclictest14540-21/usr/sbin/munin12:28:592
2606099863,33cyclictest5747-21latency_hist12:13:372
2606099862,33cyclictest30500-21idleruntime08:58:572
2606099854,34cyclictest732-21fschecks_time10:08:562
2606099854,29cyclictest19070-21fschecks_count11:38:562
2606099854,29cyclictest19070-21fschecks_count11:38:562
2606099853,33cyclictest19217-21fschecks_time08:24:522
2606099853,30cyclictest12221-21latency12:24:012
26052998463,13cyclictest16813-21timerandwakeup11:34:091
26052998055,15cyclictest6437-21kworker/u17:2-rpciod10:28:381
26052998050,20cyclictest29744-21sh11:58:371
26052998014,23cyclictest17992-21seq12:34:041
26052998014,23cyclictest17992-21seq12:34:041
26052997958,15cyclictest355-21runrttasks08:29:081
26052997954,14cyclictest0-21swapper/109:38:341
26052997858,14cyclictest355-21runrttasks11:13:581
26052997856,14cyclictest13924-21latency_hist09:33:351
26052997854,16cyclictest26028-21cyclictest07:43:491
2605299773,45cyclictest28167-21munin-run10:58:351
2605299768,37cyclictest7340-21apt-get07:48:491
26052997650,17cyclictest8975-21ssh09:23:361
26052997624,23cyclictest1131-21sh10:09:071
26052997552,17cyclictest31412-21snmpd09:48:591
26052997552,17cyclictest31412-21snmpd09:48:591
26052997350,14cyclictest25550-21fschecks_count08:43:551
26052997256,11cyclictest26028-21cyclictest12:28:571
26052997250,13cyclictest31727-21fschecks_time07:23:571
26052997248,18cyclictest16894-21latency_hist10:38:361
26052997247,17cyclictest723-21diskmemload12:07:101
2605299718,25cyclictest6065-21fschecks_count12:13:551
26052997149,12cyclictest0-21swapper/109:04:111
2605299712,50cyclictest10822-21fschecks_time07:58:551
26052997050,13cyclictest9377-21ssh09:23:571
26052997041,21cyclictest292-21dbus-daemon09:58:341
2605299703,44cyclictest20742-21timerwakeupswit10:44:101
26052997018,22cyclictest9007-21sh11:20:071
26052997018,22cyclictest9007-21sh11:20:071
26052997014,50cyclictest0-21swapper/109:13:471
26052996910,24cyclictest0-21swapper/110:18:471
26052996910,24cyclictest0-21swapper/110:18:471
26067996812,40cyclictest0-21swapper/308:24:473
2605299684,33cyclictest9887-21unixbench_singl07:54:101
26052996839,21cyclictest0-21swapper/108:53:591
26052996839,21cyclictest0-21swapper/108:53:591
2605299683,43cyclictest2970-21switchtime07:34:081
2605299683,43cyclictest2970-21switchtime07:34:081
26052996752,10cyclictest355-21runrttasks09:47:171
26052996751,10cyclictest28338-21timerandwakeup09:59:091
26052996745,13cyclictest28503-21fschecks_time07:13:581
26052996737,23cyclictest0-21swapper/110:28:361
2605299673,27cyclictest11974-21fschecks_count12:23:571
26052996650,10cyclictest17453-21/usr/sbin/munin09:39:011
26052996626,34cyclictest0-21swapper/108:08:451
2605299662,41cyclictest19508-21awk10:43:341
26052996615,12cyclictest0-21swapper/111:59:091
26052996543,16cyclictest0-21swapper/108:18:331
26052996543,16cyclictest0-21swapper/108:18:331
26052996519,34cyclictest17778-21cat08:19:091
2605299644,27cyclictest32587-21munin-run10:08:341
2605299644,27cyclictest32587-21munin-run10:08:341
26052996442,17cyclictest1470-21apt-key11:08:361
26052996442,17cyclictest1470-21apt-key11:08:361
26052996442,15cyclictest24056-21munin-plugin-st11:48:351
2605299643,42cyclictest18389-21apt-get08:23:471
26052996427,25cyclictest24507-21cat08:39:101
26052996417,32cyclictest3339-21tune2fs10:13:561
26052996417,32cyclictest3339-21tune2fs10:13:561
26052996412,20cyclictest5128-21seq07:42:231
26067996318,26cyclictest0-21swapper/311:18:493
26067996318,26cyclictest0-21swapper/311:18:493
26052996317,9cyclictest13781-21ntp_states11:29:051
26052996239,12cyclictest0-21swapper/111:51:331
2605299622,43cyclictest2039-21wc07:33:351
2605299622,43cyclictest2039-21wc07:33:351
26052996222,22cyclictest0-21swapper/107:08:591
2456826211,32sleep30-21swapper/307:03:563
2606799614,49cyclictest9265-21chrt12:19:283
2606799614,49cyclictest9265-21chrt12:19:283
2606799613,41cyclictest30211-21taskset08:58:483
2606799613,39cyclictest3695-21taskset10:14:043
2606799613,39cyclictest3695-21taskset10:14:043
26052996145,11cyclictest26028-21cyclictest08:49:111
26052996135,17cyclictest0-21swapper/111:28:371
2605299613,34cyclictest1150-21sort09:08:521
2605299613,34cyclictest1150-21sort09:08:521
2605299602,37cyclictest12682-21sed08:04:031
2606799593,39cyclictest2507-21taskset07:33:553
2606799593,39cyclictest2507-21taskset07:33:553
2605299592,43cyclictest22866-21awk08:34:081
2605299592,35cyclictest19645-21switchtime11:39:091
2605299592,35cyclictest19645-21switchtime11:39:091
2606799583,40cyclictest1244-21taskset09:08:563
2606799583,40cyclictest1244-21taskset09:08:563
26052995834,12cyclictest0-21swapper/112:08:581
184752580,6sleep30-21swapper/311:38:323
2606799576,34cyclictest0-21swapper/307:49:213
2606799574,40cyclictest14683-21taskset12:28:523
2606799573,42cyclictest0-21swapper/310:29:023
26067995711,37cyclictest0-21swapper/311:08:573
26067995711,37cyclictest0-21swapper/311:08:573
2606799563,40cyclictest29622-21taskset11:58:343
2606799563,38cyclictest895-21kworker/3:1+events09:29:073
2606799553,35cyclictest9754-21chrt09:24:043
2606799553,35cyclictest10288-21taskset07:58:343
2606799545,31cyclictest895-21kworker/3:1+events08:08:513
2606799544,37cyclictest28489-21taskset08:53:373
2606799544,34cyclictest32545-21chrt12:03:363
2606799543,37cyclictest7188-21chrt11:18:353
2606799543,36cyclictest538-21chrt12:03:573
2606799537,31cyclictest0-21swapper/310:58:563
2606799537,31cyclictest0-21swapper/310:58:563
2606799534,35cyclictest13192-21taskset12:25:163
2606799533,38cyclictest23851-21taskset10:50:233
2606799533,38cyclictest23851-21taskset10:50:233
2606799533,38cyclictest13879-21taskset11:29:083
2606799533,37cyclictest14257-21chrt10:33:453
2606799533,35cyclictest25841-21taskset08:44:023
2606799532,37cyclictest28644-21taskset09:59:343
2606799532,35cyclictest4001-21chrt07:38:533
2606799527,32cyclictest1012-21chrt11:07:293
2606799527,32cyclictest1012-21chrt11:07:293
2606799524,36cyclictest22011-21chrt08:33:373
2606799524,36cyclictest21672-21chrt11:43:513
2606799523,37cyclictest32476-21taskset07:24:113
2606799523,36cyclictest7428-21taskset10:20:453
2606799523,36cyclictest7428-21taskset10:20:453
2606799523,35cyclictest5698-21chrt12:13:373
2606799523,33cyclictest19422-21chrt09:43:373
2606799522,36cyclictest5798-21taskset07:43:563
2606799522,36cyclictest3958-21chrt09:13:553
2606799522,33cyclictest30277-21chrt07:19:013
2606799522,28cyclictest27098-21taskset09:58:283
2606799515,33cyclictest8658-21taskset10:23:493
2606799515,33cyclictest8658-21taskset10:23:493
2606799514,33cyclictest895-21kworker/3:1+events08:58:363
2606799514,33cyclictest895-21kworker/3:1+events08:58:363
2606799513,35cyclictest30734-21chrt10:04:043
2606799513,35cyclictest30734-21chrt10:04:043
2606799513,34cyclictest18020-21chrt08:19:113
2606799513,29cyclictest0-21swapper/310:58:333
2606799512,33cyclictest17358-21chrt10:38:563
2606799503,33cyclictest21235-21taskset09:46:093
2606799503,32cyclictest12973-21taskset11:28:363
2606799502,36cyclictest22252-21chrt10:48:283
2606799502,36cyclictest20314-21chrt11:40:273
2606799502,36cyclictest20314-21chrt11:40:273
2606799502,34cyclictest23708-21chrt09:50:263
2606799502,34cyclictest23708-21chrt09:50:263
2606799493,36cyclictest2862-21chrt10:13:363
2606799493,33cyclictest18723-21chrt12:34:563
2606799493,33cyclictest18723-21chrt12:34:563
2606799493,32cyclictest8251-21chrt12:18:233
2606799492,36cyclictest22439-21chrt08:33:573
2606799483,33cyclictest11263-21chrt07:59:063
2606799483,31cyclictest13526-21chrt08:08:343
2606799473,35cyclictest16518-21chrt09:38:263
2606799473,33cyclictest23899-21chrt08:38:553
2606799463,33cyclictest32621-21chrt09:04:103
2606799463,33cyclictest29712-21chrt07:18:373
2606799453,31cyclictest8007-21chrt09:20:313
2606799452,34cyclictest1532-21chrt07:29:103
2606799452,34cyclictest1532-21chrt07:29:103
2606799449,21cyclictest0-21swapper/308:18:363
2606799449,21cyclictest0-21swapper/308:18:363
2606799443,26cyclictest26543-21chrt11:52:423
2606799423,28cyclictest26765-21taskset07:09:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional