You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-18 - 23:37
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Wed Jun 18, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12881211310,73sleep212960-21/usr/sbin/munin07:02:542
143269910479,16cyclictest23956-21ssh12:08:000
143269910479,16cyclictest23956-21ssh12:08:000
14334999572,13cyclictest0-21swapper/111:17:341
137272958,58sleep012851-21/usr/sbin/munin07:03:210
132182958,62sleep113222-21/usr/sbin/munin07:03:061
14326998968,13cyclictest202-21systemd-journal10:20:390
14326998752,7cyclictest181rcu_preempt10:25:040
14344998663,17cyclictest734-21netstat08:03:022
14344998663,17cyclictest734-21netstat08:03:022
14334998667,13cyclictest365-21systemd-logind10:52:331
14334998667,13cyclictest365-21systemd-logind10:52:331
14326998618,34cyclictest15-21ksoftirqd/008:18:460
14344998565,14cyclictest593-21sh09:28:002
14326998554,25cyclictest22112-21cat10:08:110
14326998444,33cyclictest14319-21cyclictest11:08:250
14344998360,16cyclictest796-21timerwakeupswit12:23:102
14344998353,21cyclictest23966-21ls12:08:002
14344998353,21cyclictest23966-21ls12:08:002
14334998363,11cyclictest0-21swapper/107:33:061
14326998359,16cyclictest2650-21ssh12:27:350
14326998346,8cyclictest181rcu_preempt08:44:590
14344998245,8cyclictest181rcu_preempt09:41:022
14344998245,8cyclictest181rcu_preempt07:54:342
14344998244,9cyclictest181rcu_preempt11:46:572
14344998244,9cyclictest181rcu_preempt11:10:182
14344998244,10cyclictest181rcu_preempt10:16:452
14344998242,10cyclictest181rcu_preempt12:32:152
14334998263,10cyclictest0-21swapper/107:28:101
14334998263,10cyclictest0-21swapper/107:28:101
14334998262,11cyclictest0-21swapper/107:08:151
14326998250,8cyclictest181rcu_preempt11:54:160
14326998249,11cyclictest181rcu_preempt11:46:490
14344998165,9cyclictest0-21swapper/212:32:552
14344998164,11cyclictest19523-21ssh10:03:122
14344998148,24cyclictest3215-21sh10:33:192
14344998145,8cyclictest181rcu_preempt09:25:152
14344998144,8cyclictest181rcu_preempt11:07:392
14344998144,8cyclictest181rcu_preempt11:07:392
14344998143,8cyclictest181rcu_preempt09:32:372
14344998143,8cyclictest181rcu_preempt09:32:372
14326998165,9cyclictest19109-21kworker/u18:1-events_unbound12:33:080
14326998162,9cyclictest0-21swapper/011:28:020
14326998162,9cyclictest0-21swapper/011:28:020
14326998158,14cyclictest26656-21ssh12:12:570
14326998155,20cyclictest14319-21cyclictest11:17:550
14326998155,20cyclictest14319-21cyclictest11:17:550
14326998143,27cyclictest11514-21latency_hist08:37:360
14326998138,14cyclictest0-21swapper/011:23:140
210582800,21sleep1301ktimers/112:03:011
14344998057,16cyclictest29333-21latency_hist10:22:352
14334998060,10cyclictest0-21swapper/110:33:381
14326998038,9cyclictest181rcu_preempt10:33:300
14344997960,13cyclictest32658-21latency_hist11:27:352
14344997960,13cyclictest22133-21ntp_kernel_pll_09:08:012
14344997960,13cyclictest22133-21ntp_kernel_pll_09:08:012
14344997950,21cyclictest15850-21modprobe07:12:322
14344997939,23cyclictest25899-21ssh09:14:202
14344997939,11cyclictest4683-21ssh11:34:482
14326997959,14cyclictest13773-21sh10:52:500
14326997958,7cyclictest181rcu_preempt10:43:480
14326997951,8cyclictest181rcu_preempt10:29:370
14326997951,10cyclictest181rcu_preempt10:41:390
14326997942,8cyclictest181rcu_preempt08:03:090
14326997942,8cyclictest181rcu_preempt08:03:090
14326997936,15cyclictest181rcu_preempt09:08:200
14326997936,15cyclictest181rcu_preempt09:08:200
14326997919,34cyclictest15-21ksoftirqd/011:36:500
14344997862,10cyclictest6068-21idleruntime10:37:582
14344997857,11cyclictest0-21swapper/208:52:352
14344997842,9cyclictest181rcu_preempt08:43:072
14344997841,8cyclictest181rcu_preempt09:58:592
14344997840,9cyclictest181rcu_preempt08:38:152
14326997853,10cyclictest181rcu_preempt12:06:100
14326997848,21cyclictest2286-21ssh09:32:240
14326997832,8cyclictest181rcu_preempt09:51:240
14344997758,15cyclictest11339-21kworker/u18:0+events_unbound08:57:572
14344997752,7cyclictest181rcu_preempt08:22:552
14344997743,8cyclictest181rcu_preempt11:37:562
14344997740,7cyclictest181rcu_preempt09:53:362
14344997740,7cyclictest181rcu_preempt09:53:362
14326997750,18cyclictest26775-21sh09:17:330
14326997749,19cyclictest24454-21interrupts10:12:580
14326997734,11cyclictest181rcu_preempt11:48:060
14344997661,8cyclictest0-21swapper/210:28:442
14344997658,8cyclictest0-21swapper/210:48:572
14344997658,8cyclictest0-21swapper/210:48:572
14344997641,9cyclictest181rcu_preempt08:08:092
14344997641,9cyclictest181rcu_preempt08:08:092
14344997641,7cyclictest181rcu_preempt11:19:142
14344997641,7cyclictest181rcu_preempt11:19:142
14344997639,7cyclictest181rcu_preempt09:21:232
14334997655,14cyclictest356-21runrttasks09:22:341
14334997644,24cyclictest365-21systemd-logind10:08:511
14326997655,15cyclictest21252-21diskmemload12:21:230
14326997654,12cyclictest0-21swapper/009:32:580
14326997654,12cyclictest0-21swapper/009:32:580
14326997653,17cyclictest26076-21latency_hist07:42:340
14326997652,7cyclictest181rcu_preempt09:18:050
14326997646,22cyclictest6469-21ntp_kernel_err11:38:000
14326997646,22cyclictest30168-21switchtime07:53:070
14326997637,8cyclictest0-21swapper/009:43:010
14326997637,8cyclictest0-21swapper/009:43:010
14344997547,7cyclictest181rcu_preempt12:16:102
14344997544,10cyclictest181rcu_preempt09:42:402
14344997544,10cyclictest181rcu_preempt09:42:402
14344997534,7cyclictest181rcu_preempt11:15:362
14344997534,10cyclictest181rcu_preempt10:45:032
14344997513,25cyclictest39-21ksoftirqd/212:18:152
14334997561,8cyclictest0-21swapper/109:08:071
14334997561,8cyclictest0-21swapper/109:08:071
14334997559,10cyclictest8366-21modprobe12:37:321
14334997559,10cyclictest29099-21systemd-run12:17:331
14326997560,8cyclictest0-21swapper/010:59:380
14326997550,14cyclictest0-21swapper/007:33:070
14344997456,11cyclictest0-21swapper/209:47:572
14344997447,8cyclictest181rcu_preempt11:55:402
14344997442,8cyclictest181rcu_preempt11:49:422
14344997439,26cyclictest21080-21sshd07:26:552
14344997438,15cyclictest2512-21ssh11:31:492
14344997438,15cyclictest2512-21ssh11:31:492
14344997437,18cyclictest19299-21ssh11:59:482
14344997437,18cyclictest19299-21ssh11:59:482
14334997457,12cyclictest21252-21diskmemload11:57:331
14326997454,14cyclictest2350-21netstat08:08:020
14326997454,14cyclictest2350-21netstat08:08:020
14326997449,19cyclictest13494-21sh10:52:350
14326997449,19cyclictest13494-21sh10:52:350
14326997438,21cyclictest26802-21ntpq07:43:050
14326997425,28cyclictest15-21ksoftirqd/010:05:240
14326997419,28cyclictest15-21ksoftirqd/009:40:130
14344997353,15cyclictest0-21swapper/210:23:322
14344997351,12cyclictest0-21swapper/208:17:552
14344997340,9cyclictest181rcu_preempt10:53:432
14344997331,9cyclictest181rcu_preempt07:35:152
14334997358,8cyclictest0-21swapper/111:43:111
14334997357,9cyclictest0-21swapper/109:13:361
14334997350,15cyclictest2986-21sh12:27:541
14334997349,16cyclictest13929-21processes09:53:081
14334997349,16cyclictest13929-21processes09:53:081
14334997343,21cyclictest17741-21modprobe08:57:321
1433499733,32cyclictest31-21ksoftirqd/111:33:101
14326997350,17cyclictest19375-21fschecks_time11:02:590
14326997350,17cyclictest19375-21fschecks_time11:02:590
14326997349,16cyclictest20171-21fschecks_time09:02:560
14326997348,15cyclictest2756-21ssh12:27:420
1432699734,36cyclictest15-21ksoftirqd/007:48:010
14326997338,7cyclictest181rcu_preempt07:14:290
14326997336,11cyclictest14319-21cyclictest08:00:030
14326997336,11cyclictest14319-21cyclictest08:00:030
14326997334,9cyclictest181rcu_preempt09:54:570
14326997334,9cyclictest181rcu_preempt09:54:570
14326997321,30cyclictest15-21ksoftirqd/011:13:440
183052720,7sleep10-21swapper/110:02:321
14344997253,9cyclictest0-21swapper/208:13:102
14344997247,7cyclictest181rcu_preempt07:39:282
14344997246,7cyclictest181rcu_preempt11:00:332
14344997246,7cyclictest181rcu_preempt10:08:152
14344997231,9cyclictest181rcu_preempt08:36:202
14334997249,16cyclictest7239-21kworker/u19:1+events_unbound08:27:321
14326997246,17cyclictest30923-21switchtime09:23:540
14326997242,10cyclictest181rcu_preempt08:24:170
14326997235,11cyclictest58-21kcompactd007:26:420
162812710,7sleep30-21swapper/310:57:353
14344997156,9cyclictest0-21swapper/208:32:352
14344997152,10cyclictest0-21swapper/208:57:362
14344997139,7cyclictest181rcu_preempt09:03:072
14344997131,9cyclictest181rcu_preempt07:32:122
14344997131,9cyclictest181rcu_preempt07:32:122
14334997156,10cyclictest16544-21/usr/sbin/munin07:13:041
14334997152,13cyclictest27947-21sh11:18:071
14334997152,13cyclictest27947-21sh11:18:071
14334997150,10cyclictest18490-21/usr/sbin/munin11:58:091
14334997150,10cyclictest18490-21/usr/sbin/munin11:58:091
14326997147,12cyclictest181rcu_preempt11:58:270
14326997147,12cyclictest181rcu_preempt11:58:270
14326997143,7cyclictest181rcu_preempt08:52:040
14326997138,7cyclictest181rcu_preempt07:11:010
14326997137,8cyclictest181rcu_preempt08:39:080
14326997137,8cyclictest181rcu_preempt07:29:480
14326997137,8cyclictest181rcu_preempt07:29:480
14326997135,8cyclictest181rcu_preempt07:18:390
14326997120,26cyclictest15-21ksoftirqd/009:59:310
14344997057,7cyclictest0-21swapper/212:03:032
14344997028,9cyclictest181rcu_preempt07:57:432
14344997028,9cyclictest181rcu_preempt07:57:432
14334997054,10cyclictest8044-21/usr/sbin/munin09:42:541
14334997054,10cyclictest8044-21/usr/sbin/munin09:42:541
1433499704,28cyclictest31-21ksoftirqd/110:40:411
14326997043,19cyclictest9797-21latency_hist08:32:350
14344996951,10cyclictest0-21swapper/207:17:392
1434499694,26cyclictest39-21ksoftirqd/207:51:342
14334996953,9cyclictest0-21swapper/108:17:321
14334996949,15cyclictest5317-21sh09:37:491
14334996947,16cyclictest202-21systemd-journal12:27:341
14334996938,22cyclictest3439-21cat09:33:101
14334996938,22cyclictest3439-21cat09:33:101
14334996937,23cyclictest8754-21grep08:27:581
1433499692,33cyclictest31-21ksoftirqd/110:25:591
14344996849,7cyclictest181rcu_preempt07:43:492
1433499684,25cyclictest31-21ksoftirqd/109:06:501
14334996838,21cyclictest29195-21modprobe07:52:321
1432699686,25cyclictest15-21ksoftirqd/008:57:500
14334996751,10cyclictest29090-21idleruntime-cro10:22:321
14334996750,10cyclictest20311-21ssh11:03:391
14334996750,10cyclictest20311-21ssh11:03:391
14334996747,13cyclictest29858-21latency12:18:001
14334996747,13cyclictest26105-21latency_hist12:12:351
14334996747,13cyclictest26105-21latency_hist12:12:351
1433499672,35cyclictest31-21ksoftirqd/109:51:221
1433499672,24cyclictest31-21ksoftirqd/111:09:491
14326996747,11cyclictest0-21swapper/008:57:350
1433499667,24cyclictest31-21ksoftirqd/111:42:301
14334996650,10cyclictest16921-21ntp_kernel_err10:58:011
14334996648,12cyclictest19644-21awk09:02:321
14334996646,13cyclictest25271-21sendmail_mailqu07:38:071
1433499662,25cyclictest31-21ksoftirqd/108:01:581
1433499662,25cyclictest31-21ksoftirqd/108:01:581
14344996532,7cyclictest181rcu_preempt07:12:462
1433499657,31cyclictest31-21ksoftirqd/111:31:331
1433499657,31cyclictest31-21ksoftirqd/111:31:331
1433499655,26cyclictest31-21ksoftirqd/110:30:531
14334996547,10cyclictest0-21swapper/110:02:521
14334996546,13cyclictest30236-21sh11:22:541
1433499652,34cyclictest31-21ksoftirqd/108:35:261
1433499652,31cyclictest31-21ksoftirqd/109:30:451
1433499652,25cyclictest31-21ksoftirqd/109:24:471
14326996534,15cyclictest0-21swapper/008:13:100
14334996449,10cyclictest14319-21cyclictest07:42:571
14334996447,11cyclictest24773-21sendmail_mailst10:13:071
1433499642,31cyclictest31-21ksoftirqd/107:24:551
1433499642,22cyclictest31-21ksoftirqd/108:44:511
1433499641,32cyclictest31-21ksoftirqd/108:47:521
1433499641,30cyclictest31-21ksoftirqd/108:08:371
1433499641,30cyclictest31-21ksoftirqd/108:08:371
1433499634,48cyclictest8397-21ssh10:42:481
14334996343,13cyclictest13829-21cpuspeed10:52:501
1433499634,24cyclictest31-21ksoftirqd/108:03:121
1433499634,24cyclictest31-21ksoftirqd/108:03:121
1433499632,27cyclictest31-21ksoftirqd/108:18:451
1433499623,26cyclictest31-21ksoftirqd/108:38:011
1433499622,28cyclictest31-21ksoftirqd/107:52:441
1433499622,27cyclictest31-21ksoftirqd/111:49:591
1433499601,22cyclictest31-21ksoftirqd/107:18:211
1296126024,23sleep30-21swapper/307:02:553
14350995036,8cyclictest451rcuc/308:10:113
14350995036,8cyclictest451rcuc/308:10:113
14350995022,20cyclictest4317-21chrt11:33:403
1435099462,32cyclictest24495-21taskset09:12:443
14350994526,12cyclictest8709-21chrt11:42:353
1435099452,26cyclictest28321-21chrt11:18:113
1435099452,26cyclictest28321-21chrt11:18:113
1435099444,30cyclictest17657-21taskset11:57:483
1435099444,30cyclictest17657-21taskset11:57:483
1435099443,28cyclictest6716-21taskset12:33:123
1435099443,27cyclictest22489-21taskset09:08:103
1435099443,27cyclictest22489-21taskset09:08:103
14350994430,9cyclictest323022sleep311:27:123
14350994429,9cyclictest451rcuc/309:05:433
1435099433,31cyclictest31117-21chrt07:57:483
1435099433,31cyclictest31117-21chrt07:57:483
1435099433,31cyclictest11905-21taskset11:47:513
1435099433,30cyclictest3638-21taskset08:12:543
1435099433,30cyclictest31397-21taskset10:25:273
1435099433,30cyclictest26442-21chrt10:17:333
1435099433,29cyclictest5102-21chrt10:36:403
1435099433,29cyclictest30718-21taskset12:18:523
1435099433,28cyclictest2672-21taskset09:32:473
1435099433,28cyclictest2672-21taskset09:32:473
1435099433,28cyclictest21028-21taskset10:07:323
1435099433,27cyclictest18159-21taskset08:57:373
1435099432,30cyclictest11310-21chrt08:37:323
299232420,6sleep30-21swapper/309:22:373
1435099423,31cyclictest30003-21taskset07:53:023
1435099423,30cyclictest24197-21taskset11:12:233
1435099423,30cyclictest17665-21chrt10:00:113
1435099423,29cyclictest8461-21taskset10:42:523
1435099423,29cyclictest418-21taskset12:23:023
1435099423,29cyclictest20125-21taskset11:03:153
1435099423,29cyclictest20125-21taskset11:03:153
1435099423,29cyclictest1951-21taskset09:30:483
1435099423,28cyclictest14618-21taskset11:52:373
14350994229,8cyclictest451rcuc/312:12:403
1435099422,30cyclictest7553-21taskset09:42:323
1435099422,30cyclictest16988-21taskset10:58:043
1435099414,27cyclictest22779-21taskset07:32:333
1435099414,27cyclictest22779-21taskset07:32:333
1435099414,27cyclictest10294-21chrt11:44:183
1435099413,30cyclictest16511-21chrt07:12:453
1435099413,29cyclictest464-21taskset11:27:523
1435099413,29cyclictest464-21taskset11:27:523
1435099413,29cyclictest28988-21taskset09:21:073
1435099413,29cyclictest23542-21taskset12:07:493
1435099413,29cyclictest23542-21taskset12:07:493
1435099413,29cyclictest23032-21chrt07:32:503
1435099413,29cyclictest1708-21taskset10:30:113
1435099413,29cyclictest12452-21taskset09:50:553
1435099413,29cyclictest12409-21chrt10:48:563
1435099413,29cyclictest12409-21chrt10:48:563
1435099413,28cyclictest7797-21taskset08:24:083
1435099413,28cyclictest6823-21taskset10:38:143
1435099413,28cyclictest2808-21taskset12:27:483
1435099413,28cyclictest26199-21taskset11:14:503
1435099413,26cyclictest26665-21chrt10:17:373
1435099412,31cyclictest13582-21taskset09:52:583
1435099412,31cyclictest13582-21taskset09:52:583
1435099412,30cyclictest8170-21chrt09:42:563
1435099412,30cyclictest8170-21chrt09:42:563
1435099412,29cyclictest20617-21chrt12:02:513
1435099412,29cyclictest18813-21taskset08:58:183
1435099403,28cyclictest6367-21taskset08:21:293
1435099403,28cyclictest29239-21chrt07:52:343
1435099403,28cyclictest13195-21taskset08:42:473
1435099403,27cyclictest8345-21taskset08:27:443
1435099403,27cyclictest1440-21taskset08:05:293
1435099403,27cyclictest1440-21taskset08:05:293
1435099402,29cyclictest22013-21taskset10:08:093
1435099402,29cyclictest18209-21taskset07:17:473
1435099402,29cyclictest11731-21chrt08:37:533
1435099394,26cyclictest14813-21taskset08:47:493
1435099393,27cyclictest19780-21taskset07:22:373
1435099393,25cyclictest26251-21chrt07:42:463
1435099383,26cyclictest24665-21chrt07:37:483
1435099383,25cyclictest14530-21chrt07:07:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional