You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 20:28
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sun Mar 22, 2026 12:44:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12084218213,44sleep212105-21systemd-run07:04:462
10731215313,107sleep10-21swapper/107:00:211
12300991142,35cyclictest23191-21runrttasks11:53:100
123009911315,27cyclictest352-21runrttasks10:47:540
123009911235,66cyclictest0-21swapper/011:09:470
123009911235,66cyclictest0-21swapper/011:09:470
11489211131,50sleep30-21swapper/307:00:563
12300991084,32cyclictest31012-21timerwakeupswit12:05:200
12300991084,32cyclictest31012-21timerwakeupswit12:05:200
123009910430,67cyclictest0-21swapper/009:12:020
12300991042,41cyclictest24108-21ssh11:54:570
12300991042,27cyclictest22585-21latency_hist09:09:460
123009910238,58cyclictest352-21runrttasks11:15:250
123009910238,58cyclictest352-21runrttasks11:15:250
12300991023,35cyclictest16194-21grep11:40:210
12300991023,29cyclictest32260-21sed11:14:450
123009910135,57cyclictest0-21swapper/010:54:470
123009910135,57cyclictest0-21swapper/010:54:470
12300991013,25cyclictest15148-21munin-plugin-st12:34:420
123009910131,63cyclictest0-21swapper/009:39:460
123529910059,9cyclictest181rcu_preempt08:05:052
123529910059,9cyclictest181rcu_preempt08:05:052
12300991004,30cyclictest11907-21fschecks_count10:40:080
123009910042,51cyclictest0-21swapper/009:34:460
123009910042,51cyclictest0-21swapper/009:34:460
123009910041,52cyclictest0-21swapper/007:20:080
12300991003,35cyclictest7539-21grep08:25:160
12300991002,30cyclictest352-21runrttasks10:08:490
1230099993,31cyclictest1-21systemd09:59:440
12300999923,7cyclictest0-21swapper/011:59:580
1230099984,32cyclictest6461-21ssh10:30:170
1230099984,32cyclictest6461-21ssh10:30:170
12300999842,50cyclictest13669-21ssh11:36:270
12300999842,50cyclictest13669-21ssh11:36:270
1230099983,34cyclictest28737-21/usr/sbin/munin09:20:030
1230099983,34cyclictest12336-21proc_pri09:45:580
1230099983,32cyclictest3884-21fschecks_time12:15:050
1230099983,32cyclictest3884-21fschecks_time12:15:050
1230099982,32cyclictest21041-21latency_hist07:29:430
12300999737,54cyclictest0-21swapper/011:30:200
12300999736,54cyclictest22726-21ssh10:04:000
12300999736,54cyclictest22726-21ssh10:04:000
1230099973,32cyclictest3413-21proc_pri10:25:160
1230099973,31cyclictest352-21runrttasks08:57:340
1230099973,30cyclictest31230-21/usr/sbin/munin08:00:120
1230099973,30cyclictest31230-21/usr/sbin/munin08:00:120
1230099973,28cyclictest8179-21/usr/sbin/munin09:40:110
1230099973,28cyclictest8179-21/usr/sbin/munin09:40:110
12300999731,57cyclictest0-21swapper/011:24:440
12300999722,10cyclictest0-21swapper/009:15:170
12300999718,30cyclictest12069-21uname08:40:100
1230099964,32cyclictest18032-21latency_hist07:19:470
1230099964,31cyclictest27027-21mailstats10:10:170
1230099964,31cyclictest27027-21mailstats10:10:170
1230099964,29cyclictest877-21fschecks_time12:10:060
1230099963,33cyclictest9205-21latency_hist12:24:450
1230099963,32cyclictest10354-21fschecks_time08:35:050
1230099963,31cyclictest16503-21ssh09:53:140
1230099963,28cyclictest352-21runrttasks09:26:410
1230099963,24cyclictest2555-21cat10:24:460
1230099963,24cyclictest2555-21cat10:24:460
1230099962,35cyclictest14097-21switchtime07:09:530
1230099962,33cyclictest22637-21modprobe07:34:430
1230099962,31cyclictest15941-21cron07:14:430
1230099962,31cyclictest15941-21cron07:14:430
1230099954,35cyclictest24327-21mailstats11:00:180
12300999539,48cyclictest0-21swapper/009:04:440
1230099953,36cyclictest3499-21latency_hist08:14:460
1230099953,33cyclictest24368-21apt-key07:39:450
12300999532,56cyclictest12459-21latency_hist07:04:490
1230099953,24cyclictest6189-21/usr/sbin/munin11:25:110
1230099953,24cyclictest6189-21/usr/sbin/munin11:25:110
1230099952,33cyclictest662-21fschecks_time08:05:050
1230099952,33cyclictest662-21fschecks_time08:05:050
1230099952,24cyclictest13569-21tune2fs08:45:070
12300999521,27cyclictest277-21dbus-daemon07:49:450
1230099944,32cyclictest29244-21/usr/sbin/munin10:15:070
1230099943,32cyclictest19164-21unixbench_singl11:45:210
1230099943,27cyclictest8680-21/usr/sbin/munin10:35:190
1230099943,26cyclictest30967-21cron07:59:430
1230099943,26cyclictest30967-21cron07:59:430
1230099943,25cyclictest15174-21cut08:50:050
12300999414,23cyclictest11765-21sh12:28:370
12352999349,26cyclictest181rcu_preempt08:34:452
12352999349,26cyclictest181rcu_preempt08:34:452
1230099933,31cyclictest5239-21/usr/sbin/munin08:20:180
1230099933,28cyclictest20569-21/usr/sbin/munin10:55:160
1230099932,35cyclictest28260-21fschecks_time07:50:070
1230099923,31cyclictest25063-21fschecks_count07:41:050
1230099914,29cyclictest9978-21seq08:34:460
1230099914,29cyclictest9978-21seq08:34:460
1230099912,31cyclictest3634-21/usr/sbin/munin08:15:040
12352999045,7cyclictest181rcu_preempt09:55:192
1232899893,64cyclictest11408-21apt-get09:45:111
1235299884,66cyclictest352-21runrttasks11:20:232
12352998840,17cyclictest181rcu_preempt12:29:552
12352998825,14cyclictest0-21swapper/209:24:502
12328998871,11cyclictest12269-21cyclictest07:20:071
12352998651,9cyclictest181rcu_preempt11:40:312
12352998448,8cyclictest181rcu_preempt10:00:232
12352998448,8cyclictest181rcu_preempt10:00:232
12352998428,13cyclictest0-21swapper/211:54:462
12352998355,13cyclictest181rcu_preempt10:33:572
12352998355,13cyclictest181rcu_preempt10:33:572
12352998352,7cyclictest181rcu_preempt07:49:462
12352998349,8cyclictest181rcu_preempt10:50:192
12352998349,8cyclictest181rcu_preempt10:50:192
12352998346,11cyclictest181rcu_preempt08:39:462
12352998330,13cyclictest0-21swapper/209:09:212
1235299832,46cyclictest39-21ksoftirqd/211:00:052
12352998267,10cyclictest21807-21kworker/u17:2-rpciod11:10:112
12352998237,16cyclictest181rcu_preempt10:15:142
12352998233,8cyclictest0-21swapper/209:31:472
12352998233,8cyclictest0-21swapper/209:31:472
12352998230,8cyclictest181rcu_preempt07:35:182
12328998266,10cyclictest200-21systemd-journal12:19:431
12328998266,10cyclictest200-21systemd-journal12:19:431
12352998166,10cyclictest19397-21kworker/u17:0-xprtiod10:35:112
12352998157,8cyclictest181rcu_preempt08:20:002
12352998141,17cyclictest181rcu_preempt10:39:572
12352998127,13cyclictest181rcu_preempt09:38:492
12328998164,11cyclictest25135-21unixbench_multi11:55:211
1232899813,55cyclictest12589-21apt-get07:04:591
12352998055,16cyclictest4892-21ssh12:15:542
12352998055,16cyclictest4892-21ssh12:15:542
12352998052,18cyclictest19397-21kworker/u17:0-xprtiod10:14:442
12352998052,18cyclictest19397-21kworker/u17:0-xprtiod10:14:442
12352998046,9cyclictest181rcu_preempt07:55:182
12352998046,9cyclictest181rcu_preempt07:55:182
12352998044,8cyclictest181rcu_preempt10:45:072
12352998041,10cyclictest181rcu_preempt11:05:132
12352998041,10cyclictest181rcu_preempt11:05:132
1235299803,46cyclictest12034-21apt-get07:04:592
12352998031,14cyclictest0-21swapper/209:02:122
12352998031,11cyclictest0-21swapper/209:48:462
12352998031,10cyclictest0-21swapper/211:28:472
12352998031,10cyclictest0-21swapper/211:28:472
12352997957,15cyclictest25085-21fschecks_time07:41:052
12352997953,9cyclictest181rcu_preempt11:31:492
12352997951,8cyclictest181rcu_preempt12:14:262
12352997950,12cyclictest181rcu_preempt12:05:512
12352997950,12cyclictest181rcu_preempt12:05:512
12352997945,10cyclictest181rcu_preempt08:00:022
12352997945,10cyclictest181rcu_preempt08:00:022
12352997943,12cyclictest181rcu_preempt07:25:192
12352997939,24cyclictest19660-21ssh11:46:362
12352997935,11cyclictest181rcu_preempt09:10:032
12352997930,14cyclictest181rcu_preempt10:26:592
12328997961,13cyclictest15363-21fschecks_count11:40:041
12328997961,11cyclictest9592-21cpu12:24:591
12328997926,11cyclictest291rcuc/112:30:131
1235899782,35cyclictest15169-21taskset07:10:213
1235899782,35cyclictest15169-21taskset07:10:213
12352997855,8cyclictest181rcu_preempt10:22:202
12352997855,8cyclictest181rcu_preempt10:22:202
12352997852,17cyclictest8501-21sh09:40:062
12352997852,17cyclictest8501-21sh09:40:062
12352997849,13cyclictest181rcu_preempt12:20:032
12352997848,13cyclictest181rcu_preempt07:49:472
12352997838,8cyclictest181rcu_preempt08:29:462
12352997834,10cyclictest12649-21latency11:35:092
12352997834,10cyclictest12649-21latency11:35:092
12328997857,13cyclictest19876-21cpu09:04:591
12352997756,16cyclictest12269-21cyclictest08:15:192
12352997747,8cyclictest181rcu_preempt10:56:092
12352997747,8cyclictest181rcu_preempt10:05:122
12352997746,8cyclictest181rcu_preempt09:15:452
12352997743,12cyclictest181rcu_preempt09:24:462
12352997741,13cyclictest181rcu_preempt08:54:462
12352997735,13cyclictest181rcu_preempt08:45:022
12328997752,18cyclictest1-21systemd11:11:491
1235899762,28cyclictest2917-21chrt10:25:013
12358997618,43cyclictest0-21swapper/311:20:103
12352997650,19cyclictest19477-21latency_hist07:24:452
12352997640,13cyclictest181rcu_preempt09:50:062
12352997634,10cyclictest181rcu_preempt07:15:152
12352997541,8cyclictest181rcu_preempt08:55:192
12352997540,13cyclictest181rcu_preempt12:00:192
12352997540,10cyclictest181rcu_preempt07:10:232
12352997540,10cyclictest181rcu_preempt07:10:232
12352997535,17cyclictest181rcu_preempt11:55:122
12352997534,14cyclictest181rcu_preempt12:25:492
1235299752,39cyclictest39-21ksoftirqd/208:40:102
12328997556,13cyclictest3786-21sh11:20:441
12328997546,21cyclictest16295-21apt-key07:14:481
324652740,9sleep1301ktimers/110:20:041
324652740,9sleep1301ktimers/110:20:041
1235899742,36cyclictest12698-21chrt11:35:103
1235899742,36cyclictest12698-21chrt11:35:103
12352997451,13cyclictest0-21swapper/208:14:432
1235299742,39cyclictest39-21ksoftirqd/207:30:192
12328997418,38cyclictest15820-21rm10:45:371
74692730,23sleep17427-21ssh12:20:201
1235899734,27cyclictest25464-21chrt11:55:583
12352997336,12cyclictest181rcu_preempt11:14:472
12352997336,12cyclictest181rcu_preempt11:14:472
12328997351,15cyclictest29289-21netstat09:20:091
12328997341,21cyclictest0-21swapper/109:40:201
12328997341,21cyclictest0-21swapper/109:40:201
12328997312,24cyclictest0-21swapper/111:45:071
12358997218,46cyclictest0-21swapper/307:09:453
1235899714,30cyclictest29870-21taskset12:04:463
1235899713,26cyclictest20209-21taskset09:05:073
12328997152,13cyclictest12476-21fschecks_time11:35:051
12328997152,13cyclictest12476-21fschecks_time11:35:051
12328997150,15cyclictest1-21systemd11:51:491
12328997147,16cyclictest12269-21cyclictest09:53:371
1232899714,45cyclictest352-21runrttasks08:36:231
12328997127,22cyclictest31-21ksoftirqd/108:47:431
1235899703,30cyclictest27886-21chrt11:06:033
1235899703,30cyclictest27886-21chrt11:06:033
1235899703,25cyclictest32226-21taskset10:19:543
1235899703,25cyclictest32226-21taskset10:19:543
1235899702,29cyclictest8240-21chrt12:22:003
12328997054,10cyclictest17863-21systemd-run07:19:441
12328997053,10cyclictest19515-21diskmemload12:05:031
12328997053,10cyclictest19515-21diskmemload12:05:031
12328997034,13cyclictest181rcu_preempt08:30:221
12328997034,13cyclictest181rcu_preempt08:30:221
12328997026,8cyclictest301ktimers/111:33:261
12328997021,11cyclictest301ktimers/109:35:061
12328996956,7cyclictest0-21swapper/109:34:431
12328996956,7cyclictest0-21swapper/109:34:431
1232899695,32cyclictest31-21ksoftirqd/111:26:131
1232899695,32cyclictest31-21ksoftirqd/111:26:131
12328996939,21cyclictest9509-21sh10:35:191
1235899684,29cyclictest4633-21chrt08:15:213
12328996854,10cyclictest12269-21cyclictest08:00:181
12328996854,10cyclictest12269-21cyclictest08:00:181
12328996854,10cyclictest12269-21cyclictest07:45:061
12328996848,12cyclictest12269-21cyclictest10:14:521
12328996845,17cyclictest29867-21latency_hist12:04:461
1232899684,39cyclictest17843-21fschecks_count10:50:041
1232899684,39cyclictest17843-21fschecks_count10:50:041
12328996840,19cyclictest12269-21cyclictest10:34:471
12328996840,19cyclictest12269-21cyclictest10:34:471
1232899683,30cyclictest19970-21fschecks_time07:25:051
1232899682,51cyclictest24278-21sh07:39:441
12328996824,9cyclictest301ktimers/110:59:451
1232899682,46cyclictest1174-21ntp_kernel_pll_12:10:131
1232899682,45cyclictest14425-21timerandwakeup07:09:581
12328996822,8cyclictest301ktimers/109:10:061
12328996746,15cyclictest5500-21latency_hist10:29:461
12328996745,15cyclictest2120-21latency_hist09:29:461
1232899673,24cyclictest31-21ksoftirqd/109:15:471
1232899672,39cyclictest31-21ksoftirqd/111:01:021
1235899664,24cyclictest21696-21taskset10:55:213
12328996652,10cyclictest12269-21cyclictest08:20:161
1232899663,45cyclictest31069-21latency_hist07:59:451
1232899663,45cyclictest31069-21latency_hist07:59:451
12328996610,38cyclictest20045-21/usr/sbin/munin09:59:451
1232899655,47cyclictest32511-21ssh11:14:551
1232899655,47cyclictest32511-21ssh11:14:551
12328996450,9cyclictest18371-21fschecks_count09:00:081
12328996446,12cyclictest27983-21/usr/sbin/munin07:50:181
12328996443,12cyclictest0-21swapper/108:10:061
1232899644,25cyclictest31-21ksoftirqd/110:40:541
1232899643,51cyclictest24787-21apt-get07:39:581
1232899643,45cyclictest5030-21latency_hist08:19:461
12328996425,7cyclictest301ktimers/108:05:041
12328996425,7cyclictest301ktimers/108:05:041
12328996342,13cyclictest0-21swapper/110:00:071
12328996342,13cyclictest0-21swapper/110:00:071
1232899633,45cyclictest26323-21cpu10:09:591
1232899633,45cyclictest26323-21cpu10:09:591
1232899632,43cyclictest24155-21switchtime10:05:191
1235899623,41cyclictest6472-21chrt11:25:053
1235899623,41cyclictest6472-21chrt11:25:053
12328996244,11cyclictest0-21swapper/111:05:061
12328996244,11cyclictest0-21swapper/111:05:061
12328996219,27cyclictest31-21ksoftirqd/108:56:081
12328996217,15cyclictest15222-21cat08:50:071
12358996112,17cyclictest0-21swapper/308:14:433
1232899613,36cyclictest352-21runrttasks07:29:501
1235899605,37cyclictest0-21swapper/307:40:233
12328996038,12cyclictest0-21swapper/108:24:541
12358995911,13cyclictest0-21swapper/310:10:113
12358995911,13cyclictest0-21swapper/310:10:113
1235899584,37cyclictest6626-21taskset08:24:453
1235899584,35cyclictest18005-21taskset09:55:133
1235899582,44cyclictest15580-21taskset10:45:203
1232899582,26cyclictest31-21ksoftirqd/108:42:051
1235899573,39cyclictest25954-21taskset09:15:003
1235899573,37cyclictest4988-21taskset09:34:463
1235899573,37cyclictest4988-21taskset09:34:463
1235899573,37cyclictest29915-21chrt10:15:163
1235899563,39cyclictest495-21taskset08:05:003
1235899563,39cyclictest495-21taskset08:05:003
1235899563,35cyclictest4180-21chrt12:15:123
1235899563,35cyclictest4180-21chrt12:15:123
1235899563,35cyclictest29813-21taskset09:20:193
1235899554,33cyclictest16464-21chrt07:14:503
1235899553,33cyclictest17955-21taskset10:50:063
1235899553,33cyclictest17955-21taskset10:50:063
1235899543,38cyclictest32181-21taskset11:14:433
1235899543,33cyclictest18448-21taskset09:00:083
1235899543,33cyclictest16353-21chrt08:54:463
1235899543,33cyclictest15046-21chrt09:50:153
1235899542,35cyclictest21593-21chrt07:30:053
1235899533,37cyclictest32022-21chrt09:25:053
1235899533,35cyclictest23776-21chrt07:35:203
1235899533,35cyclictest18489-21chrt11:45:083
1235899533,35cyclictest17736-21chrt11:44:353
1235899533,33cyclictest11592-21taskset10:39:483
1235899532,36cyclictest7631-21chrt09:39:033
1235899524,35cyclictest10083-21chrt08:34:533
1235899524,31cyclictest702-21chrt12:10:013
1235899523,34cyclictest15237-21chrt12:34:453
1235899522,35cyclictest10267-21chrt12:25:163
1235899514,31cyclictest11469-21chrt09:45:043
1235899513,34cyclictest29335-21chrt07:54:443
1235899513,33cyclictest21322-21chrt10:00:203
1235899513,33cyclictest21322-21chrt10:00:203
1235899513,32cyclictest8184-21chrt08:29:433
1235899513,31cyclictest6278-21chrt10:30:123
1235899513,31cyclictest6278-21chrt10:30:123
1235899513,30cyclictest9197-21chrt10:35:103
1235899512,36cyclictest31187-21chrt12:05:233
1235899512,36cyclictest31187-21chrt12:05:233
1235899503,34cyclictest11445-21chrt11:33:403
1235899503,34cyclictest11149-21chrt09:44:473
1235899503,34cyclictest11149-21chrt09:44:473
1235899502,31cyclictest23494-21chrt10:05:023
1235899494,30cyclictest26871-21taskset07:45:123
1235899493,35cyclictest23058-21chrt09:10:053
1235899492,33cyclictest18652-21chrt07:20:173
1235899492,31cyclictest1076-21chrt11:15:203
1235899492,31cyclictest1076-21chrt11:15:203
1235899483,33cyclictest16559-21chrt08:54:543
1235899474,33cyclictest26372-21chrt11:04:463
1235899472,33cyclictest23688-21chrt11:54:363
1235899463,31cyclictest31478-21chrt08:00:063
1235899463,31cyclictest31478-21chrt08:00:063
1235899462,31cyclictest29980-21taskset07:55:083
1235899462,31cyclictest29980-21taskset07:55:083
1235899453,30cyclictest13667-21chrt08:45:103
12358994518,18cyclictest0-21swapper/307:29:433
1235899433,26cyclictest11716-21taskset08:39:593
1235899423,27cyclictest8477-21chrt08:29:593
1235899423,27cyclictest8477-21chrt08:29:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional