You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 21:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sat Jan 10, 2026 12:44:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1774521930,11sleep017750-21fschecks_count07:04:520
190609911558,12cyclictest12983-21latency_hist08:29:291
2770721050,3sleep0241irq_work/010:57:590
190609910286,10cyclictest15979-21munin-plugin-st08:39:251
190609910286,10cyclictest15979-21munin-plugin-st08:39:251
19060999447,10cyclictest181rcu_preempt09:09:411
19054999356,8cyclictest181rcu_preempt11:00:170
19054999153,9cyclictest181rcu_preempt07:35:020
19054999153,9cyclictest181rcu_preempt07:35:020
19054999044,22cyclictest181rcu_preempt09:47:290
19054999044,22cyclictest181rcu_preempt09:47:290
19063998973,10cyclictest30854-21ssh10:09:422
19063998973,10cyclictest30854-21ssh10:09:422
19060998966,16cyclictest19044-21cyclictest08:49:471
19054998916,41cyclictest15-21ksoftirqd/007:19:490
19054998857,11cyclictest181rcu_preempt09:00:250
19060998753,11cyclictest181rcu_preempt07:39:361
19060998753,11cyclictest181rcu_preempt07:39:361
19054998757,21cyclictest23394-21modprobe11:44:250
19054998757,21cyclictest23394-21modprobe11:44:250
19054998752,8cyclictest181rcu_preempt09:50:260
1905499873,46cyclictest15-21ksoftirqd/010:54:040
183362877,56sleep218410-21timerwakeupswit07:05:112
19054998546,10cyclictest181rcu_preempt09:35:150
19054998546,10cyclictest181rcu_preempt09:35:150
19060998443,9cyclictest0-21swapper/110:29:421
19054998452,11cyclictest181rcu_preempt10:02:500
19054998452,11cyclictest181rcu_preempt10:02:500
19054998449,7cyclictest181rcu_preempt12:32:450
19063998361,16cyclictest30235-21sh11:00:532
19054998261,15cyclictest6299-21latency_hist09:29:290
19054998261,15cyclictest6299-21latency_hist09:29:290
19054998259,8cyclictest181rcu_preempt11:26:310
19054998228,10cyclictest0-21swapper/011:52:040
19054998228,10cyclictest0-21swapper/011:52:040
1906399816,46cyclictest39-21ksoftirqd/210:16:522
1906399816,46cyclictest39-21ksoftirqd/210:16:522
19063998162,12cyclictest23740-21ntp_offset09:00:062
19063998151,8cyclictest181rcu_preempt08:17:282
1906399814,39cyclictest39-21ksoftirqd/210:54:362
19060998136,8cyclictest181rcu_preempt07:30:021
1906099812,44cyclictest1350-21/usr/sbin/munin10:14:411
1906099812,44cyclictest1350-21/usr/sbin/munin10:14:411
19054998155,10cyclictest20356-21sendmail11:39:250
19063998050,7cyclictest181rcu_preempt12:17:252
19063998050,7cyclictest181rcu_preempt12:17:252
19054998051,9cyclictest7953-21ssh11:18:540
19054998042,7cyclictest181rcu_preempt12:15:180
19054998042,7cyclictest181rcu_preempt12:15:180
19054998041,10cyclictest181rcu_preempt07:49:420
1905499802,67cyclictest21563-21apt-get08:54:400
1905499802,67cyclictest21563-21apt-get08:54:400
19054998023,22cyclictest0-21swapper/010:41:280
19063997958,15cyclictest23177-21proc_pri10:49:582
19060997955,18cyclictest803-21sendmail-mta09:35:031
19060997955,18cyclictest803-21sendmail-mta09:35:031
19054997957,16cyclictest1103-21latency_hist10:14:280
19054997957,16cyclictest1103-21latency_hist10:14:280
19054997944,8cyclictest181rcu_preempt08:19:270
19054997942,8cyclictest181rcu_preempt08:24:380
1905499793,48cyclictest15-21ksoftirqd/008:39:420
19063997859,13cyclictest1040-21unixbench_singl07:50:022
19063997856,16cyclictest298-21in:imuxsock09:24:122
19063997855,16cyclictest20356-21sendmail11:39:252
1906399784,29cyclictest39-21ksoftirqd/209:09:592
19063997841,16cyclictest181rcu_preempt11:21:412
19063997821,35cyclictest39-21ksoftirqd/209:34:032
19060997855,15cyclictest0-21swapper/108:49:251
19060997852,20cyclictest352-21runrttasks08:14:411
19060997850,14cyclictest181rcu_preempt10:37:471
19060997850,14cyclictest181rcu_preempt10:37:471
19060997847,13cyclictest181rcu_preempt09:17:111
19060997847,13cyclictest181rcu_preempt09:17:111
19054997844,8cyclictest181rcu_preempt12:05:370
19054997843,8cyclictest181rcu_preempt10:25:240
19063997757,14cyclictest30085-21sh11:54:502
19060997747,11cyclictest181rcu_preempt11:44:391
19054997733,14cyclictest0-21swapper/008:08:380
19054997733,14cyclictest0-21swapper/008:08:380
19063997660,9cyclictest23501-21ssh09:57:122
1906399762,36cyclictest39-21ksoftirqd/208:58:522
1906399762,36cyclictest39-21ksoftirqd/208:58:522
19060997652,15cyclictest6177-21latency_hist12:09:291
19060997638,11cyclictest181rcu_preempt11:24:571
19060997637,8cyclictest181rcu_preempt11:34:521
1906099762,45cyclictest23452-21fschecks_time08:59:541
19054997660,10cyclictest2155-21ssh11:09:270
19054997660,10cyclictest2155-21ssh11:09:270
19054997658,12cyclictest300-21rs:main0
19054997658,12cyclictest300-21rs:main0
19054997657,13cyclictest13086-21idleruntime-cro10:34:250
19054997656,7cyclictest181rcu_preempt11:57:190
19054997640,9cyclictest181rcu_preempt09:23:340
19054997640,8cyclictest0-21swapper/009:05:010
19054997640,8cyclictest0-21swapper/009:05:010
19054997632,9cyclictest0-21swapper/011:20:190
19054997632,14cyclictest0-21swapper/008:30:590
19063997556,12cyclictest27728-21ssh10:04:412
19063997532,21cyclictest39-21ksoftirqd/212:06:192
19063997519,26cyclictest39-21ksoftirqd/209:44:202
19060997549,8cyclictest22847-21ssh10:49:481
19060997540,11cyclictest181rcu_preempt09:44:281
19060997533,13cyclictest181rcu_preempt09:06:091
19060997533,13cyclictest181rcu_preempt09:06:091
19054997556,14cyclictest24441-21latency_hist09:59:280
19054997548,8cyclictest29743-21ssh10:07:390
19054997540,6cyclictest181rcu_preempt10:48:420
19054997539,29cyclictest0-21swapper/009:14:270
19054997535,8cyclictest181rcu_preempt12:39:280
19054997522,30cyclictest15-21ksoftirqd/008:49:260
19054997518,34cyclictest15-21ksoftirqd/007:14:280
19054997513,33cyclictest15-21ksoftirqd/008:04:280
19063997455,13cyclictest26396-21diskmemload12:13:122
19063997455,13cyclictest26396-21diskmemload12:13:122
1906399744,34cyclictest39-21ksoftirqd/208:19:462
1906399742,30cyclictest39-21ksoftirqd/208:30:432
19060997453,14cyclictest17721-21apt-key08:44:271
19060997447,8cyclictest31-21ksoftirqd/110:57:491
19060997442,9cyclictest5162-21ssh11:14:261
19054997456,12cyclictest21358-21apt-key08:54:300
19054997450,10cyclictest181rcu_preempt10:21:540
19054997435,10cyclictest181rcu_preempt12:00:010
19054997425,26cyclictest0-21swapper/011:34:210
19063997356,9cyclictest0-21swapper/210:39:292
19063997356,9cyclictest0-21swapper/210:39:292
19063997350,16cyclictest8160-21latency_hist11:19:302
1906399734,43cyclictest2670-21fschecks_count11:09:472
19063997333,9cyclictest181rcu_preempt11:30:022
1906399732,32cyclictest39-21ksoftirqd/207:43:302
1906399732,32cyclictest39-21ksoftirqd/207:43:302
1906399732,31cyclictest39-21ksoftirqd/212:24:592
1906399732,31cyclictest39-21ksoftirqd/212:24:592
1906399732,25cyclictest39-21ksoftirqd/210:40:412
19060997345,11cyclictest9908-21cstates08:19:421
19060997327,22cyclictest18305-21ssh12:29:431
1906099732,40cyclictest831-21ntpq11:59:521
19054997352,16cyclictest8966-21sh12:14:270
19054997352,16cyclictest8966-21sh12:14:270
19054997332,17cyclictest181rcu_preempt10:15:040
19054997332,17cyclictest181rcu_preempt10:15:040
19063997256,12cyclictest19044-21cyclictest12:00:132
19063997254,11cyclictest352-21runrttasks11:49:552
19063997254,11cyclictest352-21runrttasks11:49:552
19063997252,14cyclictest17606-21latency_hist08:44:292
19063997241,8cyclictest181rcu_preempt09:38:282
19063997241,8cyclictest181rcu_preempt09:38:282
19063997231,20cyclictest39-21ksoftirqd/212:32:272
1906399722,29cyclictest39-21ksoftirqd/207:30:002
1906399721,35cyclictest39-21ksoftirqd/207:14:242
19060997258,10cyclictest19044-21cyclictest11:21:111
1906099724,24cyclictest352-21runrttasks08:06:121
1906099724,24cyclictest352-21runrttasks08:06:121
19060997240,12cyclictest4890-21ssh09:26:081
19060997240,12cyclictest4890-21ssh09:26:081
19060997235,8cyclictest28919-21ssh10:59:521
19060997230,14cyclictest181rcu_preempt12:09:481
19060997230,14cyclictest181rcu_preempt12:09:481
19054997243,7cyclictest181rcu_preempt07:32:340
1905499724,32cyclictest15-21ksoftirqd/007:24:360
19054997234,9cyclictest181rcu_preempt12:25:470
19054997234,9cyclictest181rcu_preempt12:25:470
19054997216,43cyclictest30385-21journal-offline07:40:370
19054997216,43cyclictest30385-21journal-offline07:40:370
1906399717,31cyclictest39-21ksoftirqd/209:05:122
1906399717,31cyclictest39-21ksoftirqd/209:05:122
19063997154,11cyclictest20895-21ssh11:39:452
19063997154,11cyclictest20895-21ssh11:39:452
19063997152,13cyclictest5015-21processes10:19:582
19063997150,15cyclictest32084-21fschecks_count11:04:482
19063997150,15cyclictest32084-21fschecks_count11:04:482
1906399712,28cyclictest39-21ksoftirqd/212:21:502
1906399712,27cyclictest39-21ksoftirqd/212:36:112
19063997118,35cyclictest39-21ksoftirqd/211:28:192
19063997118,12cyclictest381ktimers/210:31:032
19060997134,13cyclictest181rcu_preempt08:55:231
19060997134,13cyclictest181rcu_preempt08:55:231
1906099713,31cyclictest22195-21latency_hist10:49:281
19060997129,9cyclictest181rcu_preempt10:24:341
19060997125,12cyclictest0-21swapper/107:55:121
19060997110,26cyclictest31-21ksoftirqd/112:27:081
19060997110,26cyclictest31-21ksoftirqd/112:27:081
19054997142,7cyclictest181rcu_preempt09:17:210
19054997142,7cyclictest181rcu_preempt09:17:210
19054997139,7cyclictest181rcu_preempt11:09:370
19054997133,10cyclictest181rcu_preempt12:20:110
1906399704,28cyclictest39-21ksoftirqd/207:59:372
1906399703,31cyclictest24394-21awk11:44:572
19063997010,37cyclictest39-21ksoftirqd/207:26:032
19060997054,10cyclictest19044-21cyclictest08:14:281
19060997054,10cyclictest19044-21cyclictest08:14:281
19060997026,9cyclictest181rcu_preempt11:15:021
19060997010,29cyclictest16587-21kworker/1:0-events_power_efficient07:29:051
19054997054,11cyclictest10077-21fschecks_time08:19:480
19054997038,8cyclictest181rcu_preempt09:39:560
1905499703,30cyclictest15-21ksoftirqd/011:46:080
19054997013,31cyclictest15-21ksoftirqd/007:14:310
19063996953,10cyclictest20786-21sh09:53:022
1906399694,33cyclictest39-21ksoftirqd/208:54:102
19060996943,8cyclictest181rcu_preempt07:19:401
19060996943,17cyclictest9270-21sh12:14:421
19060996943,17cyclictest9270-21sh12:14:421
19060996939,13cyclictest16656-21ssh10:39:461
19060996935,14cyclictest2037-21ssh09:21:371
1906099693,33cyclictest15401-21systemctl11:29:591
19060996933,11cyclictest277-21dbus-daemon10:24:251
19060996930,12cyclictest181rcu_preempt08:02:561
19060996919,39cyclictest18601-21sh09:49:381
19054996946,15cyclictest7007-21ntp_offset09:29:550
19054996936,13cyclictest181rcu_preempt10:38:150
19054996936,13cyclictest181rcu_preempt10:38:150
19054996934,11cyclictest181rcu_preempt07:56:100
19054996922,9cyclictest181rcu_preempt07:44:480
19063996848,13cyclictest0-21swapper/208:29:292
1906399684,32cyclictest39-21ksoftirqd/207:44:522
19063996834,10cyclictest181rcu_preempt07:36:262
19063996834,10cyclictest181rcu_preempt07:36:262
1906399683,35cyclictest25000-21cut09:59:512
1906399683,35cyclictest25000-21cut09:59:512
1906399683,31cyclictest39-21ksoftirqd/208:44:462
1906399683,28cyclictest39-21ksoftirqd/210:44:522
19063996822,9cyclictest371rcuc/209:44:562
19063996822,9cyclictest371rcuc/209:44:562
1906399682,25cyclictest39-21ksoftirqd/208:09:572
1906399682,25cyclictest39-21ksoftirqd/208:09:572
1906399681,30cyclictest39-21ksoftirqd/207:54:442
19063996810,25cyclictest39-21ksoftirqd/209:19:082
19063996810,25cyclictest39-21ksoftirqd/209:19:082
19060996850,13cyclictest27723-21cpu10:04:411
19060996843,12cyclictest181rcu_preempt12:20:011
19060996842,18cyclictest19673-21needreboot07:09:521
1906099683,30cyclictest13591-21/usr/sbin/munin08:29:591
1906099683,25cyclictest31-21ksoftirqd/107:50:231
19060996832,21cyclictest24336-21sh09:59:251
19060996832,13cyclictest181rcu_preempt11:39:531
19060996832,13cyclictest181rcu_preempt11:39:531
19060996832,13cyclictest181rcu_preempt10:14:281
19060996832,13cyclictest181rcu_preempt10:14:281
19060996831,13cyclictest181rcu_preempt09:29:521
1906399675,29cyclictest39-21ksoftirqd/207:20:002
19063996746,14cyclictest7889-21netstat10:24:512
1906399672,28cyclictest39-21ksoftirqd/207:18:562
1906099677,23cyclictest0-21swapper/110:03:401
1906099677,23cyclictest0-21swapper/110:03:401
19060996748,14cyclictest13173-21kworker/u17:1-ext4-rsv-conversion07:09:301
19060996746,15cyclictest19044-21cyclictest07:14:331
19060996728,8cyclictest181rcu_preempt11:04:551
19060996728,8cyclictest181rcu_preempt11:04:551
1906399663,29cyclictest39-21ksoftirqd/209:24:442
1906399663,29cyclictest39-21ksoftirqd/209:24:442
1906399662,24cyclictest39-21ksoftirqd/208:39:052
1906399662,24cyclictest39-21ksoftirqd/208:39:052
19060996644,9cyclictest181rcu_preempt12:34:291
1906099663,29cyclictest15733-21expr09:44:431
1906099663,29cyclictest15733-21expr09:44:431
1906099662,21cyclictest29990-21fschecks_time11:54:471
1906399653,47cyclictest6302-21latency_hist08:09:282
1906399653,47cyclictest6302-21latency_hist08:09:282
19060996537,8cyclictest26396-21diskmemload11:54:211
19060996537,8cyclictest26396-21diskmemload11:54:211
1906099652,36cyclictest30957-21apt-get07:44:371
19060996515,40cyclictest29129-21cat07:39:281
19060996515,40cyclictest29129-21cat07:39:281
25272640,7sleep30-21swapper/307:55:003
19054996327,9cyclictest181rcu_preempt08:11:070
19054996327,9cyclictest181rcu_preempt08:11:070
184112536,28sleep30-21swapper/307:05:123
1907199525,34cyclictest23522-21chrt11:44:283
1907199525,34cyclictest23522-21chrt11:44:283
1907199514,37cyclictest30376-21taskset09:14:563
1907199514,37cyclictest30376-21taskset09:14:563
1907199514,37cyclictest16109-21chrt10:39:253
1907199514,37cyclictest16109-21chrt10:39:253
1907199513,38cyclictest29389-21chrt11:54:253
1907199513,38cyclictest29389-21chrt11:54:253
1907199513,35cyclictest6888-21taskset09:29:523
19071995111,30cyclictest24699-21taskset10:53:033
19071995036,9cyclictest451rcuc/308:44:383
1907199503,36cyclictest6654-21taskset12:09:503
1907199503,36cyclictest6654-21taskset12:09:503
1907199503,36cyclictest13590-21chrt08:29:593
1907199503,34cyclictest21546-21chrt09:54:293
1907199502,38cyclictest17030-21chrt12:26:533
1907199502,38cyclictest17030-21chrt12:26:533
1907199502,35cyclictest6808-21chrt08:09:463
1907199502,35cyclictest6808-21chrt08:09:463
1907199494,35cyclictest3581-21taskset09:24:453
1907199494,35cyclictest3581-21taskset09:24:453
1907199494,35cyclictest20988-21chrt12:34:273
1907199494,33cyclictest28423-21chrt10:59:283
1907199493,36cyclictest3124-21chrt11:09:583
1907199493,36cyclictest29911-21chrt11:54:463
1907199493,35cyclictest31718-21chrt10:10:023
1907199493,35cyclictest31718-21chrt10:10:023
1907199492,37cyclictest8669-21chrt10:25:353
296962483,32sleep30-21swapper/307:39:503
296962483,32sleep30-21swapper/307:39:503
1907199484,33cyclictest21275-21chrt12:34:393
1907199484,33cyclictest14985-21taskset12:24:253
1907199483,36cyclictest26770-21chrt09:09:413
1907199483,35cyclictest8795-21taskset11:19:553
1907199483,35cyclictest16990-21taskset10:39:543
1907199483,34cyclictest1516-21chrt09:20:113
1907199482,37cyclictest22059-21taskset09:54:493
1907199482,35cyclictest6560-21chrt11:15:053
1907199474,34cyclictest5431-21chrt08:04:523
1907199474,34cyclictest5431-21chrt08:04:523
1907199473,34cyclictest31962-21chrt11:04:433
1907199473,34cyclictest31962-21chrt11:04:433
1907199473,34cyclictest21522-21chrt10:47:363
1907199473,34cyclictest17611-21chrt08:44:253
1907199472,35cyclictest6461-21chrt10:22:513
1907199472,35cyclictest25399-21chrt11:46:433
1907199464,32cyclictest15233-21chrt08:34:573
1907199464,32cyclictest15233-21chrt08:34:573
1907199463,34cyclictest23980-21chrt09:00:133
1907199463,34cyclictest17326-21taskset09:46:513
1907199463,34cyclictest17326-21taskset09:46:513
1907199463,33cyclictest9907-21chrt08:19:423
1907199463,33cyclictest11861-21chrt09:38:403
1907199463,33cyclictest11861-21chrt09:38:403
1907199462,35cyclictest2335-21chrt12:02:443
1907199462,34cyclictest12451-21taskset09:39:333
1907199462,30cyclictest21578-21chrt08:54:423
1907199462,30cyclictest21578-21chrt08:54:423
1907199455,30cyclictest29186-21chrt07:39:283
1907199455,30cyclictest29186-21chrt07:39:283
1907199454,31cyclictest3074-21taskset10:16:553
1907199454,31cyclictest3074-21taskset10:16:553
1907199453,32cyclictest21526-21chrt07:14:503
1907199453,31cyclictest16447-21chrt11:31:553
1907199452,33cyclictest29713-21chrt10:07:343
1907199452,33cyclictest29330-21chrt11:00:133
1907199452,30cyclictest26596-21chrt09:09:293
1907199452,30cyclictest26596-21chrt09:09:293
1907199443,31cyclictest4029-21chrt08:00:023
1907199443,31cyclictest10382-21taskset12:15:093
1907199443,31cyclictest10382-21taskset12:15:093
1907199443,30cyclictest12769-21chrt11:25:023
1907199442,32cyclictest21064-21chrt08:54:253
1907199432,31cyclictest11605-21chrt10:30:213
1907199423,28cyclictest31333-21taskset07:44:513
1907199423,28cyclictest25609-21chrt10:00:053
1907199423,28cyclictest25609-21chrt10:00:053
19071994225,10cyclictest451rcuc/307:10:003
1907199422,31cyclictest4728-21chrt12:06:053
1907199422,27cyclictest17662-21taskset11:34:353
19071994210,21cyclictest11612-21chrt08:24:483
1907199413,27cyclictest24266-21chrt07:24:263
1907199412,29cyclictest27096-21taskset07:30:053
1907199403,27cyclictest8179-21taskset08:14:403
1907199393,26cyclictest24514-21chrt07:24:383
1907199392,27cyclictest32564-21taskset07:49:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional