You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 06:38
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Mar 13, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2500321298,80sleep025053-21cpuspeed19:00:400
264659911838,12cyclictest0-21swapper/020:10:580
26465991143,30cyclictest900-21diskmemload22:25:230
26465991143,30cyclictest900-21diskmemload22:25:230
26465991139,72cyclictest21255-21apt-get23:25:370
264659911328,19cyclictest0-21swapper/023:30:250
264659911328,19cyclictest0-21swapper/023:30:250
264659911136,11cyclictest0-21swapper/000:12:250
264839910854,12cyclictest181rcu_preempt22:00:252
26465991083,33cyclictest16986-21/usr/sbin/munin22:25:540
26465991082,41cyclictest4441-21fschecks_count21:10:420
26465991082,41cyclictest4441-21fschecks_count21:10:420
26465991073,45cyclictest10850-21grep00:00:590
264659910633,11cyclictest0-21swapper/021:20:470
264659910555,44cyclictest900-21diskmemload22:00:250
26465991054,27cyclictest352-21runrttasks23:37:180
264659910537,13cyclictest6471-21kworker/u17:0+xprtiod23:59:110
26465991053,41cyclictest14446-21fschecks_count20:05:410
264659910533,11cyclictest0-21swapper/022:01:340
26465991052,36cyclictest32699-21grep19:20:580
264659910435,12cyclictest0-21swapper/019:50:210
264659910429,20cyclictest0-21swapper/021:15:400
26465991042,26cyclictest29055-21date22:45:240
26465991024,36cyclictest28033-21latency_hist19:10:230
26465991024,36cyclictest28033-21latency_hist19:10:230
26465991023,40cyclictest24933-21cstates00:25:380
26465991022,36cyclictest13626-21switchtime00:05:570
264659910137,8cyclictest0-21swapper/020:35:250
26363210126,44sleep20-21swapper/219:05:232
2646599993,44cyclictest31280-21unixbench_multi20:55:580
2646599993,44cyclictest31280-21unixbench_multi20:55:580
2646599993,40cyclictest25922-21latency_hist22:40:240
2646599963,43cyclictest19667-21wc20:20:550
26465999530,18cyclictest26456-21cyclictest19:30:250
26477999476,12cyclictest1-21systemd22:15:221
26483999161,9cyclictest29094-21ssh23:37:582
2554728912,51sleep125574-21/usr/sbin/munin19:01:021
2647799882,49cyclictest31-21ksoftirqd/121:08:341
2646599886,75cyclictest27696-21kworker/u18:1+ext4-rsv-conversion22:55:360
2646599886,75cyclictest27696-21kworker/u18:1+ext4-rsv-conversion22:55:360
26477998764,17cyclictest544-21cron23:00:211
26477998764,17cyclictest544-21cron23:00:211
26477998711,46cyclictest31-21ksoftirqd/123:41:511
26477998711,46cyclictest31-21ksoftirqd/123:41:511
26483998645,10cyclictest181rcu_preempt20:21:542
26465998660,18cyclictest27007-21latency_hist20:45:240
254782860,15sleep3451rcuc/320:40:253
26483998548,9cyclictest181rcu_preempt19:40:382
2647799855,47cyclictest31-21ksoftirqd/122:44:061
26465998530,28cyclictest181rcu_preempt23:50:500
26483998445,9cyclictest181rcu_preempt19:30:432
26483998445,9cyclictest181rcu_preempt19:30:432
26477998453,10cyclictest6005-21/usr/sbin/munin23:00:541
26477998422,14cyclictest0-21swapper/121:30:351
26483998363,14cyclictest2605-21apt-key22:55:242
26483998363,14cyclictest2605-21apt-key22:55:242
26483998355,20cyclictest4110-21fschecks_time23:50:442
2648399832,61cyclictest2021-21apt-get22:00:432
26477998365,12cyclictest200-21systemd-journal00:01:421
26477998356,8cyclictest181rcu_preempt22:36:331
26477998340,12cyclictest181rcu_preempt21:10:301
26477998340,12cyclictest181rcu_preempt21:10:301
26477998215,34cyclictest31-21ksoftirqd/123:56:531
26465998261,14cyclictest28650-21latency_hist20:50:240
26465998261,14cyclictest26456-21cyclictest22:51:420
26465998261,14cyclictest26456-21cyclictest22:51:420
26465998250,11cyclictest181rcu_preempt21:50:330
2515728214,39sleep30-21swapper/319:00:483
2647799813,39cyclictest31-21ksoftirqd/122:50:101
26477998111,37cyclictest31-21ksoftirqd/119:10:141
26477998111,37cyclictest31-21ksoftirqd/119:10:141
2646599813,60cyclictest9333-21grepecks_count19:50:440
26483998061,13cyclictest26946-21latency_hist23:35:252
26483998061,13cyclictest26946-21latency_hist23:35:252
26483998047,9cyclictest25323-21ssh22:38:422
26483998041,12cyclictest181rcu_preempt23:12:532
26483998035,13cyclictest0-21swapper/221:52:392
26477998062,10cyclictest3461-21ssh23:50:201
2647799805,35cyclictest31-21ksoftirqd/121:58:591
26477998015,43cyclictest31-21ksoftirqd/120:54:021
26483997942,8cyclictest30244-21/usr/sbin/munin23:40:582
26483997942,8cyclictest30244-21/usr/sbin/munin23:40:582
2647799799,50cyclictest31-21ksoftirqd/123:30:541
2647799799,50cyclictest31-21ksoftirqd/123:30:541
26477997951,10cyclictest181rcu_preempt20:05:531
26477997912,43cyclictest31-21ksoftirqd/119:22:461
26465997958,13cyclictest15-21ksoftirqd/021:35:520
206572790,22sleep2381ktimers/200:18:022
26483997864,8cyclictest0-21swapper/221:15:342
26483997858,13cyclictest5787-21latency_hist19:40:242
26483997858,13cyclictest5787-21latency_hist19:40:242
26483997846,11cyclictest181rcu_preempt20:27:272
26483997846,11cyclictest181rcu_preempt20:27:272
26483997831,26cyclictest26995-21mailstats22:40:552
26477997862,10cyclictest2749-21apt-key22:55:241
26477997862,10cyclictest2749-21apt-key22:55:241
2647799782,44cyclictest31-21ksoftirqd/119:15:251
26483997748,8cyclictest181rcu_preempt20:30:452
26483997745,8cyclictest181rcu_preempt00:29:432
26483997743,11cyclictest181rcu_preempt20:50:252
26477997760,11cyclictest15032-21/usr/sbin/munin23:15:321
26477997759,13cyclictest4895-21latency_hist22:05:241
26477997759,12cyclictest14609-21ntp_offset22:20:511
26477997759,12cyclictest14609-21ntp_offset22:20:511
26465997741,20cyclictest245802latency_hist00:25:240
2646599773,32cyclictest13815-21ntpq21:25:500
26465997729,32cyclictest27887-21ssh21:50:220
26483997649,8cyclictest181rcu_preempt19:45:562
26477997654,15cyclictest20714-21latency_hist23:25:241
26477997649,19cyclictest30778-21cat20:55:471
26477997649,19cyclictest30778-21cat20:55:471
2647799763,41cyclictest31-21ksoftirqd/123:50:381
26483997552,8cyclictest181rcu_preempt23:50:242
26483997546,20cyclictest4959-21sh22:05:252
26483997544,8cyclictest181rcu_preempt21:07:342
26483997535,9cyclictest181rcu_preempt19:15:322
2648399753,56cyclictest3020-21apt-get22:55:402
2648399753,56cyclictest3020-21apt-get22:55:402
2647799757,43cyclictest31-21ksoftirqd/120:27:051
2647799757,43cyclictest31-21ksoftirqd/120:27:051
2647799755,44cyclictest31-21ksoftirqd/119:51:321
26477997547,8cyclictest181rcu_preempt21:35:501
26477997546,10cyclictest181rcu_preempt20:42:131
26477997514,39cyclictest31-21ksoftirqd/120:30:291
26465997549,9cyclictest181rcu_preempt22:30:550
26465997549,9cyclictest181rcu_preempt22:30:550
26465997543,11cyclictest0-21swapper/023:45:210
26465997543,11cyclictest0-21swapper/023:45:210
26465997536,29cyclictest0-21swapper/021:30:380
26483997457,12cyclictest27005-21kernelversion19:05:462
26483997457,12cyclictest27005-21kernelversion19:05:462
26483997449,9cyclictest181rcu_preempt22:46:552
26483997442,9cyclictest181rcu_preempt19:22:132
26483997440,8cyclictest181rcu_preempt00:23:402
26483997437,8cyclictest181rcu_preempt23:19:122
2648399743,35cyclictest20783-21ssh22:30:552
2648399743,35cyclictest20783-21ssh22:30:552
26483997423,9cyclictest0-21swapper/223:00:472
2647799748,39cyclictest31-21ksoftirqd/121:17:431
26477997441,10cyclictest181rcu_preempt00:10:411
2647799742,47cyclictest31-21ksoftirqd/123:26:121
2647799742,32cyclictest31-21ksoftirqd/122:25:481
26465997454,12cyclictest0-21swapper/023:20:210
26465997449,18cyclictest21103-21latency_hist23:25:250
26465997446,9cyclictest181rcu_preempt20:15:380
26465997440,16cyclictest181rcu_preempt21:05:350
2646599742,30cyclictest13784-21latency_hist22:20:240
26483997344,8cyclictest181rcu_preempt00:11:512
2647799739,45cyclictest31-21ksoftirqd/119:35:171
2647799739,45cyclictest31-21ksoftirqd/119:35:171
2647799738,13cyclictest31-21ksoftirqd/120:35:361
2647799737,29cyclictest31-21ksoftirqd/100:31:581
2647799737,29cyclictest31-21ksoftirqd/100:31:581
2647799736,31cyclictest31-21ksoftirqd/123:35:391
2647799736,29cyclictest31-21ksoftirqd/119:55:431
26477997357,10cyclictest300-21rs:main1
2647799735,28cyclictest31-21ksoftirqd/119:15:241
26477997351,12cyclictest0-21swapper/121:45:441
26477997348,9cyclictest181rcu_preempt20:01:461
26477997348,9cyclictest181rcu_preempt20:01:461
26477997336,7cyclictest181rcu_preempt21:25:411
2647799733,33cyclictest31-21ksoftirqd/123:05:551
2647799733,33cyclictest31-21ksoftirqd/100:15:561
2647799733,32cyclictest31-21ksoftirqd/121:41:251
2647799732,49cyclictest31-21ksoftirqd/122:06:331
26477997313,40cyclictest31-21ksoftirqd/119:48:291
2647799731,31cyclictest31-21ksoftirqd/120:20:381
26465997351,12cyclictest15-21ksoftirqd/019:30:570
26465997351,12cyclictest15-21ksoftirqd/019:30:570
26465997349,14cyclictest30247-21ssh22:45:590
26483997256,8cyclictest0-21swapper/219:55:572
26483997247,9cyclictest22240-21rm21:40:332
26483997240,11cyclictest181rcu_preempt20:10:502
26483997239,12cyclictest181rcu_preempt19:25:412
26483997234,8cyclictest0-21swapper/223:23:212
2648399722,35cyclictest30162-21latency_hist20:55:232
2647799726,29cyclictest31-21ksoftirqd/123:10:381
26477997256,10cyclictest26455-21rm00:27:311
26477997231,11cyclictest181rcu_preempt21:23:451
26477997225,33cyclictest13679-21cron22:20:211
26465997257,9cyclictest26456-21cyclictest21:44:070
26465997238,13cyclictest181rcu_preempt23:07:560
26465997238,12cyclictest1587-21ssh23:45:550
26483997155,12cyclictest26456-21cyclictest22:20:582
26483997155,12cyclictest26456-21cyclictest22:20:582
26483997146,9cyclictest10052-21rm22:13:382
26483997141,14cyclictest181rcu_preempt20:05:502
26483997136,10cyclictest181rcu_preempt21:32:092
26483997134,9cyclictest181rcu_preempt20:05:232
26483997134,9cyclictest181rcu_preempt20:05:232
2647799717,28cyclictest31-21ksoftirqd/120:15:341
2647799712,38cyclictest31-21ksoftirqd/100:21:031
2647799712,34cyclictest31-21ksoftirqd/122:35:231
2647799712,34cyclictest31-21ksoftirqd/122:35:231
26477997123,36cyclictest6190-21cut19:40:431
26465997149,12cyclictest0-21swapper/023:00:430
2646599712,60cyclictest12647-21runrttasks20:00:450
2646599712,60cyclictest12647-21runrttasks20:00:450
2649099708,51cyclictest0-21swapper/322:35:443
26483997054,10cyclictest14151-21runrttasks21:25:572
26483997052,13cyclictest19539-21sh22:30:002
26483997045,8cyclictest181rcu_preempt20:15:372
26483997036,10cyclictest26456-21cyclictest20:36:082
2647799707,26cyclictest31-21ksoftirqd/119:25:521
2647799704,28cyclictest31-21ksoftirqd/121:05:221
26477997042,12cyclictest181rcu_preempt20:45:451
2647799703,30cyclictest31-21ksoftirqd/120:10:491
26465997049,14cyclictest26456-21cyclictest00:20:240
26465997045,19cyclictest25352-21latency_hist20:40:230
26483996951,9cyclictest181rcu_preempt22:20:232
26483996951,12cyclictest900-21diskmemload23:56:162
26483996949,8cyclictest181rcu_preempt19:50:482
26483996949,14cyclictest30041-21idleruntime-cro19:15:212
2648399694,15cyclictest381ktimers/200:00:352
26483996940,10cyclictest58-21kcompactd021:10:482
26483996940,10cyclictest58-21kcompactd021:10:482
26483996933,11cyclictest181rcu_preempt00:05:462
2647799691,30cyclictest31-21ksoftirqd/119:35:501
2647799691,30cyclictest31-21ksoftirqd/119:35:501
26465996951,13cyclictest7676-21latency_hist22:10:240
26465996950,13cyclictest0-21swapper/000:30:510
26465996950,13cyclictest0-21swapper/000:30:510
2646599693,35cyclictest352-21runrttasks23:14:130
26465996933,28cyclictest0-21swapper/020:00:240
26483996845,11cyclictest181rcu_preempt20:57:262
26483996845,11cyclictest181rcu_preempt20:57:262
26483996841,18cyclictest11646-21ssh23:10:152
26483996835,11cyclictest181rcu_preempt21:45:252
2647799684,37cyclictest13430-21ntp_offset00:05:511
26465996837,22cyclictest0-21swapper/020:30:220
26465996837,22cyclictest0-21swapper/020:30:220
26465996836,11cyclictest181rcu_preempt21:00:450
2646599683,43cyclictest30228-21latency_hist20:55:240
26483996751,9cyclictest900-21diskmemload21:35:362
26483996750,9cyclictest22381-21diskmemload23:26:392
26483996745,8cyclictest2627-21kworker/u19:1+rpciod21:21:372
26483996726,10cyclictest0-21swapper/200:35:092
26483996726,10cyclictest0-21swapper/200:35:092
26465996750,10cyclictest297-21systemd-logind19:40:220
26465996750,10cyclictest297-21systemd-logind19:40:220
26483996635,8cyclictest0-21swapper/220:40:432
2648399662,38cyclictest454-21tail21:00:582
2646599662,21cyclictest28583-21ntp_states19:10:340
26465996547,12cyclictest31666-21idleruntime-cro19:20:220
2646599646,28cyclictest15-21ksoftirqd/019:40:380
2646599643,30cyclictest7967-21/usr/sbin/munin22:10:410
26490996044,10cyclictest451rcuc/320:50:563
2649099584,38cyclictest22536-21chrt21:40:433
26490995843,10cyclictest451rcuc/321:10:543
26490995843,10cyclictest451rcuc/321:10:543
2649099583,40cyclictest8718-21taskset21:17:033
2649099572,42cyclictest8209-21taskset23:56:443
2649099564,37cyclictest17567-21taskset22:25:503
26490995634,14cyclictest451rcuc/320:24:263
2649099563,38cyclictest29738-21taskset00:33:193
2649099563,38cyclictest29738-21taskset00:33:193
2649099554,37cyclictest18026-21taskset23:20:323
2649099553,35cyclictest2929-21taskset19:30:433
2649099553,35cyclictest2929-21taskset19:30:433
2649099552,34cyclictest10512-21chrt23:07:033
2649099543,40cyclictest8148-21chrt19:45:533
2649099543,38cyclictest23786-21chrt00:23:263
2649099543,37cyclictest796-21taskset21:58:303
2649099542,38cyclictest362-21taskset19:20:593
2649099542,38cyclictest31687-21chrt19:20:223
2649099534,37cyclictest18904-21chrt20:20:243
2649099534,36cyclictest8609-21taskset22:10:503
2649099534,36cyclictest6502-21chrt23:00:473
2649099533,39cyclictest21262-21chrt22:31:253
2649099533,39cyclictest21262-21chrt22:31:253
2649099533,38cyclictest5014-21taskset22:05:313
2649099533,37cyclictest11023-21chrt00:01:133
2649099533,36cyclictest28467-21chrt21:50:443
26490995333,13cyclictest451rcuc/323:32:583
26490995333,13cyclictest451rcuc/323:32:583
2649099523,36cyclictest4645-21chrt23:50:563
2649099523,35cyclictest21012-21chrt21:37:513
2649099522,35cyclictest29719-21chrt22:45:483
2649099514,36cyclictest12294-21chrt21:23:403
2649099514,32cyclictest4135-21chrt22:56:423
2649099514,32cyclictest4135-21chrt22:56:423
2649099514,32cyclictest28208-21taskset23:35:573
2649099513,36cyclictest30121-21chrt19:15:243
2649099513,34cyclictest26535-21chrt00:27:443
2649099513,33cyclictest4775-21chrt19:35:493
2649099513,33cyclictest4775-21chrt19:35:493
2649099513,32cyclictest13960-21taskset22:20:273
2649099513,32cyclictest13960-21taskset22:20:273
2649099512,35cyclictest23906-21chrt23:30:173
2649099512,35cyclictest18531-21chrt21:34:223
2649099512,34cyclictest15743-21taskset00:10:253
26490995029,13cyclictest451rcuc/321:28:523
2649099502,33cyclictest17626-21chrt23:20:083
2649099494,35cyclictest3595-21chrt23:50:233
2649099493,34cyclictest1985-21taskset22:00:293
2649099493,31cyclictest32757-21chrt22:50:503
2649099493,31cyclictest32757-21chrt22:50:503
2649099484,32cyclictest1449-21chrt21:05:443
2649099483,34cyclictest450-21chrt21:00:583
2649099483,31cyclictest14482-21taskset20:05:423
2649099473,34cyclictest28988-21taskset22:45:233
2649099473,32cyclictest10715-21chrt19:55:253
2649099473,31cyclictest32145-21chrt23:43:103
2649099473,31cyclictest32145-21chrt23:43:103
2649099473,30cyclictest25583-21taskset21:45:453
26490994726,13cyclictest451rcuc/320:30:293
2649099472,35cyclictest13428-21chrt00:05:513
2649099464,32cyclictest12193-21chrt23:10:403
2649099464,28cyclictest26098-21chrt20:40:533
26490994633,9cyclictest451rcuc/319:07:193
26490994633,9cyclictest451rcuc/319:07:193
26490994629,9cyclictest451rcuc/320:04:043
26490994629,9cyclictest451rcuc/320:04:043
2649099454,30cyclictest19195-21taskset00:15:443
2649099453,32cyclictest1845-21chrt19:25:573
2649099443,32cyclictest10846-21taskset19:55:373
2649099443,29cyclictest10980-21chrt22:15:283
26490994430,9cyclictest451rcuc/320:48:363
26490994426,11cyclictest451rcuc/320:57:103
26490994426,11cyclictest451rcuc/320:57:103
2649099442,29cyclictest7467-21chrt19:45:253
26490994422,14cyclictest451rcuc/320:26:003
26490994422,14cyclictest451rcuc/320:26:003
26490994128,9cyclictest451rcuc/320:10:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional