You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-24 - 08:20
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Wed Jun 24, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2510421419,91sleep125123-21fschecks_count18:53:521
26405991223,75cyclictest32613-21apt-get23:38:462
26299211712,65sleep026312-21systemd-run18:58:280
26405991114,52cyclictest23988-21ssh00:18:412
264059910887,15cyclictest5698-21fschecks_time22:53:492
2612721026,16sleep20-21swapper/218:56:422
2640599973,35cyclictest28225-21ssh21:43:412
2639699967,25cyclictest32613-21apt-get23:38:540
26405999540,49cyclictest0-21swapper/222:08:302
26405999540,49cyclictest0-21swapper/222:08:302
2639699943,75cyclictest30602-21apt-get20:48:540
2640599933,37cyclictest11371-21mailstats19:49:092
26405999167,11cyclictest181rcu_preempt20:28:292
26405999167,11cyclictest181rcu_preempt20:28:292
26405999154,26cyclictest0-21swapper/220:48:572
2640599903,36cyclictest32683-21ls19:14:032
2640599903,32cyclictest352-21runrttasks00:12:032
2640599903,32cyclictest21002-21ssh00:13:372
26405999027,55cyclictest0-21swapper/222:23:522
26405999027,55cyclictest0-21swapper/222:23:522
2640599902,38cyclictest14853-21unixbench_multi19:59:032
2640599893,36cyclictest24020-21awk23:23:542
2640599893,36cyclictest24020-21awk23:23:542
2640599893,32cyclictest13902-21latency_hist22:13:312
2640599893,27cyclictest17512-21/usr/sbin/munin23:13:442
26401998974,8cyclictest0-21swapper/122:08:491
26396998913,34cyclictest14100-21apt-get22:13:450
26405998838,43cyclictest27103-21sh00:23:442
26405998834,46cyclictest0-21swapper/223:18:582
2640599883,36cyclictest22157-21latency_hist21:33:302
26405998757,22cyclictest16014-21netstat20:03:512
2640599874,32cyclictest2756-21grep21:54:032
2640599874,32cyclictest2756-21grep21:54:032
2640599874,32cyclictest2412-21/usr/sbin/munin22:48:492
26405998736,43cyclictest0-21swapper/221:23:502
2640599873,36cyclictest17761-21latency_hist00:08:282
26405998730,49cyclictest0-21swapper/222:33:042
26405998729,52cyclictest26384-21cyclictest20:53:562
26405998729,52cyclictest26384-21cyclictest20:53:562
2640599872,34cyclictest2657-21cpu19:23:442
26401998766,16cyclictest17762-21fschecks_count23:13:491
2640599864,32cyclictest26623-21awk20:38:262
2640599863,33cyclictest20152-21sh20:18:272
2640599863,31cyclictest31206-21ssh21:48:382
26405998632,48cyclictest0-21swapper/223:53:532
2640599862,35cyclictest31902-21ssh22:43:382
2640599862,35cyclictest31902-21ssh22:43:382
2640199862,42cyclictest31-21ksoftirqd/120:28:381
2640599854,34cyclictest6253-21cstates23:48:442
2640599854,32cyclictest1235-21cut19:18:482
2640599853,36cyclictest27057-21grep22:34:042
2640599853,35cyclictest12589-21fschecks_time19:53:482
2640599853,35cyclictest12589-21fschecks_time19:53:482
2640599853,33cyclictest30137-21latency_hist19:08:292
2640599853,33cyclictest25412-21sort21:38:592
2640599853,32cyclictest17448-21fschecks_time20:08:582
2640599853,31cyclictest29678-21grep20:44:212
2640599853,30cyclictest4665-21/usr/sbin/munin21:58:492
2640599853,28cyclictest13371-21ssh21:18:442
2640599852,32cyclictest4142-21grep23:44:052
26396998568,11cyclictest2374-21systemd-journal19:40:580
2625928527,37sleep30-21swapper/318:58:273
2640599844,31cyclictest22629-21cat21:33:512
2640599843,37cyclictest12652-21ssh23:04:212
2640599843,32cyclictest4658-21grep21:03:502
2640599843,32cyclictest14868-21grep22:14:182
2640599842,33cyclictest6130-21fschecks_time19:33:482
2641199832,55cyclictest14139-21chrt22:13:473
2640599834,31cyclictest14550-21/usr/sbin/munin23:08:542
2640599834,31cyclictest14550-21/usr/sbin/munin23:08:542
2640599834,30cyclictest20649-21fschecks_time20:18:502
2640599833,36cyclictest31761-21latency_hist19:13:302
2640599833,32cyclictest7474-21/usr/sbin/munin19:38:532
2640599833,30cyclictest27007-21/usr/sbin/munin23:28:542
2640599832,25cyclictest28451-21latency_hist20:43:292
2640599832,25cyclictest28451-21latency_hist20:43:292
26396998366,11cyclictest277-21dbus-daemon22:24:000
26396998366,11cyclictest277-21dbus-daemon22:24:000
26396998319,28cyclictest4477-21fschecks_count19:28:480
26396998319,28cyclictest4477-21fschecks_count19:28:480
26405998261,12cyclictest0-21swapper/223:59:012
26405998261,12cyclictest0-21swapper/223:59:012
2640599824,31cyclictest1428-21/usr/sbin/munin20:58:392
2640599824,30cyclictest10021-21unixbench_multi19:44:032
2640599824,25cyclictest26784-21fschecks_count18:58:482
2640599824,25cyclictest26784-21fschecks_count18:58:482
2640599823,32cyclictest29086-21cstates22:38:452
2640599823,30cyclictest11072-21awk21:14:012
2640599822,32cyclictest8471-21grep21:09:052
2640599822,32cyclictest17894-21netstat22:19:452
2640199823,29cyclictest31-21ksoftirqd/123:03:591
2639699822,65cyclictest10893-21apt-get22:09:000
2640599813,32cyclictest4453-21cut19:28:482
2640599813,32cyclictest4453-21cut19:28:482
2640599812,36cyclictest30140-21sendmail_mailst23:34:002
26401998166,10cyclictest26384-21cyclictest19:08:541
26401998163,11cyclictest277-21dbus-daemon23:28:281
26401998163,11cyclictest277-21dbus-daemon23:28:281
26396998162,12cyclictest1018-21/usr/sbin/munin19:19:020
26396998151,8cyclictest181rcu_preempt22:59:530
2640599803,35cyclictest352-21runrttasks20:30:462
26396998054,17cyclictest2239-21ssh21:53:530
26396998054,17cyclictest2239-21ssh21:53:530
2640599793,29cyclictest8645-21fschecks_count22:58:502
26401997963,10cyclictest23365-21kworker/u18:020:28:261
26401997963,10cyclictest23365-21kworker/u18:020:28:261
26396997961,12cyclictest25029-21latency_hist21:38:290
26401997861,12cyclictest26384-21cyclictest00:11:481
26401997860,12cyclictest2374-21systemd-journal23:51:241
26401997857,15cyclictest2257-21latency_hist22:48:291
26401997857,15cyclictest2257-21latency_hist22:48:291
26396997856,15cyclictest20230-21latency_hist23:18:280
2640199777,40cyclictest30414-21apt-config20:48:301
26401997729,16cyclictest181rcu_preempt21:18:391
26401997657,13cyclictest21113-21sh21:31:061
26401997656,13cyclictest1303-21unixbench_singl23:39:211
26401997655,15cyclictest16725-21sh21:23:541
26396997659,11cyclictest352-21runrttasks22:33:490
26396997654,16cyclictest18328-21ssh21:26:420
26401997557,12cyclictest1860-21ssh20:58:541
26401997551,9cyclictest20975-21/usr/sbin/munin00:14:021
26396997556,13cyclictest298-21in:imuxsock20:43:260
26396997556,13cyclictest298-21in:imuxsock20:43:260
26396997549,20cyclictest544-21cron00:18:260
49872740,7sleep30-21swapper/321:58:483
26401997457,11cyclictest14990-21kernelversion23:08:521
26401997457,11cyclictest14990-21kernelversion23:08:521
26401997453,15cyclictest1120-21diskmemload21:13:531
26401997449,9cyclictest28194-21ssh21:43:391
26396997458,10cyclictest15975-21ssh21:23:270
26396997451,16cyclictest2374-21systemd-journal23:23:000
26396997450,15cyclictest1136-21kworker/u18:0-flush-0:3221:03:290
2639699743,36cyclictest26940-21apt-get00:23:370
26401997356,11cyclictest26825-21ssh21:40:271
26401997346,10cyclictest4287-21ssh21:58:131
26401997346,10cyclictest4287-21ssh21:58:131
2640199734,32cyclictest31-21ksoftirqd/119:33:421
26401997258,9cyclictest58-21kcompactd023:19:101
26401997255,11cyclictest27961-21systemd-run19:03:281
26401997255,11cyclictest27961-21systemd-run19:03:281
26401997254,12cyclictest30053-21open_files23:33:591
26401997253,13cyclictest13769-21latency_hist19:58:291
26401997253,13cyclictest13769-21latency_hist19:58:291
2640199725,28cyclictest31-21ksoftirqd/120:18:421
26401997245,9cyclictest181rcu_preempt22:17:521
26396997253,13cyclictest31515-21fschecks_time21:48:490
26396997248,18cyclictest19191-21ssh22:22:010
26401997158,7cyclictest0-21swapper/122:59:011
26401997156,10cyclictest26384-21cyclictest21:38:121
26401997155,10cyclictest24879-21timerandwakeup00:19:031
26401997145,10cyclictest4233-21ssh23:44:061
26401997144,9cyclictest181rcu_preempt20:08:261
26401997138,13cyclictest181rcu_preempt21:58:581
26401997126,13cyclictest181rcu_preempt22:53:521
26401997123,31cyclictest0-21swapper/121:48:491
26396997150,15cyclictest544-21cron22:58:270
26401997055,10cyclictest26384-21cyclictest19:48:571
26401997045,12cyclictest181rcu_preempt22:28:221
26401997045,12cyclictest181rcu_preempt22:28:221
26401997031,16cyclictest181rcu_preempt23:31:421
26396997055,10cyclictest26384-21cyclictest20:32:580
26401996953,10cyclictest9542-21ssh21:12:021
26401996951,10cyclictest0-21swapper/122:06:011
26401996951,10cyclictest0-21swapper/122:06:011
26401996946,16cyclictest0-21swapper/123:53:421
26401996943,11cyclictest181rcu_preempt22:35:521
26401996943,11cyclictest181rcu_preempt19:18:421
26401996938,11cyclictest181rcu_preempt22:51:401
26396996955,9cyclictest30797-21ntp_offset19:08:550
26396996951,12cyclictest30006-21ssh23:33:560
26396996950,14cyclictest28414-21ssh21:43:500
26401996855,9cyclictest58-21kcompactd019:59:231
26401996854,10cyclictest26384-21cyclictest00:08:051
2640199684,44cyclictest25832-21latency_hist22:33:291
26401996835,12cyclictest181rcu_preempt21:07:161
26401996834,10cyclictest181rcu_preempt23:59:001
26401996834,10cyclictest181rcu_preempt23:59:001
2640199683,26cyclictest29990-21munin-run19:08:271
26401996820,40cyclictest291rcuc/119:28:411
26401996820,40cyclictest291rcuc/119:28:411
26396996852,11cyclictest19676-21snmpd20:56:280
26396996852,11cyclictest19676-21snmpd20:56:280
26396996847,15cyclictest10711-21latency_hist22:08:310
26396996847,15cyclictest10711-21latency_hist22:08:310
26401996749,11cyclictest0-21swapper/122:19:381
26401996735,13cyclictest181rcu_preempt20:13:511
26401996725,13cyclictest181rcu_preempt20:08:581
26396996753,8cyclictest0-21swapper/023:23:470
26396996753,8cyclictest0-21swapper/023:23:470
26396996743,16cyclictest7310-21latency_hist19:38:290
2639699673,52cyclictest28227-21apt-get20:43:420
26396996729,31cyclictest0-21swapper/000:04:040
2640199664,43cyclictest352-21runrttasks20:40:141
2640199664,43cyclictest352-21runrttasks20:40:141
26401996635,16cyclictest181rcu_preempt19:24:001
26401996630,9cyclictest181rcu_preempt20:53:541
26401996630,9cyclictest181rcu_preempt20:53:541
26396996650,10cyclictest5789-21munin-run23:48:270
26396996648,13cyclictest24348-21kworker/u18:0+events_unbound19:08:270
26396996642,14cyclictest1120-21diskmemload21:43:300
26396996637,15cyclictest181rcu_preempt20:18:490
2639699663,49cyclictest18586-21ntp_kernel_err00:08:550
26396996622,37cyclictest10210-21latency_hist21:13:300
26401996543,9cyclictest181rcu_preempt19:14:471
2640199654,21cyclictest27313-21/usr/sbin/munin00:23:501
26401996541,9cyclictest181rcu_preempt19:40:031
2640199653,45cyclictest352-21runrttasks19:46:551
26401996525,13cyclictest181rcu_preempt20:49:061
26396996551,10cyclictest26384-21cyclictest21:18:290
26396996549,12cyclictest26384-21cyclictest21:08:260
26396996549,11cyclictest0-21swapper/022:49:020
26396996548,12cyclictest26613-21latency_hist20:38:290
26396996546,14cyclictest1120-21diskmemload23:48:420
26396996545,11cyclictest0-21swapper/021:58:470
26396996543,12cyclictest0-21swapper/021:28:490
26396996542,17cyclictest10443-21latency_hist19:48:310
26396996541,15cyclictest9370-21fschecks_count23:53:530
26396996541,10cyclictest181rcu_preempt19:24:010
2639699653,50cyclictest18792-21/usr/sbin/munin20:14:050
2639699653,28cyclictest12353-21tune2fs23:58:470
2639699653,28cyclictest12353-21tune2fs23:58:470
26396996525,28cyclictest12089-21basename19:53:270
26396996521,20cyclictest15611-21sed23:09:040
26396996521,20cyclictest15611-21sed23:09:040
26401996442,8cyclictest181rcu_preempt22:40:141
26396996448,11cyclictest352-21runrttasks20:23:380
26396996448,11cyclictest352-21runrttasks20:23:380
26396996445,13cyclictest14271-21sendmail-msp23:08:260
26396996443,12cyclictest0-21swapper/019:59:020
26396996441,12cyclictest0-21swapper/019:13:500
26396996436,7cyclictest181rcu_preempt18:58:400
26396996436,7cyclictest181rcu_preempt18:58:400
26396996427,16cyclictest181rcu_preempt22:43:440
26396996427,16cyclictest181rcu_preempt22:43:440
26396996419,8cyclictest0-21swapper/022:38:410
26396996413,12cyclictest1120-21diskmemload22:29:050
26401996330,13cyclictest181rcu_preempt20:33:521
26396996341,13cyclictest1120-21diskmemload23:29:020
2639699633,26cyclictest18549-21grep20:13:270
26396996248,10cyclictest26384-21cyclictest00:19:070
2639699603,38cyclictest15793-21fschecks_time20:03:460
2641199594,41cyclictest1041-21chrt19:18:393
26396995910,37cyclictest12785-21latency19:53:520
26396995910,37cyclictest12785-21latency19:53:520
2641199564,39cyclictest24755-21taskset22:30:543
2641199543,40cyclictest13713-21chrt00:00:393
2641199543,40cyclictest13713-21chrt00:00:393
2641199543,38cyclictest19472-21chrt21:28:443
2641199534,37cyclictest6368-21taskset19:33:543
2641199533,36cyclictest10134-21chrt23:00:573
2641199527,34cyclictest11675-21chrt23:58:163
2641199524,34cyclictest6095-21chrt23:48:353
2641199523,34cyclictest22027-21chrt20:23:403
2641199523,34cyclictest22027-21chrt20:23:403
2641199523,32cyclictest5798-21chrt19:33:303
2641199523,32cyclictest5798-21chrt19:33:303
2641199522,36cyclictest16532-21chrt21:23:503
2641199513,35cyclictest17884-21chrt00:08:293
2641199513,34cyclictest26625-21chrt20:38:263
26411995115,27cyclictest9695-21chrt22:06:423
26411995115,27cyclictest9695-21chrt22:06:423
2641199506,35cyclictest11697-21chrt23:03:573
2641199503,38cyclictest22675-21chrt22:28:263
2641199503,38cyclictest22675-21chrt22:28:263
2641199503,37cyclictest917-21kworker/3:1+events19:08:273
2641199503,36cyclictest30100-21chrt23:34:003
2641199503,35cyclictest15867-21chrt20:03:473
2641199503,35cyclictest13340-21chrt21:18:393
2641199503,34cyclictest20826-21chrt23:18:483
2641199502,37cyclictest7228-21chrt21:08:293
2641199493,34cyclictest18849-21chrt00:09:023
2641199493,33cyclictest2886-21chrt19:23:503
2641199493,31cyclictest3246-21chrt23:43:453
2641199492,38cyclictest27527-21chrt22:35:203
2641199492,35cyclictest30536-21chrt21:47:333
2641199492,33cyclictest3100-21chrt21:54:513
2641199492,33cyclictest3100-21chrt21:54:513
2641199483,35cyclictest27283-21chrt21:41:453
2641199483,34cyclictest513-21chrt20:54:023
2641199483,34cyclictest513-21chrt20:54:023
2641199483,34cyclictest30082-21chrt22:39:093
2641199483,34cyclictest14977-21taskset23:08:513
2641199483,34cyclictest14977-21taskset23:08:513
2641199483,33cyclictest769-21chrt19:18:273
2641199483,32cyclictest1404-21taskset22:46:303
2641199483,32cyclictest1404-21taskset22:46:303
2641199483,32cyclictest12264-21chrt19:53:303
2641199482,36cyclictest2596-21chrt20:59:133
2641199482,35cyclictest19522-21chrt20:14:043
2641199482,35cyclictest14397-21chrt19:58:533
2641199474,33cyclictest18883-21chrt22:21:093
2641199473,33cyclictest27857-21chrt23:29:553
2641199473,33cyclictest27081-21chrt18:58:543
2641199473,33cyclictest27081-21chrt18:58:543
2641199473,32cyclictest12309-21chrt22:09:583
2641199472,35cyclictest1155-21chrt23:39:193
2641199472,34cyclictest23975-21chrt20:28:513
2641199472,34cyclictest23392-21chrt00:17:453
2641199463,33cyclictest23666-21chrt21:34:573
2641199463,32cyclictest13771-21chrt19:58:293
2641199463,32cyclictest13771-21chrt19:58:293
2641199462,35cyclictest2954-21chrt22:48:533
2641199462,33cyclictest29419-21chrt20:44:183
2641199453,33cyclictest31753-21chrt21:48:553
2641199453,32cyclictest9823-21taskset21:12:513
2641199452,28cyclictest8219-21taskset22:58:303
2641199443,33cyclictest11385-21taskset21:14:053
2641199443,30cyclictest18841-21taskset23:14:393
2641199442,30cyclictest25107-21taskset00:19:113
2641199433,30cyclictest25277-21chrt23:26:063
2641199433,30cyclictest25277-21chrt23:26:063
2641199423,28cyclictest8550-21taskset19:39:583
2641199423,26cyclictest28340-21taskset00:24:543
2641199413,29cyclictest28350-21taskset20:43:293
2641199413,29cyclictest28350-21taskset20:43:293
2641199413,28cyclictest30431-21chrt19:08:473
2641199413,27cyclictest20426-21chrt20:18:423
2641199403,28cyclictest17663-21chrt20:09:023
2641199402,27cyclictest31841-21taskset20:52:083
2641199392,27cyclictest9108-21chrt19:43:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional