You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 07:15
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Dec 20, 2025 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1473321217,91sleep00-21swapper/019:04:570
160289910776,21cyclictest16720-21latency_hist22:29:340
1145421060,2sleep30-21swapper/323:50:053
2077521040,6sleep00-21swapper/000:20:460
1588721009,63sleep215912-21modprobe19:09:322
16031999948,45cyclictest0-21swapper/119:39:321
1602899983,52cyclictest15-21ksoftirqd/021:02:560
16028999566,19cyclictest23047-21latency_hist19:29:350
16028999465,20cyclictest7031-21latency_hist21:59:340
16028999427,41cyclictest15-21ksoftirqd/022:00:170
1602899924,50cyclictest28055-21apt-config19:44:350
16031999174,11cyclictest16963-21apt-get20:49:511
1603199913,37cyclictest7118-21/usr/sbin/munin21:59:561
16031999046,25cyclictest16020-21cyclictest20:59:491
1603199904,33cyclictest26983-21latency_hist00:39:341
1603199903,34cyclictest19019-21fschecks_time00:14:541
1603499894,29cyclictest4271-21apt-get23:30:152
1603199894,33cyclictest22029-21latency_hist00:24:331
1603199893,37cyclictest26895-21fschecks_count21:19:521
1603199893,37cyclictest26895-21fschecks_count21:19:521
1603199893,35cyclictest350-21runrttasks00:09:511
1603199893,33cyclictest3668-21latency_hist20:09:341
16034998867,15cyclictest16365-21timerwakeupswit00:05:072
1603499884,39cyclictest25658-21cut00:34:532
1603499882,26cyclictest19604-21/usr/sbin/munin00:15:072
16034998818,30cyclictest15472-21cat00:04:352
16034998817,27cyclictest10511-21sh22:09:512
1603199885,31cyclictest29963-21sort21:29:491
1603199883,38cyclictest10130-21latency_hist22:09:331
1603199883,35cyclictest19525-21unixbench_singl20:55:101
1603199883,35cyclictest12125-21fschecks_time20:34:541
1603199883,34cyclictest17397-21proc_pri22:30:041
1603199882,39cyclictest4197-21latency20:09:571
1603199882,33cyclictest479-21latency_hist19:59:341
1603199882,33cyclictest4072-21fschecks_count21:49:531
1603199882,33cyclictest4072-21fschecks_count21:49:531
1602899884,51cyclictest15-21ksoftirqd/023:02:090
1603499873,39cyclictest9953-21unixbench_multi23:45:082
16034998715,32cyclictest22134-21cat19:25:042
1603199875,24cyclictest21807-21/usr/sbin/munin21:04:431
1603199874,35cyclictest540-21latency_hist23:19:331
1603199874,32cyclictest5810-21/usr/sbin/munin23:35:041
1603199873,43cyclictest26628-21fschecks_time19:39:541
1603199873,38cyclictest350-21runrttasks21:24:491
1603199873,36cyclictest16600-21cat20:49:341
1603199873,35cyclictest20194-21apt-key22:39:351
1603199873,35cyclictest20194-21apt-key22:39:351
1603199873,34cyclictest4872-21proc_pri23:30:551
1603199873,34cyclictest20586-21fschecks_time22:40:231
1603199873,32cyclictest29665-21latency_hist19:49:341
16031998729,51cyclictest0-21swapper/119:14:551
16034998630,48cyclictest0-21swapper/220:43:442
1603199866,17cyclictest219112apt-get22:44:421
1603199864,40cyclictest1129-21kernelversion23:19:561
1603199864,32cyclictest23767-21fschecks_time22:49:521
1603199863,36cyclictest10204-21cat20:29:361
1603199863,35cyclictest23788-21latency_hist00:29:351
1603199863,34cyclictest14675-21find00:00:061
1603199863,33cyclictest20615-21switchtime19:20:041
1603199862,37cyclictest3944-21cat23:29:341
1603199862,28cyclictest26949-21fschecks_count22:59:531
1602899865,54cyclictest15-21ksoftirqd/022:10:480
1603499854,35cyclictest17677-21ntp_offset00:10:002
1603499853,40cyclictest6777-21sendmail20:19:322
1603499853,40cyclictest6777-21sendmail20:19:322
1603499853,39cyclictest23710-21grep00:29:342
1603499853,38cyclictest28696-21latency19:45:562
16034998516,29cyclictest12745-21cat22:15:072
1603199854,37cyclictest10422-21cpuspeed22:09:491
1603199854,32cyclictest1464-21timerandwakeup21:40:051
1603199853,35cyclictest7785-21grep20:20:071
1603199853,35cyclictest6990-21latency_hist20:19:371
1603199853,35cyclictest6990-21latency_hist20:19:371
1603199853,34cyclictest32085-21ps21:35:051
1603199853,32cyclictest23095-21/usr/sbin/munin19:30:031
1603199853,32cyclictest11206-21unixbench_singl20:30:091
1603199853,32cyclictest11206-21unixbench_singl20:30:091
1603199853,31cyclictest2057-21latency_hist20:04:341
1603199853,24cyclictest29753-21latency_hist23:09:331
1603199852,36cyclictest24396-21grep21:10:081
16028998566,12cyclictest26478-21sendmail22:59:310
16034998439,36cyclictest2185-21cat21:44:342
16034998437,39cyclictest0-21swapper/220:45:432
1603499843,33cyclictest350-21runrttasks20:35:102
1603499843,23cyclictest7786-21/usr/sbin/munin22:00:032
1603499843,23cyclictest20860-21/usr/sbin/munin21:00:052
1603199844,37cyclictest15514-21/usr/sbin/munin00:04:581
1603199844,35cyclictest13691-21fschecks_count20:39:511
16031998436,40cyclictest0-21swapper/121:45:121
16031998436,40cyclictest0-21swapper/121:45:121
1603199843,36cyclictest25245-21expr22:54:501
1603199843,33cyclictest9295-21fschecks_time23:44:541
1603199843,32cyclictest21704-21fschecks_count19:24:531
1603199843,31cyclictest31316-21latency_hist23:14:351
1603199843,30cyclictest15276-21sort22:24:491
1603199843,29cyclictest5694-21fschecks_count21:54:521
1603199842,36cyclictest12172-21latency_hist23:54:341
1603199842,33cyclictest16203-21/usr/sbin/munin19:09:501
1603199842,24cyclictest14503-21grep22:20:051
16028998411,51cyclictest15-21ksoftirqd/021:22:510
16028998411,51cyclictest15-21ksoftirqd/021:22:510
1603499837,9cyclictest0-21swapper/221:58:572
1603499834,37cyclictest350-21runrttasks19:15:122
1603499833,41cyclictest3102-21grep20:05:092
1603499833,38cyclictest29983-21sshd23:09:432
1603499833,35cyclictest15192-21/usr/sbin/munin22:24:562
1603499833,25cyclictest13886-21/usr/sbin/munin22:19:522
1603499832,31cyclictest19336-21grep22:35:092
1603499832,31cyclictest19336-21grep22:35:092
16031998356,19cyclictest12165-21fschecks_count22:14:531
1603199833,33cyclictest25030-21/usr/sbin/munin21:14:501
1603199833,33cyclictest25030-21/usr/sbin/munin21:14:501
1603199833,32cyclictest29787-21/usr/sbin/munin19:49:591
1603199832,27cyclictest350-21runrttasks23:44:141
1602899834,52cyclictest15-21ksoftirqd/020:24:090
1602899833,29cyclictest15-21ksoftirqd/019:55:090
1602899832,38cyclictest15-21ksoftirqd/022:45:270
16034998237,39cyclictest21342-21latency_hist19:24:342
1603499823,37cyclictest31723-21tune2fs21:34:552
1603499823,33cyclictest27919-21wc19:44:342
1603499823,31cyclictest350-21runrttasks20:29:032
1603499823,26cyclictest1-21systemd19:59:322
1603499823,13cyclictest0-21swapper/221:24:532
1603499822,27cyclictest1056-21/usr/sbin/munin23:19:542
1603499822,24cyclictest18931-21uname20:54:582
16034998213,29cyclictest22001-21cat22:44:482
16031998233,41cyclictest0-21swapper/123:55:431
1602899823,42cyclictest15-21ksoftirqd/023:42:490
1603499818,33cyclictest23880-21latency21:09:572
1603499814,8cyclictest0-21swapper/222:04:362
1603499814,33cyclictest17405-21proc_pri22:30:042
16034998135,39cyclictest3836-21apt-key23:29:342
1603499813,38cyclictest24744-21grep00:30:092
1603499813,38cyclictest12832-21taskset23:55:012
1603499813,36cyclictest29966-21cpuspeed21:29:492
1603499813,22cyclictest5150-21sh20:14:322
1603499812,25cyclictest31191-21ls19:54:332
1603499812,24cyclictest27655-21grep23:00:082
1603499812,24cyclictest26438-21basename21:19:312
1603499812,24cyclictest26438-21basename21:19:312
16031998127,47cyclictest0-21swapper/100:34:331
16028998147,23cyclictest5148-21apt20:14:320
16028998124,38cyclictest15-21ksoftirqd/023:53:460
1603499804,66cyclictest10877-21processes20:30:052
1603499804,66cyclictest10877-21processes20:30:052
1603499804,38cyclictest707-21latency_hist23:19:352
1603499803,37cyclictest8477-21latency_hist20:24:342
1603499803,28cyclictest26041-21grep22:55:102
1603499803,23cyclictest26056-21modprobe19:39:312
1603499803,23cyclictest17829-21/usr/sbin/munin20:51:042
1603499803,22cyclictest10676-21dpkg23:49:412
1603499802,24cyclictest2062-21latency_hist20:04:342
1602899803,43cyclictest15-21ksoftirqd/023:24:590
1602899802,44cyclictest15-21ksoftirqd/000:03:420
16034997935,36cyclictest0-21swapper/221:09:352
1603499793,37cyclictest17191-21unixbench_singl19:10:062
1603499793,36cyclictest22043-21latency_hist00:24:332
1603499793,34cyclictest6538-21switchtime23:35:052
1603499793,33cyclictest29917-21latency_hist23:09:352
1603499793,33cyclictest23840-21switchtime19:30:062
1603499793,31cyclictest350-21runrttasks21:47:572
1603499793,31cyclictest350-21runrttasks21:47:572
1603499793,25cyclictest20299-21apt-get22:39:442
16034997928,45cyclictest0-21swapper/223:40:072
1603499792,25cyclictest4461-21/usr/sbin/munin21:50:042
1603499792,25cyclictest4461-21/usr/sbin/munin21:50:042
16028997963,10cyclictest1550-21timerwakeupswit21:40:060
1602899793,48cyclictest15-21ksoftirqd/019:53:580
1602899793,28cyclictest15-21ksoftirqd/021:52:450
1602899793,28cyclictest15-21ksoftirqd/021:52:450
1603499783,34cyclictest350-21runrttasks21:22:142
1603499783,34cyclictest350-21runrttasks21:22:142
1603199783,24cyclictest16082-21grep19:09:361
1602899785,39cyclictest15-21ksoftirqd/022:29:590
1602899783,58cyclictest350-21runrttasks22:39:460
1602899783,53cyclictest15-21ksoftirqd/021:06:030
1602899783,47cyclictest15-21ksoftirqd/023:13:180
1602899782,51cyclictest15-21ksoftirqd/023:06:530
1602899782,47cyclictest15-21ksoftirqd/021:27:120
1603499773,30cyclictest23986-21/usr/sbin/munin22:49:572
16028997757,12cyclictest9693-21mailstats23:45:050
16028997755,15cyclictest19791-21latency_hist19:19:350
16028997655,15cyclictest23868-21cpu00:29:500
16028997654,16cyclictest16764-21latency_hist20:49:350
1602899763,46cyclictest15-21ksoftirqd/022:20:150
1602899753,40cyclictest18323-21modprobe20:54:320
1602899753,37cyclictest15-21ksoftirqd/021:46:350
1602899753,37cyclictest15-21ksoftirqd/021:46:350
1602899752,40cyclictest15-21ksoftirqd/019:33:130
1602899746,46cyclictest15-21ksoftirqd/020:04:580
16028997418,34cyclictest12920-21cut23:55:020
16028997354,13cyclictest2120-21idleruntime-cro23:24:310
1602899734,34cyclictest29986-21cpuspeed21:29:490
1602899734,20cyclictest15-21ksoftirqd/019:38:130
1602899733,42cyclictest15-21ksoftirqd/021:16:170
1602899733,42cyclictest15-21ksoftirqd/021:16:170
16028997324,11cyclictest161ktimers/020:04:320
1602899724,41cyclictest15-21ksoftirqd/000:39:080
1602899723,50cyclictest20625-21kworker/u18:0+ext4-rsv-conversion19:45:180
16028997232,24cyclictest15-21ksoftirqd/022:15:040
1602899722,39cyclictest15-21ksoftirqd/022:50:270
1602899714,56cyclictest13223-21ls20:39:310
1602899714,49cyclictest16800-21grep19:10:020
1602899713,44cyclictest15-21ksoftirqd/022:05:500
16028997116,35cyclictest15-21ksoftirqd/020:30:050
16028997116,35cyclictest15-21ksoftirqd/020:30:050
1602899706,44cyclictest15-21ksoftirqd/023:32:290
1602899705,28cyclictest15-21ksoftirqd/022:34:590
1602899705,28cyclictest15-21ksoftirqd/022:34:590
1602899702,38cyclictest15-21ksoftirqd/021:35:340
1602899695,27cyclictest15-21ksoftirqd/023:36:200
16028996945,15cyclictest364-21snmpd19:19:460
1602899694,37cyclictest15-21ksoftirqd/020:15:390
1602899694,37cyclictest15-21ksoftirqd/020:15:390
1602899693,37cyclictest15-21ksoftirqd/023:17:150
1602899693,31cyclictest15-21ksoftirqd/000:24:550
1602899692,35cyclictest15-21ksoftirqd/020:56:240
1602899685,43cyclictest16297-21threads00:05:060
1602899683,44cyclictest15-21ksoftirqd/021:13:280
1602899683,37cyclictest15-21ksoftirqd/000:17:020
1602899672,55cyclictest14079-21proc_pri20:40:010
16041996511,33cyclictest0-21swapper/322:40:063
1602899655,31cyclictest15-21ksoftirqd/020:24:520
1602899634,26cyclictest17131-21/usr/sbin/munin00:10:030
16041996039,14cyclictest451rcuc/319:34:593
1591925811,29sleep30-21swapper/319:09:333
1604199574,29cyclictest0-21swapper/320:50:233
1604199553,36cyclictest25089-21chrt22:54:353
1604199533,34cyclictest20084-21chrt20:59:353
1604199525,35cyclictest10536-21chrt20:29:553
1604199525,35cyclictest10536-21chrt20:29:553
1604199523,38cyclictest18623-21taskset00:14:343
1604199523,37cyclictest30435-21taskset19:50:043
1604199523,36cyclictest18151-21chrt22:34:313
1604199522,37cyclictest9409-21chrtset20:25:063
16041995210,37cyclictest0-21swapper/323:30:203
1604199514,32cyclictest29530-21taskset19:49:313
16041995138,9cyclictest451rcuc/321:45:063
16041995138,9cyclictest451rcuc/321:45:063
16041995137,9cyclictest451rcuc/300:14:473
16041995136,10cyclictest451rcuc/319:56:293
1604199513,37cyclictest15412-21taskset00:04:343
1604199513,35cyclictest3313-21taskset20:06:443
1604199512,33cyclictest10490-21chrt22:09:503
16041995035,8cyclictest451rcuc/323:07:383
16041995034,9cyclictest451rcuc/321:39:223
1604199503,36cyclictest984-21chrt23:19:533
1604199503,34cyclictest21608-21chrt00:21:063
1604199503,32cyclictest14940-21chrt22:24:313
1604199502,35cyclictest24323-21taskset21:10:073
1604199493,33cyclictest19638-21chrt19:19:323
1604199492,35cyclictest30086-21taskset23:09:493
1604199492,35cyclictest26594-21chrt21:19:343
1604199492,35cyclictest26594-21chrt21:19:343
1604199492,34cyclictest28299-21chrt23:04:353
1604199492,34cyclictest24646-21chrt00:30:083
1604199483,32cyclictest7714-21chrt22:00:023
1604199483,31cyclictest30047-21chrt21:29:503
16041994832,9cyclictest451rcuc/323:38:473
1604199482,34cyclictest13930-21chrt20:39:553
1604199473,35cyclictest21873-21chrt21:04:493
1604199473,34cyclictest31973-21chrt23:14:593
1604199473,32cyclictest16113-21taskset20:45:093
1604199472,32cyclictest914-21taskset19:59:563
1604199464,30cyclictest16354-21chrt00:05:073
1604199463,34cyclictest26255-21chrt00:35:063
1604199463,34cyclictest2049-21chrt21:44:323
16041994632,9cyclictest451rcuc/321:56:513
16041994631,10cyclictest451rcuc/320:15:013
16041994631,10cyclictest451rcuc/320:15:013
1604199462,31cyclictest12467-21chrt22:15:003
1604199453,29cyclictest28953-21taskset21:25:053
16041994532,8cyclictest451rcuc/319:09:493
16041994531,9cyclictest451rcuc/320:13:003
1604199452,30cyclictest27331-21taskset19:40:083
1604199443,28cyclictest22859-21taskset19:29:313
1604199443,26cyclictest12229-21chrt20:34:573
16041994431,9cyclictest451rcuc/322:49:143
1604199442,28cyclictest26845-21chrt21:19:513
1604199442,28cyclictest26845-21chrt21:19:513
16041994310,26cyclictest0-21swapper/323:59:343
1604199423,30cyclictest20342-21taskset19:19:563
16041994229,9cyclictest451rcuc/323:28:123
1604199413,29cyclictest9194-21chrt22:04:563
16041994127,9cyclictest451rcuc/323:42:333
16041994123,8cyclictest0-21swapper/322:24:433
1604199402,26cyclictest7856-21chrt20:20:083
1604199393,27cyclictest3906-21taskset21:49:463
1604199393,27cyclictest3906-21taskset21:49:463
16041993925,8cyclictest451rcuc/300:28:193
1604199383,26cyclictest24093-21chrt19:30:083
1604199383,25cyclictest25132-21taskset22:54:473
1604199383,25cyclictest19425-21taskset22:35:123
1604199383,25cyclictest19425-21taskset22:35:123
16041993824,9cyclictest451rcuc/321:02:393
1604199374,24cyclictest9215-21taskset23:44:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional