You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 05:39
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sat Dec 27, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2596721217,81sleep125983-21systemd-run19:09:421
1461021070,6sleep30-21swapper/322:26:263
26063999750,17cyclictest22591-21latency22:40:070
26063999750,17cyclictest22591-21latency22:40:070
2606499963,43cyclictest1839-21grep23:00:051
62072950,6sleep00-21swapper/019:45:160
250382949,64sleep20-21swapper/219:06:252
2606499913,9cyclictest0-21swapper/123:29:531
26064999050,32cyclictest4613-21sh23:04:591
2606499903,37cyclictest16830-21runrttasks20:19:511
2606499903,37cyclictest16830-21runrttasks20:19:511
26066998972,11cyclictest9785-21modprobe22:19:412
2606499897,37cyclictest12831-21/usr/sbin/munin00:15:011
2606499893,38cyclictest29100-21awk20:55:171
26063998968,15cyclictest26059-21cyclictest19:29:400
26063998968,15cyclictest26059-21cyclictest19:29:400
2606499884,10cyclictest0-21swapper/120:04:551
2606499883,38cyclictest7355-21seq00:05:451
2606499883,38cyclictest346-21runrttasks19:19:571
2606499883,38cyclictest346-21runrttasks19:19:571
2606499883,37cyclictest16916-21threads22:30:181
2606499882,39cyclictest16963-21grep21:35:211
26066998764,14cyclictest32661-21sh23:53:232
2606499873,9cyclictest0-21swapper/122:35:081
2606499873,38cyclictest26388-21latency_hist20:49:431
2606499873,37cyclictest27825-21/usr/sbin/munin23:46:181
2606499873,37cyclictest27825-21/usr/sbin/munin23:46:181
26063998732,30cyclictest181rcu_preempt20:25:150
26063998732,30cyclictest181rcu_preempt20:25:150
2606499864,48cyclictest7329-21ssh23:09:541
2606499864,40cyclictest28220-21/usr/sbin/munin21:55:071
2606499863,39cyclictest31389-21ssh22:00:141
2606499863,38cyclictest3671-21latency_hist21:14:431
2606499863,38cyclictest27844-21munin-node19:14:501
2606499863,38cyclictest27844-21munin-node19:14:501
2606499863,38cyclictest18947-21runrttasks20:25:091
2606499863,38cyclictest18947-21runrttasks20:25:091
2606499863,37cyclictest28000-21taskset20:54:431
2606499863,37cyclictest24680-21cat00:34:441
2606499863,37cyclictest24680-21cat00:34:441
2606499863,37cyclictest13861-21cut20:10:021
2606499863,36cyclictest31893-21grep21:05:041
2606499863,36cyclictest21767-21/usr/sbin/munin23:35:051
2606499863,10cyclictest0-21swapper/122:24:541
26063998671,10cyclictest26059-21cyclictest23:45:590
26063998671,10cyclictest26059-21cyclictest23:45:590
2606499854,33cyclictest22127-21/usr/sbin/munin22:40:001
2606499854,33cyclictest22127-21/usr/sbin/munin22:40:001
2606499853,37cyclictest346-21runrttasks22:16:131
2606499853,37cyclictest29124-21runrttasks22:51:161
2606499853,37cyclictest29124-21runrttasks22:51:161
2606499853,37cyclictest22004-21ssh21:45:021
2606499853,36cyclictest8474-21modprobe19:54:411
2606499853,35cyclictest4003-21/usr/sbin/munin21:15:281
2606499853,35cyclictest31443-21taskset22:55:051
2606499853,35cyclictest19028-21fschecks_count00:25:021
2606499853,35cyclictest13056-21ssh21:29:581
2606499853,35cyclictest13056-21ssh21:29:581
2606499853,34cyclictest12831-21modprobe22:24:411
2606499853,34cyclictest12831-21modprobe22:24:411
26063998538,41cyclictest0-21swapper/019:09:440
2606499844,34cyclictest346-21runrttasks23:53:491
26064998434,39cyclictest0-21swapper/122:44:471
26064998434,39cyclictest0-21swapper/122:44:471
2606499843,41cyclictest19894-21grep21:40:161
2606499843,36cyclictest917-21fschecks_time19:30:021
2606499843,36cyclictest25711-21/usr/sbin/munin23:40:171
2606499843,35cyclictest21575-21latency_hist20:34:431
2606499843,35cyclictest21575-21latency_hist20:34:431
2606499843,35cyclictest11516-21ssh21:26:341
26066998360,14cyclictest23109-21latency_hist20:39:432
26066998357,16cyclictest29639-21apt19:19:402
26066998357,16cyclictest29639-21apt19:19:402
2606499835,32cyclictest16111-21unixbench_multi20:15:181
2606499834,36cyclictest5264-21modprobe19:44:401
2606499834,34cyclictest27584-21modprobe19:14:411
2606499834,34cyclictest24711-21/usr/sbin/munin21:50:071
2606499834,34cyclictest24711-21/usr/sbin/munin21:50:071
2606499833,40cyclictest3207-21grep19:35:161
2606499833,37cyclictest15970-21/usr/sbin/munin00:20:541
2606499833,37cyclictest15970-21/usr/sbin/munin00:20:541
2606499833,36cyclictest23281-21/usr/sbin/munin20:40:131
2606499833,35cyclictest10576-21ssh00:10:131
26064998329,46cyclictest0-21swapper/100:00:061
26064998329,46cyclictest0-21swapper/100:00:061
26064998311,29cyclictest320-21dbus-daemon19:25:141
26064998311,29cyclictest320-21dbus-daemon19:25:141
2606399833,47cyclictest4024-21apt-get21:14:590
26063998312,37cyclictest24131-21seq23:38:570
26066998260,15cyclictest2-21kthreadd20:44:412
26066998258,18cyclictest3786-21apt-key21:14:452
2606699822,44cyclictest39-21ksoftirqd/221:14:562
2606499824,34cyclictest26775-21sh00:37:551
2606499824,31cyclictest10188-21/usr/sbin/munin23:15:031
2606499823,38cyclictest6436-21grep19:45:191
2606499823,36cyclictest16007-21/usr/sbin/munin23:25:181
2606499823,36cyclictest16007-21/usr/sbin/munin23:25:181
2606499823,35cyclictest13736-21/usr/sbin/munin23:20:081
2606499823,34cyclictest21929-21fschecks_time20:35:031
2606499823,28cyclictest6888-21latency_hist22:14:431
26063998262,14cyclictest18308-21latency_hist20:24:430
26063998262,14cyclictest18308-21latency_hist20:24:430
26063998258,16cyclictest519-21diskmemload21:55:090
26063998220,39cyclictest320-21dbus-daemon00:24:410
26063998220,39cyclictest320-21dbus-daemon00:24:410
26066998157,13cyclictest0-21swapper/223:29:442
26066998153,19cyclictest6857-21apt-key00:04:452
2606499813,37cyclictest7560-21/usr/sbin/munin21:20:071
2606499813,36cyclictest30986-21grep21:00:411
26066998061,13cyclictest6423-21sh23:08:002
26066998056,16cyclictest0-21swapper/223:40:052
26066998051,21cyclictest28229-21fschecks_count23:46:032
26066998051,21cyclictest28229-21fschecks_count23:46:032
26066998051,21cyclictest13395-21latency21:30:072
26066998051,21cyclictest13395-21latency21:30:072
26066998013,50cyclictest717-21diskmemload22:58:202
2606499803,9cyclictest0-21swapper/123:54:561
26063998037,27cyclictest0-21swapper/022:50:000
26063998037,27cyclictest0-21swapper/022:50:000
26063998014,28cyclictest12707-21ssh22:24:230
26063998014,28cyclictest12707-21ssh22:24:230
26066997957,15cyclictest7305-21fschecks_time21:20:012
2606499794,35cyclictest10760-21latency20:00:061
2606499794,28cyclictest1469-21fschecks_time22:05:041
2606499794,28cyclictest1469-21fschecks_time22:05:041
2606499792,30cyclictest8810-21ls19:54:571
2606499792,30cyclictest8810-21ls19:54:571
26066997847,22cyclictest346-21runrttasks20:50:042
2606699782,33cyclictest39-21ksoftirqd/221:05:392
26063997854,18cyclictest320-21dbus-daemon22:19:420
26066997758,13cyclictest519-21diskmemload22:51:112
26066997758,13cyclictest519-21diskmemload22:51:112
26066997751,19cyclictest3572-40ogrotate23:59:412
26066997750,14cyclictest0-21swapper/219:59:442
26066997750,14cyclictest0-21swapper/219:59:442
26063997756,15cyclictest27673-21latency_hist22:49:430
26063997756,15cyclictest27673-21latency_hist22:49:430
2606399773,40cyclictest1206-21fschecks_count23:55:030
26063997723,22cyclictest0-21swapper/000:35:060
26066997656,12cyclictest26059-21cyclictest00:00:582
2606699765,46cyclictest39-21ksoftirqd/219:20:582
2606699765,46cyclictest39-21ksoftirqd/219:20:582
26066997654,16cyclictest10175-21fschecks_count00:10:032
26066997653,17cyclictest26059-21cyclictest20:30:042
26066997653,17cyclictest26059-21cyclictest20:30:042
26066997652,18cyclictest1513-21fschecks_time22:05:042
26066997652,18cyclictest1513-21fschecks_time22:05:042
26063997649,19cyclictest8601-21latency_hist19:54:430
26063997634,37cyclictest0-21swapper/023:05:140
2606399762,34cyclictest2039-21munin-plugin-st19:34:400
26066997557,13cyclictest26508-21sh22:46:182
26066997557,13cyclictest26508-21sh22:46:182
26066997546,21cyclictest10630-21kernelversion23:15:062
2606699754,42cyclictest39-21ksoftirqd/219:44:352
26063997537,26cyclictest20061-21apt-get20:29:530
26063997537,26cyclictest20061-21apt-get20:29:530
2606399753,39cyclictest21892-21apt-get00:29:520
2606399753,39cyclictest21892-21apt-get00:29:520
26066997456,12cyclictest329-21systemd-logind22:24:422
26066997456,12cyclictest329-21systemd-logind22:24:422
26063997454,14cyclictest346-21runrttasks19:42:010
26066997352,15cyclictest27721-21cpu21:54:562
26066997351,10cyclictest0-21swapper/220:24:432
26066997351,10cyclictest0-21swapper/220:24:432
26066997346,19cyclictest15439-21latency_hist00:19:442
26063997320,8cyclictest0-21swapper/022:12:080
26063997310,23cyclictest1918-21/usr/sbin/munin23:00:060
26066997250,16cyclictest19509-21ntp_offset23:30:102
2606699724,24cyclictest39-21ksoftirqd/220:18:332
2606699722,39cyclictest39-21ksoftirqd/223:28:132
26063997256,11cyclictest26059-21cyclictest21:13:370
26063997241,20cyclictest19222-21apt-get22:35:060
26063997238,14cyclictest0-21swapper/020:15:100
2606699715,30cyclictest39-21ksoftirqd/222:40:422
2606699715,30cyclictest39-21ksoftirqd/222:40:422
26066997140,22cyclictest12738-21latency_hist21:29:432
26066997133,24cyclictest519-21diskmemload00:29:582
26066997133,24cyclictest519-21diskmemload00:29:582
2606699712,38cyclictest39-21ksoftirqd/222:29:502
26066997118,35cyclictest39-21ksoftirqd/221:47:032
26063997139,23cyclictest18927-21ssh23:29:530
2606399712,39cyclictest16879-21unixbench_multi21:35:200
2606399712,33cyclictest29824-21latency_hist20:59:440
26063997116,47cyclictest24731-21ssh23:39:510
26063997115,22cyclictest11878-21seq21:27:360
26066997047,13cyclictest0-21swapper/219:59:582
26066997045,12cyclictest0-21swapper/219:14:442
2606699703,55cyclictest30684-21ssh21:59:532
26063997051,13cyclictest23801-21ntp_kernel_pll_20:40:090
26063997047,16cyclictest872-21latency_hist22:04:440
2606399703,35cyclictest10966-21grep00:10:190
2606399703,27cyclictest14097-21ssh00:15:460
26063997013,22cyclictest7193-21seq00:05:160
26066996949,10cyclictest0-21swapper/200:29:442
2606699692,56cyclictest29993-21apt-get21:00:002
26063996945,15cyclictest519-21diskmemload22:30:110
2606399692,29cyclictest20538-21runrttasks21:42:010
26063996916,42cyclictest29926-21/usr/sbin/munin19:19:510
26063996916,42cyclictest29926-21/usr/sbin/munin19:19:510
26066996846,12cyclictest0-21swapper/219:39:432
26066996838,22cyclictest16257-21cstates00:20:462
26066996838,22cyclictest16257-21cstates00:20:462
2606699683,27cyclictest39-21ksoftirqd/221:51:592
2606699683,27cyclictest39-21ksoftirqd/221:51:592
26066996824,8cyclictest0-21swapper/221:43:522
26063996842,20cyclictest31904-21kworker/u17:3+nfsiod22:25:420
2606399683,54cyclictest7868-21ssh21:20:140
2606399683,54cyclictest10095-21munin-run19:59:410
2606399683,54cyclictest10095-21munin-run19:59:410
26063996822,9cyclictest33022chrt19:35:280
26063996816,42cyclictest20167-21ssh00:26:130
26063996811,22cyclictest22901-21seq20:38:290
2607499672,54cyclictest30023-21chrt21:00:073
26066996752,9cyclictest13738-21sh23:20:082
2606699673,37cyclictest39-21ksoftirqd/222:12:522
26066996722,21cyclictest39-21ksoftirqd/222:25:252
26063996748,13cyclictest14291-21switchtime20:10:140
2606399673,48cyclictest31594-21grep23:50:190
2606399673,46cyclictest13545-21latency_hist20:09:440
2606399673,45cyclictest25827-21tail20:45:170
26063996728,11cyclictest0-21swapper/022:07:470
26063996728,11cyclictest0-21swapper/022:07:470
2606399671,34cyclictest0-21swapper/021:44:560
2606699665,21cyclictest39-21ksoftirqd/219:27:472
2606699665,21cyclictest39-21ksoftirqd/219:27:472
26066996645,13cyclictest0-21swapper/220:45:172
26066996644,15cyclictest0-21swapper/219:49:412
2606699664,25cyclictest39-21ksoftirqd/223:10:552
26066996642,13cyclictest0-21swapper/219:30:152
26066996639,19cyclictest14353-21threads20:10:142
2606399667,31cyclictest14332-21rm21:31:090
2606399667,31cyclictest14332-21rm21:31:090
26063996641,19cyclictest9885-21kworker/u17:1+nfsiod00:01:200
26063996641,19cyclictest9885-21kworker/u17:1+nfsiod00:01:200
26063996639,20cyclictest2316-21kworker/u17:0+nfsiod23:11:400
26063996638,17cyclictest31041-21apt-get22:54:560
2606399663,30cyclictest736-21latency_hist21:09:440
26063996615,12cyclictest0-21swapper/023:27:030
26063996615,12cyclictest0-21swapper/023:27:030
2483426611,35sleep30-21swapper/319:06:053
26066996549,10cyclictest9654-21systemd-run00:09:402
26066996546,8cyclictest181rcu_preempt21:36:132
26066996546,12cyclictest0-21swapper/220:04:562
26066996541,17cyclictest4107-21latency_hist23:04:432
26066996536,8cyclictest181rcu_preempt22:37:072
2606699652,26cyclictest39-21ksoftirqd/200:35:102
26063996545,14cyclictest30181-21entropy21:00:160
2606699643,36cyclictest39-21ksoftirqd/219:53:262
2606699642,38cyclictest39-21ksoftirqd/223:36:562
2606399643,29cyclictest14192-21head23:20:160
2606399636,24cyclictest14850irq/292-2188000.ethernet21:51:100
2606399636,24cyclictest14850irq/292-2188000.ethernet21:51:100
2606399633,39cyclictest27341-21timerwakeupswit20:50:170
2606399633,31cyclictest27823-21latency_hist19:14:450
2606699625,21cyclictest39-21ksoftirqd/220:54:472
26066996228,8cyclictest181rcu_preempt20:26:392
26066996228,8cyclictest181rcu_preempt20:26:392
2606399623,29cyclictest10465-21fschecks_count23:15:030
26063996134,19cyclictest10379-21apt-get19:59:540
2606399612,24cyclictest29216-21unixbench_multi19:15:190
2606399612,24cyclictest29216-21unixbench_multi19:15:190
2607499582,39cyclictest0-21swapper/322:34:553
26074995742,8cyclictest451rcuc/322:09:283
26074995742,8cyclictest451rcuc/322:09:283
2607499572,35cyclictest40672sleep321:14:583
26074995714,31cyclictest0-21swapper/322:50:243
26074995714,31cyclictest0-21swapper/322:50:243
224922562,6sleep30-21swapper/323:35:123
26074995539,9cyclictest451rcuc/319:16:193
26074995539,9cyclictest451rcuc/319:16:193
2607499553,35cyclictest2038-21chrt23:00:093
2607499552,40cyclictest10181-21chrt22:19:453
2607499542,33cyclictest18441-21chrt20:24:523
2607499542,33cyclictest18441-21chrt20:24:523
26074995338,9cyclictest129492sleep321:29:533
26074995338,9cyclictest129492sleep321:29:533
2607499533,37cyclictest10091-21chrt19:59:413
2607499533,37cyclictest10091-21chrt19:59:413
2607499533,36cyclictest26707-21chrt00:37:433
2607499524,37cyclictest21474-21chrt21:44:403
2607499523,35cyclictest19785-21chrt23:30:163
2607499523,34cyclictest12260-21taskset00:13:543
2607499517,35cyclictest16458-21taskset22:30:063
2607499503,35cyclictest586-21chrt23:54:293
2607499503,34cyclictest13774-21chrt23:20:113
2607499503,33cyclictest21374-21chrt00:29:403
2607499502,38cyclictest22932-21taskset00:31:023
2607499502,38cyclictest22932-21taskset00:31:023
2607499502,36cyclictest31739-21taskset19:25:023
2607499502,36cyclictest31739-21taskset19:25:023
2607499502,35cyclictest17951-21chrt21:38:083
2607499502,34cyclictest1198-21chrt22:59:403
26074994935,8cyclictest451rcuc/320:12:573
26074994933,9cyclictest451rcuc/320:02:083
2607499493,33cyclictest7005-21chrt00:04:463
2607499493,33cyclictest31301-21chrt22:00:123
2607499493,33cyclictest2981-21chrt23:57:573
2607499492,35cyclictest26220-21chrt22:45:273
2607499492,35cyclictest26220-21chrt22:45:273
2607499483,34cyclictest7394-21chrt23:09:583
2607499483,34cyclictest6482-21chrt19:45:193
2607499483,32cyclictest2574-21taskset19:35:023
2607499483,31cyclictest12221-21chrt23:18:003
2607499482,35cyclictest10492-21chrt22:20:223
2607499482,35cyclictest10492-21chrt22:20:223
2607499482,34cyclictest22776-21chrt21:45:173
2607499482,33cyclictest7282-21taskset21:20:013
2607499473,31cyclictest4839-21chrt19:40:173
2607499464,32cyclictest5080-21taskset23:05:113
2607499463,33cyclictest18758-21chrt23:29:433
2607499463,33cyclictest18758-21chrt23:29:433
2607499463,31cyclictest3528-21chrt21:14:413
26074994530,8cyclictest451rcuc/320:21:403
26074994530,8cyclictest451rcuc/320:21:403
2607499452,30cyclictest14267-21taskset00:16:173
2607499443,31cyclictest27158-21chrt21:54:043
2607499443,31cyclictest27158-21chrt21:54:043
2607499443,29cyclictest27571-21chrt23:44:433
2607499443,27cyclictest18498-21taskset00:24:423
2607499443,27cyclictest18498-21taskset00:24:423
2607499443,25cyclictest20090-21chrt20:29:583
2607499443,25cyclictest20090-21chrt20:29:583
2607499442,30cyclictest22327-21chrt22:40:003
2607499442,30cyclictest22327-21chrt22:40:003
2607499433,27cyclictest29550-21chrt21:57:293
2607499433,27cyclictest10888-21taskset21:25:163
26074994329,8cyclictest451rcuc/319:33:573
26074994327,11cyclictest451rcuc/320:45:003
2607499432,28cyclictest6368-21chrt00:04:173
2607499432,28cyclictest6368-21chrt00:04:173
2607499423,29cyclictest5906-21chrt22:12:113
2607499423,28cyclictest29099-21taskset20:55:173
2607499414,29cyclictest28516-21chrt23:46:103
2607499414,29cyclictest28516-21chrt23:46:103
2607499413,26cyclictest12932-21chrt20:05:163
2607499412,24cyclictest24865-21chrt20:44:443
2607499403,28cyclictest27618-21taskset19:14:413
2607499403,25cyclictest7146-21taskset19:49:533
26074994025,9cyclictest451rcuc/321:07:443
2607499393,26cyclictest27845-21taskset20:54:403
2607499383,27cyclictest22008-21chrt20:35:043
2607499383,26cyclictest16204-21taskset20:15:243
2607499363,24cyclictest30488-21taskset19:20:123
2607499363,24cyclictest30488-21taskset19:20:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional