You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 13:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun Nov 30, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10296991204,39cyclictest26939-21/usr/sbin/munin21:24:190
10296991204,39cyclictest26939-21/usr/sbin/munin21:24:190
10047211410,75sleep010059-21modprobe19:08:520
10296991134,29cyclictest20174-21taskset21:13:010
10296991084,29cyclictest24566-21timerandwakeup21:19:300
103489910645,13cyclictest0-21swapper/220:04:122
103489910644,15cyclictest0-21swapper/223:18:512
103489910644,15cyclictest0-21swapper/223:18:512
10296991063,75cyclictest19033-21ssh00:04:090
103489910546,14cyclictest24174-21fschecks_count19:49:152
10296991044,24cyclictest2930-21/usr/sbin/munin22:39:260
10296991044,24cyclictest2930-21/usr/sbin/munin22:39:260
10296991043,27cyclictest7348-21latency_hist21:48:550
103489910240,13cyclictest0-21swapper/222:24:132
103489910240,13cyclictest0-21swapper/222:24:132
103489910220,35cyclictest26314-21latency_hist21:23:552
103269910277,19cyclictest19579-21sh00:05:061
102969910237,60cyclictest0-21swapper/019:18:530
10296991023,32cyclictest20362-21rm23:09:210
102969910211,10cyclictest16955-21cat21:04:290
102969910136,55cyclictest0-21swapper/019:23:540
10296991012,23cyclictest30414-21/usr/sbin/munin20:09:240
102969910120,76cyclictest0-21swapper/019:39:060
10296991003,34cyclictest22095-21fschecks_time00:09:150
10296991003,29cyclictest32529-21latency_hist00:28:530
10296991002,24cyclictest1937-21ssh21:38:520
1029699993,40cyclictest31412-21snmpd23:08:540
1029699993,40cyclictest31412-21snmpd23:08:540
1029699993,32cyclictest10114-21munin-plugin-st21:53:510
1029699993,27cyclictest2878-21ps21:39:260
1029699993,27cyclictest2878-21ps21:39:260
1029699993,22cyclictest21661-21ssh22:14:060
1029699992,27cyclictest32567-21latency_hist22:33:540
1034899983,41cyclictest7649-21/usr/sbin/munin21:49:172
1034899983,38cyclictest17170-21unixbench_multi21:04:312
10296999843,47cyclictest27917-21ssh23:23:510
1029699983,21cyclictest32611-21ssh23:30:540
10296999830,59cyclictest0-21swapper/023:44:280
10296999811,21cyclictest5787-21apt-get20:33:540
1029699974,29cyclictest27386-21fschecks_count22:24:130
1029699974,29cyclictest27386-21fschecks_count22:24:130
1029699973,81cyclictest0-21swapper/023:54:190
1029699973,26cyclictest460-21fschecks_time22:34:130
1029699973,13cyclictest0-21swapper/019:34:270
246362960,6sleep10-21swapper/119:49:261
1034899963,54cyclictest30876-21proc_pri00:24:272
1029699963,30cyclictest26432-21timerwakeupswit19:54:280
1029699963,30cyclictest26432-21timerwakeupswit19:54:280
1029699963,22cyclictest24965-21rm22:19:210
10296999627,58cyclictest0-21swapper/022:09:020
10296999627,58cyclictest0-21swapper/022:09:020
1029699962,28cyclictest355-21runrttasks00:36:320
1029699962,26cyclictest25431-21cat19:53:550
1029699962,22cyclictest25959-21ssh00:15:230
1029699954,22cyclictest29655-21/usr/sbin/munin21:29:180
1029699953,32cyclictest31412-21snmpd20:34:400
1029699953,32cyclictest2173-21head20:19:300
1029699953,30cyclictest18632-21latency_hist00:03:540
1029699953,29cyclictest355-21runrttasks23:52:190
1029699953,29cyclictest355-21runrttasks23:52:190
1029699953,22cyclictest355-21runrttasks21:58:420
1029699953,22cyclictest355-21runrttasks21:58:420
1029699953,20cyclictest15913-21apt-get19:24:040
10296999528,59cyclictest0-21swapper/023:34:150
1029699952,26cyclictest32023-21/usr/sbin/munin20:14:170
1029699943,42cyclictest8517-21taskset22:49:000
1029699942,28cyclictest23442-21latency_hist21:18:540
1029699942,27cyclictest8856-21grep20:39:310
1029699942,27cyclictest18701-21latency_hist22:08:540
1034899934,38cyclictest21635-21/usr/sbin/munin22:14:172
1034899933,31cyclictest12748-21cpuspeed19:14:122
1029699934,30cyclictest9763-21fschecks_time20:44:160
1029699934,26cyclictest12977-21/usr/sbin/munin20:54:420
10296999335,50cyclictest0-21swapper/020:04:270
1029699933,44cyclictest11915-21/usr/sbin/munin22:54:210
1029699933,44cyclictest11915-21/usr/sbin/munin22:54:210
1029699933,33cyclictest14891-21processes22:59:260
1029699933,29cyclictest6808-21latency_hist23:43:530
1034899923,41cyclictest24518-21cstates22:19:102
1029699924,29cyclictest28242-21/usr/sbin/munin23:24:310
1029699923,18cyclictest207-21systemd-journal23:18:510
1029699923,18cyclictest207-21systemd-journal23:18:510
1029699922,33cyclictest28758-21latency_hist20:03:550
1029699922,30cyclictest2824-21/usr/sbin/munin20:24:270
1029699922,26cyclictest17501-21/usr/sbin/munin19:29:230
10296999217,11cyclictest0-21swapper/019:48:540
1034899912,28cyclictest355-21runrttasks21:56:372
1034899912,28cyclictest355-21runrttasks21:56:372
10326999166,13cyclictest27089-21kworker/u17:3+xprtiod23:49:091
10326999166,13cyclictest27089-21kworker/u17:3+xprtiod23:49:091
1029699914,30cyclictest32681-21/usr/sbin/munin00:29:280
1029699913,31cyclictest10888-21fschecks_time19:09:280
10296999129,54cyclictest0-21swapper/020:59:300
10296999129,54cyclictest0-21swapper/020:59:300
10296999116,26cyclictest15568-21diskmemload22:02:580
10326999059,21cyclictest2879-21latency_hist22:38:551
10326999026,45cyclictest0-21swapper/121:34:291
1029699903,32cyclictest5726-21/usr/sbin/munin22:44:270
1029699902,27cyclictest11134-21/usr/sbin/munin20:49:170
1029699893,21cyclictest27966-21/usr/sbin/munin00:19:200
1029699893,21cyclictest27966-21/usr/sbin/munin00:19:200
1034899883,42cyclictest900-21grep22:34:242
10348998768,14cyclictest10276-21cyclictest20:33:532
10348998664,15cyclictest32111-21ssh21:34:142
10326998664,15cyclictest3019-21sh21:39:291
10326998664,15cyclictest3019-21sh21:39:291
10326998420,29cyclictest3461-21ssh23:37:221
10326998268,9cyclictest0-21swapper/120:54:171
10326998261,15cyclictest3264-21fschecks_time22:39:151
10326998261,15cyclictest3264-21fschecks_time22:39:151
10326998234,35cyclictest29932-21ssh22:29:041
10348998161,14cyclictest10276-21cyclictest22:31:492
10348998063,11cyclictest10495-21sshd23:49:222
10348998063,11cyclictest10495-21sshd23:49:222
10348998058,15cyclictest10656-21cpu19:09:232
1032699803,58cyclictest12426-21apt-get23:54:061
10326998033,39cyclictest0-21swapper/120:09:091
10348997912,32cyclictest6323-21sshd20:34:212
10326997936,32cyclictest5745-21apt-get22:44:091
10348997842,14cyclictest181rcu_preempt20:39:072
10348997745,17cyclictest17367-21ssh00:00:362
10326997753,15cyclictest20952-21kworker/u17:1-rpciod21:33:531
1032699772,49cyclictest5787-21apt-get20:34:051
10326997720,26cyclictest11950-21ssh22:54:221
10326997720,26cyclictest11950-21ssh22:54:221
10348997660,10cyclictest26431-21unixbench_singl23:19:292
10326997656,14cyclictest355-21runrttasks00:14:221
10326997648,19cyclictest6811-21latency_hist23:43:531
1006127612,40sleep110083-21munin-run19:08:531
10348997560,9cyclictest39-21ksoftirqd/200:14:212
10348997554,14cyclictest2388-21ssh23:34:282
1036499743,39cyclictest5752-21chrt22:44:053
10348997457,10cyclictest1671-21kworker/u17:0-xprtiod00:34:292
10348997451,16cyclictest20748-21sh21:14:032
10348997449,13cyclictest0-21swapper/223:13:522
10348997443,10cyclictest181rcu_preempt23:44:052
10326997456,11cyclictest5872-21apt-key20:33:521
1032699743,53cyclictest18448-21kworker/u18:2+flush-179:021:54:151
1032699743,53cyclictest18448-21kworker/u18:2+flush-179:021:54:151
1036499732,38cyclictest14522-21chrt22:59:163
10348997351,12cyclictest0-21swapper/222:59:272
1032699737,26cyclictest28752-21ssh00:20:151
1032699737,26cyclictest28752-21ssh00:20:151
10326997355,12cyclictest309-21systemd-logind19:33:531
10326997351,12cyclictest0-21swapper/123:18:531
10326997351,12cyclictest0-21swapper/123:18:531
1032699733,23cyclictest21335-21grep19:39:261
10348997251,15cyclictest1-21systemd23:58:512
10326997250,14cyclictest20905-21cstates21:14:111
10326997233,33cyclictest0-21swapper/122:59:041
1034899715,46cyclictest28558-21latency_hist20:03:542
10348997151,13cyclictest31412-21snmpd21:25:532
10348997151,13cyclictest31412-21snmpd21:25:532
10326997137,26cyclictest32706-21ssh00:29:031
1032699712,30cyclictest26895-21ssh21:24:181
1032699712,30cyclictest26895-21ssh21:24:181
10326997117,23cyclictest15294-21mailstats20:59:271
10326997117,23cyclictest15294-21mailstats20:59:271
1036499703,37cyclictest19310-21taskset23:08:243
1036499703,37cyclictest19310-21taskset23:08:243
1036499702,26cyclictest27218-21chrt19:59:083
1034899705,43cyclictest355-21runrttasks21:45:192
10348997043,19cyclictest691-21cron21:33:512
1034899703,46cyclictest19108-21/usr/sbin/munin19:34:232
10326997049,11cyclictest0-21swapper/123:29:091
10326997046,13cyclictest0-21swapper/122:18:551
10326997032,34cyclictest0-21swapper/121:49:071
10326997010,26cyclictest0-21swapper/123:44:241
1036499693,29cyclictest5777-21chrt20:33:513
1036499692,33cyclictest30327-21taskset23:27:363
10348996955,10cyclictest10276-21cyclictest19:29:182
1034899695,44cyclictest355-21runrttasks20:56:372
10348996950,12cyclictest31176-21ssh23:29:072
10348996946,12cyclictest0-21swapper/222:44:162
10348996939,11cyclictest0-21swapper/220:59:172
10348996939,11cyclictest0-21swapper/220:59:172
1034899693,45cyclictest4188-21idleruntime-cro20:28:522
10348996928,9cyclictest181rcu_preempt20:49:072
10348996920,36cyclictest22650-21cat00:09:282
10326996952,10cyclictest27605-21latency22:24:171
10326996952,10cyclictest27605-21latency22:24:171
10326996948,15cyclictest1-21systemd20:17:561
10326996939,21cyclictest8834-21grep20:39:311
1032699693,37cyclictest20666-21latency_hist21:13:551
10326996932,16cyclictest19631-21latency_hist23:08:541
10326996932,16cyclictest19631-21latency_hist23:08:541
10326996927,31cyclictest0-21swapper/100:34:071
1036499683,52cyclictest22523-21chrt19:44:143
1036499683,33cyclictest24785-21chrt22:19:173
1036499682,33cyclictest17524-21taskset19:29:073
1036499682,31cyclictest12262-21chrt20:51:043
10348996851,11cyclictest11985-21rm22:54:242
10348996851,11cyclictest11985-21rm22:54:242
1034899683,45cyclictest8824-21fschecks_time22:49:152
1034899683,44cyclictest2576-21idleruntime21:39:172
1034899683,44cyclictest2576-21idleruntime21:39:172
1034899683,43cyclictest30753-21latency_hist23:28:542
1034899683,42cyclictest17800-21switchtime23:04:272
1034899683,42cyclictest17800-21switchtime23:04:272
1034899683,39cyclictest19547-21fschecks_time00:05:052
10326996845,14cyclictest25757-21fschecks_count19:54:141
10326996845,14cyclictest25757-21fschecks_count19:54:141
10326996839,21cyclictest12679-21cat20:53:541
1036499674,51cyclictest13194-21chrt20:54:283
1036499673,42cyclictest0-21swapper/323:54:063
1036499673,32cyclictest14021-21chrt21:59:243
1036499672,34cyclictest13893-21taskset22:58:523
1036499672,34cyclictest13893-21taskset22:58:523
1036499672,31cyclictest13204-21chrt19:14:253
10348996748,12cyclictest58-21kcompactd020:09:222
10348996743,13cyclictest0-21swapper/219:44:302
10348996729,13cyclictest181rcu_preempt19:57:102
10348996729,13cyclictest181rcu_preempt19:57:102
1032699676,45cyclictest17601-21diskmemload23:59:201
10326996750,11cyclictest3462-21grep20:24:251
10326996745,12cyclictest0-21swapper/122:49:151
10326996744,16cyclictest28473-21fschecks_count23:24:151
10326996713,25cyclictest21818-21ssh00:09:051
10326996713,23cyclictest292-21dbus-daemon19:48:531
10326996713,22cyclictest12218-21munin-node20:23:531
1036499663,33cyclictest1045-21taskset21:36:103
1036499663,32cyclictest3456-21chrt22:39:193
1036499663,32cyclictest3456-21chrt22:39:193
1036499662,33cyclictest30529-21taskset22:29:223
1036499662,32cyclictest21268-21taskset23:10:473
1036499662,29cyclictest8935-21chrt20:39:423
1034899664,39cyclictest16012-21cstates19:24:102
1034899664,30cyclictest355-21runrttasks19:23:062
10348996640,8cyclictest39-21ksoftirqd/223:42:252
10348996630,12cyclictest181rcu_preempt22:02:312
1034899662,43cyclictest20556-21latency_hist21:13:542
10348996620,36cyclictest19959-21grep22:09:302
10348996620,36cyclictest19959-21grep22:09:302
1032699668,28cyclictest0-21swapper/119:09:261
10326996631,29cyclictest0-21swapper/122:04:091
10326996610,28cyclictest0-21swapper/121:59:151
88872658,31sleep28890-21diskstats19:04:172
1036499653,50cyclictest3707-21chrt23:38:103
1036499653,32cyclictest3381-21chrt20:24:233
1036499653,31cyclictest20306-21chrt00:05:213
1036499653,30cyclictest9171-21chrt21:50:533
1036499653,28cyclictest15269-21taskset20:59:273
1036499653,28cyclictest15269-21taskset20:59:273
1036499652,32cyclictest785-21chrt20:17:023
1036499652,26cyclictest26142-21chrt23:19:263
10348996551,10cyclictest10276-21cyclictest00:19:232
10348996551,10cyclictest10276-21cyclictest00:19:232
10348996538,10cyclictest181rcu_preempt22:05:222
1034899653,42cyclictest355-21runrttasks20:23:162
1034899653,40cyclictest20952-21fschecks_count19:39:152
1034899653,34cyclictest2930-21/usr/sbin/munin22:39:162
1034899653,34cyclictest2930-21/usr/sbin/munin22:39:162
1034899652,35cyclictest2606-21latency_hist00:33:532
10326996545,14cyclictest1-21systemd19:15:181
1032699653,39cyclictest29718-21grep20:04:301
10326996526,30cyclictest30480-21cut00:24:161
10326996520,33cyclictest7186-21ssh21:48:191
10326996518,32cyclictest16777-21ntp_offset21:04:231
10326996511,24cyclictest27936-21munin-run23:23:511
1036499643,49cyclictest31439-21chrt23:29:143
1036499643,30cyclictest2374-21chrt20:21:283
1036499643,29cyclictest27451-21chrt00:19:073
1036499643,29cyclictest27451-21chrt00:19:073
1036499642,32cyclictest11132-21chrt22:53:533
1036499642,31cyclictest27568-21taskset21:24:393
1036499642,31cyclictest27568-21taskset21:24:393
1036499642,31cyclictest22494-21chrt23:13:553
1036499642,31cyclictest22494-21chrt23:13:553
10326996441,13cyclictest0-21swapper/119:24:181
10326996431,22cyclictest14306-21apt-get19:19:041
10326996430,22cyclictest19126-21apt-get19:34:051
10326996423,30cyclictest9622-21ls20:44:111
1036499634,47cyclictest20078-21taskset22:09:453
1036499634,47cyclictest20078-21taskset22:09:453
1036499633,47cyclictest10732-21chrt21:54:183
1036499633,47cyclictest10732-21chrt21:54:183
1036499633,35cyclictest6199-21taskset23:42:033
1036499633,30cyclictest30190-21chrt00:24:013
1036499633,30cyclictest28845-21chrt22:26:303
1036499633,30cyclictest28845-21chrt22:26:303
1036499633,30cyclictest2435-21chrt21:39:143
1036499633,30cyclictest2435-21chrt21:39:143
1036499633,30cyclictest24091-21taskset00:13:153
1036499632,32cyclictest17864-21chrt22:06:293
1036499632,31cyclictest22049-21chrt22:14:193
1036499632,30cyclictest8330-21chrt23:45:033
1036499632,30cyclictest387-21chrt22:34:123
1036499632,29cyclictest25584-21chrt19:54:063
1036499632,29cyclictest25584-21chrt19:54:063
1036499632,29cyclictest19716-21chrt21:11:383
1036499632,29cyclictest17356-21taskset21:05:593
1036499632,29cyclictest1607-21chrt00:30:293
10348996340,13cyclictest0-21swapper/220:14:152
1034899633,41cyclictest355-21runrttasks20:44:052
10326996349,8cyclictest0-21swapper/122:19:291
10326996334,22cyclictest19766-21apt-get23:09:041
10326996321,28cyclictest27396-21fschecks_count19:59:171
1036499624,28cyclictest18135-21chrt00:02:543
1036499623,30cyclictest6472-21chrt21:45:513
1036499623,28cyclictest9682-21chrt20:44:123
1036499623,28cyclictest15917-21chrt19:24:053
1036499622,34cyclictest30266-21taskset21:29:383
1036499622,30cyclictest25699-21taskset21:22:323
1036499622,30cyclictest11975-21taskset23:52:323
1036499622,30cyclictest11975-21taskset23:52:323
1036499622,29cyclictest19129-21chrt19:34:053
1036499622,22cyclictest6803-21chrt20:34:403
10326996245,12cyclictest25535-21sh21:22:001
10326996240,12cyclictest0-21swapper/122:09:291
10326996240,12cyclictest0-21swapper/122:09:291
1036499613,28cyclictest27280-21chrt00:18:553
1036499613,27cyclictest31909-21chrt20:13:543
1036499613,27cyclictest15578-21chrt19:23:123
1036499612,28cyclictest21481-21taskset19:39:283
1036499603,28cyclictest4306-21chrt00:35:563
1036499602,27cyclictest24980-21chrt19:49:453
1036499602,26cyclictest22073-21chrt21:15:063
1036499584,25cyclictest29822-21chrt20:04:353
1036499582,26cyclictest11663-21chrt19:10:083
1036499553,40cyclictest0-21swapper/319:08:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional