You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 09:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Thu Apr 02, 2026 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2183421457,18sleep00-21swapper/018:59:360
228619912650,54cyclictest2234-21latency_hist19:39:052
22847991223,24cyclictest20584-21apt-get00:24:200
22847991196,29cyclictest25733-21/usr/sbin/munin22:44:120
21864211512,73sleep121895-21sed18:59:371
228619910941,38cyclictest30031-21diskmemload23:32:222
22847991093,22cyclictest11449-21taskset00:09:150
228619910837,36cyclictest0-21swapper/219:19:332
228619910837,36cyclictest0-21swapper/219:19:332
22847991083,26cyclictest32580-21ntpq23:49:310
22847991083,26cyclictest32580-21ntpq23:49:310
228619910735,35cyclictest0-21swapper/219:24:172
228619910735,35cyclictest0-21swapper/219:24:172
22847991073,31cyclictest9248-21grep23:09:390
22847991073,26cyclictest5220-21cpu23:04:170
22847991072,29cyclictest27586-21ntpq20:54:300
228479910650,49cyclictest0-21swapper/019:14:020
228479910645,54cyclictest22839-21cyclictest23:37:340
228479910645,54cyclictest22839-21cyclictest23:37:340
22847991063,35cyclictest352-21runrttasks22:19:300
22847991062,29cyclictest4994-21ssh23:58:310
22847991053,28cyclictest1-21systemd19:25:300
22847991053,28cyclictest1-21systemd19:25:300
228619910339,39cyclictest30031-21diskmemload23:51:522
228619910339,39cyclictest30031-21diskmemload23:51:522
228479910343,50cyclictest0-21swapper/019:49:020
22847991033,42cyclictest20657-21netstat23:29:370
228479910232,38cyclictest181rcu_preempt21:49:230
228479910228,67cyclictest0-21swapper/022:00:270
228479910228,67cyclictest0-21swapper/022:00:270
228619910149,43cyclictest5617-21cstates23:59:182
22861991003,50cyclictest25799-21ssh22:44:242
22847991008,11cyclictest15350irq/295-2188000.ethernet21:04:130
228479910042,51cyclictest0-21swapper/000:19:000
22847991003,23cyclictest28052-21latency_hist19:19:020
22847991002,27cyclictest2208-21/usr/sbin/munin22:59:350
22847991002,27cyclictest2208-21/usr/sbin/munin22:59:350
22847991002,25cyclictest23395-21timerandwakeup22:39:370
22855999967,24cyclictest20676-21kworker/u18:1+nfsiod23:10:401
2284799994,34cyclictest8999-21taskset00:04:260
2284799994,29cyclictest886-21/usr/sbin/munin21:09:270
2284799994,29cyclictest886-21/usr/sbin/munin21:09:270
2284799994,28cyclictest352-21runrttasks22:06:510
2284799994,28cyclictest352-21runrttasks22:06:510
2284799993,34cyclictest26314-21cut23:39:220
22847999916,10cyclictest0-21swapper/020:44:310
2284799984,30cyclictest15006-21sh21:32:310
2284799984,30cyclictest15006-21sh21:32:310
2284799983,31cyclictest16927-21cat23:24:040
2284799983,25cyclictest15791-21fschecks_time20:19:240
2284799983,24cyclictest5454-21switchtime22:09:380
22847999817,28cyclictest471-21sendmail19:34:020
22861999733,31cyclictest0-21swapper/223:55:052
22861999730,35cyclictest0-21swapper/222:09:042
22861999730,35cyclictest0-21swapper/222:09:042
2284799973,28cyclictest352-21runrttasks23:28:380
2284799973,28cyclictest352-21runrttasks23:28:380
2284799973,27cyclictest27914-21apt-get21:54:120
2284799973,27cyclictest27914-21apt-get21:54:120
22861999662,22cyclictest13850-21ssh23:19:022
2284799963,32cyclictest18576-21latency_hist21:39:010
2284799963,25cyclictest19917-21unixbench_multi21:39:390
2284799963,25cyclictest19917-21unixbench_multi21:39:390
2284799962,34cyclictest2504-1kworker/0:0H+mmc_complete20:09:050
2284799962,34cyclictest2504-1kworker/0:0H+mmc_complete20:09:050
22855999571,17cyclictest2575-21ssh22:59:261
22855999571,17cyclictest2575-21ssh22:59:261
22847999542,45cyclictest0-21swapper/021:04:040
2284799953,30cyclictest973-21tune2fs19:34:230
2284799952,21cyclictest6533-21unixbench_singl23:59:370
2286199943,78cyclictest0-21swapper/222:34:132
22847999439,46cyclictest0-21swapper/000:24:050
2284799943,30cyclictest2538-21grep19:39:230
2284799943,30cyclictest12591-21awk21:29:020
2284799943,26cyclictest16870-21fschecks_time22:29:230
22847999431,54cyclictest0-21swapper/023:14:180
22847999417,23cyclictest14651-21grep22:24:390
22861999343,38cyclictest0-21swapper/222:29:372
22861999327,35cyclictest0-21swapper/219:09:212
22855999376,10cyclictest27626-21modprobe21:54:021
2284799934,82cyclictest19882-21grep22:34:240
2284799934,28cyclictest352-21runrttasks20:50:220
2284799934,28cyclictest352-21runrttasks20:50:220
2284799934,27cyclictest20351-21/usr/sbin/munin20:34:280
2284799934,26cyclictest29654-21modprobe19:24:020
2284799934,26cyclictest29654-21modprobe19:24:020
2284799933,31cyclictest18801-21sort20:29:190
2284799933,31cyclictest10404-21latency_hist22:19:040
2284799933,31cyclictest10404-21latency_hist22:19:040
2284799933,26cyclictest31419-21modprobe22:54:020
2284799933,26cyclictest23016-21/usr/sbin/munin19:04:310
22847999312,10cyclictest0-21swapper/020:09:150
22847999312,10cyclictest0-21swapper/020:09:150
22861999237,36cyclictest0-21swapper/221:09:402
22861999237,36cyclictest0-21swapper/221:09:402
2284799923,34cyclictest7302-21/usr/sbin/munin21:19:260
2284799923,20cyclictest15171-21chrt20:18:070
2284799923,20cyclictest15171-21chrt20:18:070
22847999219,21cyclictest22358-21latency21:44:270
22847999210,9cyclictest0-21swapper/019:54:140
22861999146,38cyclictest298-21in:imuxsock19:44:032
22861999123,29cyclictest0-21swapper/221:14:182
2284799913,29cyclictest29725-21find23:44:350
2284799913,27cyclictest24381-21ssh00:29:350
2284799913,27cyclictest24381-21ssh00:29:350
2284799913,25cyclictest9442-21grep19:59:350
2284799913,25cyclictest23638-21latency_hist20:44:040
22847999113,21cyclictest3936-21sh21:14:140
2286199902,81cyclictest0-21swapper/219:49:142
2286199902,81cyclictest0-21swapper/219:49:142
2284799902,28cyclictest32729-21grep22:54:380
22861998949,34cyclictest352-21runrttasks21:44:292
22861998943,40cyclictest11123-21needreboot22:19:292
2286199893,61cyclictest20031-21apt-get00:24:162
22847998933,49cyclictest0-21swapper/020:24:180
2284799893,31cyclictest5594-21grep19:49:190
2284799893,31cyclictest5594-21grep19:49:190
22861998858,20cyclictest13893-21sh22:24:242
22861998849,32cyclictest0-21swapper/200:09:292
2286199883,75cyclictest0-21swapper/222:49:132
22855998872,10cyclictest2400-21latency_hist23:54:051
22855998872,10cyclictest2400-21latency_hist23:54:051
2285599883,46cyclictest4653-21/usr/sbin/munin21:14:351
22861998644,35cyclictest0-21swapper/223:04:292
22861998565,15cyclictest23753-21latency_hist20:44:052
22861998544,35cyclictest2952-21systemd-journal20:08:492
22861998544,35cyclictest2952-21systemd-journal20:08:492
22861998541,38cyclictest30031-21diskmemload21:59:592
22861998541,38cyclictest30031-21diskmemload21:59:592
2286199852,71cyclictest0-21swapper/220:29:152
22861998467,11cyclictest10198-21fschecks_time21:24:252
22861998441,37cyclictest8790-21sh22:14:462
22861998441,37cyclictest8790-21sh22:14:462
2286199843,73cyclictest0-21swapper/223:19:162
2286199843,70cyclictest0-21swapper/200:19:102
22861998341,36cyclictest352-21runrttasks23:37:362
22861998341,36cyclictest352-21runrttasks23:37:362
22861998341,36cyclictest30031-21diskmemload00:29:482
22861998341,36cyclictest30031-21diskmemload00:29:482
2286199833,67cyclictest28680-21apt-get20:59:132
2286199832,76cyclictest0-21swapper/220:19:172
2286199832,75cyclictest0-21swapper/221:29:142
2286199832,75cyclictest0-21swapper/221:29:142
2286199832,72cyclictest22538-21ssh22:39:162
2286199832,70cyclictest0-21swapper/220:49:142
2286199832,70cyclictest0-21swapper/220:49:142
22855998358,19cyclictest29433-21threads20:59:361
22855998355,9cyclictest181rcu_preempt19:59:051
22861998253,20cyclictest585-21latency_hist19:34:042
22861998242,34cyclictest7749-21sh19:54:332
22855998259,16cyclictest12050-21kworker/u17:3-xprtiod00:29:151
22855998259,16cyclictest12050-21kworker/u17:3-xprtiod00:29:151
22855998250,14cyclictest181rcu_preempt23:29:291
22861998151,21cyclictest22148-21apt-key20:39:062
22861998137,38cyclictest0-21swapper/221:34:262
22861998130,43cyclictest0-21swapper/221:04:062
2286199812,70cyclictest0-21swapper/219:59:142
22855998168,9cyclictest32707-21kworker/u19:3+flush-179:022:44:261
22855998156,18cyclictest22144-21fschecks_count21:44:231
22861998034,38cyclictest0-21swapper/221:49:212
22855998058,15cyclictest25187-21latency_hist22:44:041
22855998055,18cyclictest1890-21sh22:04:241
22855998055,18cyclictest1890-21sh22:04:241
22855998054,11cyclictest181rcu_preempt22:29:031
2286199793,71cyclictest0-21swapper/223:44:162
22861997935,37cyclictest24481-21mailstats20:44:342
22861997835,35cyclictest24376-21latency_hist19:09:042
2286199782,71cyclictest0-21swapper/219:44:152
2286199782,62cyclictest0-21swapper/220:14:182
2286199782,62cyclictest0-21swapper/220:14:182
22855997856,14cyclictest17722-21kworker/u17:3-xprtiod23:35:141
22855997856,14cyclictest17722-21kworker/u17:3-xprtiod23:35:141
2286199774,64cyclictest26045-21apt-get23:39:132
2286199773,68cyclictest0-21swapper/223:24:152
2286199773,68cyclictest0-21swapper/223:24:152
22855997751,20cyclictest17386-21fschecks_time20:24:241
22855997750,10cyclictest181rcu_preempt20:54:161
22861997638,31cyclictest0-21swapper/223:12:032
2286199763,62cyclictest22865-21latency_hist19:04:062
2286199763,61cyclictest0-21swapper/220:54:162
22861997635,35cyclictest7098-21sh21:19:222
22855997660,11cyclictest14597-21latency23:19:261
22855997649,18cyclictest32468-21sh21:07:291
22855997630,12cyclictest0-21swapper/119:29:121
22855997626,37cyclictest25436-21apt-get20:49:161
22855997626,37cyclictest25436-21apt-get20:49:161
22861997531,37cyclictest20439-21sh21:40:532
22861997531,37cyclictest20439-21sh21:40:532
22861997531,37cyclictest0-21swapper/222:54:122
22855997561,10cyclictest22839-21cyclictest22:00:381
22855997561,10cyclictest22839-21cyclictest22:00:381
22855997548,10cyclictest181rcu_preempt22:54:321
2285599754,51cyclictest15761-21tune2fs20:19:231
22855997542,15cyclictest29313-21ssh22:49:311
22855997541,13cyclictest181rcu_preempt19:09:331
22861997450,16cyclictest28790-21sh21:54:372
22861997450,16cyclictest28790-21sh21:54:372
22861997441,23cyclictest13609-21cron20:14:022
22861997441,23cyclictest13609-21cron20:14:022
22855997454,13cyclictest10989-21unixbench_singl21:24:401
2285599745,35cyclictest17770-21kworker/u19:1+flush-179:019:49:241
2285599745,35cyclictest17770-21kworker/u19:1+flush-179:019:49:241
22855997452,13cyclictest0-21swapper/123:39:371
2285599742,52cyclictest21277-21tail20:34:371
213882746,48sleep30-21swapper/318:59:183
2286199734,53cyclictest26971-21fschecks_time19:14:232
22861997336,30cyclictest0-21swapper/200:04:352
22855997346,11cyclictest181rcu_preempt22:33:021
22861997224,41cyclictest17539-21cat20:24:272
22861997223,43cyclictest0-21swapper/200:14:212
2285599726,57cyclictest20584-21apt-get00:24:161
22855997238,9cyclictest181rcu_preempt20:39:151
22855997235,13cyclictest181rcu_preempt21:59:011
22855997235,13cyclictest181rcu_preempt21:59:011
22855997230,12cyclictest181rcu_preempt20:44:391
2285599722,28cyclictest31-21ksoftirqd/119:19:151
2285599722,28cyclictest31-21ksoftirqd/119:19:151
22855997222,32cyclictest19495-21ntpq21:39:311
22855997222,32cyclictest19495-21ntpq21:39:311
2286899713,53cyclictest15023-21taskset21:32:333
2286899713,53cyclictest15023-21taskset21:32:333
2286899712,55cyclictest5627-21taskset21:16:193
22855997140,13cyclictest181rcu_preempt20:14:331
22855997140,13cyclictest181rcu_preempt20:14:331
22855997139,9cyclictest181rcu_preempt00:10:341
22855997136,13cyclictest181rcu_preempt22:13:491
2285599713,43cyclictest8361-21latency_hist00:04:051
22855997126,19cyclictest11771-21apt-get20:09:151
22855997126,19cyclictest11771-21apt-get20:09:151
2286899703,52cyclictest16273-21chrt20:19:363
2286899703,52cyclictest1570-21chrt22:04:113
2286899703,52cyclictest1570-21chrt22:04:113
22868997013,37cyclictest0-21swapper/300:24:163
2286199704,56cyclictest3603-21ssh23:00:262
2286199704,56cyclictest3603-21ssh23:00:262
2286199703,56cyclictest4651-21/usr/sbin/munin22:09:132
22855997055,10cyclictest22839-21cyclictest19:37:171
22855997046,18cyclictest0-21swapper/120:09:041
22855997046,18cyclictest0-21swapper/120:09:041
2285599704,43cyclictest17355-21/usr/sbin/munin00:19:341
2286899694,50cyclictest27469-21chrt19:14:353
2286899692,54cyclictest5757-21chrt23:04:293
2286899692,53cyclictest7657-21taskset19:54:293
22855996930,12cyclictest181rcu_preempt23:06:181
22855996930,12cyclictest181rcu_preempt22:14:221
22855996930,12cyclictest181rcu_preempt22:14:221
2286899683,54cyclictest20092-21chrt23:29:213
2286899683,54cyclictest20092-21chrt23:29:213
2286899683,51cyclictest778-21taskset21:09:053
2286899683,51cyclictest0-21swapper/321:44:163
2286899682,53cyclictest27142-21chrt23:39:383
2286899682,52cyclictest6876-21taskset22:12:503
2286899682,52cyclictest18218-21chrt22:31:143
2286899682,51cyclictest19182-21taskset20:29:283
22855996849,15cyclictest17156-21kworker/u19:0+events_unbound22:39:011
22855996834,13cyclictest181rcu_preempt23:54:351
22855996831,13cyclictest7710-21ssh21:19:371
22855996828,16cyclictest181rcu_preempt19:14:241
2286899673,52cyclictest26241-21chrt22:45:163
2286899673,51cyclictest19095-21chrt21:39:223
2286899673,51cyclictest19095-21chrt21:39:223
2286899673,50cyclictest15005-21taskset22:25:353
2286899672,52cyclictest3748-21chrt19:44:043
2286899672,52cyclictest2709-21taskset21:11:373
2286899672,52cyclictest2709-21taskset21:11:373
2286899672,52cyclictest15695-21taskset23:20:583
2286899672,52cyclictest13267-21taskset23:17:303
2286899672,51cyclictest3480-21chrt22:06:383
2286899672,51cyclictest3480-21chrt22:06:383
2286899672,51cyclictest18628-21chrt00:20:023
22855996748,12cyclictest0-21swapper/119:44:401
2285599673,44cyclictest15736-21ssh21:34:051
2285599673,44cyclictest15736-21ssh21:34:051
22855996733,8cyclictest12964-21ssh23:16:391
22855996731,13cyclictest181rcu_preempt23:46:401
2286899664,48cyclictest28896-21chrt21:54:383
2286899664,48cyclictest28896-21chrt21:54:383
2286899663,51cyclictest1945-21chrt23:53:243
2286899663,51cyclictest1945-21chrt23:53:243
2286899663,50cyclictest28354-21chrt20:58:283
2286899663,50cyclictest23654-21chrt19:04:353
2286899663,50cyclictest10832-21chrt23:14:013
2286899662,51cyclictest5183-21chrt19:47:323
2286899662,51cyclictest25600-21chrt23:38:413
2286899662,51cyclictest25600-21chrt23:38:413
2286899662,51cyclictest23726-21chrt00:29:183
2286899662,51cyclictest23726-21chrt00:29:183
2286899662,51cyclictest2050-21taskset19:38:373
2286899662,51cyclictest13842-21chrt00:13:223
2286899662,50cyclictest26365-21chrt21:50:413
2286899662,49cyclictest9812-21chrt21:24:053
2286899662,49cyclictest31313-21chrt22:53:553
22855996649,10cyclictest0-21swapper/100:04:311
22855996643,15cyclictest14798-21if_eth000:14:241
2285599663,47cyclictest352-21runrttasks21:36:091
2285599663,16cyclictest10588-21ssh22:19:121
2286899653,50cyclictest12281-21chrt20:09:193
2286899653,50cyclictest12281-21chrt20:09:193
2286899653,49cyclictest20899-21chrt20:34:303
2286899653,48cyclictest30603-21chrt23:46:033
2286899653,47cyclictest19520-21chrt22:34:103
2286899652,51cyclictest2719-21chrt23:54:213
2286899652,50cyclictest797-21chrt22:55:543
2286899652,50cyclictest6725-21taskset00:00:013
2286899652,50cyclictest25443-21chrt20:49:173
2286899652,50cyclictest25443-21chrt20:49:173
2286899652,50cyclictest17133-21chrt20:24:143
2286899652,50cyclictest14912-21chrt20:14:463
2286899652,50cyclictest14912-21chrt20:14:463
2286899652,49cyclictest30460-21chrt19:24:323
2286899652,49cyclictest30460-21chrt19:24:323
2286899652,49cyclictest14418-21taskset00:14:113
22855996543,17cyclictest22839-21cyclictest20:04:041
22855996534,10cyclictest181rcu_preempt20:29:141
22855996524,13cyclictest181rcu_preempt23:26:171
22855996524,13cyclictest181rcu_preempt23:26:171
2286899643,48cyclictest6853-21chrt19:53:233
2286899643,48cyclictest6853-21chrt19:53:233
2286899643,48cyclictest32744-21chrt19:31:573
2286899643,47cyclictest24122-21chrt22:41:033
2286899642,50cyclictest3467-21chrt23:00:033
2286899642,50cyclictest3467-21chrt23:00:033
2286899642,50cyclictest16041-21chrt21:34:193
2286899642,49cyclictest9996-21chrt20:02:023
2286899642,49cyclictest11071-21taskset20:04:333
2286899642,49cyclictest11071-21taskset20:04:333
2285599648,12cyclictest181rcu_preempt21:09:231
2285599648,12cyclictest181rcu_preempt21:09:231
2285599643,43cyclictest29951-21cpu19:24:171
2285599643,43cyclictest29951-21cpu19:24:171
2285599643,32cyclictest3313-21grep19:39:381
87522630,3sleep3451rcuc/322:14:413
87522630,3sleep3451rcuc/322:14:413
2286899633,48cyclictest28688-21chrt20:59:173
2286899633,47cyclictest24956-21chrt20:46:253
2286899632,51cyclictest21269-21chrt23:29:543
2286899632,49cyclictest24881-21chrt19:09:133
2286899632,48cyclictest9374-21chrt00:04:343
2286899632,48cyclictest22271-21chrt20:39:533
2286899632,48cyclictest12429-21chrt22:21:423
2285599624,42cyclictest24377-21idleruntime-cro19:09:021
2286899613,45cyclictest29313-21chrt19:19:463
2286899613,45cyclictest29313-21chrt19:19:463
2286899612,48cyclictest11215-21taskset21:25:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional