You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-18 - 18:51
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Fri Apr 18, 2025 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15682991414,75cyclictest22812-21ssh10:10:172
15682991414,75cyclictest22812-21ssh10:10:172
15677991412,59cyclictest19822-21taskset07:20:201
15673991332,93cyclictest567-21ssh11:30:330
15127213117,83sleep115209-21grep07:05:471
156739913060,63cyclictest26231-21grep09:14:530
156739912858,62cyclictest373-21snmpd09:30:310
15682991236,71cyclictest26966-21apt-get11:20:092
156869911672,22cyclictest371-21ntpd09:30:103
15686991164,86cyclictest32290-21apt-get11:30:093
15673991164,99cyclictest13791-21taskset10:55:110
15673991164,99cyclictest13791-21taskset10:55:110
15682991137,97cyclictest22687-21apt-get10:10:212
156829911257,15cyclictest181rcu_preempt12:32:172
156739911138,12cyclictest0-21swapper/008:55:320
156739911128,24cyclictest22917-21cpu10:10:280
156739911124,24cyclictest4908-21grep08:10:390
15673991094,41cyclictest29900-21fschecks_count12:20:150
15673991094,41cyclictest29900-21fschecks_count12:20:150
15677991083,33cyclictest21747-21cat11:10:151
156739910864,29cyclictest0-21swapper/007:25:090
156739910841,14cyclictest15663-21cyclictest12:34:570
15673991064,42cyclictest7757-21fschecks_count09:40:160
15673991064,42cyclictest7757-21fschecks_count09:40:160
15673991064,30cyclictest6848-21fschecks_count11:40:180
156779910543,39cyclictest0-21swapper/111:30:441
15677991053,86cyclictest22820-21latency_hist10:10:181
15677991053,86cyclictest22820-21latency_hist10:10:181
15673991054,40cyclictest13451-21cp10:54:560
15673991054,35cyclictest26750-21fschecks_time07:40:180
15673991043,40cyclictest27975-21fschecks_time11:20:390
2928321030,7sleep029280-21ssh09:19:580
156829910242,13cyclictest0-21swapper/207:15:202
156829910242,13cyclictest0-21swapper/207:15:202
156779910222,45cyclictest27537-21apt-get11:20:091
156779910221,33cyclictest12898-21unixbench_singl11:50:271
15677991013,38cyclictest24746-21latency_hist07:34:571
15677991013,32cyclictest18352-21unixbench_singl08:50:421
156829910073,18cyclictest316-21in:imuxsock12:39:522
156829910047,12cyclictest181rcu_preempt10:57:542
156829910047,12cyclictest181rcu_preempt10:57:542
15677991003,41cyclictest6610-21apt-get11:40:091
15677991003,34cyclictest19277-21apt07:19:531
15677991003,34cyclictest19277-21apt07:19:531
1567799994,39cyclictest32530-21idleruntime-cro10:29:531
1567799993,37cyclictest1476-21munin-plugin-st09:29:531
1567799983,37cyclictest26986-21lockfile-create11:19:541
15682999770,14cyclictest6857-21kworker/u17:2-ext4-rsv-conversion10:34:572
15682999770,14cyclictest6857-21kworker/u17:2-ext4-rsv-conversion10:34:572
1567799973,40cyclictest2857-21tail09:30:451
1567799973,38cyclictest7458-21fschecks_time08:20:201
1567799973,29cyclictest31738-21cut07:55:171
15682999655,13cyclictest25310-21kworker/u17:3+xprtiod09:15:052
1567799963,38cyclictest17229-21munin-plugin-st07:14:541
1567799963,38cyclictest17229-21munin-plugin-st07:14:541
1567799963,35cyclictest5585-21/usr/sbin/munin12:35:321
1567799963,29cyclictest30806-21grep10:24:561
1567799963,29cyclictest30199-21fschecks_time07:50:211
1567799963,29cyclictest30199-21fschecks_time07:50:211
15673999645,13cyclictest181rcu_preempt11:58:320
1568299952,62cyclictest19793-21apt-get07:20:202
1567799953,34cyclictest24464-21grep12:10:271
1567799953,34cyclictest24464-21grep12:10:271
15686999472,15cyclictest17782-21sh11:59:533
1567799943,43cyclictest27064-21wc07:40:281
1567799943,37cyclictest358-21runrttasks08:10:101
1567799943,35cyclictest22638-21grep07:25:541
1567799943,33cyclictest29776-21fschecks_time09:20:241
15673999445,24cyclictest1419550irq/304-mma8451_consumer009:24:560
153012944,14sleep20-21swapper/207:06:342
15677999337,11cyclictest0-21swapper/109:00:201
1567799933,36cyclictest5771-21grep08:15:171
1567799924,58cyclictest26493-21grep10:15:331
1567799924,58cyclictest26493-21grep10:15:331
1567799924,41cyclictest18913-21mailstats12:00:301
1567799924,41cyclictest18913-21mailstats12:00:301
1567799924,35cyclictest17723-21unixbench_singl11:00:341
1567799924,31cyclictest29203-21ssh09:19:561
1567799923,37cyclictest29674-21apt12:20:041
1567799923,37cyclictest29674-21apt12:20:041
1567799923,37cyclictest24166-21ssh09:10:201
15682999154,10cyclictest181rcu_preempt07:25:452
15682999153,14cyclictest181rcu_preempt10:29:552
1567799914,33cyclictest18566-21ssh10:00:351
1567799914,33cyclictest18566-21ssh10:00:351
1567799913,36cyclictest9958-21grep08:25:451
1567799913,36cyclictest9958-21grep08:25:451
1567799913,34cyclictest5685-21latency_hist10:39:551
1567799913,31cyclictest11758-21grep10:50:411
1567799912,36cyclictest32437-21latency_hist11:29:561
1567799912,33cyclictest23778-21timerandwakeup10:10:481
1568699902,66cyclictest21648-21modprobe07:25:093
15682999042,10cyclictest0-21swapper/211:29:562
1567799904,34cyclictest21092-21latency_hist11:09:561
1567799904,34cyclictest21092-21latency_hist11:09:561
1567799903,35cyclictest32764-21/usr/sbin/munin10:30:081
1567799903,35cyclictest32764-21/usr/sbin/munin10:30:081
1567799903,34cyclictest19893-21grep08:55:351
1567799903,33cyclictest27660-21vmstat12:15:341
1567799903,33cyclictest2171-21modprobe12:29:521
1567799903,31cyclictest13506-21grep08:35:351
1567799902,33cyclictest8533-21grep09:40:481
1567799902,33cyclictest8533-21grep09:40:481
1567399905,44cyclictest10915-21cat08:30:080
15686998924,26cyclictest0-21swapper/309:15:153
15682998945,15cyclictest181rcu_preempt09:21:062
1568299893,35cyclictest22460-21fschecks_time09:05:202
1567799894,32cyclictest14902-21unixbench_multi10:55:571
1567799894,32cyclictest14902-21unixbench_multi10:55:571
1567799894,31cyclictest4633-21/usr/sbin/munin09:35:241
1567799894,31cyclictest4633-21/usr/sbin/munin09:35:241
1567799894,31cyclictest17116-21idleruntime-cro08:49:531
15677998934,14cyclictest0-21swapper/107:35:351
15677998934,14cyclictest0-21swapper/107:35:351
1567799893,35cyclictest15025-21wc09:55:141
1567799893,35cyclictest15025-21wc09:55:141
1567799893,33cyclictest12254-21modprobe08:34:531
1567799893,32cyclictest12466-21fschecks_time09:50:321
1567799893,30cyclictest6486-21cut10:40:221
1567799893,30cyclictest6486-21cut10:40:221
1567799893,29cyclictest3298-21switchtime12:30:251
15677998924,29cyclictest17999-21date11:59:561
1567799892,35cyclictest14794-21sendmail_mailst08:40:311
1567799892,35cyclictest14794-21sendmail_mailst08:40:311
1568699883,66cyclictest10909-21/usr/sbin/munin08:30:083
15686998824,35cyclictest14144-21/usr/sbin/munin08:40:323
15686998824,35cyclictest14144-21/usr/sbin/munin08:40:323
1567799883,33cyclictest28592-21taskset07:45:321
1567799883,30cyclictest11800-21idleruntime-cro09:49:541
15677998814,40cyclictest31-21ksoftirqd/111:37:471
15677998814,40cyclictest31-21ksoftirqd/111:37:471
1568699873,52cyclictest18350-21grep08:50:423
15686998733,28cyclictest0-21swapper/307:19:583
15686998725,24cyclictest0-21swapper/309:09:543
15686998722,42cyclictest24729-21latency_hist07:34:573
1568299873,39cyclictest20318-21modprobe08:59:532
1567799873,33cyclictest2137-21latency_hist08:04:561
1568699863,73cyclictest13750-21apt-get10:55:293
1568699863,73cyclictest13750-21apt-get10:55:293
1568699863,43cyclictest358-21runrttasks12:09:273
15686998617,35cyclictest0-21swapper/311:15:293
15682998654,8cyclictest1419550irq/304-mma8451_consumer010:15:172
15682998654,8cyclictest1419550irq/304-mma8451_consumer010:15:172
15682998651,9cyclictest13804-21ssh11:52:412
15682998651,8cyclictest181rcu_preempt11:40:192
15682998638,13cyclictest181rcu_preempt08:00:242
15682998634,10cyclictest181rcu_preempt12:22:392
15682998634,10cyclictest181rcu_preempt12:22:392
1568299862,21cyclictest726-21df11:30:362
1567799863,44cyclictest23304-21cat12:09:571
1567799863,32cyclictest3038-21switchtime08:05:301
15673998665,13cyclictest373-21snmpd09:05:150
15673998647,12cyclictest23405-21ssh12:10:040
15673998647,12cyclictest23405-21ssh12:10:040
1567399862,53cyclictest16747-21grep08:45:360
15686998552,9cyclictest0-21swapper/309:59:533
15686998552,9cyclictest0-21swapper/309:59:533
15686998541,22cyclictest0-21swapper/308:05:323
1568699853,48cyclictest8759-21apt-key08:24:563
15686998523,42cyclictest19790-21seq11:06:263
15686998523,42cyclictest19790-21seq11:06:263
15682998550,8cyclictest181rcu_preempt10:47:312
1568299853,38cyclictest13370-21sed10:54:562
15682998526,28cyclictest1-21systemd12:04:542
15682998526,28cyclictest1-21systemd12:04:542
1567799852,31cyclictest10448-21cron10:49:531
15686998458,19cyclictest28702-21unixbench_multi11:20:543
15686998431,22cyclictest0-21swapper/312:00:033
15686998431,22cyclictest0-21swapper/312:00:033
15686998426,29cyclictest27411-21kworker/u17:0+xprtiod12:29:563
15686998411,34cyclictest11926-21ssh11:50:073
1568299848,34cyclictest0-21swapper/207:10:272
1568299848,34cyclictest0-21swapper/207:10:272
15673998445,31cyclictest0-21swapper/008:09:530
15673998439,24cyclictest181rcu_preempt08:00:460
1567399843,31cyclictest8525-21ssh12:39:550
1568699837,51cyclictest0-21swapper/310:10:283
15686998356,15cyclictest0-21swapper/309:10:363
1568299835,42cyclictest13448-21rm09:51:142
15682998347,13cyclictest181rcu_preempt07:45:422
1568299834,43cyclictest3281-21unixbench_singl08:05:322
15682998342,15cyclictest181rcu_preempt12:29:552
1568299833,48cyclictest24839-21ssh11:15:222
1568299833,47cyclictest10683-21latency_hist08:29:562
1568299833,47cyclictest10683-21latency_hist08:29:562
1568299833,46cyclictest23077-21modprobe12:09:532
1568299833,44cyclictest30196-21fschecks_time07:50:212
1568299833,44cyclictest30196-21fschecks_time07:50:212
1568299833,35cyclictest17592-21cut08:50:222
15682998325,13cyclictest1419550irq/304-mma8451_consumer012:15:172
1567799834,33cyclictest358-21runrttasks11:45:421
1567799833,32cyclictest23833-21latency_hist09:09:591
1567399833,40cyclictest13774-21runrttasks11:52:360
1567399833,31cyclictest5767-21fschecks_count08:15:170
15686998261,13cyclictest311-21systemd-logind08:59:543
15686998260,15cyclictest1-21systemd07:19:543
15686998241,19cyclictest0-21swapper/311:00:353
15686998234,29cyclictest15663-21cyclictest09:41:363
15686998234,29cyclictest15663-21cyclictest09:41:363
15686998221,19cyclictest0-21swapper/310:42:163
15686998221,19cyclictest0-21swapper/310:42:163
15682998258,8cyclictest181rcu_preempt10:20:122
15682998252,10cyclictest181rcu_preempt10:40:092
15682998252,10cyclictest181rcu_preempt10:40:092
1568299823,39cyclictest0-21swapper/208:45:132
15673998267,9cyclictest15-21ksoftirqd/009:57:400
15673998267,9cyclictest15-21ksoftirqd/009:57:400
1567399825,49cyclictest32427-21grep07:55:320
15673998240,24cyclictest17721-21kworker/u17:2+xprtiod11:10:150
1568699814,47cyclictest26270-21latency_hist12:14:553
1568699814,47cyclictest26270-21latency_hist12:14:553
15686998142,17cyclictest1419550irq/304-mma8451_consumer008:45:363
15686998132,24cyclictest12562-21kworker/u18:1+events_unbound07:54:543
15686998132,24cyclictest12562-21kworker/u18:1+events_unbound07:54:543
15686998124,23cyclictest358-21runrttasks07:57:053
15682998158,13cyclictest0-21swapper/207:34:572
15682998154,20cyclictest15663-21cyclictest11:45:442
15682998143,10cyclictest181rcu_preempt11:58:492
1568299813,44cyclictest6188-21mailstats08:15:292
15682998114,58cyclictest381ktimers/208:45:102
15682998114,58cyclictest381ktimers/208:45:102
1567399818,49cyclictest22819-21latency_hist10:10:180
1567399818,49cyclictest22819-21latency_hist10:10:180
1567399817,37cyclictest0-21swapper/010:27:270
1567399814,26cyclictest13873-21apt08:39:530
1567399812,40cyclictest8569-21ssh10:44:530
1567399812,40cyclictest8569-21ssh10:44:530
15686998059,14cyclictest11723-21latency_hist11:49:573
15686998035,12cyclictest12169-21ssh10:51:493
1568699802,48cyclictest5585-21/usr/sbin/munin12:35:163
1568699802,34cyclictest20905-21fschecks_time09:00:213
15686998023,46cyclictest19793-21apt-get07:20:143
15686998018,27cyclictest5262-21ntpq09:35:333
15686998018,27cyclictest5262-21ntpq09:35:333
1568299805,38cyclictest22108-21latency_hist09:04:572
15682998045,10cyclictest1419550irq/304-mma8451_consumer011:14:572
1568299804,36cyclictest26410-21grep09:14:582
1568299803,49cyclictest20471-21latency_hist10:04:572
1568299803,49cyclictest20471-21latency_hist10:04:572
15682998032,15cyclictest1419550irq/304-mma8451_consumer011:00:132
1567399803,55cyclictest14449-21fschecks_time08:40:200
1567399803,55cyclictest14449-21fschecks_time08:40:200
1567399803,43cyclictest11683-21latency_hist11:49:560
42742790,15sleep347-21ksoftirqd/309:34:443
15686997954,12cyclictest0-21swapper/309:24:583
1568699793,52cyclictest2462-21/usr/sbin/munin12:30:133
15686997919,28cyclictest30304-21sh12:20:243
15686997919,28cyclictest30304-21sh12:20:243
15682997941,18cyclictest181rcu_preempt09:31:162
15682997940,13cyclictest181rcu_preempt09:36:242
15682997940,13cyclictest181rcu_preempt09:36:242
1568299793,63cyclictest10992-21cpuspeed08:30:142
1568299793,44cyclictest7435-21fschecks_time08:20:192
15682997933,11cyclictest181rcu_preempt09:42:532
15682997933,11cyclictest181rcu_preempt09:42:532
15682997930,12cyclictest181rcu_preempt09:56:292
15682997930,12cyclictest181rcu_preempt09:56:292
15673997952,21cyclictest3757-21kworker/u18:0-rpciod10:00:520
15673997952,21cyclictest3757-21kworker/u18:0-rpciod10:00:520
15673997941,23cyclictest30878-21ssh11:26:070
15673997936,12cyclictest0-21swapper/007:15:340
15673997936,12cyclictest0-21swapper/007:15:340
1567399793,53cyclictest5743-21unixbench_singl09:35:420
1567399793,53cyclictest5743-21unixbench_singl09:35:420
1567399793,33cyclictest11807-21munin-plugin-st09:49:540
149772796,49sleep315036-21timerandwakeup07:05:453
15686997855,16cyclictest15663-21cyclictest07:44:573
1568699784,38cyclictest13879-21awk08:39:533
1568699783,49cyclictest373-21snmpd10:04:403
1568699783,49cyclictest373-21snmpd10:04:403
15686997829,30cyclictest0-21swapper/308:29:553
15686997829,30cyclictest0-21swapper/308:29:553
15686997818,29cyclictest0-21swapper/309:50:103
15682997856,7cyclictest181rcu_preempt09:25:552
15682997850,8cyclictest181rcu_preempt07:44:252
15682997850,12cyclictest181rcu_preempt08:14:572
1568299782,44cyclictest25770-21sed07:35:362
1568299782,44cyclictest25770-21sed07:35:362
1567399783,40cyclictest9683-21mailstats08:25:420
1567399783,40cyclictest9683-21mailstats08:25:420
1567399783,38cyclictest30845-21cat10:24:570
1567399782,39cyclictest28299-21latency_hist10:19:570
1567399782,39cyclictest28299-21latency_hist10:19:570
1568699774,44cyclictest7028-21latency_hist08:19:573
1568699773,50cyclictest25485-21mailstats07:35:333
1568699773,50cyclictest25485-21mailstats07:35:333
15686997719,14cyclictest17231-21modprobe07:14:543
15686997719,14cyclictest17231-21modprobe07:14:543
15682997755,8cyclictest181rcu_preempt11:05:412
15682997755,8cyclictest181rcu_preempt11:05:412
15682997743,16cyclictest181rcu_preempt11:39:572
15673997740,11cyclictest27411-21kworker/u17:0-ext4-rsv-conversion11:39:570
1567399773,42cyclictest2371-21latency_hist12:29:560
1567399773,36cyclictest20773-21head07:20:470
15686997654,12cyclictest0-21swapper/310:30:273
15686997654,12cyclictest0-21swapper/310:30:273
1568699764,44cyclictest3441-21ssh10:35:153
1568699764,39cyclictest358-21runrttasks10:49:133
1568699762,43cyclictest6735-21cstates11:40:143
15686997623,27cyclictest4173-21grep08:10:213
15686997618,34cyclictest0-21swapper/310:20:363
15682997660,11cyclictest0-21swapper/207:55:312
1568299763,38cyclictest13445-21tail08:35:342
15682997627,14cyclictest181rcu_preempt11:35:192
15682997626,14cyclictest181rcu_preempt12:10:062
15682997626,14cyclictest181rcu_preempt12:10:062
15673997650,20cyclictest23572-21diskmemload12:00:280
15673997650,20cyclictest23572-21diskmemload12:00:280
1567399763,49cyclictest5918-21latency_hist10:39:570
1567399763,43cyclictest13257-21grep09:50:470
15673997627,9cyclictest1419550irq/304-mma8451_consumer008:24:560
15673997625,28cyclictest1419550irq/304-mma8451_consumer009:00:340
1568699753,46cyclictest22005-21ps11:10:233
1568699753,41cyclictest358-21runrttasks10:07:083
1568699753,41cyclictest358-21runrttasks10:07:083
1568699753,36cyclictest1325-21cut08:00:423
15673997560,10cyclictest15-21ksoftirqd/011:02:220
1567399753,40cyclictest23860-21processes07:30:310
1568699743,41cyclictest358-21runrttasks11:35:043
1568699743,41cyclictest358-21runrttasks11:35:043
15686997418,28cyclictest0-21swapper/309:49:223
15686997413,47cyclictest0-21swapper/307:45:213
15686997413,30cyclictest27423-21cat12:15:313
1567399743,39cyclictest28303-21/usr/sbin/munin07:45:380
1567399742,47cyclictest4493-21unixbench_multi11:35:310
1567399742,40cyclictest25139-21threads11:15:290
15686997323,40cyclictest358-21runrttasks10:18:183
15686997323,40cyclictest358-21runrttasks10:18:183
15682997343,8cyclictest181rcu_preempt10:35:502
15673997345,9cyclictest181rcu_preempt12:08:440
15673997336,13cyclictest181rcu_preempt10:30:130
15673997336,13cyclictest181rcu_preempt10:30:130
1568299722,31cyclictest11843-21latency_hist09:49:562
1567399728,16cyclictest0-21swapper/007:10:170
1567399728,16cyclictest0-21swapper/007:10:170
15673997250,7cyclictest181rcu_preempt12:17:110
1567399724,37cyclictest358-21runrttasks08:50:210
15686997125,22cyclictest0-21swapper/311:29:553
1567399712,39cyclictest26236-21modprobe07:39:540
1567399712,39cyclictest26236-21modprobe07:39:540
1567399712,23cyclictest10669-21latency_hist10:49:560
15686997026,20cyclictest0-21swapper/310:27:143
1567399703,35cyclictest1470-21latency_hist09:29:560
1567399702,40cyclictest30638-21switchtime07:50:330
1567399702,40cyclictest30638-21switchtime07:50:330
1441927012,29sleep00-21swapper/007:05:260
1567399693,26cyclictest21006-21modprobe11:09:530
1567399693,26cyclictest21006-21modprobe11:09:530
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional