You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 22:52
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Nov 18, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22234215313,101sleep222298-21fschecks_time07:05:042
22234215313,101sleep222298-21fschecks_time07:05:042
23543991023,33cyclictest27985-21diskmemload11:59:342
23543991023,33cyclictest27985-21diskmemload11:59:342
23527999955,17cyclictest181rcu_preempt09:51:210
23527999955,17cyclictest181rcu_preempt09:51:210
23527999662,12cyclictest181rcu_preempt07:59:540
231852966,16sleep10-21swapper/107:07:021
231852966,16sleep10-21swapper/107:07:021
2354399953,8cyclictest0-21swapper/210:09:542
2354399953,8cyclictest0-21swapper/210:09:542
23534999477,10cyclictest30476-21diskmemload09:29:401
2354399912,31cyclictest19567-21diskmemload09:47:412
23527999164,21cyclictest30421-21latency_hist07:29:400
23543999048,37cyclictest23520-21cyclictest12:22:462
23543999048,37cyclictest23520-21cyclictest12:22:462
23527999051,16cyclictest181rcu_preempt12:26:020
23527999048,19cyclictest181rcu_preempt12:20:180
23527999048,19cyclictest181rcu_preempt12:20:180
2352799894,40cyclictest18899-21apt-get10:45:290
2354399883,39cyclictest2220-21expr09:15:152
2354399882,29cyclictest355-21runrttasks10:41:282
2354399874,74cyclictest0-21swapper/208:44:502
23543998740,40cyclictest0-21swapper/210:25:062
2354399873,9cyclictest0-21swapper/211:04:532
2354399873,9cyclictest0-21swapper/211:04:532
2354399873,8cyclictest0-21swapper/211:09:512
2354399873,40cyclictest355-21runrttasks07:15:092
2354399872,30cyclictest5664-21threads07:50:572
2354399872,30cyclictest3046150ssh12:03:562
232062877,63sleep00-21swapper/007:07:180
232062877,63sleep00-21swapper/007:07:180
2354399863,71cyclictest0-21swapper/208:19:502
2354399863,40cyclictest30474-21cron09:09:372
2354399863,31cyclictest25175-21fschecks_time10:55:002
2354399863,31cyclictest25175-21fschecks_time10:55:002
23527998664,16cyclictest16099-21latency_hist08:24:400
2337928612,48sleep30-21swapper/307:09:373
2337928612,48sleep30-21swapper/307:09:373
2354399854,64cyclictest10932-21ssh10:29:552
2354399854,36cyclictest26598-21grep08:55:152
2354399854,36cyclictest26598-21grep08:55:152
2354399854,35cyclictest27499-21/usr/sbin/munin09:00:142
2354399853,38cyclictest3261-21grep07:45:002
2354399853,37cyclictest6324-21ssh11:20:002
2354399853,37cyclictest6324-21ssh11:20:002
2354399852,40cyclictest355-21runrttasks10:35:192
2354399852,40cyclictest355-21runrttasks10:35:192
23534998555,21cyclictest1437-21sh11:10:131
23527998561,15cyclictest30476-21diskmemload11:22:150
23527998561,15cyclictest30476-21diskmemload11:22:150
2354399845,68cyclictest0-21swapper/207:19:492
23543998449,8cyclictest181rcu_preempt11:01:332
2354399843,73cyclictest8943-21ssh11:24:552
2354399843,38cyclictest5456-21cut10:20:002
2354399843,36cyclictest7429-21grep07:55:142
2354399843,36cyclictest23930-21taskset11:50:232
2354399843,35cyclictest31818-21ls12:05:142
2354399843,35cyclictest13046-21/usr/sbin/munin09:35:172
2354399843,35cyclictest13046-21/usr/sbin/munin09:35:172
2354399843,34cyclictest16596-21cstates12:34:592
23527998447,12cyclictest181rcu_preempt10:01:240
23527998440,14cyclictest181rcu_preempt10:58:210
23527998440,14cyclictest181rcu_preempt10:58:210
2354399834,36cyclictest15098-21ssh11:35:202
2354399833,8cyclictest0-21swapper/210:14:542
2354399833,38cyclictest20427-21irqstats11:45:042
2354399833,38cyclictest20427-21irqstats11:45:042
2354399833,38cyclictest16277-21cat12:34:392
2354399833,35cyclictest23050-21apt-key09:54:392
2354399833,35cyclictest23050-21apt-key09:54:392
2354399833,33cyclictest12904-21/usr/sbin/munin08:14:502
2354399833,32cyclictest12646-21modprobe08:14:362
2354399833,30cyclictest355-21runrttasks08:28:422
2354399833,29cyclictest11878-21awk09:34:372
2354399833,10cyclictest0-21swapper/210:05:392
23534998360,17cyclictest13311-21sh09:35:311
23534998360,17cyclictest13311-21sh09:35:311
23534998332,12cyclictest181rcu_preempt12:19:391
23527998356,11cyclictest181rcu_preempt12:05:410
2354399823,8cyclictest0-21swapper/211:14:502
2354399823,39cyclictest17177-21ssh09:43:542
2354399823,39cyclictest17177-21ssh09:43:542
2354399823,38cyclictest7272-21ssh09:25:062
2354399823,38cyclictest7047-21/usr/sbin/munin12:19:372
2354399823,37cyclictest17380-21ssh11:39:572
2354399823,37cyclictest10922-21runrttasks12:25:052
2354399823,36cyclictest9692-21/usr/sbin/munin08:05:152
2354399823,35cyclictest23471-21/usr/sbin/munin09:55:442
2354399823,35cyclictest23471-21/usr/sbin/munin09:55:442
2354399823,34cyclictest8686-21uname08:00:172
2354399823,33cyclictest30981-21sh09:09:432
2354399823,27cyclictest21112-21cstates08:39:562
2354399822,38cyclictest21762-21idleruntime-cro10:49:362
2354399822,37cyclictest20718-21latency_hist08:39:392
23534998235,26cyclictest0-21swapper/110:35:251
23534998235,26cyclictest0-21swapper/110:35:251
23527998258,16cyclictest16290-21ssh12:34:390
23527998258,10cyclictest181rcu_preempt07:32:110
2352799824,69cyclictest19152-21sh12:39:360
2352799823,45cyclictest10604-21apt-config10:29:400
2354399813,33cyclictest29011-21runrttasks07:24:542
2354399813,11cyclictest0-21swapper/208:29:522
2354399812,38cyclictest11908-21idleruntime11:29:592
2354399812,36cyclictest2741-21latency_hist07:44:402
2354399812,36cyclictest2741-21latency_hist07:44:402
23534998129,19cyclictest30578-21/usr/sbin/munin11:05:011
23534998129,19cyclictest30578-21/usr/sbin/munin11:05:011
23527998160,13cyclictest207-21systemd-journal10:18:240
23527998154,18cyclictest30476-21diskmemload10:40:040
23527998146,12cyclictest181rcu_preempt12:17:100
2352799814,45cyclictest15-21ksoftirqd/008:59:480
2354399803,37cyclictest25553-21cron08:54:362
2354399803,37cyclictest25553-21cron08:54:362
2354399803,32cyclictest22607-21netstat10:50:062
2354399802,34cyclictest1107-21latency_hist07:39:412
2354399802,34cyclictest1107-21latency_hist07:39:412
2354399802,31cyclictest1702-21ls12:10:012
23534998064,9cyclictest0-21swapper/109:54:401
23534998064,9cyclictest0-21swapper/109:54:401
23534998060,14cyclictest25796-21rm10:55:151
23534998060,14cyclictest25796-21rm10:55:151
23534998055,19cyclictest28239-21switchtime09:00:131
23534998050,23cyclictest0-21swapper/108:19:381
23527998066,10cyclictest23520-21cyclictest12:10:140
23527998048,17cyclictest181rcu_preempt09:41:160
23527998048,17cyclictest181rcu_preempt09:41:160
23527998042,12cyclictest181rcu_preempt09:39:150
23527998042,12cyclictest181rcu_preempt09:39:150
2352799803,53cyclictest15-21ksoftirqd/009:45:070
2352799803,51cyclictest32727-21modprobe11:09:370
2352799803,51cyclictest32727-21modprobe11:09:370
322922797,8sleep00-21swapper/010:09:570
322922797,8sleep00-21swapper/010:09:570
2354399793,37cyclictest26391-21sh09:59:532
23534997958,16cyclictest23520-21cyclictest10:49:131
23534997939,33cyclictest10453-21mktemp10:29:391
23534997935,31cyclictest0-21swapper/108:29:491
23534997931,9cyclictest181rcu_preempt08:35:011
2353499792,30cyclictest17144-21grep12:35:141
23527997949,21cyclictest27329-21latency_hist07:19:410
23527997948,13cyclictest181rcu_preempt08:30:050
2352799793,52cyclictest23279-21mailstats08:45:170
2354399783,37cyclictest368-21snmpd09:19:502
2354399783,33cyclictest23982-21uniq07:10:032
2354399783,31cyclictest30810-21fschecks_count07:29:582
23534997857,15cyclictest11299-21/usr/sbin/munin08:09:581
23534997854,18cyclictest30476-21diskmemload10:10:011
23534997854,18cyclictest30476-21diskmemload10:10:011
23534997829,20cyclictest0-21swapper/110:05:531
23534997826,19cyclictest0-21swapper/109:48:081
23534997819,40cyclictest0-21swapper/107:59:411
23527997857,14cyclictest665-21kworker/u17:0+rpciod10:24:370
23527997856,14cyclictest23520-21cyclictest07:13:490
23527997852,10cyclictest181rcu_preempt07:55:100
23527997850,8cyclictest181rcu_preempt08:25:080
23527997844,11cyclictest181rcu_preempt07:50:550
23527997838,29cyclictest0-21swapper/011:00:150
23527997833,8cyclictest181rcu_preempt11:34:400
23527997753,15cyclictest30472-21apt-get09:09:420
23527997745,23cyclictest16953-21ssh11:39:390
23527997739,17cyclictest181rcu_preempt12:02:290
2352799773,40cyclictest15-21ksoftirqd/011:49:400
2352799773,40cyclictest15-21ksoftirqd/011:49:400
23527997733,10cyclictest0-21swapper/009:24:390
2352799772,53cyclictest30233-21open_files10:06:070
23534997659,10cyclictest6149-21grep10:20:141
23534997639,24cyclictest31032-21apt-get09:10:331
23527997654,8cyclictest181rcu_preempt09:54:480
23527997654,8cyclictest181rcu_preempt09:54:480
23527997653,15cyclictest3829-21ssh09:19:390
23527997645,10cyclictest181rcu_preempt11:16:430
23527997644,10cyclictest181rcu_preempt10:39:070
23527997644,10cyclictest181rcu_preempt10:39:070
23534997532,31cyclictest27005-21ssh10:00:111
23527997547,19cyclictest11878-21switchtime10:30:430
23527997541,12cyclictest181rcu_preempt09:30:070
23534997455,13cyclictest4101-21mailstats11:15:111
23534997454,14cyclictest12022-21latency_hist09:34:401
23534997441,23cyclictest30954-21apt-get12:04:501
23534997440,21cyclictest22618-21apt-get08:44:501
23534997436,29cyclictest30589-21/usr/sbin/munin07:29:471
2353499740,27cyclictest21153-21df08:39:571
2352799747,37cyclictest355-21runrttasks11:56:270
2352799747,37cyclictest355-21runrttasks11:56:270
2352799744,30cyclictest30479-21munin-run09:09:370
23527997437,13cyclictest181rcu_preempt08:55:010
23527997437,13cyclictest181rcu_preempt08:55:010
23527997437,12cyclictest181rcu_preempt07:20:100
2353499739,50cyclictest22942-21apt-get09:54:571
2353499739,50cyclictest22942-21apt-get09:54:571
23534997350,17cyclictest24576-21latency_hist10:54:391
23534997340,22cyclictest7390-21apt-get12:19:541
23534997340,22cyclictest7390-21apt-get12:19:541
23534997339,20cyclictest27395-21apt-get07:19:481
23527997349,9cyclictest181rcu_preempt11:25:330
23527997332,13cyclictest181rcu_preempt08:50:030
23527997332,13cyclictest181rcu_preempt08:50:030
23527997331,14cyclictest181rcu_preempt09:26:170
23527997330,9cyclictest181rcu_preempt07:40:020
23527997330,9cyclictest181rcu_preempt07:40:020
23534997256,10cyclictest2523-21sh09:15:291
23534997255,11cyclictest0-21swapper/112:34:321
23534997244,22cyclictest30476-21diskmemload11:22:171
23534997244,22cyclictest30476-21diskmemload11:22:171
23534997237,23cyclictest22812-21apt-get11:49:501
23534997227,37cyclictest355-21runrttasks12:25:411
23527997249,9cyclictest181rcu_preempt08:09:400
23527997240,9cyclictest181rcu_preempt07:37:500
23527997240,9cyclictest181rcu_preempt07:37:500
23527997239,8cyclictest181rcu_preempt10:50:300
23527997217,19cyclictest15-21ksoftirqd/011:10:150
23534997154,11cyclictest27286-21systemd08:59:381
23534997154,11cyclictest27286-21systemd08:59:381
23534997150,15cyclictest207-21systemd-journal07:14:501
23534997142,20cyclictest11247-21modprobe11:29:361
23534997133,25cyclictest0-21swapper/111:34:501
23534997128,31cyclictest9427-21modprobe08:04:371
23527997151,7cyclictest181rcu_preempt11:50:180
23527997143,7cyclictest181rcu_preempt08:35:000
2352799713,33cyclictest17870-21uname11:40:110
23527997127,9cyclictest181rcu_preempt07:45:140
2354899703,52cyclictest3047-21taskset11:14:363
23534997051,12cyclictest28222-21netstat11:00:061
23534997049,15cyclictest0-21swapper/107:54:371
23534997038,21cyclictest29001-21apt-get07:24:491
23534997034,22cyclictest10227-21awk08:05:081
23534997030,22cyclictest4315-21ssh10:18:011
2352799703,31cyclictest355-21runrttasks08:14:570
23527997031,9cyclictest181rcu_preempt08:09:550
2354899692,52cyclictest15665-21taskset11:35:553
2354899692,51cyclictest19361-21chrt08:34:493
23534996943,19cyclictest0-21swapper/110:44:361
23534996939,21cyclictest25565-21sh08:54:361
23534996939,21cyclictest25565-21sh08:54:361
23534996935,26cyclictest10953-21ssh10:30:001
2353499693,46cyclictest16012-21unixbench_singl09:40:161
2353499693,46cyclictest16012-21unixbench_singl09:40:161
23534996932,28cyclictest26082-21awk11:55:051
23534996932,28cyclictest26082-21awk11:55:051
23534996929,28cyclictest3494-21/usr/sbin/munin07:45:051
2354899683,49cyclictest3015-21chrt09:17:263
2354899682,52cyclictest9170-21taskset10:25:443
2354899682,51cyclictest8424-21taskset11:24:123
2354899682,51cyclictest8424-21taskset11:24:123
23534996846,13cyclictest17505-21fschecks_time11:40:011
23534996837,24cyclictest4268-21/usr/sbin/munin09:19:591
23534996833,25cyclictest28696-21ssh12:00:011
23534996827,30cyclictest355-21runrttasks11:31:121
23527996843,8cyclictest181rcu_preempt08:40:160
2354899673,50cyclictest26995-21chrt10:00:103
2354899673,49cyclictest6012-21taskset07:52:253
2354899672,52cyclictest11271-21chrt10:30:263
2354899672,51cyclictest6402-21taskset12:17:543
2353499675,40cyclictest30507-21http09:09:371
23534996747,13cyclictest1831-21latency12:10:031
23534996735,21cyclictest16616-21/usr/sbin/munin08:25:071
23534996733,25cyclictest20424-21/usr/sbin/munin11:45:031
23534996733,25cyclictest20424-21/usr/sbin/munin11:45:031
23534996730,26cyclictest23818-21cat07:09:581
23534996729,29cyclictest1300-21cat07:39:411
23534996729,29cyclictest1300-21cat07:39:411
2354899663,49cyclictest3010-21taskset07:44:523
23534996631,26cyclictest14721-21/usr/sbin/munin08:19:581
2354899653,50cyclictest7412-21chrt12:19:533
2354899653,50cyclictest7412-21chrt12:19:533
2354899653,48cyclictest10573-21chrt12:24:563
2354899652,50cyclictest25663-21taskset07:14:473
2354899652,50cyclictest15238-21taskset10:36:423
2354899652,50cyclictest15238-21taskset10:36:423
2354899652,49cyclictest6878-21chrt09:24:563
2354899652,49cyclictest4022-21chrt10:17:043
2354899652,49cyclictest1841-21chrt07:40:023
2354899652,49cyclictest1841-21chrt07:40:023
2354899652,48cyclictest21849-21taskset09:51:143
2354899652,48cyclictest21849-21taskset09:51:143
2354899652,48cyclictest1216-21chrt12:09:393
23534996532,22cyclictest1679-21/usr/sbin/munin07:39:591
23534996532,22cyclictest1679-21/usr/sbin/munin07:39:591
2354899643,48cyclictest11595-21chrt11:29:513
2354899643,47cyclictest4590-21taskset11:15:413
2354899642,49cyclictest24499-21chrt10:54:253
2354899642,49cyclictest1876-21chrt10:14:133
2354899642,49cyclictest1876-21chrt10:14:133
2354899642,49cyclictest11470-21chrt08:09:593
2354899642,49cyclictest11355-21taskset09:32:503
2354899642,48cyclictest26363-21taskset10:56:323
2354899642,48cyclictest26363-21taskset10:56:323
2354899642,48cyclictest17810-21chrt12:35:583
2354899633,47cyclictest16402-21chrt09:41:263
2354899633,47cyclictest16402-21chrt09:41:263
2354899633,46cyclictest22977-21chrt11:49:583
2354899632,48cyclictest32498-21chrt11:08:523
2354899632,48cyclictest32498-21chrt11:08:523
2354899632,48cyclictest30439-21chrt09:09:243
2354899632,48cyclictest21243-21taskset11:45:403
2354899632,48cyclictest21243-21taskset11:45:403
2354899632,47cyclictest28664-21taskset09:02:113
2354899632,47cyclictest19556-21taskset09:47:383
2354899624,44cyclictest19653-21chrt10:45:303
2354899623,47cyclictest14543-21taskset12:30:193
2354899623,46cyclictest15583-21chrt08:20:443
2354899622,47cyclictest6409-21taskset10:20:453
2354899622,47cyclictest18052-21taskset10:41:533
2354899622,46cyclictest900-21chrt07:37:103
2354899622,46cyclictest900-21chrt07:37:103
2354899622,46cyclictest26953-21chrt08:56:163
2354899622,46cyclictest26953-21chrt08:56:163
2354899622,46cyclictest18849-21chrt08:31:253
2354899622,45cyclictest28015-21chrt07:20:093
2354899622,45cyclictest17161-21chrt11:39:443
2354899612,48cyclictest22771-21chrt08:44:583
2354899612,47cyclictest6467-21chrt07:54:523
2354899612,47cyclictest26770-21chrt11:55:413
2354899612,47cyclictest26770-21chrt11:55:413
2354899612,47cyclictest13327-21chrt09:35:343
2354899612,47cyclictest13327-21chrt09:35:343
2354899612,47cyclictest10395-21taskset11:26:563
2354899612,46cyclictest9807-21chrt08:04:573
2354899612,46cyclictest32313-21taskset09:11:023
2354899612,46cyclictest23506-21chrt09:54:513
2354899612,46cyclictest23506-21chrt09:54:513
2354899612,45cyclictest31602-21chrt10:09:023
2354899603,44cyclictest13942-21chrt08:15:423
2354899602,47cyclictest2689-21chrt12:10:533
2354899602,45cyclictest5463-21chrt09:21:243
2354899602,45cyclictest24799-21chrt07:11:273
2354899602,45cyclictest20991-21chrt08:39:473
2354899593,44cyclictest31851-21chrt07:33:343
2354899593,44cyclictest29680-21chrt07:25:113
2354899592,46cyclictest28831-21taskset11:00:253
2354899592,45cyclictest8097-21taskset07:59:503
2354899592,45cyclictest24204-21chrt08:49:443
2354899592,45cyclictest24204-21chrt08:49:443
2354899592,44cyclictest29607-21chrt12:00:443
2354899592,44cyclictest17441-21chrt08:29:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional