You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 09:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Dec 05, 2025 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2293621590,22sleep1291rcuc/119:03:491
243699911665,46cyclictest24347-21cyclictest20:18:322
243699911665,46cyclictest24347-21cyclictest20:18:322
243699911041,15cyclictest0-21swapper/222:29:122
243629910947,46cyclictest31462-21diskmemload22:35:251
598621080,7sleep30-21swapper/321:19:083
598621080,7sleep30-21swapper/321:19:083
24362991064,83cyclictest176-21jbd2/mmcblk2p2-819:23:571
24362991064,83cyclictest176-21jbd2/mmcblk2p2-819:23:571
243699910538,12cyclictest0-21swapper/219:33:352
243699910538,12cyclictest0-21swapper/219:33:352
24369991043,47cyclictest30450-21latency_hist22:58:342
243699910345,9cyclictest0-21swapper/223:58:452
24369991033,39cyclictest8675-21latency_hist19:58:342
2436999994,44cyclictest27182-21switchtime23:49:062
2436999984,39cyclictest19416-21cpuspeed22:38:512
2436999963,43cyclictest355-21runrttasks23:33:502
2436999963,43cyclictest355-21runrttasks23:33:502
24362999430,33cyclictest0-21swapper/121:43:531
24362999430,33cyclictest0-21swapper/121:43:531
24369999366,17cyclictest4904-21timerwakeupswit19:44:092
24362999349,37cyclictest355-21runrttasks20:14:081
24362999349,37cyclictest355-21runrttasks20:14:081
24362999229,33cyclictest0-21swapper/100:33:581
24362999229,33cyclictest0-21swapper/100:33:581
2436999913,43cyclictest24992-21apt20:48:322
24362999158,23cyclictest17419-21fschecks_time20:23:551
24362999126,34cyclictest0-21swapper/123:57:371
24362999126,34cyclictest0-21swapper/123:57:371
2436999909,56cyclictest12326-21apt-get23:23:582
24369999060,21cyclictest4757-21sh00:07:182
2436299903,71cyclictest16247-21apt-get00:28:461
24369998951,23cyclictest10963-21kworker/u18:2+flush-179:022:50:352
24362998963,16cyclictest20860-21netstat20:34:011
24358998760,19cyclictest32360-21switchtime19:29:060
24358998760,19cyclictest32360-21switchtime19:29:060
24358998757,22cyclictest871-21sendmail-mta22:44:090
24369998671,10cyclictest24347-21cyclictest21:19:052
24369998671,10cyclictest24347-21cyclictest21:19:052
2436999864,33cyclictest30053-21apt-get19:24:212
2436999864,33cyclictest30053-21apt-get19:24:212
24369998557,20cyclictest13287-21sh22:28:322
24369998556,21cyclictest31792-40mandb23:58:342
24369998556,21cyclictest31792-40mandb23:58:342
24362998558,19cyclictest13545-21unixbench_multi23:24:461
24362998542,37cyclictest24210-21kworker/u19:1+events_unbound21:53:311
2435899859,28cyclictest27794-21ssh22:53:500
2435899853,61cyclictest15630-21sshd20:18:570
24369998466,12cyclictest31462-21diskmemload00:16:572
24369998459,18cyclictest10568-21latency_hist22:23:352
24362998462,14cyclictest15546-21sh22:31:581
24362998439,33cyclictest0-21swapper/121:58:571
24358998410,64cyclictest30053-21apt-get19:23:570
24358998410,64cyclictest30053-21apt-get19:23:570
24362998342,34cyclictest0-21swapper/121:34:081
2435899837,65cyclictest871-21sendmail-mta23:58:540
24358998355,19cyclictest0-21swapper/022:23:340
2435899833,71cyclictest355-21runrttasks21:13:080
2435899833,71cyclictest355-21runrttasks21:13:080
24369998263,13cyclictest31462-21diskmemload23:09:022
24362998223,50cyclictest369-21ntpd23:49:001
24358998259,15cyclictest2961-21ssh23:07:000
235072829,40sleep00-21swapper/019:04:180
24362998159,16cyclictest3870-21latency_hist19:43:341
24362998140,32cyclictest0-21swapper/120:58:341
24362998140,32cyclictest0-21swapper/120:58:341
2436299812,58cyclictest32647-21cpu23:59:411
24369998059,13cyclictest13313-21cron00:23:312
24369998059,13cyclictest13313-21cron00:23:312
24362998042,32cyclictest0-21swapper/123:04:091
24362998038,34cyclictest0-21swapper/123:29:111
24362998022,48cyclictest0-21swapper/123:13:201
24362997939,33cyclictest0-21swapper/119:34:091
2436299793,48cyclictest16929-21apt20:23:321
24362997928,42cyclictest30561-21memory21:04:011
24358997925,33cyclictest181rcu_preempt23:37:000
24358997925,33cyclictest181rcu_preempt23:37:000
24358997917,28cyclictest0-21swapper/021:29:030
24369997860,12cyclictest31462-21diskmemload21:39:062
24369997847,11cyclictest24228-21kworker/u18:0-rpciod22:08:572
24362997838,33cyclictest0-21swapper/100:24:131
24362997838,32cyclictest0-21swapper/119:09:001
24362997830,41cyclictest0-21swapper/121:13:581
24358997858,14cyclictest24347-21cyclictest20:38:350
24358997854,17cyclictest31462-21diskmemload00:24:050
24358997847,22cyclictest29000-21latency_hist23:53:340
2436999774,40cyclictest3298-21grep21:14:112
24362997741,28cyclictest0-21swapper/123:44:121
24362997735,34cyclictest0-21swapper/121:19:081
24362997735,34cyclictest0-21swapper/121:19:081
24362997729,40cyclictest7154-21sh00:11:001
2435899779,28cyclictest20868-21ssh23:38:490
24358997722,17cyclictest0-21swapper/000:13:480
24374997622,41cyclictest32629-21taskset21:09:073
24374997622,41cyclictest32629-21taskset21:09:073
24369997622,18cyclictest181rcu_preempt20:34:102
24362997637,31cyclictest0-21swapper/121:08:541
24362997637,31cyclictest0-21swapper/121:08:541
2436299763,67cyclictest0-21swapper/122:58:471
24362997634,36cyclictest388-21rm22:04:361
24362997634,36cyclictest0-21swapper/123:23:351
24362997634,34cyclictest0-21swapper/121:43:341
2435899766,40cyclictest32185-21latency_hist23:58:360
2435899766,40cyclictest32185-21latency_hist23:58:360
24358997650,20cyclictest1177-21kernelversion19:33:570
24358997650,18cyclictest25388-21timerwakeupswit19:09:080
24358997649,17cyclictest9224-21latency_hist23:18:350
24358997649,17cyclictest9224-21latency_hist23:18:350
2435899763,30cyclictest12115-21acpi20:08:450
24369997556,12cyclictest5651-21sh00:08:522
24369997550,16cyclictest12322-21fschecks_count20:08:552
2436999753,33cyclictest3796-21munin-run19:43:322
24362997536,35cyclictest24347-21cyclictest20:29:241
24362997536,33cyclictest355-21runrttasks22:13:401
24362997534,36cyclictest0-21swapper/120:39:031
2436299752,59cyclictest0-21swapper/119:58:471
24358997559,10cyclictest11796-21ssh23:23:310
24358997552,16cyclictest24347-21cyclictest23:28:550
24358997547,14cyclictest0-21swapper/019:48:490
2435899752,40cyclictest9422-1kworker/0:0H+mmc_complete19:23:350
24362997434,33cyclictest0-21swapper/120:59:131
2436299742,46cyclictest8355-21chrt19:55:241
2435899748,36cyclictest13302-21rm00:23:290
2435899748,36cyclictest13302-21rm00:23:290
24358997448,19cyclictest11395-21cut22:25:040
2420127411,39sleep224234-21munin-run19:08:322
24369997353,13cyclictest207-21systemd-journal21:05:112
24369997346,17cyclictest21071-21kworker/u18:3-rpciod00:34:102
24369997346,17cyclictest21071-21kworker/u18:3-rpciod00:34:102
24362997332,34cyclictest0-21swapper/121:54:091
24362997331,35cyclictest0-21swapper/123:34:051
24362997331,35cyclictest0-21swapper/123:34:051
24362997329,36cyclictest0-21swapper/100:03:541
2435899738,24cyclictest0-21swapper/022:14:250
24358997354,13cyclictest13882-21fschecks_time22:28:580
24358997349,16cyclictest24063-21ssh23:44:000
24358997349,13cyclictest0-21swapper/020:24:070
2435899733,35cyclictest4097-21runrttasks23:08:590
24358997318,22cyclictest0-21swapper/020:29:230
24358997311,24cyclictest0-21swapper/021:27:220
24369997249,16cyclictest5632-21mailstats22:14:042
24369997246,15cyclictest0-21swapper/223:29:082
24369997243,12cyclictest181rcu_preempt21:43:532
24369997243,12cyclictest181rcu_preempt21:43:532
2436299723,62cyclictest0-21swapper/122:43:461
2436299723,58cyclictest0-21swapper/121:23:461
24362997232,33cyclictest0-21swapper/123:38:551
24362997231,34cyclictest0-21swapper/100:13:561
24362997222,43cyclictest6661-21cat23:13:491
24362997222,43cyclictest6661-21cat23:13:491
2435899729,21cyclictest0-21swapper/021:48:470
24358997251,14cyclictest316-21rs:main0
24358997251,14cyclictest316-21rs:main0
24358997226,33cyclictest1-21systemd22:43:320
2435899722,31cyclictest32589-21/usr/sbin/munin22:04:100
2302427210,36sleep30-21swapper/319:03:533
24369997152,11cyclictest0-21swapper/219:14:092
24369997152,11cyclictest0-21swapper/219:14:092
2436999714,44cyclictest6935-21sshd19:52:532
2436999712,34cyclictest1020-21fschecks_count19:33:542
24362997155,10cyclictest11639-21ntp_kernel_err00:19:001
24362997155,10cyclictest11639-21ntp_kernel_err00:19:001
24362997149,16cyclictest11746-21proc_pri21:29:121
2436299713,63cyclictest0-21swapper/119:13:491
2436299713,63cyclictest0-21swapper/119:13:491
24362997126,36cyclictest10841-21if_eth020:03:571
24362997125,39cyclictest31462-21diskmemload22:48:521
24362997112,44cyclictest4734-21latency_hist22:13:341
2435899718,30cyclictest16732-21ssh22:33:560
2435899717,29cyclictest16976-21grep00:29:070
2435899717,27cyclictest28911-21ssh21:58:530
24358997149,15cyclictest0-21swapper/023:24:300
2435899714,52cyclictest27907-21cron19:18:310
2435899714,52cyclictest27907-21cron19:18:310
24369997011,35cyclictest10311-21grep23:19:092
24362997055,11cyclictest24347-21cyclictest19:46:451
24362997051,15cyclictest24347-21cyclictest19:23:051
2436299703,62cyclictest0-21swapper/122:53:481
24362997031,33cyclictest0-21swapper/122:19:101
24362997027,35cyclictest0-21swapper/122:38:501
2435899706,24cyclictest0-21swapper/000:10:080
24358997045,15cyclictest20136-21ssh21:43:580
24358997045,15cyclictest20136-21ssh21:43:580
2436999695,38cyclictest355-21runrttasks20:28:182
24369996946,12cyclictest0-21swapper/221:34:072
2436999694,45cyclictest355-21runrttasks00:25:472
24369996942,10cyclictest16878-21ssh00:29:062
24369996940,11cyclictest181rcu_preempt20:59:132
24362996953,9cyclictest0-21swapper/120:43:551
2436299693,51cyclictest31462-21diskmemload22:27:501
24358996946,17cyclictest0-21swapper/019:54:020
2435899694,32cyclictest22021-21/usr/sbin/munin20:38:440
2435899693,28cyclictest5369-21latency_hist00:08:350
24358996921,20cyclictest0-21swapper/022:48:580
24369996848,10cyclictest0-21swapper/222:38:322
24369996839,20cyclictest22954-21grep22:44:092
2436999683,48cyclictest19725-21grep20:29:242
24369996820,27cyclictest1956-21latency_hist21:13:342
24369996820,27cyclictest1956-21latency_hist21:13:342
2436299684,57cyclictest25483-21fschecks_time20:48:571
2436299684,57cyclictest25483-21fschecks_time20:48:571
2436299683,60cyclictest0-21swapper/119:28:471
2436299683,60cyclictest0-21swapper/119:28:471
2436299683,55cyclictest13580-21latency_hist20:13:341
24358996846,16cyclictest13995-21cpu21:33:470
24358996840,22cyclictest0-21swapper/021:53:570
2435899683,46cyclictest23855-21fschecks_count20:43:540
24374996730,23cyclictest7467-21taskset00:11:593
24369996751,11cyclictest18001-21kworker/u17:1-xprtiod23:14:132
24369996751,11cyclictest18001-21kworker/u17:1-xprtiod23:14:132
24369996749,12cyclictest1452-21runrttasks22:07:412
24369996744,13cyclictest0-21swapper/220:48:572
24369996744,13cyclictest0-21swapper/220:48:572
24369996732,11cyclictest2739-21rm23:06:162
24369996719,14cyclictest181rcu_preempt21:54:122
2435899677,23cyclictest19271-21basename00:33:590
2435899677,23cyclictest19271-21basename00:33:590
24358996736,25cyclictest31459-21munin-run21:08:320
2435899672,52cyclictest30011-21latency_hist21:03:350
24358996723,8cyclictest0-21swapper/021:14:110
24374996628,25cyclictest23636-21taskset22:45:273
24369996649,9cyclictest0-21swapper/223:38:462
24369996649,12cyclictest24347-21cyclictest22:59:092
24369996649,11cyclictest22559-21ntp_offset20:39:032
24369996649,11cyclictest13696-21systemd-run21:33:342
24369996640,13cyclictest0-21swapper/221:24:112
2436999663,45cyclictest9525-21grep19:59:062
24369996629,9cyclictest0-21swapper/221:53:322
24369996619,36cyclictest24948-21latency19:08:582
2436299662,52cyclictest5665-21apt-get19:48:441
2435899667,19cyclictest25452-21tune2fs20:48:570
2435899667,19cyclictest25452-21tune2fs20:48:570
2435899665,25cyclictest5280-21cpuspeed21:18:490
2435899665,25cyclictest5280-21cpuspeed21:18:490
24358996648,11cyclictest17060-21diskmemload21:38:490
24358996636,23cyclictest0-21swapper/019:38:550
2435899663,45cyclictest2505-21kernelversion22:08:590
2435899663,42cyclictest31697-21ssh22:59:120
24374996531,22cyclictest18565-21chrt20:28:333
24374996530,23cyclictest1365-21chrt19:34:043
24374996527,26cyclictest13994-21taskset21:33:463
24374996527,25cyclictest22043-21taskset20:38:453
24374996526,26cyclictest3297-21taskset00:04:053
2436999653,39cyclictest27067-21sort20:53:552
2436999653,39cyclictest27067-21sort20:53:552
24369996530,12cyclictest181rcu_preempt20:08:342
2435899652,45cyclictest355-21runrttasks19:44:030
24374996431,19cyclictest12695-21chrt23:24:263
24374996430,22cyclictest11919-21chrt22:25:173
24374996428,23cyclictest10341-21taskset00:17:103
24374996427,23cyclictest27018-21taskset22:52:163
24374996426,26cyclictest16162-21taskset20:19:383
2436999643,47cyclictest16998-21latency_hist20:23:342
2436999643,26cyclictest23689-21ssh23:43:512
2436999642,42cyclictest29601-21runrttasks19:23:332
24358996442,11cyclictest0-21swapper/020:08:360
24374996332,21cyclictest16395-21chrt22:33:443
24374996330,23cyclictest27859-21chrt20:54:133
24374996330,23cyclictest27859-21chrt20:54:133
24374996330,23cyclictest15145-21chrt23:28:473
24374996329,23cyclictest11308-21taskset20:04:093
24374996329,22cyclictest24541-21chrt19:08:443
24374996329,22cyclictest17947-21chrt00:31:203
24374996329,22cyclictest13816-21chrt00:23:543
24374996329,21cyclictest26989-21chrt21:54:413
24374996329,20cyclictest2159-21chrt21:13:373
24374996329,20cyclictest18713-21chrt21:41:123
24374996328,21cyclictest5989-21chrt22:14:103
24374996328,21cyclictest4639-21chrt22:13:333
24374996327,24cyclictest8656-21taskset21:24:023
24374996326,25cyclictest3001-21taskset23:07:073
24374996326,25cyclictest21775-21taskset22:43:283
24374996325,25cyclictest27619-21taskset19:14:563
24374996325,25cyclictest27619-21taskset19:14:563
24369996340,12cyclictest0-21swapper/221:58:422
2435899632,35cyclictest11131-1kworker/0:2H+mmc_complete20:58:340
2435899632,35cyclictest11131-1kworker/0:2H+mmc_complete20:58:340
24374996232,20cyclictest21126-21chrt23:38:563
24374996231,19cyclictest24281-21chrt21:50:203
24374996231,19cyclictest19252-21chrt00:33:593
24374996231,19cyclictest19252-21chrt00:33:593
24374996230,21cyclictest32488-21chrt23:59:143
24374996230,21cyclictest29484-21chrt20:59:153
24374996230,21cyclictest28993-21chrt21:58:553
24374996230,21cyclictest12653-21chrt20:09:033
24374996230,20cyclictest14668-21chrt22:29:143
24374996229,21cyclictest8322-21chrt19:54:573
24374996229,21cyclictest19979-21chrt23:37:183
24374996229,21cyclictest19979-21chrt23:37:183
24374996229,20cyclictest30922-21chrt23:56:023
24374996229,20cyclictest30922-21chrt23:56:023
24374996228,23cyclictest27956-21taskset22:53:543
24374996228,22cyclictest4598-21chrt19:44:043
24374996228,22cyclictest25956-21chrt23:48:123
24374996130,20cyclictest26536-21chrt23:48:493
24374996129,20cyclictest14070-21chrt20:13:573
24374996129,20cyclictest14070-21chrt20:13:573
24374996129,20cyclictest11253-21chrt21:28:563
24374996128,22cyclictest11117-21chrt00:18:483
24374996128,22cyclictest11117-21chrt00:18:483
24374996128,21cyclictest24830-21chrt20:46:393
24374996127,22cyclictest30062-21chrt19:23:473
24374996127,22cyclictest30062-21chrt19:23:473
24374996127,22cyclictest11651-21chrt23:23:123
24374996127,21cyclictest32736-21chrt19:30:153
24374996127,21cyclictest32736-21chrt19:30:153
24358996124,27cyclictest10396-21latency_hist20:03:350
24374996030,20cyclictest8691-21chrt22:19:093
24374996030,19cyclictest3921-21chrt23:08:533
24374996028,21cyclictest7629-21chrt23:14:123
24374996028,21cyclictest7629-21chrt23:14:123
24374996028,21cyclictest31630-21chrt22:03:493
24374996027,21cyclictest19951-21chrt20:31:203
24374995928,20cyclictest8909-21chrt19:58:463
24374995928,19cyclictest30830-21chrt21:04:093
24374995927,20cyclictest28218-21chrt19:18:453
24374995926,21cyclictest3564-21chrt19:40:483
24374995926,20cyclictest21027-21chrt21:44:243
24374995926,20cyclictest21027-21chrt21:44:243
24374995829,18cyclictest26622-21chrt20:53:323
24374995829,18cyclictest26622-21chrt20:53:323
24374995828,20cyclictest331-21chrt23:02:333
24374995825,22cyclictest20381-21chrt20:33:393
24374995625,19cyclictest6654-21chrt19:49:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional