You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 08:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Wed Jul 15, 2026 00:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
798991214,97cyclictest0-21swapper/021:50:460
798991214,96cyclictest0-21swapper/000:35:440
798991214,96cyclictest0-21swapper/000:35:440
798991194,95cyclictest0-21swapper/023:50:460
1692721180,3sleep016920-21ntp_states00:30:420
798991172,87cyclictest7194-21/usr/sbin/munin23:40:360
798991125,91cyclictest0-21swapper/023:35:300
894121030,5sleep280099cyclictest21:45:202
7989910331,50cyclictest0-21swapper/022:00:180
798999610,73cyclictest8050irq/56-dwc_otg_21:25:250
800999517,54cyclictest0-21swapper/221:25:252
79899895,51cyclictest0-21swapper/023:30:410
79899883,60cyclictest13-21rcuc/022:50:350
79899881,45cyclictest0-21swapper/022:55:320
799998716,45cyclictest2998-21missed_timers22:55:401
79899872,51cyclictest0-21swapper/022:20:360
799998629,16cyclictest17697-21apt-get22:35:211
79899852,68cyclictest0-21swapper/022:35:220
801998426,26cyclictest7950irq/56-dwc_otg_22:55:413
799998226,17cyclictest30907-21forks22:50:371
801998130,23cyclictest18040-21ssh22:35:223
799998139,22cyclictest0-21swapper/121:10:291
798998149,18cyclictest20510-21taskset23:55:440
16182800,5sleep380199cyclictest20:40:403
799997921,23cyclictest0-21swapper/123:40:351
76122790,5sleep225771-21kworker/2:223:40:352
225572780,5sleep380199cyclictest23:20:073
18022780,2sleep20-21swapper/223:33:472
799997530,23cyclictest21028-21/usr/sbin/munin00:35:431
799997530,23cyclictest21028-21/usr/sbin/munin00:35:431
302392750,4sleep3381ktimersoftd/323:29:013
302392750,4sleep3381ktimersoftd/323:29:013
801997420,24cyclictest7587-21fschecks_time23:40:353
80099732,53cyclictest14228-21ssh21:50:462
79999734,23cyclictest16422-21ssh23:50:451
80199689,32cyclictest0-21swapper/319:25:253
801996839,22cyclictest5797-21/usr/sbin/munin22:20:363
801996825,17cyclictest30890-21diskstats22:50:363
79999679,22cyclictest0-21swapper/121:25:251
79999673,22cyclictest14294-21timerwakeupswit21:50:471
798996733,18cyclictest0-21swapper/021:45:440
798996611,39cyclictest0-21swapper/022:40:470
801996529,30cyclictest14224-21taskset21:50:463
801996521,23cyclictest0-21swapper/321:25:253
801996515,23cyclictest8050irq/56-dwc_otg_23:55:443
798996539,6cyclictest0-21swapper/000:10:310
79899653,45cyclictest31566-21latency_hist20:35:190
79899653,45cyclictest31566-21latency_hist20:35:190
122412650,2sleep20-21swapper/223:45:422
800996318,26cyclictest301ktimersoftd/222:55:402
79999635,11cyclictest0-21swapper/122:00:191
799996310,24cyclictest0-21swapper/123:55:421
79899639,38cyclictest0-21swapper/022:00:210
79899639,38cyclictest0-21swapper/022:00:210
798996343,8cyclictest7950irq/56-dwc_otg_22:10:480
79899631,51cyclictest16450-21cpu21:15:330
801996234,22cyclictest21172-21latency_hist22:00:183
801996214,25cyclictest21056-21rm00:35:443
801996214,25cyclictest21056-21rm00:35:443
799996233,24cyclictest5882-21fschecks_time22:20:371
79999623,22cyclictest23-21ksoftirqd/122:00:211
79999623,22cyclictest23-21ksoftirqd/122:00:211
798996236,13cyclictest16485-21grep19:50:460
798996139,12cyclictest7950irq/56-dwc_otg_21:25:180
79899612,47cyclictest0-21swapper/022:05:370
801996031,23cyclictest16428-21/usr/sbin/munin23:50:453
798996036,6cyclictest9880-21sh21:05:180
79899602,50cyclictest0-21swapper/020:10:450
79899602,44cyclictest0-21swapper/022:25:330
79899602,44cyclictest0-21swapper/022:25:330
79899602,30cyclictest6833-21fschecks_time20:55:370
79899601,48cyclictest0-21swapper/023:10:480
801995919,24cyclictest31814-21irqstats23:30:393
798995932,15cyclictest5268-21fschecks_time21:40:360
798995932,15cyclictest5268-21fschecks_time21:40:360
798995929,15cyclictest0-21swapper/019:46:580
798995929,15cyclictest0-21swapper/019:46:580
798995928,13cyclictest0-21swapper/020:20:450
79899592,40cyclictest0-21swapper/000:20:360
79899591,42cyclictest19928-21grep20:00:450
79999586,26cyclictest0-21swapper/119:25:251
799995819,26cyclictest29343-21cron22:10:171
79899588,28cyclictest0-21swapper/023:00:460
79899588,28cyclictest0-21swapper/023:00:460
79899582,48cyclictest0-21swapper/020:38:090
79899581,46cyclictest0-21swapper/019:56:510
79899581,42cyclictest21898-21unixbench_multi20:05:450
79899581,36cyclictest0-21swapper/023:05:380
206422580,2sleep20-21swapper/221:59:292
80099575,27cyclictest16299-21open_files23:50:432
799995723,8cyclictest29068-21apt-get21:30:291
79899572,43cyclictest0-21swapper/019:41:320
798995723,11cyclictest0-21swapper/019:20:300
798995723,11cyclictest0-21swapper/019:20:300
79999565,21cyclictest7950irq/56-dwc_otg_21:20:311
798995635,5cyclictest9326-21sed19:30:450
301782560,2sleep20-21swapper/200:07:372
301782560,2sleep20-21swapper/200:07:372
801995514,21cyclictest8050irq/56-dwc_otg_22:15:393
800995511,28cyclictest8050irq/56-dwc_otg_23:55:452
79999559,14cyclictest1956-21latency22:15:391
799995525,23cyclictest31796-21ssh23:30:381
79899558,30cyclictest4805-21aten_rbpower_po20:50:330
79899557,36cyclictest12638-21missed_timers00:25:400
798995541,4cyclictest9-21ksoftirqd/021:35:310
79899551,41cyclictest0-21swapper/019:30:120
79899551,41cyclictest0-21swapper/000:15:330
79899551,36cyclictest10210-21grep19:35:350
70222550,2sleep20-21swapper/219:25:402
80099546,15cyclictest29-21rcuc/222:35:212
79999543,10cyclictest7850irq/56-dwc_otg20:25:311
79899541,40cyclictest0-21swapper/020:25:430
79899541,38cyclictest0-21swapper/019:10:210
283852540,3sleep1221ktimersoftd/100:05:391
283852540,3sleep1221ktimersoftd/100:05:391
80199531,9cyclictest8050irq/56-dwc_otg_20:10:323
80099524,15cyclictest5784-21diskstats22:20:352
80099523,15cyclictest29-21rcuc/222:50:372
79899524,29cyclictest10122-21kworker/0:100:05:220
79899524,29cyclictest10122-21kworker/0:100:05:220
798995224,5cyclictest24102-21fschecks_time00:00:370
79899522,35cyclictest0-21swapper/022:30:390
79899521,37cyclictest0-21swapper/020:15:430
79899514,25cyclictest0-21swapper/023:15:480
798995127,7cyclictest3322-21ntp_kernel_pll_19:15:400
79899511,42cyclictest0-21swapper/022:45:210
79899511,41cyclictest0-21swapper/023:21:560
79899511,36cyclictest2804-21tr20:45:280
80199503,17cyclictest0-21swapper/322:00:213
80199503,17cyclictest0-21swapper/322:00:213
79999509,24cyclictest30020-21awk20:30:331
79999509,24cyclictest30020-21awk20:30:331
79999503,7cyclictest29695-21ssh22:10:301
799995014,22cyclictest30481-21rm23:29:281
799995014,22cyclictest30481-21rm23:29:281
79899501,36cyclictest0-21swapper/023:45:420
80199495,17cyclictest7850irq/56-dwc_otg21:20:313
801994918,12cyclictest5120-21if_eth020:50:383
801994916,15cyclictest24417-21ssh23:20:563
79999499,25cyclictest11304-21ssh21:47:301
79999496,28cyclictest10207-21fschecks_count19:35:351
799994911,23cyclictest2664-21ssh00:14:021
801994817,15cyclictest21851-21ls20:05:443
801994815,18cyclictest29841-21/usr/sbin/munin20:30:323
801994815,18cyclictest29841-21/usr/sbin/munin20:30:323
79999488,13cyclictest18768-21ls20:00:171
799994814,16cyclictest8131-21ssh21:44:151
799994814,16cyclictest8131-21ssh21:44:151
799994812,17cyclictest7391-21ssh00:20:051
799994811,18cyclictest17863-21rm21:16:121
798994832,3cyclictest0-21swapper/021:30:390
79899481,7cyclictest10359-21grep21:05:340
79899481,7cyclictest10359-21grep21:05:340
80199478,22cyclictest8050irq/56-dwc_otg_21:38:393
801994715,12cyclictest11926-21rm23:05:503
801994713,20cyclictest14590-21ssh00:28:073
800994713,15cyclictest7950irq/56-dwc_otg_00:35:442
800994713,15cyclictest7950irq/56-dwc_otg_00:35:442
79999475,26cyclictest0-21swapper/119:15:311
799994713,17cyclictest10309-21sh00:22:521
799994712,21cyclictest20806-21apt-get20:05:281
79899476,38cyclictest1881-21ls20:40:450
798994722,4cyclictest8050irq/56-dwc_otg_23:25:300
798994722,4cyclictest8050irq/56-dwc_otg_23:25:300
79899471,35cyclictest1856-21ssh22:15:380
80199466,7cyclictest4566-21sed20:50:183
801994616,12cyclictest28310-21if_err_wlan000:05:373
801994616,12cyclictest28310-21if_err_wlan000:05:373
800994619,11cyclictest727-21runrttasks00:27:102
79999467,13cyclictest4417-21gpgv20:50:181
80199458,14cyclictest12970-21vmstat19:40:453
80199458,11cyclictest12029-21apt-get21:10:303
80199457,24cyclictest0-21swapper/319:35:313
801994510,11cyclictest19267-21ssh21:18:373
800994524,4cyclictest14810-21basename23:10:342
799994516,12cyclictest16889-21sshd19:54:371
799994513,12cyclictest25235-21rm00:01:051
79899452,26cyclictest0-21swapper/021:10:460
314182453,35sleep00-21swapper/019:05:280
80199449,26cyclictest23750-21ssh00:00:313
80199446,22cyclictest12816-21ssh23:46:023
80199446,19cyclictest8050irq/56-dwc_otg_00:14:123
801994416,18cyclictest31160-21ssh21:32:103
801994415,13cyclictest12610-21rm21:49:533
79999447,25cyclictest0-21swapper/123:35:321
79999447,19cyclictest16818-21ssh23:12:121
79999446,12cyclictest27868-21ssh22:45:521
79999446,11cyclictest7850irq/56-dwc_otg21:35:491
79999445,20cyclictest8050irq/56-dwc_otg_20:50:411
79999444,24cyclictest8050irq/56-dwc_otg_00:25:251
799994415,11cyclictest8019-21ssh23:01:151
799994415,11cyclictest8019-21ssh23:01:151
799994414,14cyclictest8050irq/56-dwc_otg_23:18:141
799994411,12cyclictest20336-21taskset20:02:361
80199437,25cyclictest0-21swapper/321:40:323
80199437,25cyclictest0-21swapper/321:40:323
80199436,11cyclictest7950irq/56-dwc_otg_23:12:353
80199436,11cyclictest18756-21modprobe20:00:173
80199434,25cyclictest8050irq/56-dwc_otg_19:19:323
801994314,19cyclictest12909-21ssh22:29:193
801994314,19cyclictest12909-21ssh22:29:193
801994310,18cyclictest4890-21ssh23:36:513
80099437,22cyclictest13831-21ssh22:30:252
80099434,13cyclictest61-21kswapd023:25:302
80099434,13cyclictest61-21kswapd023:25:302
800994311,3cyclictest0-21swapper/200:15:372
79999438,24cyclictest0-21swapper/122:30:311
79999437,15cyclictest20109-21latency_hist00:35:191
79999436,23cyclictest0-21swapper/120:55:331
799994310,16cyclictest23747-21ssh22:41:041
799994310,12cyclictest23475-21if_eth023:20:381
80199429,12cyclictest8050irq/56-dwc_otg_19:46:113
80199429,12cyclictest8050irq/56-dwc_otg_19:46:113
80199427,23cyclictest0-21swapper/322:10:333
801994213,13cyclictest5809-21ssh00:17:133
801994212,13cyclictest27536-21ssh22:07:003
801994210,13cyclictest11693-21latency_hist00:25:183
80099428,22cyclictest61-21kswapd023:35:302
79999425,11cyclictest13884-21seq23:09:251
79999424,24cyclictest75050snmpd19:46:071
79999424,24cyclictest75050snmpd19:46:071
799994213,17cyclictest12520-21/usr/sbin/munin19:40:411
799994212,7cyclictest0-21swapper/119:20:301
799994212,7cyclictest0-21swapper/119:20:301
801994115,19cyclictest381ktimersoftd/322:43:543
80099417,24cyclictest0-21swapper/220:50:322
800994119,10cyclictest10593-21sh23:05:312
79999417,9cyclictest8050irq/56-dwc_otg_23:45:281
79999415,11cyclictest7950irq/56-dwc_otg_20:40:361
80199409,13cyclictest9370-21taskset23:03:443
80199409,13cyclictest9370-21taskset23:03:443
80199405,10cyclictest8050irq/56-dwc_otg_22:46:493
800994018,5cyclictest15508-21aten_rbpower_vo19:50:332
799994011,11cyclictest11695-21diskmemload22:29:451
799994011,11cyclictest11695-21diskmemload22:29:451
80199398,10cyclictest0-21swapper/320:56:133
80199397,10cyclictest8050irq/56-dwc_otg_20:22:093
80199395,24cyclictest17775-21ssh00:31:223
801993923,13cyclictest1348-21latency19:10:383
800993915,5cyclictest22695-21/usr/sbin/munin22:40:372
79999395,11cyclictest0-21swapper/120:19:211
80199386,10cyclictest0-21swapper/321:04:263
80199385,20cyclictest13825-21ssh22:30:243
801993814,11cyclictest5756-21chrt19:22:073
801993814,11cyclictest5756-21chrt19:22:073
80099383,10cyclictest21103-21latency_hist22:00:202
80099383,10cyclictest21103-21latency_hist22:00:202
80099381,4cyclictest7214-21ntp_states23:00:412
80099381,4cyclictest7214-21ntp_states23:00:412
79999389,12cyclictest2085-21taskset19:11:301
79999384,11cyclictest9550irq/48-DMA15:26:471
79999381,26cyclictest22838-21cut20:10:361
80199371,4cyclictest8050irq/56-dwc_otg_20:25:303
80099375,8cyclictest7850irq/56-dwc_otg23:20:302
80099371,6cyclictest24500-21munin-run21:25:182
800993715,5cyclictest3492-21sh21:38:202
800993714,5cyclictest2988050wget20:30:322
800993714,5cyclictest2988050wget20:30:322
79999377,21cyclictest644-21runrttasks20:39:161
80199368,13cyclictest11661-21taskset21:10:083
80199368,13cyclictest11661-21taskset21:10:083
80199365,9cyclictest0-21swapper/319:51:363
80099361,24cyclictest12144-21ls19:40:352
79999362,9cyclictest7850irq/56-dwc_otg21:01:561
80199353,9cyclictest0-21swapper/320:39:223
80099353,6cyclictest8050irq/56-dwc_otg_22:10:182
80099351,23cyclictest0-21swapper/221:16:492
80099351,19cyclictest0-21swapper/221:30:432
79999356,11cyclictest0-21swapper/120:22:281
80099345,20cyclictest23764-21ssh00:00:322
80099342,5cyclictest7850irq/56-dwc_otg21:10:312
79999346,15cyclictest8050irq/56-dwc_otg_19:35:191
80199334,10cyclictest0-21swapper/320:02:433
801993312,11cyclictest25682-21chrt20:16:533
80099338,5cyclictest7950irq/56-dwc_otg_00:30:472
80099337,4cyclictest7950irq/56-dwc_otg_00:20:402
80099335,11cyclictest7950irq/56-dwc_otg_22:10:342
800993317,5cyclictest9867-21cron21:05:182
800993316,5cyclictest23779-21/usr/sbin/munin20:10:472
12792330,2sleep20-21swapper/222:15:202
80099325,7cyclictest7950irq/56-dwc_otg_20:45:342
80099324,5cyclictest29130-21unixbench_multi20:25:462
80099324,13cyclictest8050irq/56-dwc_otg_21:41:492
80099324,13cyclictest8050irq/56-dwc_otg_21:41:492
80099322,3cyclictest7850irq/56-dwc_otg20:15:352
80099321,7cyclictest20124-21grep20:00:472
79999327,3cyclictest7850irq/56-dwc_otg21:10:101
801993117,11cyclictest8162-21/usr/sbin/munin19:30:473
800993113,7cyclictest11695-21diskmemload23:18:302
80099306,9cyclictest7950irq/56-dwc_otg_22:25:362
80099306,9cyclictest7950irq/56-dwc_otg_22:25:362
80099305,6cyclictest32012-21ssh00:10:312
80099304,8cyclictest0-21swapper/219:55:472
80099301,7cyclictest27342-21grep20:20:462
4522302,22sleep10-21swapper/119:09:331
80099297,6cyclictest7950irq/56-dwc_otg_19:15:182
80099292,4cyclictest11529-21modprobe19:40:182
80099292,10cyclictest4467-21basename19:20:322
80099292,10cyclictest4467-21basename19:20:322
800992916,7cyclictest6768-21cut20:55:372
80099286,9cyclictest7950irq/56-dwc_otg_21:09:202
80099286,9cyclictest7950irq/56-dwc_otg_21:09:202
80099285,5cyclictest8050irq/56-dwc_otg_19:15:442
80099285,4cyclictest7950irq/56-dwc_otg_19:50:172
80099285,4cyclictest7950irq/56-dwc_otg_19:50:172
80099281,18cyclictest21064-21grep20:05:342
80099281,17cyclictest11695-21diskmemload22:49:552
80099262,5cyclictest738-21latency_hist20:40:182
80099262,3cyclictest0-21swapper/219:30:342
326332263,14sleep30-21swapper/319:05:553
3102263,13sleep20-21swapper/219:07:402
80099254,5cyclictest7850irq/56-dwc_otg20:40:302
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional