You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-16 - 02:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sun Aug 16, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15663991565,134cyclictest0-21swapper/022:30:240
15663991412,133cyclictest28338-21taskset21:10:370
15663991384,63cyclictest5617-21fschecks_count00:00:270
15663991333,99cyclictest0-21swapper/023:05:380
15663991332,123cyclictest4416-21apt-get23:20:120
15663991312,105cyclictest1032-21idleruntime23:15:270
15663991303,54cyclictest32362-21irqstats21:55:290
15663991252,64cyclictest13545-21apt-get00:10:210
15663991243,90cyclictest23895-21grep21:45:240
15663991243,62cyclictest12231-21if_wlan022:10:270
15663991208,82cyclictest0-21swapper/023:40:340
15663991193,93cyclictest9233-21fschecks_count23:25:240
15663991192,55cyclictest3952-21cat22:00:280
15663991182,65cyclictest7850irq/56-dwc_otg_00:20:070
15663991173,106cyclictest0-21swapper/023:00:210
15663991153,81cyclictest0-21swapper/022:25:280
15663991103,74cyclictest9717-21grep00:05:260
15663991102,91cyclictest19958-21/usr/sbin/munin22:20:240
15663991092,47cyclictest7850irq/56-dwc_otg_00:20:090
15663991074,73cyclictest2312-21grep00:35:260
15663991074,73cyclictest2312-21grep00:35:260
156639910559,14cyclictest7750irq/56-dwc_otg22:50:370
15664991047,18cyclictest7750irq/56-dwc_otg00:20:081
15664991034,78cyclictest0-21swapper/100:20:081
15663991031,83cyclictest26584-21cat00:25:260
15664991004,13cyclictest5443-21ssh23:20:301
156649910011,71cyclictest7950irq/56-dwc_otg_23:00:191
1566399964,71cyclictest0-21swapper/023:45:300
1566399964,71cyclictest0-21swapper/023:45:300
15666999439,21cyclictest26142-21ssh23:05:373
1566499943,10cyclictest13623-21unixbench-2d22:50:351
1566399942,74cyclictest0-21swapper/020:20:210
15666999344,23cyclictest21522-21ssh00:20:063
15665999214,56cyclictest4789-21ssh23:20:122
1566399922,43cyclictest840-21latency_hist23:55:050
25212910,22sleep37950irq/56-dwc_otg_21:19:503
15666999136,24cyclictest10479-21ssh00:05:373
15666999136,23cyclictest28397-21cat21:10:373
15664999115,9cyclictest28350-21ls21:10:371
1566399913,74cyclictest0-21swapper/021:30:320
1566399911,75cyclictest16277-21missed_timers19:10:280
1566399911,73cyclictest0-21swapper/022:40:160
15666999034,34cyclictest14284-1kworker/u9:221:55:203
1566399902,75cyclictest0-21swapper/019:30:330
15666998933,34cyclictest9248-21sort23:25:243
1566499899,15cyclictest5688-21fschecks_time00:00:281
15663998950,11cyclictest17942-21ssh23:35:300
1566399891,5cyclictest0-21swapper/022:37:080
15666998834,33cyclictest0-21swapper/322:10:233
15666998813,50cyclictest0-21swapper/322:20:253
1566499883,14cyclictest0-21swapper/123:35:291
1566399881,73cyclictest17356-21ntpq22:55:300
15666998739,22cyclictest23891-21grep21:45:243
1566499873,16cyclictest11729-21/usr/sbin/munin22:10:281
1566399874,70cyclictest28089-21fschecks_time21:50:230
1566499865,11cyclictest0-21swapper/123:05:371
1566499864,15cyclictest10624-21ls00:05:401
1566499863,17cyclictest28573-21sort22:30:241
15664998612,8cyclictest7750irq/56-dwc_otg21:55:171
1566399862,73cyclictest30146-21wget00:30:210
1566399861,68cyclictest19597-21fschecks_time19:20:230
1566499855,13cyclictest0-21swapper/121:45:241
1566499854,13cyclictest0-21swapper/123:25:231
1566399852,63cyclictest17803-21fschecks_time19:15:240
1566499845,72cyclictest0-21swapper/122:20:261
1566499844,15cyclictest24822-21ntp_kernel_pll_22:25:281
1566399842,69cyclictest0-21swapper/023:10:470
1566399842,65cyclictest0-21swapper/019:35:360
15666998337,16cyclictest20932-21diskstats23:00:203
1566499835,13cyclictest0-21swapper/123:15:241
1566499834,11cyclictest0-21swapper/123:40:341
15664998318,8cyclictest3945-21basename22:00:271
1566399831,5cyclictest21881-21grep19:25:270
15666998219,41cyclictest0-21swapper/300:20:093
15664998217,58cyclictest0-21swapper/100:35:271
15664998217,58cyclictest0-21swapper/100:35:271
1566399822,65cyclictest0-21swapper/021:41:370
1566399821,39cyclictest17758-21latency_hist20:45:060
1566399812,64cyclictest0-21swapper/023:33:210
1566399812,61cyclictest0-21swapper/022:06:470
1566499805,8cyclictest0-21swapper/100:10:221
1566399802,64cyclictest0-21swapper/023:57:500
1566399801,60cyclictest7138-21cut20:15:200
1566399801,60cyclictest7138-21cut20:15:200
15665997936,17cyclictest7750irq/56-dwc_otg21:10:382
1566399791,68cyclictest0-21swapper/021:18:460
1566399791,66cyclictest0-21swapper/022:15:300
1566399791,40cyclictest49172sleep021:21:400
15666997844,28cyclictest2352-21/usr/sbin/munin00:35:273
15666997844,28cyclictest2352-21/usr/sbin/munin00:35:273
15666997836,28cyclictest8138-21kworker/u8:422:30:263
15666997825,26cyclictest0-21swapper/322:50:353
15666997811,45cyclictest5196-21fschecks_time23:20:243
1566399781,67cyclictest0-21swapper/021:38:470
1566499774,10cyclictest0-21swapper/123:45:311
1566499774,10cyclictest0-21swapper/123:45:311
1566399771,57cyclictest0-21swapper/020:55:200
1566399771,52cyclictest0-21swapper/020:45:300
15665997612,46cyclictest28572-21cut22:30:242
1566399761,49cyclictest0-21swapper/019:45:280
142632760,6sleep30-21swapper/319:05:233
142632760,6sleep30-21swapper/319:05:233
269742740,2sleep30-21swapper/323:46:053
269742740,2sleep30-21swapper/323:46:053
15666997444,23cyclictest4032-21/usr/sbin/munin22:00:293
15666997443,25cyclictest597-21/usr/sbin/munin23:15:263
15666997443,24cyclictest22092-21sh23:40:323
1566399746,52cyclictest0-21swapper/020:05:190
1566399741,3cyclictest0-21swapper/021:27:220
47062730,2sleep10-21swapper/123:59:361
15665997327,13cyclictest7850irq/56-dwc_otg_00:00:222
1566399732,63cyclictest0-21swapper/022:48:590
15666997233,15cyclictest13545-21apt-get00:10:223
15666997142,23cyclictest17897-21/usr/sbin/munin23:35:293
1566599717,21cyclictest301ktimersoftd/221:55:292
15665997133,23cyclictest9811-21fschecks_time00:05:272
1566599713,31cyclictest23902-21fschecks_count21:45:242
1566599712,45cyclictest301ktimersoftd/200:20:072
1566399701,3cyclictest0-21swapper/020:01:310
1566699694,38cyclictest0-21swapper/300:00:193
1566599693,36cyclictest0-21swapper/223:15:262
15665996927,9cyclictest7850irq/56-dwc_otg_22:10:252
1566399696,55cyclictest0-21swapper/019:40:110
15666996839,23cyclictest24774-21grep22:25:273
15665996837,15cyclictest7750irq/56-dwc_otg22:50:362
15665996829,14cyclictest7750irq/56-dwc_otg23:00:212
1566599679,34cyclictest0-21swapper/222:20:262
1566599674,37cyclictest0-21swapper/222:25:282
15664996713,9cyclictest11799-21/usr/sbin/munin21:30:251
1566399671,5cyclictest0-21swapper/020:36:470
1566399671,50cyclictest0-21swapper/021:04:340
15666996623,9cyclictest26579-21missed_timers00:25:263
1566599669,37cyclictest0-21swapper/222:00:292
15665996627,14cyclictest7850irq/56-dwc_otg_23:35:302
1566399661,2cyclictest0-21swapper/020:53:270
1566399661,2cyclictest0-21swapper/020:53:270
248832650,2sleep10-21swapper/119:35:061
15666996521,27cyclictest1693-21/usr/sbin/munin23:55:253
1566599658,37cyclictest7850irq/56-dwc_otg_23:05:372
1566499651,6cyclictest11419-21idleruntime-cro22:10:051
1566399652,48cyclictest0-21swapper/020:31:120
1566399651,3cyclictest0-21swapper/020:12:420
194132640,2sleep10-21swapper/122:19:431
1566599649,32cyclictest301ktimersoftd/200:20:092
1566599647,31cyclictest301ktimersoftd/200:35:272
1566599647,31cyclictest301ktimersoftd/200:35:272
15664996421,20cyclictest18712-21kworker/1:219:20:171
1566499641,4cyclictest28898-21grep21:50:341
15664996410,4cyclictest0-21swapper/122:40:241
1566399641,3cyclictest121632sleep020:28:260
1566399641,2cyclictest0-21swapper/019:57:470
1566399631,2cyclictest0-21swapper/019:53:300
1566399621,4cyclictest0-21swapper/021:07:220
15666996129,10cyclictest7024-21apt-get20:15:173
15666996129,10cyclictest7024-21apt-get20:15:173
15665996110,35cyclictest9226-21fschecks_count23:25:242
15665996024,16cyclictest22082-21ntp_states23:40:322
1566499597,5cyclictest7850irq/56-dwc_otg_20:50:261
1566499597,5cyclictest7850irq/56-dwc_otg_20:50:261
1566599585,32cyclictest7850irq/56-dwc_otg_00:10:212
1566499581,5cyclictest7750irq/56-dwc_otg23:30:241
15666995629,16cyclictest15826-21apt-get22:15:163
15666995625,12cyclictest23918-21ps19:30:323
15666995623,21cyclictest9533-21/usr/sbin/munin20:20:253
15666995616,13cyclictest5087-21ssh21:21:583
1566599563,27cyclictest7950irq/56-dwc_otg_23:45:312
1566599563,27cyclictest7950irq/56-dwc_otg_23:45:312
7182550,2sleep10-21swapper/100:33:541
15666995527,8cyclictest12559-21users21:30:353
1566499558,3cyclictest0-21swapper/121:41:181
1566499555,6cyclictest7750irq/56-dwc_otg20:30:221
15666995424,8cyclictest21667-21irqstats19:25:243
15666995424,12cyclictest14329-21ssh23:30:353
15666995419,11cyclictest2493-21ssh22:37:483
15664995412,6cyclictest7950irq/56-dwc_otg_19:35:201
15666995311,23cyclictest0-21swapper/322:55:233
1566499532,9cyclictest7950irq/56-dwc_otg_21:25:191
15666995226,20cyclictest0-21swapper/319:10:083
15666995219,20cyclictest7950irq/56-dwc_otg_22:45:583
15666995218,14cyclictest29265-21/usr/sbin/munin19:45:283
15666995117,18cyclictest7950irq/56-dwc_otg_22:40:233
15666995116,25cyclictest23146-21runrttasks20:59:063
15666995116,24cyclictest28333-21sh21:50:263
1566499519,7cyclictest0-21swapper/120:40:261
15666995019,18cyclictest18163-21ntpq19:15:283
15666995017,13cyclictest619-21ssh00:33:433
15666994920,17cyclictest31840-21ssh23:52:253
1566499491,31cyclictest4052-21ssh21:20:331
15666994818,19cyclictest17960-21/usr/sbin/munin20:45:323
15666994815,11cyclictest7950irq/56-dwc_otg_20:04:203
1566599487,26cyclictest0-21swapper/221:35:212
15666994717,12cyclictest30541-21ssh23:11:073
15666994714,11cyclictest7950irq/56-dwc_otg_19:20:283
1566499472,3cyclictest0-21swapper/120:05:171
15666994620,16cyclictest9466-21taskset21:27:263
15666994618,20cyclictest15691-21chrt20:37:153
15666994618,11cyclictest10296-21ssh22:08:043
15666994616,12cyclictest0-21swapper/320:15:053
15666994611,18cyclictest7850irq/56-dwc_otg_21:38:523
1566599467,25cyclictest0-21swapper/220:50:172
1566599467,25cyclictest0-21swapper/220:50:172
151602463,35sleep00-21swapper/019:07:030
151602463,35sleep00-21swapper/019:07:030
15666994513,18cyclictest26618-21taskset21:07:293
1566599458,27cyclictest0-21swapper/221:30:192
1566599457,24cyclictest0-21swapper/220:35:212
15666994415,13cyclictest17531-21chrt20:42:423
15666994415,11cyclictest1227-21taskset19:58:473
15666994414,13cyclictest1180-21runrttasks19:35:363
1566599447,25cyclictest0-21swapper/222:40:192
1566499446,26cyclictest0-21swapper/120:20:171
1566599437,24cyclictest0-21swapper/219:25:122
1566499432,4cyclictest7950irq/56-dwc_otg_19:45:181
1566699425,21cyclictest1191-21snmpd20:05:163
15666994231,8cyclictest30433-21apt-get19:50:173
15666994216,10cyclictest21391-21rm21:41:403
1566499427,23cyclictest0-21swapper/119:40:111
15666994117,15cyclictest209782chrt20:50:473
15666994117,15cyclictest209782chrt20:50:473
1566599407,25cyclictest0-21swapper/219:10:202
1566599407,18cyclictest733-21ls19:55:332
1566599406,23cyclictest0-21swapper/222:45:202
1566499403,8cyclictest0-21swapper/119:10:081
1566599395,4cyclictest7750irq/56-dwc_otg23:50:342
1566599394,3cyclictest7950irq/56-dwc_otg_21:15:192
1566599392,23cyclictest7850irq/56-dwc_otg_20:20:192
1566599391,27cyclictest1059-21timerwakeupswit22:35:352
15666993824,11cyclictest13381-21expr20:30:323
1566599385,5cyclictest7750irq/56-dwc_otg21:25:342
1566599381,22cyclictest5158-21ls20:10:062
15665993810,10cyclictest301ktimersoftd/221:50:232
15666993726,8cyclictest26491-21apt-get19:40:123
15666993720,14cyclictest10980-21grep20:25:263
1566599378,19cyclictest0-21swapper/220:45:212
1566599374,13cyclictest7850irq/56-dwc_otg_22:05:352
1566599373,5cyclictest7750irq/56-dwc_otg19:15:192
1566599371,24cyclictest6849-21ls20:15:062
15665993710,6cyclictest31-21ksoftirqd/223:55:212
1566499378,16cyclictest0-21swapper/122:35:311
1566499374,9cyclictest16570-21/usr/sbin/munin22:55:361
15666993619,13cyclictest24625-21unixbench_singl21:00:343
1566499364,13cyclictest7950irq/56-dwc_otg_20:00:061
1566499362,6cyclictest0-21swapper/123:50:311
1566499361,25cyclictest3200-21ls20:05:051
1566599358,6cyclictest7850irq/56-dwc_otg_00:30:212
1566599358,4cyclictest7750irq/56-dwc_otg19:35:332
1566599357,18cyclictest0-21swapper/219:50:192
15665993513,9cyclictest7850irq/56-dwc_otg_20:45:062
1566599351,22cyclictest1923-21uniq20:00:242
1566599351,18cyclictest17275-21sed22:55:292
1566499358,15cyclictest7950irq/56-dwc_otg_23:10:271
1566499356,12cyclictest7850irq/56-dwc_otg_21:40:061
1566499354,6cyclictest7750irq/56-dwc_otg19:15:341
1566599347,14cyclictest0-21swapper/220:55:112
1566599341,4cyclictest29848-21ssh00:30:062
1566599341,23cyclictest19664-21cpuspeed21:40:192
1566499349,11cyclictest7950irq/56-dwc_otg_20:45:211
1566499348,5cyclictest7850irq/56-dwc_otg_21:15:231
1566499342,19cyclictest0-21swapper/120:59:331
15664993412,5cyclictest15369-21cat20:35:351
15664993410,4cyclictest0-21swapper/100:28:281
1566599337,7cyclictest7850irq/56-dwc_otg_23:32:202
1566599336,15cyclictest0-21swapper/219:30:112
15665993320,4cyclictest29-21rcuc/221:10:062
1566499334,7cyclictest7750irq/56-dwc_otg20:15:171
1566499334,7cyclictest7750irq/56-dwc_otg20:15:171
1566499334,6cyclictest7750irq/56-dwc_otg20:25:231
1566499334,5cyclictest12011-21apt-get22:50:051
1566499331,22cyclictest23800-21cut21:00:241
15664993310,5cyclictest7750irq/56-dwc_otg20:10:201
1566599328,15cyclictest0-21swapper/223:10:192
1566599327,5cyclictest7850irq/56-dwc_otg_20:15:282
1566599327,5cyclictest7850irq/56-dwc_otg_20:15:282
1566599322,4cyclictest7950irq/56-dwc_otg_19:20:172
1566599322,18cyclictest16196-21fschecks_time22:15:232
1566499323,13cyclictest7850irq/56-dwc_otg_19:25:251
1566599317,11cyclictest0-21swapper/221:00:192
1566599316,4cyclictest7950irq/56-dwc_otg_21:20:292
1566499311,6cyclictest31523-21tail19:50:341
1566499311,21cyclictest25545-21uniq21:05:231
1566599301,19cyclictest13138-21missed_timers20:30:282
1566599296,11cyclictest7850irq/56-dwc_otg_19:45:132
1566599290,22cyclictest27391-21memory19:40:272
151612286,15sleep10-21swapper/119:07:041
151612286,15sleep10-21swapper/119:07:041
150922273,15sleep20-21swapper/219:06:122
150922273,15sleep20-21swapper/219:06:122
15665992620,4cyclictest12136-21seq20:28:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional