You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 21:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Wed Aug 19, 2026 12:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
154229460,5sleep20-21swapper/207:05:352
126925810,5sleep30-21swapper/307:05:323
23009912239,32cyclictest21893-21/usr/sbin/munin09:20:252
2298991175,71cyclictest0-21swapper/010:05:250
2299991152,89cyclictest7850irq/56-dwc_otg_09:50:301
2299991104,7cyclictest0-21swapper/110:50:351
22989910915,75cyclictest13-21rcuc/009:55:340
22989910733,51cyclictest0-21swapper/010:30:310
22989910733,51cyclictest0-21swapper/010:30:310
2298991044,75cyclictest0-21swapper/009:50:290
2298991035,68cyclictest6196-21kworker/0:110:50:350
22989910310,70cyclictest0-21swapper/009:35:330
22999910212,67cyclictest30684-21apt-get10:10:211
22999910212,67cyclictest30684-21apt-get10:10:211
22989910210,76cyclictest0-21swapper/009:20:260
2298999914,62cyclictest31110-21ssh09:30:350
2298999910,67cyclictest0-21swapper/010:10:210
2298999910,67cyclictest0-21swapper/010:10:210
229999974,10cyclictest7750irq/56-dwc_otg09:20:241
2300999339,34cyclictest14909-21netstat09:50:272
253252910,2sleep3381ktimersoftd/312:00:343
253252910,2sleep3381ktimersoftd/312:00:343
229999913,8cyclictest7750irq/56-dwc_otg09:55:351
229999903,9cyclictest0-21swapper/109:30:401
229999893,78cyclictest15554-21ssh10:30:291
229999893,78cyclictest15554-21ssh10:30:291
2300998838,24cyclictest2689-21processes09:35:342
229999882,11cyclictest8661-21ssh11:40:311
229999874,17cyclictest2669-21ssh09:35:341
229999843,13cyclictest26840-21diskstats10:05:241
2301998319,33cyclictest37-21rcuc/309:20:253
2300998362,15cyclictest30966-21ntp_states09:30:322
229899836,57cyclictest0-21swapper/011:40:300
230199823,53cyclictest31295-21ls09:30:383
2300998135,24cyclictest19260-21ssh09:55:362
2300998131,33cyclictest7950irq/56-dwc_otg_11:40:302
2300998040,16cyclictest26846-21diskstats10:05:262
109462780,36sleep37950irq/56-dwc_otg_10:25:213
229999762,14cyclictest0-21swapper/107:25:181
229899766,22cyclictest101ktimersoftd/010:15:230
7882750,4sleep130500-1kworker/u9:012:10:271
260322750,3sleep1229999cyclictest12:01:001
260322750,3sleep1229999cyclictest12:01:001
2298997528,16cyclictest0-21swapper/012:20:210
128632730,2sleep20-21swapper/211:05:462
244372710,5sleep2230099cyclictest08:10:262
244372710,5sleep2230099cyclictest08:10:262
2301997130,21cyclictest7850irq/56-dwc_otg_09:50:303
2300997038,25cyclictest15592-21ls10:30:292
2300997038,25cyclictest15592-21ls10:30:292
229899702,56cyclictest0-21swapper/010:20:360
2300996941,22cyclictest30694-21ssh10:10:202
2300996941,22cyclictest30694-21ssh10:10:202
230099693,59cyclictest0-21swapper/210:50:352
2299996918,32cyclictest28292-21sh11:25:241
229899692,53cyclictest4380-21fschecks_count07:15:250
229899682,52cyclictest22285-21wget08:05:220
229899682,50cyclictest4409-21ssh11:35:280
230199673,42cyclictest0-21swapper/311:40:313
2300996729,25cyclictest26069-21ssh09:25:212
229899672,51cyclictest12032-21grep10:25:350
229899671,57cyclictest15070-21tr07:45:210
229899671,55cyclictest0-21swapper/009:05:070
229999668,36cyclictest0-21swapper/112:06:171
2301996525,20cyclictest7850irq/56-dwc_otg_12:20:213
229999651,7cyclictest6532-21missed_timers07:20:291
229999651,7cyclictest6532-21missed_timers07:20:291
2300996419,21cyclictest7750irq/56-dwc_otg12:35:182
229999644,14cyclictest2499-21ssh10:15:211
229899646,44cyclictest25117-21grep11:20:350
229899642,51cyclictest13948-21ssh09:50:070
2298996412,39cyclictest26681-21sh09:25:290
229899639,42cyclictest29689-21grep08:25:250
229899632,33cyclictest0-21swapper/011:30:180
229899632,33cyclictest0-21swapper/011:30:180
2300996224,10cyclictest0-21swapper/207:25:182
230099622,10cyclictest7750irq/56-dwc_otg09:05:212
229999627,5cyclictest7850irq/56-dwc_otg_07:55:241
229999624,48cyclictest8425-21grep08:55:301
229999624,48cyclictest8425-21grep08:55:301
229999621,7cyclictest27929-21latency_hist11:25:081
229899622,44cyclictest9499-21latency_hist09:00:080
229899622,44cyclictest9499-21latency_hist09:00:080
229899621,51cyclictest0-21swapper/011:10:230
229899621,47cyclictest14666-21unixbench_multi09:10:370
229899621,47cyclictest14666-21unixbench_multi09:10:370
229899612,47cyclictest0-21swapper/012:00:380
229899612,47cyclictest0-21swapper/012:00:380
229899612,34cyclictest0-21swapper/011:15:270
229899611,50cyclictest0-21swapper/009:15:350
229899611,47cyclictest21030-21ntp_kernel_err08:00:300
229999607,4cyclictest24439-21unixbench_singl10:40:361
229999603,4cyclictest7750irq/56-dwc_otg09:14:361
229999603,4cyclictest7750irq/56-dwc_otg09:14:361
2298996023,18cyclictest7850irq/56-dwc_otg_09:10:070
229899601,45cyclictest28057-21ntpq10:45:320
2300995925,18cyclictest3662-21ls10:15:372
2300995920,19cyclictest17805-21rm11:51:032
229999596,9cyclictest7950irq/56-dwc_otg_10:04:151
229899592,45cyclictest0-21swapper/010:40:320
2300995820,24cyclictest23935-21ssh11:59:452
229999587,4cyclictest0-21swapper/111:01:351
229899581,48cyclictest0-21swapper/008:30:230
229899581,42cyclictest8489-21ps07:25:310
2298995813,31cyclictest3889-21dhclient-script08:44:390
2301995717,15cyclictest7750irq/56-dwc_otg09:55:343
2300995721,15cyclictest5928-21/usr/sbin/munin07:20:262
2300995721,15cyclictest5928-21/usr/sbin/munin07:20:262
229999578,9cyclictest7950irq/56-dwc_otg_12:25:311
229999573,4cyclictest7950irq/56-dwc_otg_08:50:211
229999571,16cyclictest7950irq/56-dwc_otg_11:45:321
229899575,39cyclictest11617-21uniq07:35:250
2298995728,6cyclictest7850irq/56-dwc_otg_11:50:230
94192560,2sleep20-21swapper/210:23:192
230099566,20cyclictest0-21swapper/209:10:122
230099566,20cyclictest0-21swapper/209:10:122
2300995620,18cyclictest29784-21if_err_eth008:25:262
2300995618,24cyclictest28293-21ssh11:25:242
2300995616,21cyclictest7750irq/56-dwc_otg08:55:262
2300995616,21cyclictest7750irq/56-dwc_otg08:55:262
230099561,4cyclictest0-21swapper/207:15:182
229999567,8cyclictest7950irq/56-dwc_otg_11:36:551
229999565,10cyclictest0-21swapper/111:20:041
229899562,46cyclictest6196-21kworker/0:110:56:270
229899561,45cyclictest0-21swapper/012:15:280
230199554,25cyclictest0-21swapper/310:10:193
230199554,25cyclictest0-21swapper/310:10:193
2300995525,12cyclictest13922-21ssh11:46:292
2300995518,11cyclictest29165-21apt08:25:072
229999556,11cyclictest0-21swapper/110:35:411
229999553,9cyclictest7950irq/56-dwc_otg_11:56:581
229999553,7cyclictest6463-21taskset09:40:271
229999553,7cyclictest6463-21taskset09:40:271
229999553,5cyclictest7750irq/56-dwc_otg10:29:141
229999552,6cyclictest0-21swapper/112:20:191
229999552,3cyclictest7750irq/56-dwc_otg11:00:061
229999551,3cyclictest0-21swapper/110:23:121
2299995512,5cyclictest0-21swapper/111:50:241
229899556,40cyclictest9-21ksoftirqd/011:05:210
229899554,39cyclictest0-21swapper/007:10:260
2298995528,5cyclictest0-21swapper/010:00:300
229899552,43cyclictest21665-21grep12:35:290
229899551,46cyclictest0-21swapper/007:40:300
230199547,7cyclictest7950irq/56-dwc_otg_10:05:273
2300995424,10cyclictest0-21swapper/207:50:072
2300995423,13cyclictest27071-21ssh12:02:552
2300995423,13cyclictest27071-21ssh12:02:552
2300995422,20cyclictest25734-21ssh10:42:552
229999541,3cyclictest7950irq/56-dwc_otg_07:15:321
229999541,3cyclictest0-21swapper/112:15:201
229999541,11cyclictest30478-21ssh10:49:031
229999541,11cyclictest0-21swapper/109:03:461
229899545,40cyclictest20964-21latency11:55:290
2298995427,13cyclictest0-21swapper/012:28:040
229899542,43cyclictest20165-21ssh12:34:200
230199538,16cyclictest32178-21ssh11:30:183
230199538,16cyclictest32178-21ssh11:30:183
2300995325,12cyclictest901-21ls12:10:282
2300995324,11cyclictest17394-21ssh11:11:302
2300995323,15cyclictest19505-21timerandwakeup07:55:332
2300995319,14cyclictest22805-21ls10:40:072
229999534,7cyclictest7950irq/56-dwc_otg_11:08:281
229999534,6cyclictest32421-21diskstats11:30:231
229999534,6cyclictest32421-21diskstats11:30:231
229999531,14cyclictest18079-21grep12:30:371
229899532,42cyclictest0-21swapper/007:32:590
229899531,43cyclictest0-21swapper/010:35:320
229899531,42cyclictest31517-21systemd-journal07:53:380
230199522,8cyclictest7950irq/56-dwc_otg_10:30:293
230199522,8cyclictest7950irq/56-dwc_otg_10:30:293
2301995211,11cyclictest1795-21/usr/sbin/munin09:35:343
2300995224,17cyclictest22605-21ssh11:18:332
2300995219,14cyclictest3970-21latency_hist07:15:082
229999524,5cyclictest0-21swapper/108:25:301
229999523,6cyclictest0-21swapper/112:35:291
229999523,4cyclictest0-21swapper/109:17:131
229999522,5cyclictest31517-21systemd-journal08:33:461
229999522,4cyclictest14083-21sed09:50:081
2298995231,13cyclictest7950irq/56-dwc_otg_08:20:320
2300995125,11cyclictest19921-21ssh12:33:532
2300995120,13cyclictest12968-21diskmemload10:00:262
2300995119,13cyclictest21623-21cut09:20:072
2300995116,12cyclictest13038-21fschecks_time12:25:252
2300995116,11cyclictest29173-21latency12:05:272
229999512,5cyclictest7750irq/56-dwc_otg09:26:441
229999512,4cyclictest17696-21rm11:12:031
229999512,11cyclictest24962-21grep08:10:341
229999512,11cyclictest24962-21grep08:10:341
229999511,8cyclictest0-21swapper/109:09:161
2298995130,5cyclictest7950irq/56-dwc_otg_09:40:180
2298995130,5cyclictest7950irq/56-dwc_otg_09:40:180
2298995128,13cyclictest7850irq/56-dwc_otg_08:15:070
2298995128,13cyclictest7850irq/56-dwc_otg_08:15:070
229899512,43cyclictest0-21swapper/007:23:230
229899512,43cyclictest0-21swapper/007:23:230
2300995024,11cyclictest9936-21ssh11:03:002
2300995017,17cyclictest4967-21ssh12:15:292
229999503,3cyclictest7850irq/56-dwc_otg_07:41:521
229999502,5cyclictest0-21swapper/107:47:141
229999501,5cyclictest1191-21snmpd08:43:481
2298995030,12cyclictest0-21swapper/012:10:210
229899501,34cyclictest0-21swapper/008:35:190
309302490,34sleep30-21swapper/310:49:533
230199493,15cyclictest12968-21diskmemload10:50:363
2300994922,12cyclictest10882-21ssh12:22:272
2300994920,11cyclictest28827-21rm10:46:032
2300994918,23cyclictest9979-21ssh09:45:102
2300994916,13cyclictest5458-21ssh10:57:212
2300994916,12cyclictest26416-21ssh11:22:462
229999496,22cyclictest0-21swapper/107:50:201
229899491,42cyclictest0-21swapper/007:58:530
229899491,39cyclictest0-21swapper/011:04:460
2298994910,20cyclictest0-21swapper/011:45:180
2300994822,17cyclictest1490-21ssh11:31:242
2300994822,17cyclictest1490-21ssh11:31:242
2300994818,15cyclictest23625-21taskset08:08:512
2300994817,20cyclictest7420-21ssh11:39:422
2300994817,11cyclictest12450-21ssh10:26:172
2300994816,11cyclictest9328-21ssh09:44:242
2300994816,11cyclictest9328-21ssh09:44:242
229999486,8cyclictest129012sleep107:40:071
229899486,31cyclictest0-21swapper/011:29:540
229899481,38cyclictest0-21swapper/012:05:110
2298994812,21cyclictest26956-21awk08:15:370
2300994718,12cyclictest20355-21/usr/sbin/munin08:00:262
229999471,3cyclictest0-21swapper/107:31:271
229899472,36cyclictest0-21swapper/008:53:320
2300994618,12cyclictest32368-21uptime08:30:352
2300994616,19cyclictest10773-21taskset07:31:042
229899461,29cyclictest5320-21grep08:45:360
16952462,36sleep00-21swapper/007:05:590
229999451,3cyclictest7850irq/56-dwc_otg_08:36:101
229999451,3cyclictest0-21swapper/107:12:511
230199444,3cyclictest7950irq/56-dwc_otg_11:15:193
2300994417,13cyclictest31517-21systemd-journal08:38:472
2300994414,11cyclictest11051-21taskset09:03:582
229999441,13cyclictest0-21swapper/108:08:541
118772440,5sleep2230099cyclictest07:35:282
2300994325,15cyclictest13366-21df_inode07:40:232
2300994316,11cyclictest0-21swapper/208:15:352
2300994312,15cyclictest5970-21chrt08:50:152
230199427,25cyclictest0-21swapper/311:25:213
2300994214,11cyclictest0-21swapper/208:47:082
2300994214,10cyclictest0-21swapper/208:41:562
239702410,1sleep30-21swapper/310:00:573
2300994111,13cyclictest16738-21chrt07:50:142
229999411,11cyclictest0-21swapper/108:49:581
230199397,22cyclictest0-21swapper/308:00:213
2301993918,7cyclictest20708-21fschecks_time11:55:263
230199391,24cyclictest21019-21ssh12:35:213
230199373,11cyclictest7950irq/56-dwc_otg_09:40:213
230199373,11cyclictest7950irq/56-dwc_otg_09:40:213
230199367,26cyclictest0-21swapper/308:45:213
2301993620,9cyclictest16646-21ssh09:13:573
2301993620,9cyclictest16646-21ssh09:13:573
2301993617,13cyclictest30840-21sh12:07:163
2301993615,5cyclictest28151-21memory08:20:283
2301993615,4cyclictest12968-21diskmemload09:46:493
229999362,20cyclictest28339-21wc08:20:321
229999361,13cyclictest0-21swapper/108:03:501
2301993518,9cyclictest14715-21idleruntime-cro07:45:073
2301993516,9cyclictest7950irq/56-dwc_otg_09:15:343
2301993516,5cyclictest15293-21cat11:10:073
230199351,30cyclictest6183-21ls08:50:223
230199351,22cyclictest26076-21ssh09:25:223
2301993510,6cyclictest7850irq/56-dwc_otg_08:15:073
2301993510,6cyclictest7850irq/56-dwc_otg_08:15:073
229999351,3cyclictest7750irq/56-dwc_otg08:15:311
2301993414,6cyclictest12583-21sh12:25:103
230199336,14cyclictest0-21swapper/307:45:203
2301993313,5cyclictest7950irq/56-dwc_otg_09:10:073
2301993313,4cyclictest381ktimersoftd/307:12:343
230199329,4cyclictest0-21swapper/311:35:253
230199328,6cyclictest7609-21switchtime10:20:343
230199326,5cyclictest7850irq/56-dwc_otg_07:30:203
230199324,5cyclictest7850irq/56-dwc_otg_08:35:213
2301993214,9cyclictest11271-21ssh11:05:083
230199321,22cyclictest8034-21uniq08:55:253
230199321,22cyclictest8034-21uniq08:55:253
2301993212,10cyclictest7727-21ssh12:19:143
230199321,19cyclictest31697-21ls08:30:273
230199321,18cyclictest2646-21cut08:40:243
230199321,12cyclictest7850irq/56-dwc_otg_07:55:083
2301993211,11cyclictest381ktimersoftd/308:05:213
230199319,4cyclictest0-21swapper/312:10:353
230199319,4cyclictest0-21swapper/311:10:223
230199318,4cyclictest7950irq/56-dwc_otg_11:20:343
230199318,3cyclictest0-21swapper/309:05:073
230199317,12cyclictest0-21swapper/310:44:313
230199312,21cyclictest7950irq/56-dwc_otg_10:15:203
2301993120,7cyclictest381ktimersoftd/307:20:213
2301993120,7cyclictest381ktimersoftd/307:20:213
230199311,13cyclictest7950irq/56-dwc_otg_07:55:183
230199309,4cyclictest0-21swapper/311:49:583
230199306,14cyclictest0-21swapper/308:25:213
230199302,4cyclictest0-21swapper/311:50:353
230199302,14cyclictest0-21swapper/312:30:263
2301993013,9cyclictest5084-21sh10:56:403
2301993012,11cyclictest213882chrt10:37:333
230199291,7cyclictest7850irq/56-dwc_otg_07:25:353
230199282,5cyclictest0-21swapper/307:17:533
19272285,15sleep10-21swapper/107:08:591
230199267,9cyclictest0-21swapper/308:15:213
230199252,11cyclictest0-21swapper/307:36:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional