You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 22:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue Jul 15, 2025 12:48:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1424421050,5sleep055199cyclictest10:50:040
551998435,9cyclictest8050irq/56-dwc_otg_10:35:190
280332820,2sleep00-21swapper/010:21:460
219962820,3sleep20-21swapper/212:28:262
55199817,26cyclictest0-21swapper/008:45:190
146982810,2sleep014694-21awk11:35:290
259972770,1sleep20-21swapper/209:30:362
551997143,2cyclictest0-21swapper/007:15:150
113442710,2sleep311331-21ssh11:30:333
3243199707,51cyclictest32380-21apt-get06:25:170
551996818,41cyclictest19277-21latency_hist10:10:050
551996818,41cyclictest19277-21latency_hist10:10:050
551996619,36cyclictest32648-21tune2fs12:00:220
551996526,7cyclictest26217-21head11:50:400
560996418,28cyclictest25751-21ntpq11:05:283
56099631,22cyclictest0-21swapper/307:20:193
553996210,31cyclictest0-21swapper/107:30:171
55199622,31cyclictest0-21swapper/009:40:040
551996215,5cyclictest9-21ksoftirqd/011:05:050
560996120,13cyclictest3334-21munin-run12:05:043
55199612,7cyclictest32612-21systemd-journal12:25:540
55199602,42cyclictest31813-21grep08:35:370
192172600,2sleep30-21swapper/309:20:383
551995931,13cyclictest8050irq/56-dwc_otg_12:35:330
551995931,13cyclictest8050irq/56-dwc_otg_12:35:330
55199591,7cyclictest24611-21munin-plugin-st11:50:040
551995915,26cyclictest9550irq/80-mmc011:20:050
180392590,6sleep155399cyclictest12:23:171
560995833,8cyclictest14370-21apt-get07:50:163
55199585,6cyclictest0-21swapper/008:55:340
551995829,5cyclictest26417-21grep08:20:350
551995815,5cyclictest6000-21wc07:25:320
70132570,1sleep20-21swapper/211:25:252
560995722,22cyclictest17196-21/usr/sbin/munin10:05:343
560995722,22cyclictest17196-21/usr/sbin/munin10:05:343
551995735,5cyclictest13003-21fschecks_time10:00:260
56099566,32cyclictest0-21swapper/308:45:103
56099562,10cyclictest7850irq/56-dwc_otg10:45:183
55399566,26cyclictest0-21swapper/111:05:181
55199567,32cyclictest0-21swapper/007:45:350
551995630,5cyclictest8050irq/56-dwc_otg_09:55:360
551995627,12cyclictest8050irq/56-dwc_otg_10:25:320
55199561,5cyclictest2964-21ssh09:45:260
55199561,31cyclictest0-21swapper/010:15:050
560995516,24cyclictest7418-21ssh12:10:103
55199554,30cyclictest5985-21ssh09:50:150
55199552,27cyclictest419550ssh11:20:380
560995420,15cyclictest3808-21/usr/sbin/munin08:50:193
560995420,12cyclictest7165-21munin-run12:10:053
560995413,26cyclictest8050irq/56-dwc_otg_09:57:323
551995429,4cyclictest7850irq/56-dwc_otg09:25:210
551995426,4cyclictest10307-21grep09:05:340
551995414,16cyclictest0-21swapper/011:25:330
560995325,16cyclictest8964-21ssh09:54:463
560995324,15cyclictest14525-21dpkg10:50:063
560995324,15cyclictest14525-21dpkg10:50:063
560995316,21cyclictest7950irq/56-dwc_otg_12:23:343
56099531,21cyclictest0-21swapper/309:00:213
56099531,21cyclictest0-21swapper/309:00:213
55199532,28cyclictest3832-21aten_rbpower_cu08:50:190
55199531,27cyclictest15506-21unixbench_singl07:50:420
560995217,20cyclictest5650-21ssh11:23:503
55399524,28cyclictest7850irq/56-dwc_otg11:25:331
55399522,4cyclictest8050irq/56-dwc_otg_11:20:201
55199522,28cyclictest23632-21ntp_kernel_pll_10:15:320
55199522,10cyclictest0-21swapper/012:40:170
55199521,25cyclictest7611-21wget12:10:230
551995210,24cyclictest0-21swapper/009:30:240
560995126,11cyclictest22459-21sort08:10:303
560995120,18cyclictest604-21ssh12:42:153
560995118,17cyclictest1868-21ssh09:44:213
560995112,11cyclictest32627-21ssh11:15:263
55199515,23cyclictest8940-21aten_rbpower_vo07:35:190
55199512,33cyclictest0-21swapper/008:05:240
55199512,33cyclictest0-21swapper/008:05:240
551995117,17cyclictest30010-21tail08:30:330
551995117,17cyclictest30010-21tail08:30:330
551995115,20cyclictest19105-21unixbench_singl08:00:340
55199511,4cyclictest32499-21/usr/sbin/munin08:40:310
55199511,38cyclictest0-21swapper/010:41:540
55199511,29cyclictest24852-21kworker/0:108:25:230
3243399514,8cyclictest8050irq/56-dwc_otg_06:25:182
56099506,16cyclictest0-21swapper/311:35:193
560995024,19cyclictest19333-21ssh11:41:323
560995024,19cyclictest19333-21ssh11:41:323
560995022,17cyclictest29877-21ssh09:36:213
560995020,10cyclictest12523-21ssh09:10:513
560995011,17cyclictest22592-21unixbench_singl09:25:343
55199509,3cyclictest4714-21expr12:05:320
55199503,7cyclictest8050irq/56-dwc_otg_07:30:170
55199503,20cyclictest15273-21runrttasks10:50:290
55199503,20cyclictest15273-21runrttasks10:50:290
551995030,2cyclictest0-21swapper/009:00:190
551995030,2cyclictest0-21swapper/009:00:190
551995024,5cyclictest11119-21ssh09:10:170
55199502,40cyclictest0-21swapper/012:24:190
551995019,8cyclictest0-21swapper/011:55:170
214042500,1sleep20-21swapper/210:12:142
560994920,16cyclictest29629-21ls11:10:353
560994920,16cyclictest29629-21ls11:10:353
560994919,18cyclictest30785-21ssh11:57:203
560994918,11cyclictest21471-21ssh11:00:113
560994917,12cyclictest16797-21/usr/sbin/munin07:55:293
55899497,25cyclictest0-21swapper/211:10:192
55899497,25cyclictest0-21swapper/211:10:192
55399494,10cyclictest0-21swapper/108:25:191
553994911,9cyclictest8050irq/56-dwc_otg_10:35:191
55199491,7cyclictest29068-21ssh11:10:280
55199491,7cyclictest29068-21ssh11:10:280
551994911,22cyclictest11863-21unixbench_multi07:40:320
560994819,20cyclictest16062-21rm09:16:243
560994817,11cyclictest12264-21/usr/sbin/munin12:15:303
560994815,14cyclictest9221-21ssh11:28:353
55399481,5cyclictest0-21swapper/110:20:171
55199482,27cyclictest4370-21grep07:20:320
55199481,26cyclictest10582-21fschecks_time11:30:230
56099477,25cyclictest0-21swapper/308:05:233
56099477,25cyclictest0-21swapper/308:05:233
560994719,13cyclictest20869-21ssh12:26:093
560994716,10cyclictest5573-21ssh09:49:493
560994714,12cyclictest18347-21ssh10:55:243
560994714,10cyclictest0-21swapper/310:44:443
560994713,26cyclictest31085-21ssh10:25:433
55899477,27cyclictest0-21swapper/212:30:192
55199473,5cyclictest8050irq/56-dwc_otg_11:05:180
55199472,7cyclictest8050irq/56-dwc_otg_12:30:180
55199472,30cyclictest0-21swapper/008:10:170
551994718,5cyclictest7950irq/56-dwc_otg_12:20:040
551994714,14cyclictest0-21swapper/011:00:050
55199471,26cyclictest0-21swapper/010:31:030
32434994723,8cyclictest32351-31find06:25:133
56099467,25cyclictest0-21swapper/307:40:113
56099467,24cyclictest0-21swapper/308:40:203
560994621,16cyclictest12570-21ssh10:00:093
560994619,10cyclictest30334-21ssh12:39:363
560994619,10cyclictest30334-21ssh12:39:363
560994618,13cyclictest27243-21ssh09:33:243
560994614,13cyclictest10035-21vmstat07:35:343
560994613,22cyclictest14088-21chrt07:49:533
558994623,13cyclictest661-21runrttasks12:40:322
55199464,23cyclictest0-21swapper/007:55:200
560994517,8cyclictest0-21swapper/311:45:263
560994516,11cyclictest24882-21ssh12:31:273
560994515,13cyclictest28061-21ssh11:54:313
56099447,23cyclictest0-21swapper/308:30:193
56099447,23cyclictest0-21swapper/308:30:193
56099445,25cyclictest0-21swapper/307:30:193
560994424,5cyclictest19714-21wc08:05:063
560994419,13cyclictest22074-21ssh10:13:553
560994414,10cyclictest28335-21taskset08:26:083
560994411,8cyclictest0-21swapper/310:34:343
560994411,21cyclictest28028-21ssh10:21:463
560994410,23cyclictest6058-21chrt10:37:073
55399447,23cyclictest0-21swapper/108:15:111
55199441,30cyclictest14460-21acpi09:15:160
56099437,26cyclictest0-21swapper/308:55:203
56099437,24cyclictest0-21swapper/307:15:213
56099437,22cyclictest0-21swapper/308:15:113
560994316,12cyclictest6582-21chrt07:27:273
55399432,24cyclictest0-21swapper/112:30:181
553994320,8cyclictest0-21swapper/109:25:261
55399431,33cyclictest15785-21latency_hist10:05:061
551994320,4cyclictest7850irq/56-dwc_otg09:40:300
551994313,5cyclictest10771-21diskmemload11:43:480
551994313,5cyclictest10771-21diskmemload11:43:480
56099426,20cyclictest8050irq/56-dwc_otg_08:23:083
55399422,9cyclictest30715-21sed12:40:061
55399422,9cyclictest30715-21sed12:40:061
55399421,15cyclictest0-21swapper/111:30:171
551994221,9cyclictest8050irq/56-dwc_otg_08:15:090
551994215,16cyclictest20958-21ssh09:24:530
19062420,2sleep20-21swapper/211:18:202
560994115,11cyclictest23495-21ls10:15:293
558994122,7cyclictest15214-21ssh09:15:312
558994114,5cyclictest28229-21idleruntime-cro11:10:052
55399417,22cyclictest7850irq/56-dwc_otg09:35:321
55399414,29cyclictest8050irq/56-dwc_otg_12:40:171
56099407,21cyclictest0-21swapper/308:35:173
55399404,6cyclictest7141-21ssh12:10:051
55399402,7cyclictest0-21swapper/111:39:161
55399402,4cyclictest8050irq/56-dwc_otg_07:35:161
55399402,24cyclictest7850irq/56-dwc_otg08:40:181
55399402,12cyclictest7950irq/56-dwc_otg_10:15:351
55399401,29cyclictest0-21swapper/112:25:221
553994011,19cyclictest0-21swapper/109:10:041
55899398,20cyclictest11103-21apt-get09:10:162
55899391,22cyclictest0-21swapper/209:00:212
55899391,22cyclictest0-21swapper/209:00:212
55399398,3cyclictest0-21swapper/110:30:351
55399398,19cyclictest20083-21fschecks_time08:05:271
55399398,19cyclictest20083-21fschecks_time08:05:271
55399394,22cyclictest0-21swapper/107:50:161
553993933,4cyclictest9002-21ls09:05:051
553993933,4cyclictest9002-21ls09:05:051
553993921,5cyclictest15249-21ssh10:50:291
553993921,5cyclictest15249-21ssh10:50:291
55399391,4cyclictest9497-21ssh09:55:211
55399391,28cyclictest0-21swapper/109:15:321
118732390,4sleep211871-21fschecks_time12:15:252
55899387,22cyclictest0-21swapper/208:45:192
55899385,24cyclictest7950irq/56-dwc_otg_08:55:232
558993822,9cyclictest19815-21ssh10:56:382
558993819,6cyclictest21381-21aten_rbpower_vo11:45:182
55899381,24cyclictest0-21swapper/208:10:202
55399387,4cyclictest7950irq/56-dwc_otg_09:30:331
55399386,22cyclictest28573-21ssh11:10:191
55399386,22cyclictest28573-21ssh11:10:191
55399384,5cyclictest7850irq/56-dwc_otg10:40:221
55399382,5cyclictest7149-21sed09:50:331
553993818,5cyclictest19555-21modprobe08:05:051
55399381,10cyclictest14121-21apt07:50:041
40042380,3sleep24003-21df08:50:222
560993712,9cyclictest8050irq/56-dwc_otg_09:05:073
55899377,20cyclictest32612-21systemd-journal11:35:042
558993717,4cyclictest3172-21sh12:04:472
558993715,6cyclictest15666-21tune2fs12:20:232
55899371,25cyclictest30310-21ls10:25:282
558993712,15cyclictest0-21swapper/212:10:202
55399376,22cyclictest7950irq/56-dwc_otg_12:15:361
55399375,24cyclictest7850irq/56-dwc_otg11:56:521
55399374,6cyclictest0-21swapper/107:55:201
55399372,3cyclictest0-21swapper/110:29:431
553993722,5cyclictest661-21runrttasks08:55:351
553993718,9cyclictest841-21perl07:15:151
55399371,6cyclictest0-21swapper/109:22:241
55399371,5cyclictest32075-21fschecks_time09:40:231
55399371,5cyclictest20980-21ssh10:59:361
55399371,5cyclictest12372-21grep09:10:331
55399371,3cyclictest0-21swapper/111:55:051
553993711,16cyclictest3527-21basename07:20:201
553993710,18cyclictest221ktimersoftd/111:00:281
55899367,18cyclictest3994-21latency_hist10:35:062
55899362,7cyclictest14583-21df07:50:282
558993619,5cyclictest29397-21nfs4_client11:55:282
558993611,13cyclictest19218-21idleruntime-cro10:10:052
558993611,13cyclictest19218-21idleruntime-cro10:10:052
55399367,19cyclictest7850irq/56-dwc_otg12:10:211
55399364,5cyclictest0-21swapper/107:40:151
55399364,24cyclictest7850irq/56-dwc_otg11:40:191
55399364,24cyclictest7850irq/56-dwc_otg11:40:191
55399361,7cyclictest30805-21aten_rbpower_po08:35:231
55399361,5cyclictest21339-21aten_rbpower_en11:45:171
55399361,25cyclictest29500-21missed_timers08:30:271
55399361,25cyclictest29500-21missed_timers08:30:271
553993610,13cyclictest21-21rcuc/111:15:121
55899359,14cyclictest13931-21/usr/sbin/munin11:35:352
55899357,19cyclictest0-21swapper/210:48:352
55899354,21cyclictest0-21swapper/208:40:332
55899352,4cyclictest8050irq/56-dwc_otg_07:15:172
558993521,8cyclictest7950irq/56-dwc_otg_11:24:322
55399357,18cyclictest7850irq/56-dwc_otg10:10:061
55399357,18cyclictest7850irq/56-dwc_otg10:10:061
55399357,16cyclictest7850irq/56-dwc_otg09:45:261
55399356,22cyclictest0-21swapper/108:45:191
55399354,23cyclictest7950irq/56-dwc_otg_12:00:231
553993514,4cyclictest19805-21grep10:10:231
55899348,4cyclictest8050irq/56-dwc_otg_10:20:272
55899347,19cyclictest7950irq/56-dwc_otg_12:35:192
55899347,19cyclictest7950irq/56-dwc_otg_12:35:192
55899346,18cyclictest0-21swapper/208:00:332
55899346,11cyclictest8050irq/56-dwc_otg_07:35:192
55899346,10cyclictest8050irq/56-dwc_otg_09:55:362
55899343,9cyclictest0-21swapper/207:40:152
55899341,8cyclictest17282-21unixbench_multi07:55:362
558993415,12cyclictest1609550ssh10:51:042
558993415,12cyclictest1609550ssh10:51:042
558993415,10cyclictest2837-21grep09:45:252
558993414,10cyclictest0-21swapper/211:41:412
558993414,10cyclictest0-21swapper/211:41:412
558993410,12cyclictest27392-21cut08:25:252
55399342,4cyclictest0-21swapper/107:25:211
553993420,5cyclictest661-21runrttasks08:50:181
55899334,5cyclictest7950irq/56-dwc_otg_11:50:342
55899333,3cyclictest7950irq/56-dwc_otg_09:25:252
558993317,9cyclictest4104-21ssh10:35:162
558993315,10cyclictest24882-21wc11:05:052
558993312,4cyclictest26921-21cat08:25:052
55899331,21cyclictest7850irq/56-dwc_otg08:34:232
55899331,21cyclictest7850irq/56-dwc_otg08:34:232
55899331,21cyclictest0-21swapper/208:06:382
55899331,21cyclictest0-21swapper/208:06:382
55899331,19cyclictest0-21swapper/209:21:412
55899331,13cyclictest6867-21ssh09:50:302
553993315,5cyclictest8050irq/56-dwc_otg_08:20:351
55399331,4cyclictest7850irq/56-dwc_otg08:10:151
3243299336,19cyclictest60-21kswapd006:25:201
55899325,20cyclictest0-21swapper/210:15:532
55899323,6cyclictest7850irq/56-dwc_otg08:35:592
55899323,19cyclictest7950irq/56-dwc_otg_12:05:222
55899321,6cyclictest9967-21wc09:05:312
558993212,13cyclictest301ktimersoftd/210:40:202
55399328,9cyclictest0-21swapper/110:48:181
55899314,5cyclictest7850irq/56-dwc_otg10:00:252
55899312,3cyclictest0-21swapper/207:30:312
55899311,7cyclictest32042-21fschecks_time09:40:232
55899311,6cyclictest661-21runrttasks07:25:352
558993116,5cyclictest31476-21latency_hist09:40:042
55899311,22cyclictest0-21swapper/207:20:192
55899311,21cyclictest0-21swapper/208:15:312
558993110,4cyclictest13690-21unixbench_singl07:45:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional