You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 13:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Nov 14, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
280121410,2sleep20-21swapper/222:20:202
564021050,1sleep30-21swapper/323:48:293
230299917,72cyclictest7061-21grep19:20:330
2302998817,58cyclictest1521-21grep00:25:350
2302998013,53cyclictest12915-21tr23:15:260
2302998013,41cyclictest8165-21aten_rbpower_cu20:55:160
2302997912,55cyclictest18743-21wget19:55:160
2302997811,46cyclictest10215-21cron23:55:020
2302997612,53cyclictest14831-21sh00:00:210
2302997612,53cyclictest14831-21sh00:00:210
2302997612,47cyclictest21865-21/usr/sbin/munin21:20:220
2302997612,47cyclictest21865-21/usr/sbin/munin21:20:220
2302997610,45cyclictest11361-21unixbench_multi22:30:320
230299753,39cyclictest11898-21ssh21:50:160
230952740,2sleep20-21swapper/220:05:312
230299746,46cyclictest29701-21grep20:25:160
2302997411,44cyclictest21024-21apt21:20:020
2302997410,48cyclictest0-21swapper/019:15:180
230399728,45cyclictest7950irq/56-dwc_otg_19:15:261
230299728,43cyclictest28993-21latency_hist00:20:020
245642710,4sleep324569-21unixbench_multi22:05:323
230399716,15cyclictest0-21swapper/122:45:091
230399713,12cyclictest7950irq/56-dwc_otg_20:55:301
230399712,4cyclictest7950irq/56-dwc_otg_23:10:011
230299709,43cyclictest13197-21taskset21:08:580
230299708,45cyclictest0-21swapper/021:45:160
230399693,57cyclictest1146-21snmpd23:35:161
230299697,51cyclictest28630-21sh23:35:300
177552690,1sleep20-21swapper/200:04:472
177552690,1sleep20-21swapper/200:04:472
165672690,2sleep00-21swapper/022:38:140
2303996818,35cyclictest6029-21sh23:49:161
230299687,36cyclictest0-21swapper/020:40:020
2302996815,37cyclictest30872-21ssh22:56:040
202022680,2sleep10-21swapper/123:25:211
230299678,46cyclictest0-21swapper/022:15:300
230299676,45cyclictest0-21swapper/019:10:160
230299674,35cyclictest0-21swapper/019:40:160
2302996716,38cyclictest21166-21ssh22:44:530
2302996712,46cyclictest26764-21unixbench_singl22:50:340
2302996712,46cyclictest26764-21unixbench_singl22:50:340
2302996643,7cyclictest7950irq/56-dwc_otg_21:35:130
230399656,48cyclictest7950irq/56-dwc_otg_19:50:231
230299657,45cyclictest13309-21diskmemload22:10:210
230299655,47cyclictest0-21swapper/000:14:410
230299655,35cyclictest29242-21ssh21:30:210
2302996512,29cyclictest27543-21latency_hist23:35:030
2302996512,29cyclictest27543-21latency_hist23:35:030
2302996512,12cyclictest10494-21ssh23:55:080
230399646,5cyclictest15610-21unixbench_singl00:00:311
230399646,5cyclictest15610-21unixbench_singl00:00:311
230399641,5cyclictest344-21latency_hist23:00:021
230299645,48cyclictest0-21swapper/023:49:060
230299643,51cyclictest0-21swapper/000:20:230
303942630,3sleep230397-21timerandwakeup22:55:312
230399633,16cyclictest7750irq/56-dwc_otg21:20:131
230399633,16cyclictest7750irq/56-dwc_otg21:20:131
230399631,3cyclictest7950irq/56-dwc_otg_23:55:151
230299639,44cyclictest22893-21tail22:45:350
230299637,34cyclictest0-21swapper/021:25:410
230299636,44cyclictest0-21swapper/020:41:580
230299636,37cyclictest0-21swapper/000:35:340
230399629,4cyclictest0-21swapper/123:11:561
230399626,5cyclictest19564-21switchtime19:55:281
230299626,43cyclictest0-21swapper/019:36:530
230299624,42cyclictest0-21swapper/020:00:480
230299624,27cyclictest0-21swapper/021:00:160
2302996212,32cyclictest0-21swapper/022:05:010
172982620,1sleep20-21swapper/221:15:022
230299616,43cyclictest0-21swapper/023:40:430
2302996115,32cyclictest5880-21ssh22:24:530
2302996113,26cyclictest0-21swapper/022:05:140
2302996110,38cyclictest6424-21taskset21:42:250
230399605,11cyclictest7950irq/56-dwc_otg_22:25:021
230399604,7cyclictest7950irq/56-dwc_otg_22:50:191
230399604,7cyclictest7950irq/56-dwc_otg_22:50:191
230399601,4cyclictest15703-21ls19:45:231
230299609,39cyclictest17699-21ssh23:21:060
230299606,43cyclictest0-21swapper/022:27:370
2302996012,20cyclictest16105-21/usr/sbin/munin21:55:200
2302996011,12cyclictest3796-21ssh23:04:140
230399592,48cyclictest0-21swapper/120:20:161
2303995918,38cyclictest4301-21latency_hist23:05:021
230399591,4cyclictest7750irq/56-dwc_otg00:40:011
230299594,44cyclictest0-21swapper/020:14:440
230399587,42cyclictest6514-21cpuspeed20:50:191
230399587,42cyclictest6514-21cpuspeed20:50:191
230399584,5cyclictest7850irq/56-dwc_otg_21:35:211
230399582,13cyclictest24610-21unixbench_singl22:05:321
2302995812,28cyclictest21089-21ssh00:08:500
2302995810,39cyclictest0-21swapper/023:26:450
230399577,39cyclictest0-21swapper/122:14:151
230399575,10cyclictest7950irq/56-dwc_otg_22:02:421
230399572,44cyclictest7950irq/56-dwc_otg_20:06:191
230399571,9cyclictest13669-21ssh21:10:161
230299577,22cyclictest0-21swapper/020:45:180
230299574,35cyclictest0-21swapper/020:20:030
2302995722,20cyclictest7750irq/56-dwc_otg19:50:180
230399562,8cyclictest25776-21latency_hist20:15:021
230399562,4cyclictest17744-21/usr/sbin/munin22:40:291
230399562,13cyclictest0-21swapper/121:17:221
230299569,22cyclictest0-21swapper/023:10:150
230299568,11cyclictest16879-21ssh21:14:310
230299563,29cyclictest0-21swapper/020:50:220
230299563,29cyclictest0-21swapper/020:50:220
230299561,40cyclictest0-21swapper/020:20:100
230399553,8cyclictest7950irq/56-dwc_otg_21:32:351
230399552,4cyclictest7850irq/56-dwc_otg_21:48:121
230399551,4cyclictest7850irq/56-dwc_otg_22:25:211
2303995514,11cyclictest0-21swapper/123:20:131
230399551,11cyclictest7950irq/56-dwc_otg_20:03:391
230299556,38cyclictest0-21swapper/000:33:540
230299556,38cyclictest0-21swapper/000:33:540
230299555,47cyclictest6695-21rm23:07:210
230399543,5cyclictest7950irq/56-dwc_otg_20:25:231
230399542,9cyclictest7950irq/56-dwc_otg_00:28:311
230399542,6cyclictest31762-21ntp_states22:15:251
230399542,11cyclictest7950irq/56-dwc_otg_00:20:291
230399542,10cyclictest3075-21proc_pri19:10:271
230399541,45cyclictest0-21swapper/121:56:391
230399541,3cyclictest0-21swapper/123:30:161
230399541,3cyclictest0-21swapper/123:30:161
53242530,2sleep30-21swapper/300:31:253
53242530,2sleep30-21swapper/300:31:253
230599533,24cyclictest7850irq/56-dwc_otg_20:50:163
230599533,24cyclictest7850irq/56-dwc_otg_20:50:163
230399532,7cyclictest7850irq/56-dwc_otg_00:12:061
230399531,4cyclictest0-21swapper/100:32:261
230399531,4cyclictest0-21swapper/100:32:261
230399531,3cyclictest0-21swapper/119:30:021
230399531,13cyclictest0-21swapper/123:41:131
230399531,12cyclictest0-21swapper/121:52:451
230399526,15cyclictest0-21swapper/119:35:071
230399525,6cyclictest25450-21fschecks_count21:25:191
230399525,4cyclictest7750irq/56-dwc_otg00:05:561
230399523,7cyclictest7950irq/56-dwc_otg_22:36:151
230399521,5cyclictest0-21swapper/123:52:041
230399521,10cyclictest7750irq/56-dwc_otg23:15:261
230399511,6cyclictest0-21swapper/100:17:181
230399511,3cyclictest7750irq/56-dwc_otg20:35:221
230399511,3cyclictest0-21swapper/119:20:171
230399509,4cyclictest14234-21timerandwakeup19:40:291
230399502,5cyclictest32139-21runrttasks20:30:271
230399501,9cyclictest0-21swapper/121:04:011
2302995024,13cyclictest8326-21cat19:25:230
230399491,3cyclictest7950irq/56-dwc_otg_20:50:021
230399491,12cyclictest0-21swapper/122:33:471
230399491,11cyclictest13505-21sort21:10:021
230299499,28cyclictest0-21swapper/019:49:530
230299492,28cyclictest575-21latency_hist20:35:020
2302994913,21cyclictest23568-21taskset20:06:340
230399481,9cyclictest2786-21grep20:40:181
230399481,35cyclictest0-21swapper/121:41:511
2305994715,17cyclictest14375-21ssh00:00:143
2305994715,17cyclictest14375-21ssh00:00:143
2304994717,23cyclictest7950irq/56-dwc_otg_23:15:162
230299470,35cyclictest0-21swapper/019:31:170
230599467,23cyclictest0-21swapper/321:50:173
230499467,25cyclictest0-21swapper/219:30:162
230399461,3cyclictest7950irq/56-dwc_otg_20:17:141
230499456,31cyclictest0-21swapper/200:10:182
2304994522,15cyclictest31-21ksoftirqd/221:50:142
230499451,6cyclictest0-21swapper/220:00:122
19042453,11sleep00-21swapper/019:08:210
19042453,11sleep00-21swapper/019:08:210
230599447,24cyclictest0-21swapper/321:45:153
230599446,25cyclictest0-21swapper/323:30:193
230599446,25cyclictest0-21swapper/323:30:193
230599437,26cyclictest0-21swapper/323:00:153
230599437,24cyclictest0-21swapper/319:20:173
230499433,4cyclictest0-21swapper/223:45:172
11672436,25sleep31178-21/usr/sbin/munin19:05:253
11672436,25sleep31178-21/usr/sbin/munin19:05:253
230499427,25cyclictest0-21swapper/222:40:162
230499427,23cyclictest0-21swapper/220:55:162
230499426,24cyclictest0-21swapper/200:30:182
230499426,24cyclictest0-21swapper/200:30:182
230499425,14cyclictest0-21swapper/221:30:162
230499422,14cyclictest7950irq/56-dwc_otg_23:20:312
2304994213,6cyclictest31-21ksoftirqd/200:20:202
230399421,10cyclictest7950irq/56-dwc_otg_19:34:011
230599417,23cyclictest0-21swapper/322:40:163
2305994113,5cyclictest5796-21awk23:05:323
230499417,24cyclictest0-21swapper/223:35:172
230499417,22cyclictest0-21swapper/223:30:192
230499417,22cyclictest0-21swapper/223:30:192
230499417,16cyclictest31-21ksoftirqd/221:20:252
230499417,16cyclictest31-21ksoftirqd/221:20:252
230499416,8cyclictest7950irq/56-dwc_otg_21:00:172
2304994132,7cyclictest7950irq/56-dwc_otg_22:10:162
2304994122,10cyclictest301ktimersoftd/222:30:332
2304994119,5cyclictest1120-21runrttasks23:50:302
230599406,24cyclictest0-21swapper/323:55:173
2305994015,14cyclictest13309-21diskmemload22:15:213
230499407,24cyclictest0-21swapper/220:10:202
230499405,7cyclictest7950irq/56-dwc_otg_20:25:142
230499405,24cyclictest0-21swapper/200:15:182
230599399,7cyclictest31197-21idleruntime-cro20:30:013
230599397,22cyclictest0-21swapper/320:55:163
2305993917,7cyclictest3389-21/usr/sbin/munin22:20:293
230599391,10cyclictest3884-21idleruntime-cro21:40:013
230499396,17cyclictest7850irq/56-dwc_otg_21:45:182
230499393,6cyclictest25487-21ssh22:50:152
230499393,6cyclictest25487-21ssh22:50:152
230499392,24cyclictest7750irq/56-dwc_otg19:55:182
2305993819,11cyclictest28698-21ssh00:19:323
2305993815,11cyclictest7787-21ssh00:35:313
230599381,24cyclictest26155-21cut20:15:223
230499386,17cyclictest7950irq/56-dwc_otg_21:20:022
230499386,12cyclictest301ktimersoftd/222:02:122
2304993814,6cyclictest31-21ksoftirqd/200:25:332
230499381,25cyclictest655-21ls20:35:022
2304993810,16cyclictest20238-21ssh23:25:232
230599373,28cyclictest7950irq/56-dwc_otg_23:20:143
2305993712,4cyclictest29463-21latency_hist20:25:033
230499378,20cyclictest7750irq/56-dwc_otg23:10:242
230499377,8cyclictest29-21rcuc/223:01:272
230499376,18cyclictest301ktimersoftd/223:09:512
230499376,15cyclictest31-21ksoftirqd/223:57:282
230499376,12cyclictest29-21rcuc/220:41:552
230499371,15cyclictest7850irq/56-dwc_otg_19:15:162
230599369,17cyclictest11-21rcu_preempt21:05:133
230599361,25cyclictest10222-21grep21:00:223
230499367,24cyclictest0-21swapper/219:50:172
230499367,11cyclictest29-21rcuc/222:47:252
230499366,17cyclictest29-21rcuc/200:39:312
2304993616,5cyclictest15931-21sh21:55:172
230499361,5cyclictest14894-21switchtime22:35:302
230599357,8cyclictest0-21swapper/319:15:243
230599354,7cyclictest7950irq/56-dwc_otg_21:30:013
230599353,17cyclictest7950irq/56-dwc_otg_23:26:543
230599352,6cyclictest140-21systemd-journal22:01:033
230499359,11cyclictest7750irq/56-dwc_otg22:16:212
230499357,8cyclictest29-21rcuc/221:40:022
230499357,5cyclictest27985-21fschecks_count20:20:192
2304993514,9cyclictest301ktimersoftd/223:40:422
2304993514,8cyclictest6669-21ssh22:25:202
230499351,17cyclictest7753-21latency_hist21:45:022
230499351,15cyclictest31-21ksoftirqd/219:20:332
230599349,11cyclictest7850irq/56-dwc_otg_20:40:223
230599348,5cyclictest7950irq/56-dwc_otg_20:30:323
230599347,7cyclictest7950irq/56-dwc_otg_20:05:163
230599343,4cyclictest7950irq/56-dwc_otg_00:05:173
230599341,23cyclictest19695-21timerwakeupswit19:55:293
2305993410,12cyclictest0-21swapper/323:15:083
230499347,24cyclictest0-21swapper/220:50:182
230499347,24cyclictest0-21swapper/220:50:182
230499343,5cyclictest0-21swapper/219:35:142
2304993414,9cyclictest301ktimersoftd/222:05:272
2304993411,8cyclictest7750irq/56-dwc_otg00:08:502
2305993325,5cyclictest381ktimersoftd/319:25:133
2305993313,4cyclictest9810-21kworker/u8:322:30:023
230599331,26cyclictest11978-21latency19:35:223
230599331,24cyclictest10063-21ls19:30:233
230599331,21cyclictest0-21swapper/321:40:193
230599331,15cyclictest29013-21apt-get21:30:163
230499331,27cyclictest15431-21ls19:45:202
230499331,11cyclictest7850irq/56-dwc_otg_20:35:222
2304993310,5cyclictest31-21ksoftirqd/219:10:162
2304993310,5cyclictest12860-21tail21:05:302
230599328,12cyclictest0-21swapper/300:13:413
230599326,14cyclictest0-21swapper/322:10:253
2305993225,5cyclictest22878-21ssh21:21:003
2305993225,5cyclictest22878-21ssh21:21:003
230599322,4cyclictest7850irq/56-dwc_otg_19:45:273
2305993224,5cyclictest29581-21fschecks_count22:55:203
230599321,21cyclictest26475-21switchtime22:50:313
230599321,21cyclictest26475-21switchtime22:50:313
230599321,19cyclictest14420-21ssh21:10:263
230599321,19cyclictest11832-21latency_hist23:15:023
230599321,16cyclictest0-21swapper/319:14:233
230499323,3cyclictest0-21swapper/221:25:272
2304993218,6cyclictest1146-21snmpd19:25:162
5122317,11sleep20-21swapper/219:05:162
5122317,11sleep20-21swapper/219:05:162
230599316,13cyclictest7950irq/56-dwc_otg_23:35:193
230599312,5cyclictest0-21swapper/322:45:163
230599312,16cyclictest0-21swapper/319:51:053
2305993119,5cyclictest16284-21ssh21:55:223
230599311,6cyclictest6987-21df_inode23:50:213
2305993113,7cyclictest0-21swapper/323:40:293
2305993111,4cyclictest0-21swapper/321:17:493
230499311,21cyclictest7750irq/56-dwc_otg20:46:312
230599303,5cyclictest7950irq/56-dwc_otg_22:30:213
230599302,13cyclictest7950irq/56-dwc_otg_00:20:233
230599301,15cyclictest7950irq/56-dwc_otg_20:45:193
230599292,19cyclictest0-21swapper/320:35:463
230599291,7cyclictest1146-21snmpd19:40:193
2305992913,5cyclictest494-21cpuspeed00:25:213
230499297,15cyclictest0-21swapper/220:15:182
230599285,10cyclictest7850irq/56-dwc_otg_20:00:123
230499281,17cyclictest0-21swapper/219:40:152
230599273,5cyclictest7950irq/56-dwc_otg_22:35:103
230599272,5cyclictest7850irq/56-dwc_otg_20:10:223
16612253,14sleep10-21swapper/119:05:311
16612253,14sleep10-21swapper/119:05:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional