You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 00:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Fri Nov 14, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
145821680,6sleep31846099cyclictest10:50:213
18457991001,77cyclictest1529-21fschecks_time10:50:220
1845799961,82cyclictest15011-21latency_hist11:10:030
18457999211,68cyclictest1120-21runrttasks08:29:110
18457999211,68cyclictest1120-21runrttasks08:29:110
1845799911,82cyclictest8049-21fschecks_count08:10:200
1845799891,74cyclictest21404-21grep08:45:280
1845799871,67cyclictest21405-21fschecks_count09:45:200
1845799861,68cyclictest26481-21ssh10:39:090
1845799851,69cyclictest28364-21wc11:30:020
1845799841,69cyclictest4272-21sed12:25:320
1845799832,34cyclictest18105-21cat12:00:240
1845799831,70cyclictest8613-21sh11:46:270
1845799831,37cyclictest6835-21latency_hist09:25:020
1845799831,37cyclictest6835-21latency_hist09:25:020
1845799821,71cyclictest19053-21ssh10:27:560
1845799821,55cyclictest0-21swapper/008:05:180
1845799811,67cyclictest0-21swapper/009:05:120
1845799811,59cyclictest3507-21grep10:05:220
1845799811,57cyclictest0-21swapper/012:20:200
1845799811,49cyclictest26733-21ssh09:52:320
211292800,1sleep30-21swapper/309:45:163
18457998058,6cyclictest7850irq/56-dwc_otg_08:40:020
1845799802,63cyclictest32212-21ssh11:35:160
1845799801,69cyclictest7915-21ssh10:11:000
1845799801,60cyclictest0-21swapper/011:13:120
89312790,2sleep10-21swapper/108:10:311
1845799791,60cyclictest18891-21aten_rbpower_en08:40:180
1845799791,56cyclictest29549-21diskmemload12:15:160
1845799791,29cyclictest1146-21snmpd11:32:290
1845799781,68cyclictest9354-21ssh09:28:170
1845799781,68cyclictest21583-21ssh10:30:430
1845799781,65cyclictest28316-21ssh09:55:180
1845799781,65cyclictest0-21swapper/009:33:410
1845799781,64cyclictest16310-21ssh11:57:370
18457997813,53cyclictest0-21swapper/009:10:350
173892785,59sleep217464-21/usr/sbin/munin07:05:302
199262770,2sleep20-21swapper/209:43:052
1845799771,73cyclictest27805-21ssh12:14:270
1845799771,63cyclictest3932-21taskset07:59:490
1845799771,62cyclictest0-21swapper/009:42:250
1845799771,55cyclictest0-21swapper/011:21:150
1845799771,55cyclictest0-21swapper/011:21:150
1845799771,53cyclictest0-21swapper/009:02:250
1845799761,61cyclictest11552-21ssh10:16:390
1845799761,61cyclictest0-21swapper/008:05:020
1845799761,61cyclictest0-21swapper/008:05:020
1845799761,60cyclictest6103-21ssh10:55:580
1845799761,59cyclictest0-21swapper/009:39:300
1845799761,57cyclictest1406-21ssh10:02:290
1845799761,54cyclictest0-21swapper/007:51:590
1845799761,53cyclictest12391-21chrt12:37:010
1845799761,48cyclictest0-21swapper/011:18:480
281722750,1sleep10-21swapper/109:05:161
1845799751,60cyclictest0-21swapper/010:23:450
1845799751,60cyclictest0-21swapper/010:23:450
1845799743,47cyclictest471750ssh11:40:550
1845799743,47cyclictest471750ssh11:40:550
1845799741,50cyclictest12411-21ssh11:52:060
1845799731,58cyclictest29991-21ssh10:44:460
1845799731,58cyclictest0-21swapper/008:15:440
1845899721,54cyclictest6556-21ls08:05:241
1845799721,68cyclictest98572sleep011:01:410
1845799721,49cyclictest22620-21ssh12:06:020
1845799711,58cyclictest8044-21runrttasks12:31:060
1845799711,56cyclictest0-21swapper/009:16:240
1845799711,50cyclictest0-21swapper/007:46:510
1845799711,46cyclictest0-21swapper/010:47:560
1845799701,54cyclictest0-21swapper/007:39:070
1845799701,50cyclictest0-21swapper/007:16:270
1845799691,60cyclictest0-21swapper/008:20:190
1845799681,53cyclictest19741-21chrt07:11:250
1845799681,50cyclictest23342-21runrttasks07:21:140
1845899671,56cyclictest1412150ssh11:55:171
1845899671,55cyclictest18626-21ls12:00:301
1845799671,53cyclictest25378-21chrt07:29:120
1845799671,53cyclictest25378-21chrt07:29:120
240042660,1sleep10-21swapper/111:22:271
240042660,1sleep10-21swapper/111:22:271
1845899661,54cyclictest15059-21ssh11:10:051
1845899651,42cyclictest32285-21wc11:35:181
1845799651,54cyclictest0-21swapper/007:31:470
1845799651,52cyclictest30607-21taskset07:41:440
305152640,2sleep10-21swapper/107:40:401
1845899642,42cyclictest7950irq/56-dwc_otg_10:25:341
1845899641,51cyclictest1959-21ssh12:23:081
1845899641,45cyclictest23838-21ssh12:08:521
1845899631,52cyclictest16339-21ssh09:37:591
1845899631,43cyclictest3557-21ssh10:53:161
1845899631,43cyclictest32565-21ssh09:13:591
1845899631,43cyclictest21409-21ps08:45:281
1845799631,58cyclictest0-21swapper/008:34:340
1845899622,49cyclictest11610-21ssh10:16:461
1845899621,50cyclictest12241-21ssh12:36:431
1845899611,51cyclictest8661-21ssh11:46:321
1845899611,48cyclictest4742-21ssh11:40:591
1845899611,48cyclictest4742-21ssh11:40:591
1845899611,48cyclictest27568-21fschecks_time10:40:211
1845899611,48cyclictest0-21swapper/110:22:421
1845899611,48cyclictest0-21swapper/110:22:421
1845899611,45cyclictest30870-21ssh11:32:541
1845899611,43cyclictest21049-21ssh11:18:271
1845899611,36cyclictest0-21swapper/110:05:451
1845799611,47cyclictest0-21swapper/008:59:350
1845899601,45cyclictest7654-21ssh09:25:231
1845899601,44cyclictest19658-21ssh09:42:291
1845899601,43cyclictest7934-21ssh10:11:031
1845899601,43cyclictest31264-21apt-get10:00:041
1845899601,41cyclictest31782-21wc12:20:031
1845899601,38cyclictest6931-21latency_hist09:25:031
1845899601,38cyclictest6931-21latency_hist09:25:031
1845899601,35cyclictest9989-21ssh11:02:001
1845799601,45cyclictest0-21swapper/008:51:290
1845899592,53cyclictest22464-21apt-get08:50:141
1845899591,43cyclictest1146-21snmpd07:39:061
1845899591,42cyclictest25473-21runrttasks10:36:471
1845899591,39cyclictest4287-21grep12:25:321
1845899582,44cyclictest26292-21rm09:51:311
1845899581,47cyclictest30146-21ssh09:57:141
1845899581,42cyclictest12581-21ssh11:52:291
1845899581,39cyclictest30388-21ssh10:45:131
1845899581,39cyclictest21712-21ls09:45:231
1845899581,35cyclictest11220-21ssh09:30:281
1845899579,28cyclictest1120-21runrttasks07:49:271
1845899573,31cyclictest0-21swapper/111:27:181
1845899571,38cyclictest5972-21ssh10:55:391
1845899571,37cyclictest25941-21awk09:00:011
222192560,1sleep30-21swapper/307:20:173
1845899562,41cyclictest18693-21latency_hist08:40:021
1845899562,39cyclictest18840-21/usr/sbin/munin08:40:181
1845899561,45cyclictest22899-21taskset10:33:531
1845899561,38cyclictest26479-21ssh12:11:401
1845899561,35cyclictest7850irq/56-dwc_otg_07:55:021
1845899552,41cyclictest8215-21sh12:31:301
18458995517,36cyclictest795-21/usr/sbin/munin09:15:371
1845899551,40cyclictest12985-21chrt08:23:511
1845899543,34cyclictest13544-21ssh11:07:041
1845899532,41cyclictest21527-21seq07:16:121
1845899531,47cyclictest0-21swapper/108:30:081
1845899531,42cyclictest10927-21chrt08:15:461
18458995218,32cyclictest26162-21/usr/sbin/munin09:00:171
1845899521,48cyclictest26756-21ls07:30:301
1846099512,4cyclictest7750irq/56-dwc_otg07:40:123
1845899513,38cyclictest1120-21runrttasks08:02:291
1845899513,38cyclictest1120-21runrttasks08:02:291
1845899501,46cyclictest2435-21/usr/sbin/munin07:55:161
1845899501,44cyclictest18642-21/usr/sbin/munin07:10:161
1845899493,43cyclictest22346-21fschecks_count07:20:201
1845899491,44cyclictest13341-21apt-get08:25:191
1845899491,44cyclictest13341-21apt-get08:25:191
262712480,2sleep226268-21ps11:25:502
1846099487,26cyclictest0-21swapper/308:55:083
18459994812,12cyclictest301ktimersoftd/211:38:272
1845899481,44cyclictest24255-21basename07:25:211
1845899481,44cyclictest24255-21basename07:25:211
18459994720,6cyclictest7750irq/56-dwc_otg12:39:302
1846099467,25cyclictest0-21swapper/311:50:153
1846099467,24cyclictest0-21swapper/307:50:073
1846099467,24cyclictest0-21swapper/307:25:163
1846099467,24cyclictest0-21swapper/307:25:163
1845999467,25cyclictest0-21swapper/209:30:172
1845999467,25cyclictest0-21swapper/207:25:162
1845999467,25cyclictest0-21swapper/207:25:162
1845999456,31cyclictest0-21swapper/207:10:152
1845999456,25cyclictest0-21swapper/212:30:152
18459994519,11cyclictest9450irq/48-DMA09:28:482
18459994519,11cyclictest9450irq/48-DMA09:28:482
18459994511,17cyclictest21354-21/usr/sbin/munin09:45:192
99862440,2sleep20-21swapper/210:15:182
1846099447,23cyclictest0-21swapper/312:00:173
1846099447,23cyclictest0-21swapper/310:55:203
1845999449,16cyclictest5968-21date08:05:032
1845999449,16cyclictest5968-21date08:05:032
1845999448,11cyclictest17550-21ntpq10:25:282
1845999447,24cyclictest0-21swapper/209:35:082
1845999444,27cyclictest7750irq/56-dwc_otg11:50:142
18459994413,18cyclictest18474-21/usr/sbin/munin12:00:292
1846099437,22cyclictest0-21swapper/311:15:153
1845999437,22cyclictest0-21swapper/207:50:072
1845999436,24cyclictest11563-21rm11:05:082
18459994326,10cyclictest7950irq/56-dwc_otg_09:14:052
18459994311,16cyclictest6462-21irqstats08:05:232
1846099429,8cyclictest37-21rcuc/312:25:303
1846099429,26cyclictest7950irq/56-dwc_otg_09:20:143
1846099429,26cyclictest7950irq/56-dwc_otg_09:20:143
1846099427,21cyclictest0-21swapper/308:15:083
18460994220,5cyclictest432650wget08:00:233
18460994220,5cyclictest432650wget08:00:233
18460994216,12cyclictest381ktimersoftd/308:10:193
1845999428,15cyclictest3815-21switchtime10:05:272
1845999427,22cyclictest0-21swapper/207:55:162
18459994218,6cyclictest14885-21cron08:30:022
18459994218,6cyclictest14885-21cron08:30:022
181742422,34sleep00-21swapper/007:09:530
1846099417,22cyclictest0-21swapper/310:20:183
1846099417,22cyclictest0-21swapper/310:20:183
1846099415,18cyclictest37-21rcuc/310:33:353
1846099413,16cyclictest37-21rcuc/309:08:183
18460994118,5cyclictest28616-21sh09:55:233
1845999419,11cyclictest21283-21ls08:45:252
1845999417,25cyclictest0-21swapper/210:55:202
1845999416,24cyclictest0-21swapper/209:00:162
18459994115,10cyclictest301ktimersoftd/212:20:202
18459994113,5cyclictest301ktimersoftd/211:15:592
18459994110,15cyclictest1545-21ssh10:50:222
1846099407,25cyclictest0-21swapper/309:35:083
18460994018,10cyclictest9550irq/80-mmc009:55:033
1846099401,27cyclictest27274-21idleruntime-cro07:35:013
1845999407,22cyclictest28741-21ssh10:40:582
1845999407,13cyclictest0-21swapper/211:10:272
18459994017,5cyclictest7850irq/56-dwc_otg_08:40:022
18459994016,10cyclictest301ktimersoftd/210:45:082
18459994015,6cyclictest301ktimersoftd/210:00:032
18459994012,14cyclictest301ktimersoftd/209:25:262
18459994010,16cyclictest19055-21df08:40:192
166892400,8sleep21845999cyclictest11:58:312
1846099395,24cyclictest0-21swapper/310:00:173
1846099394,4cyclictest0-21swapper/311:20:293
1846099394,4cyclictest0-21swapper/311:20:293
18460993919,9cyclictest381ktimersoftd/311:25:073
1846099391,6cyclictest2913250ssh11:30:273
18460993910,7cyclictest6198-21cpuspeed08:05:183
1845999397,23cyclictest0-21swapper/210:20:182
1845999397,23cyclictest0-21swapper/210:20:182
1845999396,17cyclictest8820-21rm11:46:552
18459993919,3cyclictest7750irq/56-dwc_otg10:10:162
1846099386,15cyclictest37-21rcuc/308:48:343
1846099383,4cyclictest7950irq/56-dwc_otg_08:40:163
1846099381,23cyclictest10243-21ls11:50:023
18459993821,8cyclictest301ktimersoftd/208:30:072
18459993818,4cyclictest27592-21aten_rbpower_po07:35:182
275262370,1sleep20-21swapper/212:13:532
1846099377,22cyclictest0-21swapper/309:00:163
1846099376,11cyclictest37-21rcuc/312:20:213
1846099373,4cyclictest7750irq/56-dwc_otg11:55:253
18460993720,8cyclictest7950irq/56-dwc_otg_08:40:023
18460993715,5cyclictest8026-21sh10:11:163
1845999378,11cyclictest8027-21diskstats08:10:192
1845999375,6cyclictest31476-21idleruntime07:45:202
1845999375,15cyclictest21762-21runrttasks07:19:022
18459993721,8cyclictest301ktimersoftd/209:54:582
18459993720,8cyclictest7850irq/56-dwc_otg_10:39:142
18459993719,8cyclictest7950irq/56-dwc_otg_11:24:092
18459993719,8cyclictest7950irq/56-dwc_otg_11:24:092
18459993710,8cyclictest4067-21cat12:25:302
1846099368,19cyclictest0-21swapper/312:35:323
1846099367,9cyclictest37-21rcuc/311:01:583
1846099366,8cyclictest37-21rcuc/310:05:243
18460993622,8cyclictest381ktimersoftd/312:08:543
1846099361,17cyclictest22551-21aten_rbpower_po08:50:183
1845999366,18cyclictest28023-21aten_rbpower_en09:05:152
1845999364,19cyclictest5522-21sh09:22:102
1845999364,19cyclictest5522-21sh09:22:102
18459993620,4cyclictest7750irq/56-dwc_otg11:02:022
18459993618,7cyclictest7850irq/56-dwc_otg_10:00:062
18459993612,5cyclictest22197-21timerwakeupswit12:05:302
1846099353,18cyclictest381ktimersoftd/311:35:313
18460993521,5cyclictest7850irq/56-dwc_otg_10:20:033
18460993518,8cyclictest381ktimersoftd/310:36:293
18460993516,5cyclictest1120-21runrttasks10:45:183
18460993514,9cyclictest381ktimersoftd/309:31:433
1845999356,8cyclictest21602-21ssh10:30:462
18459993518,8cyclictest7950irq/56-dwc_otg_09:16:302
1845999351,3cyclictest7750irq/56-dwc_otg12:15:202
1846099348,13cyclictest0-21swapper/311:05:083
1846099347,15cyclictest0-21swapper/308:30:083
1846099346,19cyclictest381ktimersoftd/308:25:333
1846099346,19cyclictest381ktimersoftd/308:25:333
1846099346,15cyclictest0-21swapper/312:30:153
1846099344,8cyclictest39-21ksoftirqd/311:10:253
18460993416,6cyclictest7850irq/56-dwc_otg_07:35:213
18460993416,10cyclictest381ktimersoftd/310:28:003
1846099341,3cyclictest0-21swapper/312:10:303
18460993412,13cyclictest7950irq/56-dwc_otg_12:20:033
18460993411,11cyclictest39-21ksoftirqd/309:11:133
1845999346,10cyclictest7950irq/56-dwc_otg_09:00:022
18459993417,7cyclictest7850irq/56-dwc_otg_08:54:222
18459993414,9cyclictest31-21ksoftirqd/208:24:122
18459993414,5cyclictest11443-21latency_hist08:20:032
1846099338,9cyclictest39-21ksoftirqd/309:40:233
1846099333,13cyclictest37-21rcuc/309:25:243
1846099332,16cyclictest0-21swapper/307:11:343
18460993318,10cyclictest7750irq/56-dwc_otg11:43:433
18460993318,10cyclictest7750irq/56-dwc_otg11:43:433
18460993316,9cyclictest381ktimersoftd/309:16:503
18460993311,9cyclictest7950irq/56-dwc_otg_07:49:503
176532337,12sleep10-21swapper/107:05:331
1846099322,6cyclictest27248-21ssh10:40:173
1845999324,6cyclictest7850irq/56-dwc_otg_11:30:212
1846099313,6cyclictest7950irq/56-dwc_otg_07:55:263
1845999318,3cyclictest0-21swapper/207:45:022
1846099303,14cyclictest7950irq/56-dwc_otg_08:20:233
1846099301,18cyclictest21159-21ls07:15:293
1845999291,7cyclictest29-21rcuc/207:20:272
1845999291,19cyclictest0-21swapper/207:30:572
172732276,11sleep30-21swapper/307:05:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional