You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 23:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Wed Nov 19, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
734023260,4sleep17343-21fschecks_count07:05:251
172221030,8sleep0890099cyclictest11:55:260
2718921010,2sleep10-21swapper/110:56:051
890099916,51cyclictest0-21swapper/009:00:220
890099901,25cyclictest0-21swapper/010:35:230
233522880,2sleep20-21swapper/211:40:142
259922820,1sleep10-21swapper/111:44:031
890099802,42cyclictest0-21swapper/008:30:180
8900998015,23cyclictest985-21basename11:55:050
8900998015,23cyclictest985-21basename11:55:050
8900997844,23cyclictest7950irq/56-dwc_otg_09:30:120
890099771,29cyclictest28949-21aten_rbpower_en08:05:190
271972770,2sleep20-21swapper/210:56:062
890099768,56cyclictest0-21swapper/010:40:270
890099742,27cyclictest23166-21cron09:15:040
8900997419,15cyclictest12789-21fschecks_count07:20:250
890099721,34cyclictest0-21swapper/008:45:160
890099719,32cyclictest0-21swapper/012:08:440
8900996912,37cyclictest21932-21modprobe10:50:040
890299684,52cyclictest0-21swapper/209:28:392
72172680,3sleep27212-21fschecks_count11:15:262
8900996645,5cyclictest7850irq/56-dwc_otg_09:55:390
8900996617,26cyclictest23076-21cron11:40:050
890099651,7cyclictest31855-21grep08:10:370
890299641,14cyclictest27077-21/usr/sbin/munin08:00:222
890299637,26cyclictest0-21swapper/210:35:232
8900996346,4cyclictest7850irq/56-dwc_otg_11:15:050
8900996317,36cyclictest0-21swapper/010:05:330
890099631,49cyclictest32170-21latency_hist12:40:060
890099631,49cyclictest32170-21latency_hist12:40:060
890399626,4cyclictest0-21swapper/309:10:193
84682623,11sleep00-21swapper/007:08:080
890299611,4cyclictest973-21modprobe11:55:052
890299611,4cyclictest973-21modprobe11:55:052
8900996135,8cyclictest7850irq/56-dwc_otg_11:00:060
8900996021,20cyclictest26893-21if_wlan009:20:270
85032603,48sleep30-21swapper/307:08:363
890299595,13cyclictest0-21swapper/210:49:432
8900995925,4cyclictest0-21swapper/010:25:200
890399588,8cyclictest7950irq/56-dwc_otg_11:05:213
890299581,9cyclictest0-21swapper/211:05:242
890099583,34cyclictest0-21swapper/007:25:370
8900995824,11cyclictest9702-21cat08:40:290
890299574,5cyclictest7750irq/56-dwc_otg09:20:282
890299571,5cyclictest30770-21df_inode10:15:222
890299571,5cyclictest30770-21df_inode10:15:222
890299571,5cyclictest23763-21tune2fs09:15:232
8902995715,40cyclictest21688-21basename07:45:192
8902995713,6cyclictest25154-21grep10:05:362
890099572,5cyclictest140-21systemd-journal09:55:040
890299563,7cyclictest23184-21munin-plugin-st09:15:042
890299563,5cyclictest28950-21aten_rbpower_en08:05:192
890299562,5cyclictest18983-21ssh12:20:272
890299561,5cyclictest15160-21ntpq07:25:382
890099566,27cyclictest0-21swapper/011:30:060
8900995627,6cyclictest0-21swapper/007:55:170
890099561,6cyclictest30382-21grep11:00:360
890099561,6cyclictest30382-21grep11:00:360
8900995616,24cyclictest14576-21idleruntime-cro12:15:050
890099561,48cyclictest11380-21sed11:20:390
890299558,4cyclictest0-21swapper/212:26:522
890299555,18cyclictest7950irq/56-dwc_otg_07:55:192
890299552,4cyclictest0-21swapper/207:15:272
890299552,4cyclictest0-21swapper/207:15:272
890299551,17cyclictest0-21swapper/210:40:292
890299551,15cyclictest7395-21grep09:40:222
890299551,12cyclictest11183-21fschecks_time08:45:222
890299551,11cyclictest165572chrt09:53:462
8900995536,6cyclictest7850irq/56-dwc_otg_08:00:360
8900995527,5cyclictest7750irq/56-dwc_otg12:17:210
8900995527,5cyclictest7750irq/56-dwc_otg12:17:210
890099552,30cyclictest0-21swapper/011:20:050
890099552,29cyclictest30909-21fschecks_time10:15:240
890099552,29cyclictest30909-21fschecks_time10:15:240
890099551,8cyclictest16357-21aten_rbpower_po07:30:220
890099551,8cyclictest16357-21aten_rbpower_po07:30:220
890099551,7cyclictest7957-21latency_hist12:05:050
890099551,4cyclictest23360-21grep12:25:390
8900995513,4cyclictest29926-21modprobe11:50:050
890099551,30cyclictest24616-21sed09:15:370
890299542,10cyclictest7850irq/56-dwc_otg_08:10:372
890299541,5cyclictest18257-21latency_hist12:20:052
890299541,5cyclictest18257-21latency_hist12:20:052
890299541,3cyclictest0-21swapper/212:11:452
890299541,14cyclictest22221-21ssh10:50:162
890099545,23cyclictest4282-21fschecks_time09:35:240
890099543,26cyclictest0-21swapper/009:25:200
890099542,9cyclictest27840-21wc10:10:280
890299535,8cyclictest7850irq/56-dwc_otg_11:10:382
890299533,3cyclictest7750irq/56-dwc_otg12:31:322
890299533,14cyclictest0-21swapper/208:30:182
890299532,11cyclictest17487-21sed11:30:272
890299531,9cyclictest7750irq/56-dwc_otg12:36:452
890299531,9cyclictest7750irq/56-dwc_otg12:36:452
890299531,5cyclictest11257-21grep09:45:282
890299531,5cyclictest11257-21grep09:45:282
890299531,3cyclictest0-21swapper/209:30:112
890299531,10cyclictest4732-21latency_hist12:00:052
8902995310,35cyclictest0-21swapper/209:35:352
8900995332,6cyclictest18205-21aten_rbpower_po09:05:200
8900995332,6cyclictest18205-21aten_rbpower_po09:05:200
890099532,8cyclictest21894-21latency_hist12:25:050
890099532,7cyclictest24151-21ntpq07:50:270
890399529,5cyclictest7950irq/56-dwc_otg_10:00:183
890299526,9cyclictest7750irq/56-dwc_otg12:06:122
890299524,4cyclictest8231-21/usr/sbin/munin10:30:232
890299523,4cyclictest7850irq/56-dwc_otg_10:04:462
890299522,41cyclictest0-21swapper/211:05:062
890299522,41cyclictest0-21swapper/211:05:062
890299522,3cyclictest7750irq/56-dwc_otg11:24:042
890299521,6cyclictest0-21swapper/207:35:322
890299521,3cyclictest0-21swapper/210:20:312
890299521,11cyclictest0-21swapper/211:35:402
890299521,10cyclictest27585-21ntp_states11:45:312
890099523,26cyclictest0-21swapper/010:20:230
890099522,27cyclictest21818-21unixbench_multi10:00:340
8900995220,5cyclictest10409-21cat09:45:050
890099521,28cyclictest24500-21ssh11:40:340
890299513,3cyclictest7750irq/56-dwc_otg10:29:222
890299511,7cyclictest0-21swapper/208:15:552
890299511,7cyclictest0-21swapper/208:15:552
890299511,12cyclictest7950irq/56-dwc_otg_08:37:352
8902995110,7cyclictest17118-21switchtime07:30:342
8902995110,7cyclictest17118-21switchtime07:30:342
8900995118,21cyclictest0-21swapper/008:27:080
890099511,4cyclictest28592-21munin-plugin-st12:35:050
890099511,31cyclictest25574-21cat10:55:050
8901995018,4cyclictest30197-21sh10:15:051
8900995013,20cyclictest13705-21modprobe09:50:050
8900995013,20cyclictest13705-21modprobe09:50:050
890299491,7cyclictest12640-21aten_rbpower_po07:20:222
890299491,5cyclictest18131-21ntp_offset09:55:332
890299491,5cyclictest1146-21snmpd10:13:302
8902994913,4cyclictest18177-21aten_rbpower_en09:05:202
8902994913,4cyclictest18177-21aten_rbpower_en09:05:202
890299491,2cyclictest0-21swapper/212:04:452
890199496,30cyclictest0-21swapper/107:45:191
890099493,6cyclictest20108-21fschecks_count07:40:230
890099491,26cyclictest32729-21fschecks_count08:15:250
890099491,26cyclictest32729-21fschecks_count08:15:250
890099489,5cyclictest0-21swapper/010:35:050
8900994827,7cyclictest0-21swapper/008:55:340
8900994814,22cyclictest9450irq/48-DMA09:28:480
890399477,25cyclictest0-21swapper/307:50:183
890299471,9cyclictest0-21swapper/211:27:192
8900994721,10cyclictest0-21swapper/011:05:370
8900994718,18cyclictest0-21swapper/007:15:410
8900994718,18cyclictest0-21swapper/007:15:410
890099471,32cyclictest9447-21grep07:10:250
890299461,4cyclictest7850irq/56-dwc_otg_09:04:352
890199462,8cyclictest0-21swapper/109:35:181
890199462,7cyclictest7950irq/56-dwc_otg_09:45:181
890199462,7cyclictest7950irq/56-dwc_otg_09:45:181
8900994619,6cyclictest13-21rcuc/008:20:310
890199457,23cyclictest0-21swapper/111:45:211
8900994519,14cyclictest0-21swapper/008:38:210
890199447,23cyclictest0-21swapper/110:40:191
890399437,20cyclictest0-21swapper/308:50:183
890199437,23cyclictest0-21swapper/111:50:211
890199437,23cyclictest0-21swapper/111:50:211
890199435,9cyclictest7750irq/56-dwc_otg10:00:181
8901994318,5cyclictest24668-21sh10:05:301
193622430,1sleep10-21swapper/109:57:361
890399427,27cyclictest0-21swapper/307:40:193
890399427,25cyclictest0-21swapper/307:45:193
890399417,22cyclictest0-21swapper/311:40:193
8903994123,14cyclictest7950irq/56-dwc_otg_12:15:193
8903994123,14cyclictest7950irq/56-dwc_otg_12:15:193
890299417,3cyclictest0-21swapper/208:20:332
134432411,35sleep20-21swapper/208:50:312
890399407,7cyclictest0-21swapper/312:10:203
890399401,25cyclictest0-21swapper/311:45:213
8901994024,9cyclictest7750irq/56-dwc_otg10:15:241
8901994024,9cyclictest7750irq/56-dwc_otg10:15:241
8901994023,8cyclictest7850irq/56-dwc_otg_12:40:061
890099402,23cyclictest11-21rcu_preempt08:50:260
890099401,19cyclictest0-21swapper/007:45:280
8903993913,13cyclictest381ktimersoftd/310:50:193
890199393,6cyclictest0-21swapper/109:30:121
8901993917,7cyclictest28601-21idleruntime-cro12:35:051
890199391,27cyclictest7648-21cut09:40:261
890099391,19cyclictest0-21swapper/007:35:280
83052390,2sleep38292-21ssh12:05:193
272032390,2sleep30-21swapper/310:56:073
890399384,15cyclictest7950irq/56-dwc_otg_09:50:253
890299381,5cyclictest0-21swapper/208:25:332
890199388,6cyclictest8504-21/usr/sbin/munin12:05:221
890199387,21cyclictest0-21swapper/108:55:201
890199382,4cyclictest0-21swapper/112:39:261
890199381,6cyclictest1622-21taskset11:55:251
890199381,31cyclictest9428-21cut08:40:251
890399377,6cyclictest15540-21fschecks_time10:40:243
8903993717,6cyclictest18625-21sh12:20:193
890399371,25cyclictest1413-21ls09:30:333
890199377,17cyclictest0-21swapper/110:35:101
8901993717,6cyclictest13249-21latency_hist11:25:051
890199371,25cyclictest9907-21timerwakeupswit07:10:321
890199371,22cyclictest6633-21cron11:15:041
86552370,1sleep30-21swapper/311:16:563
890399366,5cyclictest7850irq/56-dwc_otg_08:00:193
890399363,7cyclictest0-21swapper/312:40:053
890399363,7cyclictest0-21swapper/312:40:053
8903993628,6cyclictest381ktimersoftd/311:55:193
890399362,7cyclictest12042-21grep08:45:333
890399361,8cyclictest23562-21aten_rbpower_po09:15:203
890299362,17cyclictest0-21swapper/207:13:202
8901993621,4cyclictest23-21ksoftirqd/111:30:051
890199361,4cyclictest29163-21ssh11:00:201
890199361,4cyclictest29163-21ssh11:00:201
890199361,3cyclictest0-21swapper/107:35:241
8901993612,17cyclictest0-21swapper/107:30:361
8901993612,17cyclictest0-21swapper/107:30:361
890399358,21cyclictest28599-1kworker/u9:210:11:243
890399357,6cyclictest7950irq/56-dwc_otg_08:20:273
890399356,7cyclictest2718-21tail10:20:353
890399355,17cyclictest7950irq/56-dwc_otg_08:55:203
890399355,17cyclictest0-21swapper/309:25:333
890399351,8cyclictest5153-21tune2fs12:00:233
8903993513,10cyclictest19689-21diskmemload11:21:263
8903993511,9cyclictest0-21swapper/311:35:223
890299351,20cyclictest23710-21uniq07:50:222
890199359,9cyclictest328-21fschecks_time08:15:251
890199359,9cyclictest328-21fschecks_time08:15:251
890199357,16cyclictest0-21swapper/112:00:191
890199357,14cyclictest0-21swapper/110:35:021
890199354,18cyclictest0-21swapper/107:25:421
890199353,19cyclictest7950irq/56-dwc_otg_10:53:371
8901993526,7cyclictest0-21swapper/112:20:191
890199351,7cyclictest5500-21memory10:25:301
890199351,6cyclictest7850irq/56-dwc_otg_09:25:051
8901993516,6cyclictest23077-21munin-plugin-st11:40:051
8901993514,11cyclictest15009-21aten_rbpower_po12:15:211
8901993514,11cyclictest15009-21aten_rbpower_po12:15:211
890199351,13cyclictest19235-21grep09:05:341
890199351,13cyclictest19235-21grep09:05:341
8901993510,18cyclictest0-21swapper/111:30:081
8901993510,14cyclictest0-21swapper/107:40:191
202152350,2sleep30-21swapper/309:59:513
890399349,4cyclictest26752-21fschecks_time09:20:253
890399348,5cyclictest7850irq/56-dwc_otg_08:05:343
890399344,6cyclictest7950irq/56-dwc_otg_11:50:253
890399344,6cyclictest7950irq/56-dwc_otg_11:50:253
8903993413,5cyclictest16169-21latency_hist07:30:063
890299342,6cyclictest15371-21grep08:55:332
890199349,14cyclictest15331-1kworker/u9:209:53:341
890199346,16cyclictest0-21swapper/108:35:191
890199342,7cyclictest30039-21unixbench_singl08:05:351
8901993423,9cyclictest19689-21diskmemload12:28:551
890199341,8cyclictest2783-21unixbench_singl10:20:351
8901993418,10cyclictest22108-21sh09:12:361
890199341,6cyclictest7850irq/56-dwc_otg_10:45:251
890399339,5cyclictest13688-21unixbench_singl07:20:363
890399339,4cyclictest7950irq/56-dwc_otg_10:20:053
890399339,4cyclictest7950irq/56-dwc_otg_10:20:053
890399337,16cyclictest0-21swapper/310:35:103
890399334,5cyclictest7936-21missed_timers09:40:303
8903993327,4cyclictest39-21ksoftirqd/308:40:213
8903993315,9cyclictest9826-21ssh10:31:183
8903993314,5cyclictest7308-21aten_rbpower_cu08:35:193
8903993314,4cyclictest1120-21runrttasks11:34:153
8903993313,12cyclictest381ktimersoftd/311:03:033
8903993313,12cyclictest381ktimersoftd/311:03:033
890299338,5cyclictest7750irq/56-dwc_otg08:45:052
890199338,9cyclictest0-21swapper/107:50:171
890199337,11cyclictest0-21swapper/111:05:431
890199331,5cyclictest0-21swapper/111:15:191
890399327,5cyclictest7850irq/56-dwc_otg_09:45:253
890399327,5cyclictest7850irq/56-dwc_otg_09:45:253
890399324,3cyclictest7950irq/56-dwc_otg_07:15:203
890399324,3cyclictest7950irq/56-dwc_otg_07:15:203
890399321,4cyclictest7750irq/56-dwc_otg12:27:353
8903993214,4cyclictest381ktimersoftd/310:45:223
8903993212,3cyclictest381ktimersoftd/307:35:203
890399321,22cyclictest5235-21latency_hist08:30:043
890199324,6cyclictest7850irq/56-dwc_otg_08:45:331
890199323,5cyclictest12853-21grep12:10:361
890199322,15cyclictest7950irq/56-dwc_otg_09:00:291
890199321,9cyclictest11072-21fschecks_time07:15:391
890199321,9cyclictest11072-21fschecks_time07:15:391
8901993218,4cyclictest27077-21/usr/sbin/munin08:00:321
890199321,4cyclictest23490-21ssh09:15:191
79222320,5sleep20-21swapper/207:05:342
890399317,11cyclictest0-21swapper/311:25:193
890399311,5cyclictest0-21swapper/309:35:233
8903993112,5cyclictest28771-21latency_hist12:35:063
890199311,4cyclictest12700-21/usr/sbin/munin08:50:211
890199311,14cyclictest7950irq/56-dwc_otg_07:55:321
99392300,1sleep30-21swapper/307:10:333
890399306,10cyclictest7950irq/56-dwc_otg_07:33:333
890399306,10cyclictest7950irq/56-dwc_otg_07:33:333
890399304,5cyclictest30772-21aten_rbpower_po08:10:223
890399302,14cyclictest7950irq/56-dwc_otg_11:11:593
890199309,10cyclictest6223-21proc_pri08:30:321
890199306,5cyclictest7750irq/56-dwc_otg09:25:361
890199301,12cyclictest7950irq/56-dwc_otg_08:10:241
890399292,4cyclictest0-21swapper/310:26:403
890399291,8cyclictest0-21swapper/308:31:363
890399291,6cyclictest16637-21tune2fs09:00:263
890399291,4cyclictest24664-21ls10:05:293
890199293,4cyclictest0-21swapper/107:20:171
890399286,6cyclictest7950irq/56-dwc_otg_07:55:313
890399282,6cyclictest19277-21head09:05:343
890399282,6cyclictest19277-21head09:05:343
890199281,5cyclictest7850irq/56-dwc_otg_08:20:271
8901992812,5cyclictest4501-21sensors_temp08:25:301
890299274,17cyclictest7850irq/56-dwc_otg_07:44:582
890399261,15cyclictest563-21grep08:15:283
890399261,15cyclictest563-21grep08:15:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional