You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-16 - 20:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Mon Jun 16, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17925991071,92cyclictest972-21grep09:15:340
17925991051,79cyclictest18060-21latency_hist08:40:030
17925991021,85cyclictest9879-21idleruntime-cro09:30:020
1792599991,89cyclictest27070-21/usr/sbin/munin11:30:310
1792599961,82cyclictest29715-21ssh10:45:280
1792599951,82cyclictest8782-21aten_rbpower_vo10:15:190
1792599951,77cyclictest28656-21ssh10:45:030
1792599951,37cyclictest23299-21munin-run08:55:030
1792599941,82cyclictest23276-21grep07:25:030
1792599941,77cyclictest1152-21grep07:50:320
1792599931,82cyclictest22740-21aten_rbpower_cu11:25:190
1792599921,78cyclictest25193-21ssh10:39:570
1792599921,76cyclictest12695-21/usr/sbin/munin11:55:200
198812900,2sleep20-21swapper/207:15:192
1792599901,79cyclictest13583-21ssh10:21:110
1792599901,73cyclictest11320-21ssh12:35:020
1792599891,72cyclictest24669-21/usr/sbin/munin09:50:300
1792599891,71cyclictest20576-21ssh10:31:460
1792599871,72cyclictest14730-21ssh11:12:190
1792599871,61cyclictest7197-21cron12:30:030
1792599871,61cyclictest7197-21cron12:30:030
1792599863,68cyclictest5238-21ssh11:45:240
1792599861,71cyclictest4686-21rm10:09:450
1792599851,73cyclictest15042-21ssh09:35:500
1792599847,61cyclictest0-21swapper/012:24:320
1792599841,71cyclictest0-21swapper/012:35:240
1792599841,69cyclictest20202-21sh11:20:310
1792599831,55cyclictest10777-21ssh11:51:530
1792599822,70cyclictest23891-21seq08:55:230
1792599821,71cyclictest15949-21/usr/sbin/munin11:15:210
1792599821,66cyclictest0-21swapper/011:07:030
1792599811,69cyclictest10783-21latency09:30:280
1792599811,69cyclictest10783-21latency09:30:280
1792599811,65cyclictest26048-21runrttasks12:12:000
1792599811,62cyclictest0-21swapper/012:19:430
1792599811,58cyclictest32406-21chrt10:01:240
1792599801,63cyclictest9203-21ssh11:04:260
1792599801,59cyclictest0-21swapper/008:10:350
1792599791,68cyclictest0-21swapper/011:41:020
1792599791,68cyclictest0-21swapper/011:41:020
1792599791,64cyclictest2609-21ssh10:53:390
1792599791,64cyclictest2609-21ssh10:53:390
1792599791,63cyclictest17075-21sshd10:26:380
1792599791,63cyclictest0-21swapper/008:25:280
1792599791,54cyclictest0-21swapper/009:21:420
1792599782,50cyclictest0-21swapper/007:25:220
1792599781,64cyclictest19913-21ssh09:44:290
1792599781,61cyclictest0-21swapper/009:12:400
1792599781,59cyclictest30192-21dhclient-script07:41:560
1792599781,56cyclictest19761-21ssh12:04:360
1792599771,61cyclictest0-21swapper/012:06:500
1792599771,61cyclictest0-21swapper/012:06:500
1792599771,61cyclictest0-21swapper/011:38:230
1792599771,60cyclictest0-21swapper/008:40:260
1792599771,52cyclictest0-21swapper/008:19:090
1792599771,38cyclictest5574-21basename08:05:190
1792599761,58cyclictest5068-21taskset10:56:270
1792599761,56cyclictest28557-21chrt07:39:080
235382750,5sleep11792699cyclictest12:09:511
235382750,5sleep11792699cyclictest12:09:511
1792599752,34cyclictest0-21swapper/009:10:030
1792599751,62cyclictest0-21swapper/007:59:080
1792599751,60cyclictest0-21swapper/009:00:580
1792599751,60cyclictest0-21swapper/007:13:590
1792599751,57cyclictest0-21swapper/010:12:300
1792599741,60cyclictest15976-21seq08:34:060
1792599741,56cyclictest28784-21ssh09:55:480
1792599741,54cyclictest0-21swapper/009:47:220
1792599741,32cyclictest32043-21chrt07:47:370
1792599721,55cyclictest0-21swapper/007:19:320
1792599711,57cyclictest21398-21chrt08:49:030
203362700,1sleep20-21swapper/210:31:132
17928997037,14cyclictest381ktimersoftd/308:05:163
1792599701,54cyclictest5198-21taskset08:04:530
1792599701,54cyclictest5198-21taskset08:04:530
227762680,4sleep231-21ksoftirqd/211:25:202
1792599661,53cyclictest0-21swapper/007:30:410
1792799655,18cyclictest8050irq/56-dwc_otg_12:05:192
1792799655,18cyclictest8050irq/56-dwc_otg_12:05:192
1792699659,6cyclictest7850irq/56-dwc_otg10:40:171
1792599651,52cyclictest0-21swapper/008:22:060
83772630,2sleep30-21swapper/311:49:483
1792799637,31cyclictest0-21swapper/208:35:202
253302600,34sleep39450irq/48-DMA08:55:163
17927995718,19cyclictest1588-21ssh12:22:162
17927995714,13cyclictest7821-21diskstats12:30:222
17927995615,14cyclictest15030-21ssh09:35:482
85212550,2sleep30-21swapper/309:26:333
1792699554,12cyclictest9769-21ssh11:05:151
17927995315,16cyclictest4649-21ssh09:20:442
17927995314,22cyclictest24255-21sed08:55:272
17927995314,10cyclictest1049-21/usr/sbin/munin09:15:352
17927995226,18cyclictest8050irq/56-dwc_otg_12:10:182
17927995214,11cyclictest29522-21/usr/sbin/munin10:45:252
17927995112,26cyclictest25225-21ssh09:50:392
17927995112,12cyclictest11184-21rm10:18:212
17927995017,14cyclictest20368-21ls11:20:332
17927995013,14cyclictest7442-21ssh10:12:562
17927994939,5cyclictest67912sleep211:00:172
17927994917,16cyclictest8089-21ssh11:49:132
17927994916,13cyclictest24071-21ntp_states07:25:262
17927994914,12cyclictest10607-21ssh11:51:302
17927994910,15cyclictest7850irq/56-dwc_otg11:15:022
1792699499,24cyclictest7850irq/56-dwc_otg10:15:341
1792899487,27cyclictest0-21swapper/308:45:213
1792899482,17cyclictest8050irq/56-dwc_otg_11:40:203
1792899482,17cyclictest8050irq/56-dwc_otg_11:40:203
17927994815,19cyclictest13609-21ssh10:21:162
17927994814,15cyclictest18423-21ssh12:01:472
17927994811,18cyclictest11641-21/usr/sbin/munin12:35:272
17927994718,12cyclictest4895-21date10:10:032
17927994717,16cyclictest30093-21ssh12:17:332
17927994717,12cyclictest18688-21ssh09:41:352
17926994719,19cyclictest23-21ksoftirqd/109:25:161
1792799467,26cyclictest7850irq/56-dwc_otg10:40:242
1792799466,20cyclictest7850irq/56-dwc_otg09:30:312
1792799466,20cyclictest7850irq/56-dwc_otg09:30:312
1792799464,15cyclictest29896-21ssh11:35:162
17927994614,20cyclictest16022-21cut11:15:232
17926994618,6cyclictest13229-21sh11:10:241
173702463,11sleep00-21swapper/007:06:260
1792899456,15cyclictest37-21rcuc/312:25:173
1792899456,15cyclictest37-21rcuc/312:25:173
17928994525,7cyclictest7850irq/56-dwc_otg10:55:093
1792799457,20cyclictest8050irq/56-dwc_otg_08:40:342
17927994510,10cyclictest8050irq/56-dwc_otg_08:05:032
17927994510,10cyclictest8050irq/56-dwc_otg_08:05:032
1792699457,4cyclictest7850irq/56-dwc_otg09:30:211
1792699457,4cyclictest7850irq/56-dwc_otg09:30:211
1792899447,22cyclictest0-21swapper/308:00:163
1792899447,22cyclictest0-21swapper/308:00:163
1792899443,9cyclictest7950irq/56-dwc_otg_11:00:173
1792899442,7cyclictest7850irq/56-dwc_otg11:05:173
1792799447,24cyclictest0-21swapper/208:50:202
17927994413,15cyclictest8901-21ssh09:27:352
17927994410,19cyclictest28335-21taskset07:36:232
1792699447,7cyclictest0-21swapper/112:00:341
1792899437,23cyclictest0-21swapper/307:40:163
1792899434,25cyclictest8050irq/56-dwc_otg_12:00:173
1792799437,24cyclictest0-21swapper/207:45:172
1792799437,11cyclictest8050irq/56-dwc_otg_08:05:242
17927994315,16cyclictest32556-21ssh10:01:482
17927994314,19cyclictest30082-21sshd07:40:362
1792699432,3cyclictest8050irq/56-dwc_otg_07:15:221
17926994314,23cyclictest0-21swapper/107:55:161
17926994310,17cyclictest8050irq/56-dwc_otg_10:15:021
1792899427,21cyclictest0-21swapper/308:50:203
1792899427,16cyclictest39-21ksoftirqd/309:30:263
1792899427,16cyclictest39-21ksoftirqd/309:30:263
1792899425,9cyclictest7950irq/56-dwc_otg_10:20:173
1792899421,22cyclictest0-21swapper/311:35:163
1792799428,13cyclictest17130-21ssh10:26:442
1792799427,22cyclictest0-21swapper/209:55:172
17927994216,15cyclictest3083-21ssh11:41:512
17927994216,15cyclictest3083-21ssh11:41:512
17927994215,15cyclictest12889-21latency_hist11:10:042
183192410,2sleep30-21swapper/307:10:213
1792899417,24cyclictest0-21swapper/311:15:183
1792899416,6cyclictest39-21ksoftirqd/311:25:213
1792799417,10cyclictest0-21swapper/211:30:322
1792799416,12cyclictest29041-21seq09:10:042
17927994121,7cyclictest3449-21apt-get12:25:182
17927994121,7cyclictest3449-21apt-get12:25:182
17927994114,10cyclictest14966-21ssh11:57:362
17927994113,15cyclictest23101-21ssh09:49:012
17927994111,16cyclictest7850irq/56-dwc_otg08:25:142
17927994111,15cyclictest25289-21cat10:40:032
1792899408,15cyclictest39-21ksoftirqd/311:20:333
1792899406,21cyclictest8050irq/56-dwc_otg_08:35:203
1792899405,20cyclictest12694-21ssh11:55:203
1792799406,21cyclictest25435-21/usr/sbin/munin09:00:322
17927994011,11cyclictest6259-21ssh10:59:322
1792699402,7cyclictest7950irq/56-dwc_otg_11:15:161
1792899397,24cyclictest0-21swapper/312:10:083
1792899396,14cyclictest37-21rcuc/310:45:293
1792899394,25cyclictest0-21swapper/308:40:183
1792899393,14cyclictest11-21rcu_preempt12:32:503
17928993915,4cyclictest37-21rcuc/310:50:563
17928993915,4cyclictest37-21rcuc/310:50:563
1792799398,10cyclictest3141-21chrt07:56:452
17927993915,19cyclictest1527-21ssh10:50:552
17927993915,19cyclictest1527-21ssh10:50:552
1792699395,10cyclictest7850irq/56-dwc_otg11:05:031
1792699392,24cyclictest13074-21ls11:55:251
1792699391,5cyclictest5659-21cat11:45:291
1792699391,26cyclictest23664-21cut07:25:201
1792699391,12cyclictest8050irq/56-dwc_otg_07:30:081
33282380,5sleep11792699cyclictest12:25:041
33282380,5sleep11792699cyclictest12:25:041
279912380,2sleep327993-21awk07:35:283
1792899387,27cyclictest0-21swapper/308:55:083
1792899387,22cyclictest0-21swapper/307:45:173
1792899384,8cyclictest7950irq/56-dwc_otg_09:00:203
17928993815,11cyclictest7850irq/56-dwc_otg10:15:163
17927993812,12cyclictest32477-21/usr/sbin/munin07:50:342
1792699388,17cyclictest8050irq/56-dwc_otg_09:05:211
1792899377,15cyclictest0-21swapper/312:20:173
1792899377,14cyclictest39-21ksoftirqd/312:35:243
1792899377,13cyclictest37-21rcuc/312:20:033
1792899372,16cyclictest0-21swapper/308:16:053
1792899371,25cyclictest8127-21ls08:10:313
1792899371,25cyclictest13646-21ls11:10:313
1792799378,9cyclictest8050irq/56-dwc_otg_08:10:432
17927993715,4cyclictest21562-21/usr/sbin/munin07:20:232
17926993710,20cyclictest0-21swapper/108:25:171
161002377,19sleep10-21swapper/107:05:141
1792899365,23cyclictest7950irq/56-dwc_otg_10:25:343
17928993610,8cyclictest27560-21unixbench_singl11:30:353
1792699364,22cyclictest29182-21ssh09:10:201
1792699362,6cyclictest7850irq/56-dwc_otg12:35:201
17926993616,4cyclictest955-21runrttasks11:50:491
1792899357,22cyclictest0-21swapper/311:50:223
1792899356,9cyclictest39-21ksoftirqd/310:32:063
1792899356,17cyclictest11-21rcu_preempt10:05:173
1792899355,15cyclictest37-21rcuc/310:40:213
1792899355,13cyclictest7950irq/56-dwc_otg_10:35:343
1792899354,15cyclictest29004-21latency_hist09:10:033
1792899352,6cyclictest0-21swapper/312:05:163
1792899352,6cyclictest0-21swapper/312:05:163
17928993515,8cyclictest381ktimersoftd/310:10:103
1792899351,23cyclictest28378-21timerandwakeup09:55:323
1792899351,18cyclictest0-21swapper/309:50:333
1792799359,11cyclictest12365-21taskset08:24:062
1792799359,10cyclictest10595-21taskset08:19:352
17927993513,13cyclictest28845-21latency_hist09:10:032
17927993510,11cyclictest15984-21chrt08:34:122
1792699358,14cyclictest28811-21diskmemload11:25:341
1792699358,13cyclictest8050irq/56-dwc_otg_10:00:021
1792699357,15cyclictest8050irq/56-dwc_otg_07:45:031
1792699354,7cyclictest7850irq/56-dwc_otg12:20:321
1792699354,6cyclictest7850irq/56-dwc_otg10:00:351
1792699353,6cyclictest7850irq/56-dwc_otg11:35:331
1792699353,6cyclictest0-21swapper/108:30:151
1792699352,8cyclictest25200-21apt09:00:021
1792699351,5cyclictest7950irq/56-dwc_otg_10:20:351
17926993510,5cyclictest8050irq/56-dwc_otg_12:15:221
1792899341,15cyclictest37-21rcuc/307:25:243
1792899341,14cyclictest381ktimersoftd/309:15:343
17927993412,11cyclictest19449-21chrt07:14:222
1792699348,4cyclictest0-21swapper/110:50:201
1792699348,4cyclictest0-21swapper/110:50:201
1792699342,16cyclictest0-21swapper/110:05:341
1792699341,6cyclictest0-21swapper/109:20:181
17926993414,10cyclictest20353-21sh10:31:151
1792699341,22cyclictest23135-21latency_hist07:25:031
1792699341,15cyclictest8050irq/56-dwc_otg_08:50:341
92942330,3sleep1221ktimersoftd/108:15:191
1792899339,7cyclictest37-21rcuc/309:41:353
1792899337,8cyclictest39-21ksoftirqd/308:25:293
1792899337,23cyclictest0-21swapper/309:35:183
1792899335,10cyclictest0-21swapper/310:00:173
1792899333,22cyclictest7950irq/56-dwc_otg_07:20:033
1792899333,22cyclictest11262-21ls08:20:363
1792899331,29cyclictest367-21ls07:50:223
17927993319,12cyclictest7121-21kworker/u8:007:30:092
17927993310,15cyclictest21192-21chrt08:46:312
1792699338,12cyclictest7850irq/56-dwc_otg10:25:301
1792699335,15cyclictest8050irq/56-dwc_otg_09:45:231
1792699334,5cyclictest955-21runrttasks07:10:251
17926993323,7cyclictest875-21dhclient07:50:251
1792699331,7cyclictest19476-21wget11:20:211
17926993316,4cyclictest955-21runrttasks10:55:041
1792699331,5cyclictest11184-21fschecks_count08:20:351
17926993311,4cyclictest0-21swapper/109:36:301
17926993310,13cyclictest0-21swapper/109:43:531
1792899325,15cyclictest381ktimersoftd/309:21:273
1792899321,3cyclictest0-21swapper/309:10:343
1792899321,14cyclictest0-21swapper/307:56:373
1792899320,19cyclictest0-21swapper/309:45:193
1792699326,9cyclictest0-21swapper/108:02:051
1792699326,9cyclictest0-21swapper/108:02:051
1792699324,14cyclictest8050irq/56-dwc_otg_08:10:221
1792699323,5cyclictest7850irq/56-dwc_otg12:10:241
17926993216,9cyclictest7850irq/56-dwc_otg12:30:191
1792699321,5cyclictest7850irq/56-dwc_otg11:30:191
1792699321,14cyclictest7850irq/56-dwc_otg08:45:351
1792899318,21cyclictest0-21swapper/307:23:083
1792699317,15cyclictest0-21swapper/109:00:211
1792699315,12cyclictest0-21swapper/109:51:041
1792699312,21cyclictest0-21swapper/107:39:241
17926993117,11cyclictest0-21swapper/109:15:321
1792699311,6cyclictest22375-21/usr/sbin/munin10:35:241
1792699311,23cyclictest17878-21latency_hist08:40:031
1792699310,22cyclictest0-21swapper/111:40:291
1792699310,22cyclictest0-21swapper/111:40:291
1792699302,7cyclictest18360-21cut08:40:221
17926993014,8cyclictest955-21runrttasks10:48:091
1792699301,15cyclictest32244-21latency_hist07:50:031
1792899293,4cyclictest0-21swapper/308:30:323
1792699298,4cyclictest0-21swapper/108:05:301
176382274,15sleep30-21swapper/307:09:493
160962253,14sleep20-21swapper/207:05:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional