You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-25 - 00:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Mon Aug 24, 2026 12:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759722910,1sleep20-21swapper/207:05:272
290299912540,61cyclictest24698-21ssh09:30:422
290279912073,17cyclictest0-21swapper/011:20:220
290289911851,45cyclictest8944-21mv09:55:331
290279911674,20cyclictest8739-21/usr/sbin/munin09:55:290
29027991114,76cyclictest0-21swapper/009:15:380
29030991107,88cyclictest23807-21aten_rbpower_cu09:30:263
290279911054,24cyclictest0-21swapper/009:30:420
290289910739,31cyclictest14052-21ssh11:20:241
290289910641,39cyclictest24602-21/usr/sbin/munin09:30:401
290279910664,23cyclictest28446-21apt-get10:20:160
290279910622,77cyclictest101ktimersoftd/011:00:340
29027991045,71cyclictest1561-21runrttasks09:45:240
149912970,6sleep22902999cyclictest12:00:252
29029999433,24cyclictest8854-21ssh09:55:322
29027999438,35cyclictest7850irq/56-dwc_otg_09:20:300
29027999412,39cyclictest7406-21diskmemload10:40:300
2903099933,13cyclictest9017-21wc09:55:353
116292920,2sleep2301ktimersoftd/209:15:102
29029998933,26cyclictest14041-21ssh11:20:222
2903099883,13cyclictest1924-21fw_packets09:45:293
29027998715,65cyclictest101ktimersoftd/011:10:380
2903099863,12cyclictest0-21swapper/311:10:383
55302850,2sleep00-21swapper/007:35:100
29028998542,25cyclictest24261-21kworker/u8:210:20:161
2902799857,51cyclictest9-21ksoftirqd/009:10:230
29029998426,19cyclictest0-21swapper/210:20:162
2902799846,15cyclictest0-21swapper/011:15:220
2902799836,34cyclictest0-21swapper/008:15:240
187072830,4sleep229-21rcuc/211:25:312
187072830,4sleep229-21rcuc/211:25:312
2903099824,8cyclictest7850irq/56-dwc_otg_09:25:323
2903099824,15cyclictest13498-21fschecks_count10:40:293
2903099824,11cyclictest0-21swapper/311:00:353
2903099823,16cyclictest12902-21switchtime09:15:393
29028998255,21cyclictest7775-21ssh09:10:241
29028998252,24cyclictest29827-21sshd11:00:181
29028998250,25cyclictest12611-21ls09:15:331
29028998234,24cyclictest9699-21sed11:15:101
29029998129,20cyclictest0-21swapper/209:15:382
29028998152,22cyclictest16598-21memory09:20:301
29027998133,28cyclictest20551-21sh09:25:300
29027998118,15cyclictest9-21ksoftirqd/012:05:240
29028998050,24cyclictest2047-21taskset09:45:311
2903099795,14cyclictest0-21swapper/309:20:313
2903099794,15cyclictest14030-21apt-get11:20:223
29027997829,4cyclictest0-21swapper/011:25:230
29027997829,4cyclictest0-21swapper/011:25:230
50142760,2sleep10-21swapper/112:26:211
29028997647,22cyclictest13496-21ssh10:40:291
29028997645,23cyclictest20639-21chrt09:25:321
29028997621,12cyclictest7950irq/56-dwc_otg_11:05:221
2902799753,34cyclictest0-21swapper/011:50:220
29029997443,25cyclictest16623-21sed09:20:312
2903099724,11cyclictest0-21swapper/310:20:173
29029997243,23cyclictest13173-21/usr/sbin/munin10:40:312
29027997222,47cyclictest101ktimersoftd/011:10:100
29028997033,21cyclictest22989-21ssh10:12:511
29028997033,21cyclictest22989-21ssh10:12:511
29028997032,20cyclictest11291-21cut07:50:221
277862700,2sleep10-21swapper/110:18:591
79852690,1sleep20-21swapper/211:50:472
29029996926,17cyclictest14060-21ls07:55:372
29028996931,23cyclictest30275-21ssh11:39:591
29028996834,7cyclictest15800-21ls08:00:371
29028996833,12cyclictest9543-21irqstats10:35:291
29028996832,24cyclictest13998-21ssh12:37:331
29028996831,14cyclictest23825-21ssh10:52:311
151712680,2sleep30-21swapper/308:00:293
304482670,3sleep229-21rcuc/209:40:112
29029996638,22cyclictest1542-21/usr/sbin/munin09:45:292
29029996636,24cyclictest6808-21timerandwakeup11:10:382
29029996622,20cyclictest0-21swapper/211:00:332
29028996633,21cyclictest14014-21rm10:01:291
29028996630,13cyclictest14028-21timerwakeupswit07:55:371
29028996629,19cyclictest14013-21ssh11:59:111
29028996623,23cyclictest7750irq/56-dwc_otg11:31:421
176102660,2sleep20-21swapper/210:05:392
29028996533,20cyclictest30381-21ssh12:18:381
29028996532,20cyclictest20687-21ssh10:49:231
29028996429,22cyclictest23492-21/usr/sbin/munin12:10:271
29028996429,22cyclictest23492-21/usr/sbin/munin12:10:271
29028996428,19cyclictest30685-21ssh11:40:191
29028996428,19cyclictest30685-21ssh11:40:191
29028996333,16cyclictest16942-21ssh12:01:561
29028996324,14cyclictest10935-21taskset07:49:581
29028996324,14cyclictest10935-21taskset07:49:581
29029996235,22cyclictest20624-21seq09:25:312
29028996235,15cyclictest2976-21expr11:45:291
29028996232,19cyclictest634-21ssh12:21:001
29028996230,18cyclictest28521-21sed08:35:381
29028996224,23cyclictest21221-21sh11:28:481
29028996224,23cyclictest21221-21sh11:28:481
29030996114,5cyclictest1674050wget08:05:253
29028996133,12cyclictest5005-21ssh09:50:281
29028996133,11cyclictest7333-21ssh10:32:421
29028996130,20cyclictest25378-21runrttasks08:29:561
29028996126,16cyclictest4363-21ssh10:29:521
2902799615,11cyclictest0-21swapper/007:10:210
2902799612,30cyclictest0-21swapper/009:40:230
2902799611,25cyclictest0-21swapper/011:35:230
29028996030,16cyclictest8304-21ssh11:51:221
29028996030,12cyclictest28168-21ssh10:57:531
29028996028,20cyclictest29538-21sh09:38:281
29028996026,17cyclictest10830-21kworker/u8:409:43:191
29028996026,10cyclictest7850irq/56-dwc_otg_09:05:231
2903099599,10cyclictest0-21swapper/311:58:493
29028995933,17cyclictest22833-21ssh12:10:051
29028995929,15cyclictest21742-21chrt08:19:391
29028995928,9cyclictest12276-21chrt11:17:371
29028995928,20cyclictest20020-21ssh10:10:011
2902799598,34cyclictest27484-21ssh12:15:240
29027995941,6cyclictest101ktimersoftd/011:45:310
29027995931,4cyclictest24587-21apt-get10:15:190
2902799592,39cyclictest3828-21latency_hist09:00:100
2902799592,33cyclictest30774-21grep11:40:240
2902799592,33cyclictest30774-21grep11:40:240
2902799591,34cyclictest0-21swapper/009:35:370
2903099583,12cyclictest7950irq/56-dwc_otg_10:15:333
2903099581,5cyclictest30532-21latency_hist11:40:113
29028995831,7cyclictest25640-21sshd08:30:181
29028995831,7cyclictest25640-21sshd08:30:181
29028995824,14cyclictest19864-21taskset08:13:521
29028995818,9cyclictest0-21swapper/112:33:441
2903099576,5cyclictest7750irq/56-dwc_otg08:43:063
2903099572,4cyclictest7850irq/56-dwc_otg_08:30:113
2903099572,12cyclictest7746-21apt-get09:10:223
29029995720,19cyclictest22797-21ssh10:12:302
29029995720,19cyclictest22797-21ssh10:12:302
29029995716,22cyclictest7950irq/56-dwc_otg_10:35:362
29028995727,22cyclictest30699-21chrt08:43:461
29028995724,13cyclictest3806-21munin-plugin-st09:00:101
2902799573,15cyclictest0-21swapper/008:05:220
29027995723,18cyclictest9-21ksoftirqd/008:00:270
29027995720,23cyclictest0-21swapper/007:40:150
24702570,2sleep30-21swapper/310:26:263
29029995614,21cyclictest7950irq/56-dwc_otg_11:40:252
29029995614,21cyclictest7950irq/56-dwc_otg_11:40:252
29029995611,26cyclictest0-21swapper/209:00:102
29028995623,12cyclictest0-21swapper/108:45:291
29028995525,17cyclictest0-21swapper/107:11:131
29028995523,14cyclictest0-21swapper/107:26:411
29027995527,8cyclictest31180-21fschecks_time07:15:250
29027995527,8cyclictest31180-21fschecks_time07:15:250
2902799552,24cyclictest0-21swapper/008:40:220
29027995518,34cyclictest101ktimersoftd/010:00:240
29027995514,30cyclictest9450irq/48-DMA08:56:170
2902799551,40cyclictest12847-21apt10:40:100
29028995433,17cyclictest17864-21sshd08:06:191
2902799547,29cyclictest9450irq/48-DMA08:56:170
2902799546,45cyclictest0-21swapper/008:45:240
2902799545,24cyclictest0-21swapper/008:35:110
2902799545,24cyclictest0-21swapper/008:35:110
29027995433,10cyclictest15597-21latency_hist12:40:110
29027995432,7cyclictest7850irq/56-dwc_otg_09:05:100
29027995430,6cyclictest7850irq/56-dwc_otg_12:00:100
2902799541,29cyclictest17576-21grep10:05:390
29027995411,24cyclictest11484-21fschecks_count07:50:250
29029995320,11cyclictest7950irq/56-dwc_otg_10:15:282
29029995315,14cyclictest26128-21ssh10:55:292
29028995322,18cyclictest9138-21seq07:45:041
29028995322,18cyclictest1497-21chrt07:21:301
29028995322,17cyclictest5393-21seq07:34:331
29027995313,21cyclictest18282-21switchtime10:45:360
2902799531,10cyclictest10561-21sed07:45:390
2902799531,10cyclictest10561-21sed07:45:390
2903099522,18cyclictest0-21swapper/308:54:243
29029995222,19cyclictest28629-21/usr/sbin/munin08:35:392
29029995218,8cyclictest28425-21ls12:15:372
29028995226,12cyclictest23047-21sed08:20:371
2902799526,23cyclictest0-21swapper/007:20:240
2902799522,39cyclictest0-21swapper/010:55:310
29027995214,12cyclictest0-21swapper/010:30:220
2902799521,30cyclictest1953-21sed10:25:390
29027995211,23cyclictest13-21rcuc/008:27:070
211192520,1sleep30-21swapper/310:50:093
2903099512,18cyclictest0-21swapper/308:14:243
29029995111,11cyclictest12747-21sh11:56:522
29028995126,12cyclictest0-21swapper/107:36:551
29028995123,9cyclictest0-21swapper/108:54:331
29027995124,4cyclictest19305-21switchtime08:10:370
29027995121,15cyclictest9-21ksoftirqd/009:05:260
29027995114,21cyclictest21484-21sh10:50:210
284282513,12sleep00-21swapper/007:05:540
29029995018,12cyclictest25268-21cron10:55:102
2902799501,34cyclictest0-21swapper/012:29:310
2903099497,7cyclictest7171-21latency11:50:313
29029994923,9cyclictest15521-21ntp_states08:00:342
29029994919,11cyclictest3473-21ssh11:06:422
29029994916,17cyclictest4432-21ssh12:25:372
29029994915,11cyclictest7850irq/56-dwc_otg_12:10:302
29029994915,11cyclictest7850irq/56-dwc_otg_12:10:302
29029994914,13cyclictest9412-21ssh12:31:462
29029994910,12cyclictest7850irq/56-dwc_otg_12:10:112
29028994923,14cyclictest0-21swapper/107:16:221
29028994923,14cyclictest0-21swapper/107:16:221
2902799492,7cyclictest21698-21grep10:10:390
2902799492,7cyclictest21698-21grep10:10:390
2902799492,25cyclictest0-21swapper/012:30:360
2902799491,4cyclictest23646-21ls12:10:290
2902799491,4cyclictest23646-21ls12:10:290
2903099487,12cyclictest0-21swapper/309:50:223
29029994822,11cyclictest15324-21ssh10:03:572
29029994819,11cyclictest30252-21ssh11:39:572
29029994815,12cyclictest11444-21sort07:50:252
29029994810,23cyclictest7850irq/56-dwc_otg_10:46:392
29027994815,17cyclictest5223-21sh09:50:310
29027994812,2cyclictest0-21swapper/012:00:240
29029994724,15cyclictest693-21ssh09:44:072
29029994715,12cyclictest4544-21kworker/2:011:15:322
29029994714,22cyclictest858-21sh12:21:242
29029994633,4cyclictest2438-21ssh11:45:222
29029994621,11cyclictest2600-21ssh10:26:392
29029994617,12cyclictest7300-21ssh10:32:392
29029994614,13cyclictest30288-21taskset07:11:122
29029994613,12cyclictest4610-21ssh09:50:112
29028994618,13cyclictest7850irq/56-dwc_otg_09:00:111
2903099454,6cyclictest7850irq/56-dwc_otg_11:15:233
29029994521,14cyclictest13839-21taskset12:37:162
29029994516,12cyclictest22413-21apt-get11:30:222
2903099442,8cyclictest7950irq/56-dwc_otg_07:20:213
29029994418,16cyclictest23550-21taskset08:24:402
29029994413,13cyclictest7950irq/56-dwc_otg_09:00:112
2902799442,26cyclictest12568-21kworker/0:107:55:320
2902799442,23cyclictest9-21ksoftirqd/008:54:060
2902999439,22cyclictest7950irq/56-dwc_otg_08:30:232
2902999439,22cyclictest7950irq/56-dwc_otg_08:30:232
29029994315,10cyclictest7950irq/56-dwc_otg_09:05:242
29029994314,11cyclictest0-21swapper/207:16:052
29029994314,11cyclictest0-21swapper/207:16:052
29029994313,12cyclictest0-21swapper/207:29:232
29029994212,12cyclictest6984-21seq07:36:552
2902799422,23cyclictest0-21swapper/007:25:150
29027994220,14cyclictest101ktimersoftd/008:37:560
29029994112,11cyclictest30467-21taskset08:40:452
29029994111,13cyclictest32508-21runrttasks08:48:482
29029994111,10cyclictest7950irq/56-dwc_otg_07:50:102
29029994111,10cyclictest7950irq/56-dwc_otg_07:50:102
29027994120,9cyclictest101ktimersoftd/007:37:060
2902999409,13cyclictest0-21swapper/208:51:532
29029994023,14cyclictest4533-21/usr/sbin/munin07:30:312
29029993927,9cyclictest21269-21ls08:15:382
29029993925,11cyclictest857-21ls07:20:322
29029993917,12cyclictest25198-21taskset08:27:412
2902799392,21cyclictest0-21swapper/008:20:330
2903099388,5cyclictest2096-21ls11:45:103
2903099388,5cyclictest2096-21ls11:45:103
29030993816,4cyclictest37-21rcuc/312:25:303
29029993813,11cyclictest17860-21taskset08:06:162
29029993811,9cyclictest0-21swapper/207:42:052
2903099371,22cyclictest9164-21latency_hist07:45:113
29030993629,5cyclictest4102-21ssh11:46:183
29030993617,4cyclictest26366-21sh10:55:323
29030993512,12cyclictest39-21ksoftirqd/312:05:373
29029993522,11cyclictest19087-21/usr/sbin/munin08:10:332
2903099347,13cyclictest7950irq/56-dwc_otg_11:25:403
2903099347,13cyclictest7950irq/56-dwc_otg_11:25:403
2903099346,13cyclictest39-21ksoftirqd/312:00:283
2903099345,5cyclictest7850irq/56-dwc_otg_09:35:223
2903099341,24cyclictest21105-21ls08:15:363
29030993411,11cyclictest7750irq/56-dwc_otg11:30:153
192212340,2sleep30-21swapper/310:08:343
2903099337,15cyclictest23037-1kworker/u9:010:51:513
2903099332,7cyclictest7850irq/56-dwc_otg_08:20:203
2903099331,15cyclictest7950irq/56-dwc_otg_08:35:313
2903099323,17cyclictest0-21swapper/310:00:373
2903099323,14cyclictest7950irq/56-dwc_otg_07:15:113
2903099321,3cyclictest7950irq/56-dwc_otg_11:05:283
2903099321,26cyclictest5675-21ls09:05:103
44982310,5sleep34499-21netstat07:30:303
2903099319,4cyclictest0-21swapper/312:19:553
2903099313,16cyclictest7950irq/56-dwc_otg_07:25:333
2903099312,19cyclictest0-21swapper/312:23:203
2903099311,5cyclictest308-21sort08:50:103
2903099311,25cyclictest0-21swapper/312:30:253
2903099311,21cyclictest10285-21ls07:45:363
2903099311,21cyclictest10285-21ls07:45:363
2903099311,20cyclictest6264-21missed_timers07:35:313
2903099311,20cyclictest14579-21latency_hist08:00:103
2903099311,14cyclictest0-21swapper/312:10:363
2903099311,14cyclictest0-21swapper/312:10:363
29030993111,15cyclictest0-21swapper/309:40:243
2903099305,5cyclictest7950irq/56-dwc_otg_09:05:273
2903099302,21cyclictest0-21swapper/307:19:063
2903099302,21cyclictest0-21swapper/307:19:063
2903099301,7cyclictest9366-21tune2fs10:35:273
29030993015,5cyclictest6760-21sh10:31:403
29030993010,4cyclictest0-21swapper/310:12:153
29030993010,4cyclictest0-21swapper/310:12:153
2903099299,10cyclictest0-21swapper/312:35:593
2903099293,16cyclictest1180-21runrttasks07:52:193
279052295,12sleep327199-21/usr/sbin/munin07:05:333
2903099281,6cyclictest2813-21grep08:55:323
271902283,14sleep10-21swapper/107:05:151
2903099252,4cyclictest0-21swapper/308:30:243
2903099252,4cyclictest0-21swapper/308:30:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional