You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-04 - 02:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Sun Jan 04, 2026 00:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
193482920,2sleep1221ktimersoftd/122:30:131
129722860,5sleep11152699cyclictest23:40:331
121862860,2sleep10-21swapper/123:40:131
11525997814,41cyclictest28246-21aten_rbpower_cu21:25:270
11525997812,47cyclictest16213-21aten_rbpower_po23:50:280
11525997710,47cyclictest32745-21missed_timers23:05:340
135282760,5sleep21152799cyclictest19:15:292
11525997613,48cyclictest26479-21fschecks_time19:50:310
1152599758,48cyclictest30115-21latency_hist23:00:130
315852740,2sleep30-21swapper/320:05:293
11525997211,51cyclictest25784-21wc00:15:330
1152599699,49cyclictest29114-21grep19:55:400
11525996810,44cyclictest0-21swapper/000:35:010
1152599679,36cyclictest0-21swapper/022:40:250
1152599676,49cyclictest1125-21runrttasks21:17:590
105472670,2sleep110545-21df22:05:301
162902650,2sleep20-21swapper/223:50:292
149332650,5sleep30-21swapper/319:20:123
11525996514,41cyclictest22529-21apt-get19:40:310
11525996417,28cyclictest1125-21runrttasks00:10:360
1152599636,24cyclictest0-21swapper/019:20:270
11525996317,25cyclictest20732-21latency_hist21:05:130
11525996311,31cyclictest15895-21aten_rbpower_vo22:20:280
11525996310,38cyclictest18706-21sed19:30:130
1152599629,34cyclictest30494-21cut21:30:330
1152599629,34cyclictest30494-21cut21:30:330
1152599623,45cyclictest0-21swapper/022:05:410
1152599615,39cyclictest0-21swapper/020:11:040
11525996116,38cyclictest21829-21sed19:35:410
11525996110,40cyclictest27091-21/usr/sbin/munin21:20:350
1152699606,10cyclictest0-21swapper/123:20:261
1152599602,6cyclictest8599-21latency_hist23:30:130
1152599601,32cyclictest0-21swapper/022:30:410
1152599593,45cyclictest0-21swapper/020:17:080
1152599585,35cyclictest0-21swapper/022:35:240
11525995826,15cyclictest0-21swapper/020:30:250
1152599582,47cyclictest0-21swapper/019:45:390
11525995811,26cyclictest18563-21apt-get19:30:140
11525995733,5cyclictest1854-21cut23:10:310
11525995719,13cyclictest0-21swapper/000:35:250
1152599567,36cyclictest0-21swapper/023:33:430
1152599566,37cyclictest0-21swapper/021:10:130
1152599566,35cyclictest28025-21timerandwakeup00:20:400
1152599563,41cyclictest0-21swapper/020:39:040
1152599563,39cyclictest0-21swapper/023:55:290
11525995612,25cyclictest1125-21runrttasks20:28:480
11525995611,29cyclictest13073-21chrt20:41:360
1152599555,30cyclictest0-21swapper/019:19:360
1152599555,29cyclictest17178-21latency_hist20:55:140
1152599553,41cyclictest0-21swapper/022:45:510
1152599553,34cyclictest0-21swapper/021:43:560
11525995532,5cyclictest7850irq/56-dwc_otg_21:40:120
1152599552,29cyclictest3378-21aten_rbpower_cu23:15:290
11525995521,18cyclictest0-21swapper/000:05:240
11525995510,42cyclictest5803-21chrt20:21:120
11525995434,5cyclictest0-21swapper/000:25:250
1152599542,27cyclictest128-21systemd-journal23:36:330
1152599541,35cyclictest8525-21acpi22:00:250
11525995410,32cyclictest31716-21chrt23:02:450
1152599540,42cyclictest0-21swapper/020:00:240
151892530,8sleep31152899cyclictest20:50:133
1152599539,34cyclictest0-21swapper/021:49:130
1152599536,30cyclictest0-21swapper/023:45:200
1152599535,35cyclictest0-21swapper/023:45:140
1152599534,28cyclictest0-21swapper/022:15:390
11525995311,26cyclictest15097-21chrt20:49:170
1152699526,7cyclictest7950irq/56-dwc_otg_19:55:241
1152599525,29cyclictest0-21swapper/022:50:240
1152599525,29cyclictest0-21swapper/022:50:240
1152599515,29cyclictest128-21systemd-journal23:22:190
1152599513,37cyclictest0-21swapper/021:57:230
1152599511,37cyclictest128-21systemd-journal19:11:530
1152599504,43cyclictest0-21swapper/020:08:300
1152599503,44cyclictest21017-21chrt00:01:420
11525995024,5cyclictest17473-21fschecks_count20:55:300
11525995024,5cyclictest17473-21fschecks_count20:55:300
1152699497,27cyclictest0-21swapper/121:05:251
1152599494,35cyclictest0-21swapper/021:15:030
1152699488,24cyclictest0-21swapper/100:20:281
1152599489,23cyclictest13597-21taskset22:13:530
1152599483,36cyclictest0-21swapper/021:51:570
1152899477,25cyclictest0-21swapper/323:45:203
1152899476,18cyclictest7950irq/56-dwc_otg_19:30:393
1152799477,25cyclictest0-21swapper/220:05:292
1152699477,26cyclictest0-21swapper/123:30:271
1152899467,24cyclictest0-21swapper/321:45:183
1152699467,24cyclictest0-21swapper/121:45:191
1152699467,24cyclictest0-21swapper/121:30:271
1152699467,24cyclictest0-21swapper/121:30:271
1152599465,30cyclictest0-21swapper/022:27:000
11528994423,13cyclictest15246-21wget19:20:283
11527994417,7cyclictest21384-21aten_rbpower_po22:35:272
1152699447,23cyclictest0-21swapper/119:25:271
11526994429,11cyclictest0-21swapper/121:55:221
11526994423,5cyclictest861550wget22:00:281
110242442,36sleep00-21swapper/019:07:130
1152899437,24cyclictest0-21swapper/322:55:263
1152899437,24cyclictest0-21swapper/320:40:293
1152899437,24cyclictest0-21swapper/319:10:273
1152799437,24cyclictest0-21swapper/221:25:272
1152699437,23cyclictest0-21swapper/123:05:261
1152699437,23cyclictest0-21swapper/121:00:181
1152699431,8cyclictest7750irq/56-dwc_otg20:40:271
116842420,4sleep337-21rcuc/322:07:333
1152899427,24cyclictest0-21swapper/320:25:253
1152699427,25cyclictest0-21swapper/119:15:271
1152699427,23cyclictest0-21swapper/121:25:271
11527994113,12cyclictest31-21ksoftirqd/221:10:132
1152899402,14cyclictest37-21rcuc/300:10:263
1152699407,23cyclictest0-21swapper/122:35:261
1152699407,23cyclictest0-21swapper/100:35:271
1152699402,21cyclictest7750irq/56-dwc_otg19:20:261
1152899397,9cyclictest39-21ksoftirqd/319:40:313
1152899397,22cyclictest0-21swapper/321:40:273
11528993917,5cyclictest37-21rcuc/321:20:283
11528993914,6cyclictest0-21swapper/322:00:193
1152799399,13cyclictest7950irq/56-dwc_otg_21:55:392
1152799397,24cyclictest0-21swapper/220:20:262
11527993917,5cyclictest4096-21/usr/sbin/munin23:15:402
1152799391,30cyclictest27449-21ls00:20:322
1152699397,22cyclictest0-21swapper/123:00:281
1152699391,23cyclictest0-21swapper/100:15:261
1152899387,20cyclictest0-21swapper/322:40:273
11528993817,4cyclictest25529-21fschecks_count00:15:303
1152899381,3cyclictest7950irq/56-dwc_otg_21:50:243
1152799382,4cyclictest7950irq/56-dwc_otg_23:00:252
1152799381,25cyclictest24716-21ls22:45:132
1152699381,23cyclictest32356-21ls21:35:371
1152899379,14cyclictest39-21ksoftirqd/323:05:273
1152899377,18cyclictest37-21rcuc/319:45:403
1152899371,8cyclictest7850irq/56-dwc_otg_22:15:253
1152899371,22cyclictest22060-21latency00:05:333
1152799374,15cyclictest7950irq/56-dwc_otg_19:20:312
11527993717,4cyclictest23109-21latency_hist00:10:142
11527993715,5cyclictest18559-21latency_hist19:30:142
1152699371,26cyclictest5265-21ls21:50:311
308322360,3sleep1221ktimersoftd/100:30:291
1152899367,18cyclictest39-21ksoftirqd/323:50:313
1152899366,20cyclictest28452-21uniq21:25:313
1152899366,13cyclictest7950irq/56-dwc_otg_21:05:133
1152899365,16cyclictest37-21rcuc/321:35:583
1152899365,15cyclictest7950irq/56-dwc_otg_20:20:413
1152899364,19cyclictest37-21rcuc/321:10:133
11528993617,5cyclictest26194-21aten_rbpower_en19:50:273
1152799369,5cyclictest31-21ksoftirqd/221:30:312
1152799369,5cyclictest31-21ksoftirqd/221:30:312
1152799368,21cyclictest31-21ksoftirqd/221:10:402
1152799364,14cyclictest7850irq/56-dwc_otg_22:10:422
1152799363,4cyclictest7950irq/56-dwc_otg_00:00:312
11527993616,11cyclictest7750irq/56-dwc_otg23:05:332
11526993618,6cyclictest19805-21fschecks_time22:30:321
1152899358,5cyclictest7950irq/56-dwc_otg_22:30:433
1152899357,15cyclictest7950irq/56-dwc_otg_00:30:343
1152899356,16cyclictest37-21rcuc/321:30:363
1152899356,16cyclictest37-21rcuc/321:30:363
11528993513,4cyclictest37-21rcuc/319:55:363
11528993511,12cyclictest381ktimersoftd/322:35:253
1152799356,12cyclictest7950irq/56-dwc_otg_21:20:402
11527993516,4cyclictest7850irq/56-dwc_otg_22:15:332
1152799351,5cyclictest0-21swapper/222:55:242
11527993512,5cyclictest0-21swapper/222:00:282
1152799351,14cyclictest31-21ksoftirqd/219:55:402
1152699358,3cyclictest0-21swapper/120:30:131
1152699357,5cyclictest7850irq/56-dwc_otg_21:40:331
1152699357,16cyclictest7850irq/56-dwc_otg_21:25:131
1152699352,7cyclictest0-21swapper/120:00:251
1152699352,15cyclictest7850irq/56-dwc_otg_23:25:321
11526993516,4cyclictest25014-21/usr/sbin/munin19:45:361
1152699351,24cyclictest27161-21timerwakeupswit19:50:401
300122340,4sleep31152899cyclictest20:00:303
1152899348,19cyclictest381ktimersoftd/319:40:133
1152899345,16cyclictest7950irq/56-dwc_otg_23:25:433
1152899341,11cyclictest7950irq/56-dwc_otg_23:55:333
1152799345,5cyclictest14704-21unixbench_multi20:45:412
1152799344,12cyclictest29-21rcuc/200:10:192
1152799342,4cyclictest7850irq/56-dwc_otg_19:35:262
11527993424,5cyclictest15482-21aten_rbpower_en20:50:262
1152799341,5cyclictest7950irq/56-dwc_otg_00:35:322
1152799341,25cyclictest7850irq/56-dwc_otg_00:15:332
11527993411,11cyclictest29-21rcuc/219:45:132
1152699342,3cyclictest7750irq/56-dwc_otg20:15:261
1152699341,23cyclictest32085-21latency20:05:351
1152899339,5cyclictest7950irq/56-dwc_otg_00:35:383
1152899336,10cyclictest381ktimersoftd/319:30:133
1152799338,5cyclictest31-21ksoftirqd/221:00:262
1152799333,5cyclictest5404-21fschecks_count23:20:312
1152799333,16cyclictest301ktimersoftd/221:20:132
11527993314,6cyclictest31-21ksoftirqd/222:35:132
11527993311,15cyclictest31-21ksoftirqd/223:45:142
11527993310,5cyclictest7850irq/56-dwc_otg_00:25:302
1152699338,6cyclictest7850irq/56-dwc_otg_22:40:411
1152699338,5cyclictest9450irq/48-DMA10:44:551
1152699338,5cyclictest9450irq/48-DMA10:44:551
1152699337,16cyclictest0-21swapper/100:25:261
1152699336,12cyclictest7850irq/56-dwc_otg_22:15:281
1152699332,20cyclictest0-21swapper/119:40:281
11526993313,16cyclictest0-21swapper/100:05:261
1152899327,12cyclictest39-21ksoftirqd/323:45:143
1152899327,11cyclictest7750irq/56-dwc_otg23:39:213
1152899325,14cyclictest37-21rcuc/322:21:523
1152899323,5cyclictest39-21ksoftirqd/300:25:413
1152899323,19cyclictest0-21swapper/323:22:153
1152899323,18cyclictest39-21ksoftirqd/320:54:333
11528993226,4cyclictest381ktimersoftd/323:10:253
1152799326,15cyclictest0-21swapper/223:45:202
1152799325,14cyclictest3243-21latency_hist23:15:132
1152799324,6cyclictest12122-21fschecks_time20:40:332
1152799321,22cyclictest0-21swapper/221:50:352
1152699328,4cyclictest0-21swapper/122:45:401
1152699324,14cyclictest7950irq/56-dwc_otg_19:10:391
1152699321,9cyclictest23940-21sed21:10:411
1152899313,10cyclictest37-21rcuc/321:20:133
1152899312,14cyclictest7950irq/56-dwc_otg_23:00:283
1152899311,27cyclictest18960-21latency_hist21:00:143
1152899311,27cyclictest18960-21latency_hist21:00:143
1152899311,16cyclictest0-21swapper/300:21:083
11528993110,9cyclictest37-21rcuc/320:39:043
1152799317,21cyclictest0-21swapper/221:35:292
1152799315,14cyclictest7950irq/56-dwc_otg_21:45:262
1152799315,14cyclictest29-21rcuc/222:49:072
1152799314,7cyclictest7850irq/56-dwc_otg_21:40:392
1152799313,14cyclictest7950irq/56-dwc_otg_20:10:272
1152799313,11cyclictest829350wget20:30:272
1152799312,11cyclictest29-21rcuc/219:45:322
1152799311,3cyclictest7750irq/56-dwc_otg22:20:262
11527993111,9cyclictest7850irq/56-dwc_otg_23:25:312
1152699316,7cyclictest7750irq/56-dwc_otg00:10:261
1152699316,15cyclictest0-21swapper/120:45:261
1152699315,5cyclictest7750irq/56-dwc_otg22:10:351
1152699312,22cyclictest0-21swapper/123:52:551
11526993120,8cyclictest18023-21grep23:55:281
1152899302,14cyclictest7950irq/56-dwc_otg_23:15:293
11528993014,8cyclictest37-21rcuc/322:45:553
11528993011,11cyclictest27196-21sshd22:50:333
11528993011,11cyclictest27196-21sshd22:50:333
1152799306,15cyclictest0-21swapper/219:50:262
1152799305,13cyclictest7950irq/56-dwc_otg_20:15:312
11527993024,4cyclictest18808-21idleruntime-cro21:00:132
11527993024,4cyclictest18808-21idleruntime-cro21:00:132
1152799302,10cyclictest0-21swapper/223:35:362
1152799301,4cyclictest7750irq/56-dwc_otg22:25:192
1152699301,6cyclictest7750irq/56-dwc_otg22:20:391
1152699301,6cyclictest1141-21lldpd20:30:351
11526993013,3cyclictest7850irq/56-dwc_otg_19:30:341
1152699301,27cyclictest7950irq/56-dwc_otg_21:20:131
34402290,4sleep3381ktimersoftd/320:15:353
1152899295,11cyclictest7950irq/56-dwc_otg_00:00:343
1152899292,3cyclictest0-21swapper/320:30:373
1152899291,23cyclictest18705-21timerwakeupswit22:25:413
1152799295,12cyclictest7950irq/56-dwc_otg_23:55:292
1152799294,5cyclictest7750irq/56-dwc_otg20:25:332
1152799294,16cyclictest9039-21fschecks_count23:30:312
1152799292,6cyclictest0-21swapper/200:30:282
1152799291,5cyclictest7850irq/56-dwc_otg_19:30:192
1152699295,5cyclictest7750irq/56-dwc_otg20:20:311
1152699294,6cyclictest7950irq/56-dwc_otg_23:11:321
1152899288,11cyclictest381ktimersoftd/321:57:243
1152899284,16cyclictest7850irq/56-dwc_otg_21:10:523
1152899284,10cyclictest7950irq/56-dwc_otg_23:30:543
1152799281,6cyclictest7850irq/56-dwc_otg_20:00:332
1152799281,4cyclictest7850irq/56-dwc_otg_20:35:262
1152699285,5cyclictest21170-21sshd19:35:321
1152699281,19cyclictest0-21swapper/123:47:521
1152699281,11cyclictest0-21swapper/120:50:401
112552287,8sleep10-21swapper/119:10:131
1152899275,5cyclictest7950irq/56-dwc_otg_20:10:303
1152899271,16cyclictest0-21swapper/322:10:323
1152799277,18cyclictest0-21swapper/219:10:272
1152799273,15cyclictest29-21rcuc/222:05:372
1152799271,5cyclictest7950irq/56-dwc_otg_22:50:402
1152799271,5cyclictest7950irq/56-dwc_otg_22:50:402
110772272,16sleep30-21swapper/319:07:523
106622275,9sleep20-21swapper/219:05:412
1152699269,4cyclictest7850irq/56-dwc_otg_00:00:241
1152699265,4cyclictest7750irq/56-dwc_otg23:15:401
1152699264,9cyclictest7850irq/56-dwc_otg_22:55:151
1152699263,4cyclictest7750irq/56-dwc_otg22:50:331
1152699263,4cyclictest7750irq/56-dwc_otg22:50:331
1152699262,4cyclictest0-21swapper/120:10:391
1152699261,4cyclictest10448-21irqstats20:35:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional