You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 16:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu Jun 04, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18067996843,16cyclictest35966irq/58-eth012:30:113
18059995816,32cyclictest0-21swapper/212:05:142
1805999572,42cyclictest20643-21ssh09:55:132
1803999562,49cyclictest27460-21ssh12:05:140
18067995432,15cyclictest35966irq/58-eth010:45:133
18059995432,16cyclictest3782-21diskmemload10:45:122
1804799541,43cyclictest7954-21ssh11:00:111
18059995023,20cyclictest27-21ksoftirqd/211:40:122
18047995042,3cyclictest30678-21ssh12:25:121
1804799472,5cyclictest131rcu_preempt09:35:131
1806799463,18cyclictest35966irq/58-eth011:55:133
1804799460,35cyclictest24304-21ssh10:15:131
18067994533,6cyclictest0-21swapper/309:25:133
18059994528,11cyclictest6216-21ssh10:05:122
18059994314,20cyclictest25442-21ssh11:09:572
1804799423,25cyclictest7871-21ssh09:30:131
18047994229,6cyclictest536-21usb-storage07:35:101
1803999426,30cyclictest13169-21ssh09:15:110
18039994232,7cyclictest12429-21kworker/u8:210:45:120
18067994123,12cyclictest35966irq/58-eth012:20:143
18059994123,11cyclictest27346irq/40-dwc2_hso09:20:112
18059994112,19cyclictest5198-21ssh09:27:172
1804799413,28cyclictest4004-21ssh12:10:141
18047994113,16cyclictest21-21ksoftirqd/109:05:111
18059994032,4cyclictest32767-21apt-get11:50:132
18059994029,7cyclictest13157-21apt-get09:15:122
18059994028,8cyclictest29636-21apt-get10:00:132
18059994014,15cyclictest25757-21ssh11:10:022
18059994013,16cyclictest3782-21diskmemload11:28:222
1806799394,18cyclictest35966irq/58-eth010:05:123
1806799391,23cyclictest35966irq/58-eth009:40:133
18059993913,16cyclictest3782-21diskmemload10:34:442
18047993925,3cyclictest21-21ksoftirqd/107:55:131
1806799385,18cyclictest35966irq/58-eth007:55:113
18067993815,13cyclictest35966irq/58-eth010:40:123
18059993833,2cyclictest19371-21apt-get07:45:132
1805999382,21cyclictest2677-21ssh09:45:132
18059993815,19cyclictest0-21swapper/209:35:122
18059993815,14cyclictest3782-21diskmemload09:12:092
18059993814,20cyclictest0-21swapper/208:55:122
18059993813,11cyclictest3782-21diskmemload10:51:422
18059993813,10cyclictest3782-21diskmemload10:19:172
18059993812,8cyclictest3782-21diskmemload11:18:062
18059993812,18cyclictest9287-21rm12:11:012
18059993811,7cyclictest0-21swapper/208:35:242
18059993811,19cyclictest16909-21/usr/sbin/munin12:35:202
18059993810,12cyclictest6071-21ssh12:29:042
1804799384,19cyclictest530-21kworker/u8:010:40:121
18039993823,9cyclictest151rcuc/010:35:110
1803999381,27cyclictest31304-21ssh09:25:120
1806799371,24cyclictest0-21swapper/309:45:133
1805999379,12cyclictest2813-21ssh11:31:402
18059993726,8cyclictest9837-21apt-get10:25:112
18059993713,21cyclictest25185-21kthreadcore07:15:212
18059993713,12cyclictest6221-21ssh10:21:072
18059993712,6cyclictest3782-21diskmemload09:51:332
18059993711,17cyclictest31376-21ssh11:48:462
18059993711,17cyclictest31376-21ssh11:48:462
18059993711,12cyclictest3782-21diskmemload09:44:572
18059993710,12cyclictest31110-21cut10:35:212
18059993710,10cyclictest0-21swapper/210:41:332
18047993715,10cyclictest21-21ksoftirqd/110:25:131
1805999369,11cyclictest3092-21awk10:55:292
18059993626,7cyclictest24179-21apt-get07:50:122
18059993624,9cyclictest18341-21apt-get12:00:132
18059993612,10cyclictest14117-21ssh11:39:362
18059993611,17cyclictest20316-21awk08:15:312
18059993611,15cyclictest14967-21ssh11:03:052
18059993610,18cyclictest0-21swapper/209:33:242
18059993610,16cyclictest0-21swapper/212:16:342
18047993614,17cyclictest21-21ksoftirqd/108:55:111
1804799361,26cyclictest18809-21rm10:30:111
1806799356,17cyclictest35966irq/58-eth011:45:123
1806799356,17cyclictest35966irq/58-eth011:45:123
1805999359,12cyclictest3782-21diskmemload11:21:262
1805999359,12cyclictest13463-21ssh12:32:152
18059993511,10cyclictest20353-21ssh10:10:462
1804799352,26cyclictest322-21ssh11:50:141
18047993519,14cyclictest21-21ksoftirqd/111:05:121
1805999348,12cyclictest2670-21ntp_states09:05:252
18059993423,3cyclictest21786-21ssh12:20:122
18059993417,13cyclictest30606-21grep09:00:272
18059993314,15cyclictest0-21swapper/208:10:122
1804799332,20cyclictest11493-21ssh11:20:121
1805999328,12cyclictest0-21swapper/211:57:432
18059993212,11cyclictest2002-21tr07:25:212
18059993212,11cyclictest2002-21tr07:25:202
1804799326,12cyclictest0-21swapper/110:20:111
1806799316,18cyclictest35966irq/58-eth010:20:123
18059993118,10cyclictest12311-21diskstats08:45:142
18059993117,11cyclictest0-21swapper/208:20:122
18059993116,8cyclictest18447-21awk07:40:302
18059993116,8cyclictest18447-21awk07:40:292
18059993115,12cyclictest0-21swapper/208:40:102
18059993111,15cyclictest9641-21sed07:35:022
18059993111,15cyclictest9641-21sed07:35:012
1803999311,4cyclictest822-21apt-get10:20:120
1803999311,21cyclictest31324-21ssh10:55:120
18067993016,9cyclictest35966irq/58-eth009:30:133
1805999305,17cyclictest27346irq/40-dwc2_hso08:05:142
1805999305,17cyclictest27346irq/40-dwc2_hso08:05:132
18059993017,6cyclictest32439-21/usr/sbin/munin07:55:262
18059993016,11cyclictest31573-21grep07:20:282
1804799303,21cyclictest2668-21ssh09:45:131
1803999301,18cyclictest12-21ksoftirqd/008:15:110
1806799294,12cyclictest35966irq/58-eth011:20:133
18067992916,10cyclictest35966irq/58-eth010:15:123
1806799291,20cyclictest35966irq/58-eth010:25:133
1806799291,20cyclictest35966irq/58-eth008:40:133
18059992914,8cyclictest2178-21usage_EthcTask08:30:292
18059992911,9cyclictest21819-21sed07:10:282
18039992916,6cyclictest0-21swapper/008:50:130
1806799282,16cyclictest35966irq/58-eth010:55:113
18067992816,5cyclictest0-21swapper/308:55:113
18059992817,8cyclictest29951-21sshd08:28:292
18059992814,11cyclictest5068-21grep08:00:292
18059992812,12cyclictest21589-21cron08:55:012
18059992810,15cyclictest14501-21date07:40:012
18047992812,9cyclictest0-21swapper/112:20:141
1803999289,11cyclictest0-21swapper/010:10:260
1803999282,21cyclictest4195-21ssh09:10:130
1804799272,13cyclictest9324-21ssh11:55:121
18047992715,8cyclictest18028-21cyclictest12:15:131
18047992714,7cyclictest18028-21cyclictest12:35:151
1804799271,20cyclictest29389-21ssh11:30:131
1803999272,16cyclictest9318-21ssh11:55:130
18039992714,3cyclictest12-21ksoftirqd/012:36:220
1804799262,15cyclictest18028-21cyclictest07:50:121
18047992614,9cyclictest0-21swapper/108:30:131
1804799261,19cyclictest27824-21ssh10:35:131
1803999262,18cyclictest14882-21ssh11:40:150
1803999262,16cyclictest20418-21ssh11:25:120
18039992614,3cyclictest12-21ksoftirqd/007:45:280
1806799253,16cyclictest35966irq/58-eth010:35:123
1806799253,14cyclictest0-21swapper/312:10:153
1806799252,17cyclictest35966irq/58-eth011:00:123
18067992514,5cyclictest0-21swapper/311:30:133
1803999259,3cyclictest12-21ksoftirqd/012:15:120
1803999252,17cyclictest11649-21ssh09:50:130
18039992512,4cyclictest12-21ksoftirqd/009:30:260
18039992510,4cyclictest131rcu_preempt11:00:110
1806799241,13cyclictest35966irq/58-eth009:55:143
1804799241,2cyclictest131rcu_preempt12:00:131
18047992412,8cyclictest0-21swapper/107:15:121
1803999248,8cyclictest12-21ksoftirqd/009:20:410
1803999241,18cyclictest7221-21ssh12:30:110
1804799237,11cyclictest21-21ksoftirqd/107:35:021
1804799237,11cyclictest21-21ksoftirqd/107:35:011
1804799236,11cyclictest22323-21ssh10:50:141
1804799232,17cyclictest14855-21apt-get11:40:121
1804799231,9cyclictest7334-21diskstats12:30:131
18047992313,7cyclictest10729-21memory08:40:211
1804799231,18cyclictest31293-21apt-get09:25:111
1804799231,17cyclictest2583-21find08:35:121
1804799231,17cyclictest15231-21ssh10:10:131
1803999239,8cyclictest0-21swapper/009:39:580
1803999238,5cyclictest12-21ksoftirqd/010:33:360
18039992314,6cyclictest12-21ksoftirqd/012:20:130
1803999231,19cyclictest29651-21ssh10:00:130
18039992310,4cyclictest12-21ksoftirqd/010:40:130
1806799221,12cyclictest35966irq/58-eth011:10:113
1804799221,18cyclictest12202-21find08:45:101
1803999229,7cyclictest0-21swapper/011:17:260
1803999226,4cyclictest12-21ksoftirqd/010:08:000
1803999224,13cyclictest18372-21rm12:00:130
1803999222,17cyclictest332-21ssh11:50:130
18039992211,3cyclictest12-21ksoftirqd/012:10:150
1806799212,15cyclictest0-21swapper/307:10:133
1806799211,11cyclictest35966irq/58-eth011:40:123
18067992110,8cyclictest0-21swapper/311:25:193
1805999212,9cyclictest18091-21grep07:10:012
1804799219,9cyclictest21-21ksoftirqd/110:00:131
1804799212,15cyclictest25918-21ssh11:10:141
1804799211,16cyclictest25945-21ssh09:20:251
1803999212,14cyclictest23814-21ssh11:45:120
1803999212,14cyclictest23814-21ssh11:45:110
18039992114,4cyclictest12-21ksoftirqd/011:30:410
1803999211,17cyclictest30352-21ls09:00:250
1803999211,16cyclictest25826-21sh10:50:240
1803999211,15cyclictest20659-21sh09:55:140
18039992110,8cyclictest11632-21entropy11:20:160
18039992110,3cyclictest12-21ksoftirqd/009:45:260
18039992110,3cyclictest12-21ksoftirqd/008:40:130
1803999210,15cyclictest10757-21ls08:10:020
1803999210,15cyclictest10757-21ls08:10:010
1806799203,12cyclictest0-21swapper/309:10:143
18067992010,7cyclictest35966irq/58-eth012:25:113
18067992010,7cyclictest0-21swapper/311:15:123
1804799201,9cyclictest21468-21missed_timers07:10:221
1804799201,17cyclictest20725-21uniq08:20:171
1804799201,15cyclictest31419-21cat07:20:261
1804799201,14cyclictest2549-21ssh10:55:231
1804799201,10cyclictest17006-21ntp_192.168.11410:45:251
1803999209,8cyclictest17058-21df11:05:140
1803999209,3cyclictest12-21ksoftirqd/010:29:260
1803999207,4cyclictest131rcu_preempt07:15:210
1803999207,3cyclictest12-21ksoftirqd/011:12:550
1803999203,11cyclictest0-21swapper/009:05:040
18039992014,4cyclictest0-21swapper/007:25:270
18039992014,4cyclictest0-21swapper/007:25:270
18039992014,3cyclictest14903-21tr07:40:190
18039992014,3cyclictest14903-21tr07:40:180
18039992013,3cyclictest12-21ksoftirqd/009:40:130
1803999201,13cyclictest24275-21diskmemload10:15:110
18039992010,7cyclictest1730-21load12:25:210
1803999200,14cyclictest5132-21ls07:30:180
1803999200,14cyclictest5132-21ls07:30:170
1806799198,6cyclictest0-21swapper/310:00:143
1806799193,11cyclictest0-21swapper/312:39:293
1806799193,11cyclictest0-21swapper/312:05:143
1806799192,10cyclictest0-21swapper/308:50:143
1806799192,10cyclictest0-21swapper/307:20:123
18067991912,3cyclictest0-21swapper/312:15:113
18067991911,4cyclictest0-21swapper/309:35:133
1806799191,12cyclictest0-21swapper/308:30:163
1806799191,12cyclictest0-21swapper/307:15:283
1806799191,11cyclictest0-21swapper/308:25:273
18067991910,6cyclictest0-21swapper/309:50:523
1804799195,8cyclictest0-21swapper/111:15:101
18047991913,4cyclictest18028-21cyclictest07:10:011
18047991912,5cyclictest0-21swapper/107:25:191
18047991912,5cyclictest0-21swapper/107:25:181
1804799191,14cyclictest23911-21memory11:25:231
1804799191,10cyclictest0-21swapper/109:58:471
18047991910,7cyclictest24194-21ps11:45:181
18047991910,7cyclictest24194-21ps11:45:171
1804799190,16cyclictest23139-21timerwakeupswit07:45:291
1803999192,8cyclictest131rcu_preempt11:35:100
1803999192,4cyclictest53646usb-storage07:35:040
1803999192,3cyclictest131rcu_preempt08:35:220
18039991913,4cyclictest0-21swapper/007:50:280
18039991912,4cyclictest0-21swapper/008:00:180
1803999190,13cyclictest12399-21cut08:45:150
1806799189,6cyclictest0-21swapper/312:01:383
1806799189,6cyclictest0-21swapper/311:35:513
1806799188,6cyclictest0-21swapper/309:17:093
1806799182,10cyclictest0-21swapper/311:05:133
1806799182,10cyclictest0-21swapper/309:05:043
1806799182,10cyclictest0-21swapper/308:45:123
1806799182,10cyclictest0-21swapper/308:35:143
1806799182,10cyclictest0-21swapper/307:50:123
18067991810,5cyclictest35966irq/58-eth011:50:133
18067991810,3cyclictest0-21swapper/309:23:113
1804799182,13cyclictest3460-21kthreadcore08:00:221
1804799182,12cyclictest13409-21perl09:15:151
1804799182,11cyclictest5949-21ssh11:35:131
1804799181,7cyclictest27602-21/usr/sbin/munin12:05:161
1804799181,15cyclictest4158-21/usr/sbin/munin09:10:261
1804799181,14cyclictest6552-21irqcore08:05:181
1804799181,14cyclictest6552-21irqcore08:05:171
1804799181,14cyclictest26454-21cut09:40:181
1804799181,14cyclictest15102-21irqrtprio07:40:191
1804799181,14cyclictest15102-21irqrtprio07:40:191
1804799181,13cyclictest6348-21ssh10:05:151
18047991810,5cyclictest20377-21latency_hist08:20:011
1804799180,15cyclictest26732-21/usr/sbin/munin09:00:151
1804799180,12cyclictest11098-21cut08:10:161
1803999182,11cyclictest0-21swapper/008:20:140
18039991812,3cyclictest12-21ksoftirqd/008:30:130
18039991811,5cyclictest32263-21sshd07:24:290
1806799179,6cyclictest0-21swapper/310:30:183
1806799179,6cyclictest0-21swapper/310:14:383
1806799178,6cyclictest0-21swapper/310:50:243
1806799172,9cyclictest0-21swapper/308:00:153
1806799172,9cyclictest0-21swapper/307:35:053
1806799172,10cyclictest0-21swapper/309:00:133
1806799171,9cyclictest0-21swapper/308:20:023
1806799171,10cyclictest0-21swapper/307:30:143
1806799171,10cyclictest0-21swapper/307:30:143
1804799171,12cyclictest21591-21apt-get08:55:011
1804799171,12cyclictest12025-21sed09:50:171
1803999173,11cyclictest1100-21idleruntime-cro08:00:010
1803999172,10cyclictest0-21swapper/008:25:130
18039991711,3cyclictest12-21ksoftirqd/007:10:210
1806799162,10cyclictest0-21swapper/308:20:143
1804799161,12cyclictest25638-21idleruntime08:25:161
1803999161,2cyclictest0-21swapper/008:55:040
1803999161,11cyclictest11421-21irqcore08:10:170
1806799152,7cyclictest0-21swapper/308:10:123
1806799151,11cyclictest0-21swapper/307:40:253
1806799151,11cyclictest0-21swapper/307:40:253
1803999157,5cyclictest18090-21cat07:10:010
1806799149,3cyclictest0-21swapper/307:45:473
1806799138,3cyclictest0-21swapper/308:05:063
1806799138,3cyclictest0-21swapper/308:05:063
1806799115,3cyclictest0-21swapper/307:25:123
1806799115,3cyclictest0-21swapper/307:25:123
1806799111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional