You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 08:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Jan 02, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28321996633,21cyclictest28366irq/58-eth022:30:153
28321996633,21cyclictest28366irq/58-eth022:30:153
2829899622,48cyclictest7463-21ssh22:00:131
2830999582,42cyclictest558-21ssh21:20:122
28321995628,18cyclictest28366irq/58-eth021:10:143
2828999502,8cyclictest131rcu_preempt23:00:120
2829899478,23cyclictest21-21ksoftirqd/120:15:141
2828999474,6cyclictest131rcu_preempt19:30:110
28321994524,11cyclictest28366irq/58-eth019:55:133
28289994529,11cyclictest0-21swapper/021:15:120
2829899423,27cyclictest9671-21ssh21:25:141
2829899423,27cyclictest9671-21ssh21:25:141
2829899421,33cyclictest14060-21ssh22:40:131
2828999423,28cyclictest26257-21ssh23:05:150
2832199413,22cyclictest28366irq/58-eth023:35:153
2830999411,32cyclictest7045-21needreboot00:05:252
28309994031,4cyclictest22346irq/40-dwc2_hso23:05:152
2830999401,29cyclictest8443-21ssh22:55:142
2830999392,29cyclictest9650-21ntp_192.168.11423:30:252
2830999391,28cyclictest4243-21ssh21:40:152
28289993921,8cyclictest131rcu_preempt19:10:120
2828999391,3cyclictest131rcu_preempt19:50:140
2832199382,24cyclictest28366irq/58-eth022:35:153
2832199371,22cyclictest28366irq/58-eth020:25:163
2829899373,22cyclictest14933-21ssh21:10:131
2828999372,3cyclictest131rcu_preempt20:00:130
2832199362,18cyclictest28366irq/58-eth021:25:133
2832199362,18cyclictest28366irq/58-eth021:25:133
2830999361,18cyclictest31672-21cut23:25:252
2828999363,25cyclictest14537-21diskmemload23:15:140
2828999362,24cyclictest5130-21ssh22:35:150
2832199353,15cyclictest28366irq/58-eth023:00:123
2828999351,28cyclictest26928-21ssh00:00:130
2832199342,19cyclictest28366irq/58-eth000:00:143
2830999343,25cyclictest0-21swapper/223:43:252
28309993422,9cyclictest23053-21ssh22:45:142
2832199335,7cyclictest28366irq/58-eth021:20:133
2832199332,16cyclictest28366irq/58-eth000:05:143
2830999331,27cyclictest0-21swapper/220:50:132
2828999334,18cyclictest12-21ksoftirqd/020:25:150
2828999332,24cyclictest6013-21ssh23:30:140
2830999320,26cyclictest13127-21ssh21:45:142
2828999321,28cyclictest9253-21rm23:50:120
2832199313,19cyclictest28366irq/58-eth022:00:143
2832199313,18cyclictest28366irq/58-eth000:35:143
2832199313,18cyclictest28366irq/58-eth000:35:143
2832199313,16cyclictest28366irq/58-eth020:40:133
2829899312,23cyclictest29581-21ssh23:25:161
28289993114,5cyclictest12-21ksoftirqd/022:25:140
2828999311,20cyclictest30957-21ssh21:55:130
2832199303,17cyclictest28366irq/58-eth022:50:143
28298993010,10cyclictest131rcu_preempt23:20:141
28289993023,3cyclictest12461-21ssh00:10:130
2832199291,9cyclictest0-21swapper/320:20:013
2830999296,14cyclictest27-21ksoftirqd/221:00:152
2830999292,21cyclictest0-21swapper/219:40:132
2830999292,20cyclictest0-21swapper/222:00:142
2830999292,20cyclictest0-21swapper/221:10:152
2830999292,17cyclictest27025-21ssh23:22:242
2828999292,21cyclictest10756-21ssh23:14:250
2828999291,23cyclictest4352-21ssh21:20:260
2830999283,8cyclictest0-21swapper/223:14:252
2830999282,19cyclictest0-21swapper/222:30:162
2830999282,19cyclictest0-21swapper/222:30:162
2830999281,20cyclictest22346irq/40-dwc2_hso00:15:122
2829899281,21cyclictest23892-21ssh23:40:131
2832199279,9cyclictest28366irq/58-eth023:40:133
2832199277,6cyclictest28366irq/58-eth021:15:133
2832199273,16cyclictest0-21swapper/323:25:243
2832199273,16cyclictest0-21swapper/322:57:443
2832199273,16cyclictest0-21swapper/322:42:423
2832199272,16cyclictest28366irq/58-eth023:15:133
2832199271,12cyclictest0-21swapper/322:20:173
2830999272,17cyclictest0-21swapper/219:25:122
2830999272,10cyclictest0-21swapper/219:20:132
28298992713,10cyclictest30351-21df00:20:161
2828999276,18cyclictest14537-21diskmemload22:05:160
2828999272,20cyclictest3452-21ssh00:05:140
2828999272,19cyclictest21671-21sh23:37:480
2832199263,15cyclictest0-21swapper/323:33:253
2832199262,11cyclictest28366irq/58-eth020:20:123
2830999262,9cyclictest6201-21ssh22:52:442
2830999260,20cyclictest9665-21ssh21:25:142
2830999260,20cyclictest9665-21ssh21:25:142
28298992616,7cyclictest4700-21cron22:35:011
28298992616,7cyclictest4700-21cron22:35:011
2828999262,18cyclictest31033-21ssh23:43:250
2832199253,15cyclictest28366irq/58-eth020:30:163
2832199253,15cyclictest0-21swapper/322:45:253
2830999258,15cyclictest22651-21cron22:45:012
2830999252,20cyclictest0-21swapper/219:15:132
2830999252,18cyclictest18326-21ssh23:17:462
2830999252,17cyclictest22346irq/40-dwc2_hso21:55:142
2830999252,14cyclictest22346irq/40-dwc2_hso19:35:142
2830999251,16cyclictest18969-21ls21:30:162
2829899258,11cyclictest28278-21cyclictest19:25:121
2829899252,8cyclictest131rcu_preempt00:15:121
28298992516,5cyclictest0-21swapper/123:10:261
2829899251,18cyclictest32014-21ssh22:50:151
2828999253,17cyclictest18690-21ssh21:30:130
2828999253,14cyclictest151rcuc/020:40:120
2828999252,17cyclictest7600-21ssh22:00:160
2832199243,7cyclictest0-21swapper/300:15:283
2830999245,10cyclictest14537-21diskmemload00:00:222
2830999243,12cyclictest13640-21cron22:40:012
2830999242,7cyclictest30236-21ssh00:20:142
2830999242,15cyclictest0-21swapper/223:00:042
2830999242,15cyclictest0-21swapper/200:10:042
2829899246,3cyclictest21-21ksoftirqd/100:10:131
2829899242,17cyclictest0-21swapper/122:45:151
2829899241,20cyclictest27681-21ssh21:35:161
2828999242,17cyclictest14537-21diskmemload00:29:250
28289992412,8cyclictest18702-21ntpq21:10:290
2832199233,13cyclictest0-21swapper/321:50:143
2830999232,6cyclictest0-21swapper/223:58:242
28309992312,9cyclictest18211-21ssh22:23:432
2830999231,18cyclictest22346irq/40-dwc2_hso22:10:122
2829899232,16cyclictest576-21sh21:20:131
28298992313,5cyclictest18719-21processes23:35:281
2829899231,19cyclictest6692-21ssh00:25:141
2828999236,12cyclictest12-21ksoftirqd/022:42:440
2828999232,15cyclictest31657-21ssh23:25:250
2828999232,12cyclictest28598-21ssh22:30:150
2828999232,12cyclictest28598-21ssh22:30:150
28289992312,8cyclictest5592-21ntpq22:15:270
28289992312,5cyclictest12-21ksoftirqd/020:45:130
2828999231,19cyclictest5704-21find19:20:110
28289992311,3cyclictest14537-21diskmemload00:20:180
2832199225,12cyclictest28366irq/58-eth000:20:153
2830999229,6cyclictest5173-21cat23:45:332
2830999228,11cyclictest19607-21cat23:35:322
2830999224,13cyclictest22346irq/40-dwc2_hso19:50:142
2830999222,15cyclictest0-21swapper/219:45:152
2830999221,7cyclictest8267-21tr00:25:212
28309992211,9cyclictest7974-21ssh22:17:022
28309992211,8cyclictest24246-21/usr/sbin/munin00:35:332
28309992211,8cyclictest24246-21/usr/sbin/munin00:35:332
2830999221,14cyclictest0-21swapper/220:30:162
28298992215,4cyclictest0-21swapper/123:00:121
28298992215,4cyclictest0-21swapper/121:40:171
2829899220,17cyclictest27241-21latency_hist21:35:011
2828999229,10cyclictest17110-21gpgv23:00:010
2828999225,4cyclictest131rcu_preempt22:45:260
2828999221,18cyclictest31533-21ls20:20:160
2830999217,7cyclictest23115-21df_inode20:45:152
2830999216,12cyclictest13853-21grep20:35:222
2830999211,14cyclictest22346irq/40-dwc2_hso20:10:132
2830999211,10cyclictest3848-21cut20:25:182
28309992110,7cyclictest24725-21kthreadcore21:15:202
2829899212,16cyclictest322-21ssh23:45:151
2829899211,9cyclictest0-21swapper/123:17:541
28298992111,7cyclictest28652-21cat21:15:301
2829899211,16cyclictest24278-21ssh00:35:141
2829899211,16cyclictest24278-21ssh00:35:141
2829899211,10cyclictest4218-21kthreadcore00:05:211
2828999213,15cyclictest13827-21grep21:25:310
2828999213,15cyclictest13827-21grep21:25:310
2828999211,9cyclictest31987-21apt-get22:50:150
28289992113,5cyclictest24386-21df00:35:160
28289992113,5cyclictest24386-21df00:35:160
28289992112,5cyclictest23874-21latency_hist00:35:020
2828999211,15cyclictest20590-21apt-get23:20:130
2832199208,9cyclictest0-21swapper/300:34:233
2832199203,6cyclictest0-21swapper/319:50:183
2830999209,4cyclictest0-21swapper/222:28:422
2830999208,10cyclictest16472-21cut00:30:212
2830999206,12cyclictest9837-21irqrtprio23:50:182
2830999205,12cyclictest4008-21ssh21:40:022
2830999201,7cyclictest0-21swapper/220:15:132
28309992012,5cyclictest0-21swapper/222:05:152
2830999201,10cyclictest2779-21grep20:20:292
2829899207,10cyclictest17364-21kthreadcore00:30:221
2829899205,4cyclictest12398-21grep20:00:181
2829899203,2cyclictest131rcu_preempt22:35:321
2829899202,2cyclictest131rcu_preempt22:20:121
28298992012,6cyclictest21339-21cron20:10:011
28298992012,6cyclictest11642-21cron20:00:011
2829899201,11cyclictest16748-21ntp_192.168.11421:45:251
2829899200,16cyclictest29999-21grep19:45:181
28289992013,5cyclictest0-21swapper/019:15:220
28289992011,7cyclictest32408-21cron20:55:010
2828999201,16cyclictest16653-21ls21:45:250
2828999200,17cyclictest21559-21latency23:55:260
2832199199,6cyclictest28366irq/58-eth023:10:143
2832199193,11cyclictest0-21swapper/323:59:253
2832199193,11cyclictest0-21swapper/323:20:263
2832199193,11cyclictest0-21swapper/300:27:533
2832199192,11cyclictest0-21swapper/320:55:053
2832199191,13cyclictest0-21swapper/320:05:293
2832199191,13cyclictest0-21swapper/319:15:223
2832199191,12cyclictest0-21swapper/320:10:283
2832199191,11cyclictest0-21swapper/319:25:153
28321991910,6cyclictest0-21swapper/322:05:493
2830999193,13cyclictest0-21swapper/221:05:122
2830999192,10cyclictest7740-21kthreadcore19:55:202
2830999191,8cyclictest377-21fschecks_count20:55:172
2830999191,14cyclictest16936-21grep20:05:172
2830999191,14cyclictest12399-21sed20:00:182
2830999191,14cyclictest0-21swapper/219:30:132
2830999191,10cyclictest30152-21kthreadcore19:10:212
2830999191,10cyclictest24237-21kthreadcore21:50:222
2830999191,10cyclictest18563-21irqcore20:40:192
2829899197,5cyclictest23528-21/usr/sbin/munin20:45:191
2829899194,5cyclictest2538-21sed19:50:191
2829899192,4cyclictest131rcu_preempt21:50:151
2829899192,2cyclictest131rcu_preempt22:10:121
2829899192,14cyclictest19710-21ssh22:25:151
2829899191,9cyclictest32090-21switchtime19:10:291
2829899191,2cyclictest0-21swapper/123:30:141
28298991911,6cyclictest0-21swapper/121:00:211
2829899191,15cyclictest9252-21sh23:50:121
2829899191,15cyclictest16450-21sh22:05:161
2829899191,14cyclictest26906-21apt-get00:00:141
2829899190,6cyclictest53146usb-storage20:20:041
2828999198,3cyclictest131rcu_preempt22:10:020
2828999193,11cyclictest22062-21ssh21:50:140
2828999192,2cyclictest131rcu_preempt20:15:260
28289991913,4cyclictest0-21swapper/023:45:150
2828999191,15cyclictest20165-21find19:35:130
2832199189,6cyclictest28366irq/58-eth022:25:143
2832199189,6cyclictest0-21swapper/321:40:513
2832199188,7cyclictest0-21swapper/323:50:303
2832199188,6cyclictest28366irq/58-eth021:35:153
2832199187,8cyclictest0-21swapper/300:12:043
2832199182,10cyclictest0-21swapper/323:45:153
2832199182,10cyclictest0-21swapper/320:50:123
28321991810,6cyclictest0-21swapper/323:06:443
28321991810,6cyclictest0-21swapper/321:55:593
2829899189,6cyclictest4527-21cut22:15:231
2829899183,12cyclictest9308-21mailstats19:20:271
2829899183,10cyclictest28278-21cyclictest19:35:151
2829899181,9cyclictest1215-21sed19:15:171
2829899181,9cyclictest0-21swapper/121:57:271
2829899181,14cyclictest26644-21ls00:00:021
28298991811,4cyclictest26438-21fschecks_time23:05:181
2829899181,14cyclictest10845-21kthreadcore20:30:241
2829899181,12cyclictest18825-21ntpq19:30:251
2828999182,12cyclictest14436-21ntp_states22:20:260
2828999182,12cyclictest12-21ksoftirqd/020:55:140
2828999181,3cyclictest131rcu_preempt21:05:320
28289991811,5cyclictest0-21swapper/019:55:170
2828999181,14cyclictest8508-21find20:30:150
28289991811,4cyclictest0-21swapper/019:45:180
2828999181,13cyclictest28452-21tr21:35:220
2828999180,13cyclictest16763-21cpuspeed20:05:140
2832199179,6cyclictest0-21swapper/322:11:203
2832199179,6cyclictest0-21swapper/321:31:243
2832199172,10cyclictest0-21swapper/320:45:143
2832199172,10cyclictest0-21swapper/319:45:153
2832199172,10cyclictest0-21swapper/319:40:133
2832199172,10cyclictest0-21swapper/319:35:153
28321991711,4cyclictest0-21swapper/321:45:323
2832199171,10cyclictest0-21swapper/321:00:193
2832199171,10cyclictest0-21swapper/320:00:233
28321991710,4cyclictest0-21swapper/322:15:233
2830999171,13cyclictest28313-21latency_hist19:10:012
2829899172,13cyclictest5166-21kthreadcore20:25:221
2829899171,8cyclictest10854-21kthreadcore21:05:201
2829899171,13cyclictest8529-21df22:55:151
2829899171,13cyclictest28139-21kthreadcore19:40:221
28298991710,5cyclictest0-21swapper/120:55:251
2828999178,6cyclictest25046-21ntpq20:10:260
2828999172,11cyclictest27809-21kthreadcore19:40:200
2828999172,10cyclictest0-21swapper/020:35:170
28289991711,4cyclictest23339-21kthreadcore00:15:200
28289991710,5cyclictest4802-21irqrtprio21:40:190
2829899161,12cyclictest31146-21memory20:50:221
2829899161,12cyclictest18164-21apt-get20:40:131
2829899161,12cyclictest13558-21tune2fs20:35:201
2829899160,12cyclictest21966-21sed20:10:181
2828999162,11cyclictest10872-21irqcore19:25:170
2828999161,12cyclictest5453-21grep21:00:200
2832199149,3cyclictest0-21swapper/319:24:113
2832199141,7cyclictest0-21swapper/319:35:013
2832199141,10cyclictest0-21swapper/321:05:323
2832199141,10cyclictest0-21swapper/319:15:023
2829899144,8cyclictest28278-21cyclictest19:10:011
2828999142,9cyclictest28316-21sort19:10:010
2832199111,7cyclictest0-21swapper/320:35:173
283219991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional