You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 14:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu Sep 18, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1927999602,35cyclictest0-21swapper/219:25:102
1927199571,46cyclictest0-21swapper/100:25:131
1927199471,27cyclictest29182-21ssh22:07:091
1927999465,25cyclictest17255-21ssh22:40:142
1927199461,41cyclictest12924-21apt-get23:55:131
1927999453,25cyclictest31697-21ssh22:10:142
1927199441,39cyclictest1618-21apt-get22:50:131
1927199441,38cyclictest1499-21apt-get21:30:151
1927999435,17cyclictest27-21ksoftirqd/221:10:102
1927999432,22cyclictest27-21ksoftirqd/219:15:162
1927199430,39cyclictest6294-21ssh21:55:091
1927199421,36cyclictest31405-21apt-get20:30:131
1926199422,4cyclictest0-21swapper/023:00:150
1929199413,22cyclictest25366irq/58-eth022:55:163
1927999412,25cyclictest27-21ksoftirqd/200:15:172
1926199412,3cyclictest4782-21apt-get23:50:140
1926199411,13cyclictest6290-21apt-get21:55:100
19291994022,12cyclictest25366irq/58-eth000:10:153
1927199401,35cyclictest3745-21apt-get20:35:141
1927199401,34cyclictest0-21swapper/122:20:161
1927199401,34cyclictest0-21swapper/122:20:151
1926199403,33cyclictest12929-21sshd23:55:150
19279993915,20cyclictest27-21ksoftirqd/200:25:122
1927199391,34cyclictest0-21swapper/123:15:131
1927199391,34cyclictest0-21swapper/120:40:121
1927199391,32cyclictest369-21apt-get21:10:111
1927199391,31cyclictest26877-21ssh23:05:151
1927199390,34cyclictest4791-21ssh23:50:141
1929199386,20cyclictest25366irq/58-eth022:50:133
19291993826,9cyclictest25366irq/58-eth023:30:113
1927199383,27cyclictest11995-21apt-get23:35:161
1927199380,31cyclictest19364-21apt-get23:20:151
1926199383,30cyclictest20866-21ssh00:00:100
1926199383,23cyclictest28277-21ssh00:05:170
1926199381,5cyclictest18133-21apt-get20:50:160
1929199372,23cyclictest25366irq/58-eth021:35:173
19291993714,20cyclictest0-21swapper/323:40:163
1927199371,31cyclictest0-21swapper/120:25:131
1927199370,33cyclictest0-21swapper/122:00:131
1926199372,2cyclictest0-21swapper/022:50:130
1927999362,13cyclictest27-21ksoftirqd/219:35:132
1927199361,32cyclictest17246-21apt-get22:40:151
1927199361,32cyclictest0-21swapper/119:50:161
1927199361,31cyclictest28775-21apt-get23:45:141
1927199361,27cyclictest0-21swapper/120:50:151
1927199361,26cyclictest3609-21ssh23:30:111
1927199360,31cyclictest7732-21apt-get22:15:141
1926199361,8cyclictest18760-21cut00:20:220
1929199352,21cyclictest25366irq/58-eth022:10:143
1927199351,29cyclictest0-21swapper/120:15:141
1927199351,27cyclictest0-21swapper/100:30:141
1927199351,19cyclictest13811-21ssh21:15:501
1926199354,5cyclictest15994-21sed23:35:300
1926199351,8cyclictest26650-21ssh23:24:010
1926199351,6cyclictest20437-21ssh23:40:140
1926199351,5cyclictest29682-21kthreadcore23:05:230
1926199351,3cyclictest0-21swapper/021:30:150
1926199351,10cyclictest1117-21memory19:55:220
1929199343,21cyclictest25366irq/58-eth023:15:133
1927199349,19cyclictest21-21ksoftirqd/121:25:141
1927199342,28cyclictest18427-21apt-get23:00:151
1927199341,28cyclictest0-21swapper/121:45:111
1927199341,28cyclictest0-21swapper/121:45:101
1927199340,30cyclictest31685-21apt-get22:10:141
1926199341,8cyclictest25812-21tune2fs21:25:170
1926199341,6cyclictest25764-21kthreadcore19:15:220
1926199341,5cyclictest31688-21latency_hist00:30:010
1926199341,4cyclictest3583-21apt-get23:30:110
1926199340,6cyclictest3006-21ssh23:48:060
1926199340,6cyclictest20809-21/usr/sbin/munin21:20:260
1927199333,25cyclictest388-21ssh22:30:141
1927199331,29cyclictest0-21swapper/119:15:151
1927199331,28cyclictest1376-21apt-get19:25:101
1927199331,27cyclictest0-21swapper/119:10:121
1927199331,25cyclictest0-21swapper/120:55:121
1927199330,29cyclictest20550-21apt-get19:45:141
1927199330,29cyclictest17554-21ssh00:20:151
1927199330,28cyclictest0-21swapper/122:25:111
1927199330,20cyclictest15418-21ssh22:38:101
1927199330,20cyclictest15418-21ssh22:38:091
1926199332,9cyclictest21754-21ssh22:40:310
1926199332,3cyclictest25612-21apt-get22:45:130
1926199332,28cyclictest24776-21usage_EthcTask19:45:310
1926199331,9cyclictest7325-21cut00:35:170
1926199331,9cyclictest18711-21grep21:40:200
1926199331,9cyclictest17670-21ssh23:18:130
1926199331,8cyclictest6617-21tr20:35:220
1926199331,8cyclictest18344-21tr22:20:250
1926199331,8cyclictest18344-21tr22:20:250
1926199331,6cyclictest30629-21grep21:45:280
1926199331,6cyclictest30629-21grep21:45:270
1926199331,5cyclictest18730-21ssh22:00:280
1926199331,10cyclictest26901-21ntp_states22:05:290
1926199330,8cyclictest24602-21cat22:25:140
1929199323,17cyclictest25366irq/58-eth021:00:123
1927999323,20cyclictest19252-21cyclictest21:45:122
1927999323,20cyclictest19252-21cyclictest21:45:112
1927999322,22cyclictest6314-21ssh21:55:112
1927199321,27cyclictest6999-21if_err_eth000:35:171
1927199321,23cyclictest1239-21ssh00:06:561
1926199323,22cyclictest412-21ssh22:30:160
1926199321,9cyclictest29349-21grep19:20:180
1926199321,8cyclictest9284-21irqcore21:15:200
1926199321,8cyclictest29238-21grep19:50:310
1926199321,8cyclictest11232-21kthreadcore00:15:230
1926199321,8cyclictest10282-21kthreadcore22:15:220
1926199321,7cyclictest27934-21grep23:25:170
1926199321,7cyclictest21278-21kthreadcore19:10:210
1926199321,6cyclictest1721-21cut22:10:230
1926199321,5cyclictest31912-21grep20:30:180
1926199321,3cyclictest0-21swapper/000:30:250
1926199321,10cyclictest6013-21ntp_192.168.11420:00:230
1929199316,11cyclictest25366irq/58-eth022:20:173
1929199316,11cyclictest25366irq/58-eth022:20:163
1927999312,17cyclictest21846irq/40-dwc2_hso20:10:132
1927999312,16cyclictest27-21ksoftirqd/223:00:162
1927199312,25cyclictest25611-21ssh22:45:121
1927199311,27cyclictest20449-21apt-get23:40:151
1927199311,27cyclictest0-21swapper/122:55:151
1927199311,26cyclictest0-21swapper/123:10:161
1927199311,26cyclictest0-21swapper/123:10:151
1926199317,21cyclictest15518-21sendmail19:40:010
1926199315,23cyclictest26934-21irqcore20:25:180
1926199312,8cyclictest6354-21tune2fs19:30:180
1926199312,7cyclictest13578-21grep20:45:160
1926199311,8cyclictest9443-21sed22:35:200
1926199311,8cyclictest9443-21sed22:35:190
1926199311,8cyclictest2736-21df_inode00:10:180
1926199311,8cyclictest19573-21grep19:40:290
1926199311,7cyclictest8795-21wc20:40:170
1926199311,7cyclictest24156-21cut20:20:220
1926199311,7cyclictest24156-21cut20:20:210
1926199311,6cyclictest7600-21grep20:05:170
1926199311,5cyclictest3743-21ssh21:51:580
1926199311,5cyclictest10411-21ps21:35:200
1926199311,5cyclictest10353-21/usr/sbin/munin22:55:190
1926199311,4cyclictest32231-21users21:00:300
1926199310,6cyclictest7433-21ssh23:10:340
1926199310,6cyclictest7433-21ssh23:10:330
1926199310,6cyclictest32752-21ssh21:10:030
1929199302,15cyclictest25366irq/58-eth023:50:143
1927999303,21cyclictest1524-21ssh21:30:172
1927199301,26cyclictest6190-21mailstats20:00:261
1927199301,25cyclictest0-21swapper/119:35:131
1927199300,26cyclictest27717-21apt-get21:00:131
1926199301,7cyclictest18000-21kthreadcore20:15:200
1926199301,6cyclictest5133-21wc19:25:250
1926199301,5cyclictest24787-21tr20:55:210
1926199301,3cyclictest12203-21apt-get20:10:150
19291992912,9cyclictest25466irq/59-eth020:25:143
1927999293,18cyclictest21846irq/40-dwc2_hso20:25:122
1927999293,16cyclictest21846irq/40-dwc2_hso19:50:152
1927999291,17cyclictest26828-21ssh22:05:262
1927199292,24cyclictest21204-21timerwakeupswit21:20:281
1927199291,25cyclictest2604-21/usr/sbin/munin00:10:161
1927199291,25cyclictest15664-21ntp_states20:10:271
1927199291,25cyclictest0-21swapper/123:25:151
1927199291,24cyclictest7398-21apt-get20:05:141
1927199291,24cyclictest6162-21/usr/sbin/munin19:30:171
1927199291,22cyclictest15763-21apt-get19:40:131
1927199290,27cyclictest10177-21ssh00:15:171
1927199290,26cyclictest13511-21ssh21:35:281
1926199291,6cyclictest19252-21cyclictest19:10:010
1929199285,16cyclictest25366irq/58-eth019:10:133
1929199281,14cyclictest25366irq/58-eth022:45:123
1927999282,20cyclictest0-21swapper/220:15:152
1927999282,17cyclictest21846irq/40-dwc2_hso19:10:142
1927199281,23cyclictest13351-21apt-get20:45:131
1927199281,17cyclictest1811766ssh21:40:121
1927999273,18cyclictest21846irq/40-dwc2_hso23:40:152
1927199271,23cyclictest21121-21fw_packets00:00:151
1927199270,24cyclictest22085-21cat20:20:181
1927199270,24cyclictest22085-21cat20:20:181
1927199270,22cyclictest29025-21apt-get19:20:121
1926199271,6cyclictest32528-21ssh21:10:020
1929199262,13cyclictest25366irq/58-eth023:35:133
19279992612,12cyclictest21966-21ssh22:22:012
19279992612,12cyclictest21966-21ssh22:22:002
19279992612,12cyclictest19777-21ssh23:58:582
1927999261,15cyclictest12661-21ssh23:35:202
1927199261,23cyclictest0-21swapper/119:55:121
1927199261,22cyclictest32520-21ssh21:10:011
1927199261,22cyclictest19287-21latency_hist19:10:021
1929199252,16cyclictest0-21swapper/323:55:153
1927999253,17cyclictest21846irq/40-dwc2_hso00:35:112
1927999253,16cyclictest27-21ksoftirqd/222:15:122
1927999252,20cyclictest0-21swapper/220:40:142
1927999252,15cyclictest17782-21hwlatdetect00:20:192
1927999252,11cyclictest11019-21ssh23:15:132
1927999251,19cyclictest8780-21sh22:35:162
1927999251,19cyclictest8780-21sh22:35:152
19271992510,12cyclictest2696-21sh21:50:301
1927999242,3cyclictest131rcu_preempt22:25:142
19279992415,6cyclictest30373-21sshd19:55:162
1927999241,13cyclictest14879-21kthreadcore20:45:192
1929199233,11cyclictest25366irq/58-eth021:55:113
1927999238,5cyclictest18164-21kthreadcore22:00:232
1927999236,4cyclictest131rcu_preempt23:25:142
1927999233,16cyclictest21846irq/40-dwc2_hso22:45:122
1927999232,16cyclictest21846irq/40-dwc2_hso20:35:132
19279992313,8cyclictest0-21swapper/221:20:242
19279992313,8cyclictest0-21swapper/221:00:172
1927999231,18cyclictest26553-21latency_hist20:25:022
1927999231,18cyclictest26553-21latency_hist20:25:012
1927999230,13cyclictest32706-21timerandwakeup23:45:312
19291992211,8cyclictest25366irq/58-eth000:05:173
1929199221,13cyclictest0-21swapper/321:30:163
1927999224,15cyclictest21846irq/40-dwc2_hso22:50:132
1927999223,16cyclictest21846irq/40-dwc2_hso20:50:182
1927999222,6cyclictest0-21swapper/222:55:162
1927999222,16cyclictest32506-21diskmemload23:20:162
1927999222,14cyclictest0-21swapper/200:10:162
1927999221,8cyclictest21018-21ssh00:00:142
1927999221,5cyclictest3657-21ls20:35:022
1927999221,5cyclictest0-21swapper/219:20:132
1927999221,18cyclictest9900-21kthreadcore20:05:232
1927999221,18cyclictest2742-21grep20:00:142
1927999221,17cyclictest4768-21usage_Anybus22:30:312
1927999221,17cyclictest26837-21/usr/sbin/munin23:05:282
1927999221,11cyclictest17418-21kthreadcore19:40:192
1927999221,10cyclictest32623-21latency_hist21:10:012
1927999221,10cyclictest27013-21cut21:25:212
1927999221,10cyclictest10357-21irqcore21:35:202
19279992210,10cyclictest9189-21ssh23:31:352
1929199218,8cyclictest0-21swapper/300:15:283
19291992110,5cyclictest0-21swapper/322:30:213
1927999216,12cyclictest23470-21irqrtprio20:55:182
1927999215,12cyclictest21846irq/40-dwc2_hso19:45:152
1927999213,15cyclictest3781-21kthreadcore23:10:222
1927999213,15cyclictest3781-21kthreadcore23:10:212
1927999211,9cyclictest18354-21fschecks_time21:40:182
1927999211,17cyclictest9001-21fschecks_time21:15:172
1927999211,15cyclictest19353-21latency_hist19:10:022
1927999211,11cyclictest30210-21kthreadcore00:05:252
1927999211,11cyclictest2046-21kthreadcore00:30:242
1927999211,10cyclictest5037-21fschecks_time23:50:172
1929199202,13cyclictest0-21swapper/323:00:163
19291992010,8cyclictest0-21swapper/321:29:593
1927999202,15cyclictest7780-21kthreadcore19:30:212
1929199192,11cyclictest0-21swapper/319:35:133
1929199192,11cyclictest0-21swapper/319:25:113
19291991914,3cyclictest33-21ksoftirqd/320:35:293
19291991911,5cyclictest25366irq/58-eth022:00:133
1929199191,12cyclictest0-21swapper/320:15:283
19291991910,4cyclictest0-21swapper/322:25:233
1929199189,6cyclictest0-21swapper/321:22:263
1929199189,6cyclictest0-21swapper/300:39:053
1929199188,8cyclictest0-21swapper/323:26:093
1929199188,8cyclictest0-21swapper/323:21:533
1929199187,9cyclictest0-21swapper/322:15:453
1929199182,5cyclictest0-21swapper/321:15:163
1929199182,10cyclictest0-21swapper/322:40:053
1929199182,10cyclictest0-21swapper/320:45:143
1929199182,10cyclictest0-21swapper/319:50:163
1929199182,10cyclictest0-21swapper/319:15:163
1929199181,8cyclictest25366irq/58-eth000:00:113
1929199181,13cyclictest0-21swapper/320:35:013
19291991810,6cyclictest0-21swapper/323:45:213
19291991810,6cyclictest0-21swapper/321:55:013
19291991810,6cyclictest0-21swapper/300:30:513
1927999181,14cyclictest5979-21latency_hist21:55:002
1929199179,6cyclictest0-21swapper/323:13:353
1929199179,6cyclictest0-21swapper/323:13:343
1929199179,6cyclictest0-21swapper/322:36:553
1929199179,6cyclictest0-21swapper/322:36:543
1929199179,6cyclictest0-21swapper/300:21:103
1929199179,5cyclictest25366irq/58-eth021:40:153
1929199178,6cyclictest25366irq/58-eth021:45:233
1929199178,6cyclictest25366irq/58-eth021:45:233
1929199178,6cyclictest0-21swapper/323:05:193
1929199172,10cyclictest0-21swapper/320:10:153
1929199172,10cyclictest0-21swapper/319:45:143
1929199172,10cyclictest0-21swapper/319:40:143
1929199172,10cyclictest0-21swapper/300:25:143
1929199171,10cyclictest0-21swapper/322:05:153
1929199171,10cyclictest0-21swapper/320:40:143
19291991710,4cyclictest0-21swapper/321:12:113
1929199162,10cyclictest0-21swapper/320:50:173
1929199159,4cyclictest0-21swapper/320:00:283
1929199153,10cyclictest0-21swapper/319:20:153
1929199149,3cyclictest0-21swapper/320:05:253
1929199149,3cyclictest0-21swapper/319:34:283
1929199139,3cyclictest33-21ksoftirqd/319:55:113
1929199131,6cyclictest0-21swapper/320:20:213
1929199131,6cyclictest0-21swapper/320:20:203
1929199127,3cyclictest0-21swapper/321:07:183
1929199121,9cyclictest0-21swapper/320:55:213
1929199101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional