You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-17 - 10:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Jun 17, 2026 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1262899593,45cyclictest32764-21ssh23:15:132
1261299573,29cyclictest7121-21ssh21:50:130
1261299531,42cyclictest3374-21ssh00:10:130
1262899521,30cyclictest7860-21ssh21:15:132
1262899511,28cyclictest31131-21ssh21:10:142
1261299492,16cyclictest21213-21ssh21:40:110
1263999472,28cyclictest35966irq/58-eth021:50:133
1261299476,25cyclictest17630-21ssh00:00:120
1261299473,33cyclictest16530-21ssh00:35:120
12628994625,15cyclictest3355-21ssh00:10:122
1262899461,29cyclictest6515-21ssh22:25:152
1261999464,38cyclictest5546-21ssh23:00:161
1262899452,23cyclictest28825-21diskmemload22:55:152
12619994532,9cyclictest24759-21apt-get22:35:151
12619994532,9cyclictest24759-21apt-get22:35:141
12612994539,3cyclictest30734-21diskmemload23:40:120
1261999441,31cyclictest27737-21ssh23:30:151
1263999424,23cyclictest35966irq/58-eth019:50:113
1262899421,30cyclictest24762-21ssh22:35:132
1262899421,30cyclictest24762-21ssh22:35:132
12619994235,4cyclictest8532-21sh23:55:141
1261299422,32cyclictest11453-21sh22:10:110
1261299413,12cyclictest26834-21rm00:05:170
12619994025,10cyclictest17536-21apt-get19:15:131
1261299401,14cyclictest23708-21ssh23:10:140
1262899394,8cyclictest131rcu_preempt21:05:132
1261999383,29cyclictest14619-21ssh23:05:151
12619993830,4cyclictest15360-21apt-get20:20:141
12619993829,4cyclictest11749-21apt-get20:50:131
1261999382,21cyclictest4358-21ssh23:35:111
12619993728,4cyclictest18969-21apt-get19:50:121
12619993726,5cyclictest29786-21apt-get20:35:121
12619993724,9cyclictest32752-21apt-get23:15:121
1261299373,12cyclictest21691-21ssh00:20:150
1263999362,19cyclictest35966irq/58-eth020:20:123
1262899367,22cyclictest261rcuc/222:05:142
12619993628,4cyclictest12727-21apt-get19:10:151
1263999352,20cyclictest35966irq/58-eth023:25:133
12619993425,4cyclictest9258-21apt-get19:40:131
12619993425,3cyclictest31119-21apt-get21:10:111
1261999341,25cyclictest15667-21ssh22:30:141
1261299342,14cyclictest30734-21diskmemload21:35:150
1261299341,30cyclictest18628-21rm23:25:140
1263999334,18cyclictest35966irq/58-eth023:40:133
1263999333,15cyclictest35966irq/58-eth020:05:143
1263999333,14cyclictest35966irq/58-eth023:30:143
12639993323,7cyclictest35966irq/58-eth021:55:143
12619993327,3cyclictest9523-21apt-get23:20:141
12619993327,3cyclictest7103-21apt-get21:50:141
1261299331,14cyclictest30734-21diskmemload23:00:140
1263999322,18cyclictest35966irq/58-eth020:55:143
1263999322,18cyclictest35966irq/58-eth020:55:133
1262899322,20cyclictest18611-21ssh23:25:142
1261999324,11cyclictest8979-21sh22:07:341
1261999322,24cyclictest11454-21rm22:10:101
12619993221,7cyclictest20167-21apt-get20:25:121
1261299323,23cyclictest17065-21ssh21:20:120
1261299321,22cyclictest31829-21sh23:50:130
12628993124,3cyclictest9531-21ssh23:20:122
1262899309,15cyclictest27-21ksoftirqd/220:35:132
1262899302,21cyclictest0-21swapper/222:45:152
12628993016,9cyclictest27-21ksoftirqd/223:35:132
1262899301,17cyclictest20679-21ssh22:15:162
1261999309,6cyclictest2410-21ssh21:29:501
1261999307,10cyclictest20684-21usage_Anybus21:55:301
12619993020,8cyclictest0-21swapper/100:15:131
1261299303,23cyclictest312-21ssh23:15:140
1261299302,20cyclictest9522-21ssh23:20:110
1261299301,22cyclictest1512-21ssh22:40:150
1263999294,17cyclictest35966irq/58-eth021:45:123
12639992911,7cyclictest35966irq/58-eth000:25:143
1263999291,15cyclictest35966irq/58-eth022:15:143
1262899293,9cyclictest30386-21ssh21:45:122
1262899293,18cyclictest27346irq/40-dwc2_hso19:40:122
1262899292,20cyclictest0-21swapper/220:10:142
1262899292,16cyclictest26279-21ssh21:25:152
12628992912,14cyclictest19060-21ssh21:55:232
1261999298,8cyclictest28832-21ssh21:43:401
1261999298,8cyclictest25604-21sh23:27:581
1261999297,8cyclictest20678-21ssh23:43:021
1261999296,6cyclictest0-21swapper/121:37:461
1261999295,9cyclictest0-21swapper/121:45:221
1261999295,8cyclictest0-21swapper/100:39:011
1261999294,8cyclictest0-21swapper/123:13:391
1261999294,8cyclictest0-21swapper/122:59:241
12619992912,13cyclictest0-21swapper/100:00:121
1261299292,21cyclictest10254-21ssh22:25:280
1261299292,19cyclictest7876-21ssh21:15:140
1261299292,17cyclictest25531-21ssh22:00:130
12639992814,6cyclictest35966irq/58-eth023:20:133
1262899282,20cyclictest0-21swapper/200:20:152
1261999286,8cyclictest30734-21diskmemload22:17:381
1261999285,6cyclictest0-21swapper/100:06:031
12619992823,3cyclictest17067-21apt-get21:20:141
12619992819,5cyclictest7431-21apt-get00:30:141
12619992818,8cyclictest29774-21apt-get22:20:121
12619992815,9cyclictest0-21swapper/123:45:121
12619992812,13cyclictest2955-21kthreadcore20:40:201
1261299289,12cyclictest630-21Async09:36:310
1261299284,20cyclictest15657-21ssh22:30:150
1261299281,19cyclictest16350-21ssh21:55:120
1263999273,15cyclictest35966irq/58-eth022:25:143
1262899272,14cyclictest27346irq/40-dwc2_hso20:25:132
1262899272,14cyclictest27346irq/40-dwc2_hso00:00:122
1262899271,19cyclictest0-21swapper/219:15:142
1261999278,11cyclictest304431latency19:25:231
1261999275,9cyclictest14406-21sh21:17:251
12619992714,9cyclictest22488-21diskstats19:20:131
12619992714,8cyclictest4150-21apt-get19:35:041
1261299272,20cyclictest24784-21ssh22:35:160
1261299272,20cyclictest24784-21ssh22:35:150
1262899266,10cyclictest0-21swapper/223:05:142
1262899263,6cyclictest7449-21ssh00:30:132
1262899263,14cyclictest27346irq/40-dwc2_hso19:30:122
12628992618,5cyclictest21441-21ssh00:20:022
12628992611,7cyclictest27-21ksoftirqd/223:10:132
1262899261,12cyclictest31841-21ssh23:50:132
1262899260,18cyclictest21595-21ls21:00:152
1261999267,7cyclictest11480-21ssh00:14:221
1261999265,9cyclictest6671-21rm00:29:391
12619992615,9cyclictest2822-21cut22:40:221
12619992614,8cyclictest0-21swapper/119:30:121
1261299261,19cyclictest2985-21ssh21:30:140
1261299261,18cyclictest19685-21ssh22:50:150
1261299261,17cyclictest22797-21sh23:45:140
1262899252,15cyclictest930-21snmpd19:25:132
1261999259,13cyclictest22232-21tr22:50:241
1261999256,7cyclictest27476-21ssh00:21:361
1261999255,8cyclictest30734-21diskmemload22:01:411
12619992515,7cyclictest7065-21diskstats20:45:151
12619992514,8cyclictest23873-21/usr/sbin/munin19:55:151
12619992514,8cyclictest16538-21apt-get20:55:141
12619992514,8cyclictest16538-21apt-get20:55:141
12619992512,9cyclictest11160-21tr22:45:191
12619992512,9cyclictest10650-21fschecks_count20:15:151
12619992511,9cyclictest0-21swapper/120:10:141
12619992511,10cyclictest18869-21date19:50:011
1263999242,17cyclictest35966irq/58-eth022:00:153
12639992417,4cyclictest35966irq/58-eth022:50:133
1262899243,17cyclictest27346irq/40-dwc2_hso21:40:122
1262899243,15cyclictest27346irq/40-dwc2_hso20:50:122
1262899242,18cyclictest27346irq/40-dwc2_hso22:40:152
1262899241,6cyclictest16205-21ssh22:30:192
1262899241,14cyclictest12090-21sh21:35:142
1261999246,6cyclictest9647-21ssh21:32:391
12619992415,7cyclictest6491-21ssh22:25:131
12619992413,8cyclictest29539-21sshd20:31:321
12619992412,8cyclictest31813-21apt-get23:50:111
12619992411,9cyclictest878-21latency_hist20:05:011
12619992410,11cyclictest4308-21/usr/sbin/munin20:05:241
1261299245,10cyclictest151rcuc/021:10:110
1263999239,8cyclictest35966irq/58-eth021:30:133
1263999234,10cyclictest35966irq/58-eth023:05:143
1263999232,11cyclictest35966irq/58-eth021:20:123
12639992317,3cyclictest0-21swapper/323:10:133
12639992315,5cyclictest0-21swapper/323:55:123
12639992314,7cyclictest0-21swapper/321:18:063
12639992314,6cyclictest0-21swapper/300:30:373
12639992313,8cyclictest0-21swapper/323:35:193
1263999231,14cyclictest35966irq/58-eth000:00:133
1262899232,15cyclictest0-21swapper/223:40:042
1262899231,7cyclictest30749-21ssh00:25:132
12628992313,8cyclictest0-21swapper/222:00:152
12628992312,9cyclictest755-21ssh23:31:142
12628992311,4cyclictest17306-21kthreadcore00:35:182
12628992310,5cyclictest7307-21fschecks_count21:50:162
12619992310,9cyclictest27102-21tr21:05:201
12619992310,9cyclictest25225-21usage_Anybus21:00:261
1261299231,18cyclictest9488-21ls19:40:160
12639992216,4cyclictest0-21swapper/322:06:153
1262899228,6cyclictest30158-21irqcore22:20:172
1262899228,4cyclictest27-21ksoftirqd/223:47:462
1262899227,5cyclictest17096-21rm21:20:152
1262899223,13cyclictest27346irq/40-dwc2_hso19:10:132
12628992213,3cyclictest0-21swapper/220:00:182
12628992212,8cyclictest0-21swapper/223:59:242
12628992210,9cyclictest17197-21ssh22:11:272
1261299226,9cyclictest12-21ksoftirqd/022:15:150
12612992213,4cyclictest0-21swapper/020:20:130
1261299221,17cyclictest32094-21cut19:30:150
1261299221,17cyclictest10572-21apt-get22:45:130
1263999218,8cyclictest35966irq/58-eth000:35:133
12639992115,4cyclictest0-21swapper/322:20:183
12639992115,4cyclictest0-21swapper/321:25:163
12639992114,5cyclictest0-21swapper/300:20:263
12639992113,5cyclictest0-21swapper/323:50:133
12639992113,5cyclictest0-21swapper/321:44:113
12639992112,7cyclictest0-21swapper/323:04:363
12639992112,6cyclictest0-21swapper/322:58:253
1263999211,13cyclictest35966irq/58-eth023:15:133
12628992112,7cyclictest0-21swapper/221:30:122
12628992111,5cyclictest0-21swapper/200:05:162
1261299213,11cyclictest0-21swapper/019:35:130
12612992113,5cyclictest0-21swapper/019:45:250
1261299211,15cyclictest17691-21cut19:15:170
1261299210,18cyclictest5787-21ls20:40:300
12639992014,4cyclictest0-21swapper/322:44:573
12639992014,4cyclictest0-21swapper/322:10:383
12639992014,4cyclictest0-21swapper/321:10:113
12639992014,4cyclictest0-21swapper/319:55:153
12639992014,4cyclictest0-21swapper/300:17:303
12639992014,4cyclictest0-21swapper/300:05:253
12639992013,5cyclictest0-21swapper/322:45:483
12639992013,5cyclictest0-21swapper/321:35:133
12639992013,5cyclictest0-21swapper/319:45:333
12639992013,4cyclictest0-21swapper/322:35:173
12639992013,4cyclictest0-21swapper/322:35:173
12639992013,4cyclictest0-21swapper/321:05:143
12639992012,6cyclictest0-21swapper/320:45:223
12639992011,6cyclictest0-21swapper/320:15:243
1262899208,5cyclictest17910-21grep19:45:292
1262899202,3cyclictest131rcu_preempt19:55:172
1262899201,9cyclictest22455-21tr22:50:232
1262899201,5cyclictest0-21swapper/220:30:122
12628992013,5cyclictest5370-21ssh23:00:042
12628992011,7cyclictest0-21swapper/220:55:202
12628992011,7cyclictest0-21swapper/220:55:192
1262899201,16cyclictest6900-21latency_hist20:45:022
1262899201,16cyclictest25945-21find19:20:242
1262899201,10cyclictest17045-21kthreadcore20:20:192
1261999207,10cyclictest12643-21sed19:10:011
1261299203,6cyclictest12-21ksoftirqd/000:15:310
12612992013,5cyclictest0-21swapper/019:25:200
1261299201,14cyclictest20306-21timerwakeupswit20:55:280
1261299201,14cyclictest20306-21timerwakeupswit20:55:280
1261299201,10cyclictest30320-21grep21:25:310
12639991914,3cyclictest0-21swapper/300:10:233
12639991913,4cyclictest0-21swapper/323:45:253
12639991912,5cyclictest0-21swapper/322:30:193
12639991912,5cyclictest0-21swapper/321:00:123
12639991911,5cyclictest0-21swapper/320:30:013
12639991911,5cyclictest0-21swapper/319:40:283
1263999191,13cyclictest0-21swapper/319:20:133
1262899192,7cyclictest8971-21kthreadcore20:45:212
1262899191,9cyclictest12648-21latency_hist19:10:022
1262899191,9cyclictest1147-21fschecks_time20:05:172
1262899191,9cyclictest10765-21grep20:15:162
1262899191,10cyclictest5676-21cut19:35:192
1262899190,10cyclictest19160-21fschecks_time19:50:152
1261299198,3cyclictest131rcu_preempt23:05:200
12612991913,3cyclictest0-21swapper/021:45:130
12612991912,5cyclictest28807-21apt-get22:55:150
1261299191,16cyclictest1911-21sshd20:39:330
1261299191,14cyclictest23983-21unixbench_multi20:25:280
1263999186,8cyclictest321rcuc/320:35:133
12639991812,4cyclictest0-21swapper/320:00:233
12639991812,4cyclictest0-21swapper/319:40:023
12639991812,4cyclictest0-21swapper/319:30:123
12639991812,4cyclictest0-21swapper/319:15:223
12639991812,4cyclictest0-21swapper/319:15:013
1261299188,7cyclictest12012-21kthreadcore20:15:200
1261299183,12cyclictest25137-21uniq20:30:160
1261299182,13cyclictest1796-21kthreadcore20:05:210
1261299182,11cyclictest26123-21apt-get21:05:120
1261299181,9cyclictest12488-21latency_hist19:10:010
1261299181,5cyclictest22896-21ps19:20:150
12612991812,4cyclictest13617-21sort20:50:210
1261299181,14cyclictest7102-21cut20:45:150
1261299181,14cyclictest10520-21ps23:55:210
1261299181,13cyclictest844-21munin-node22:20:150
1261299181,11cyclictest31234-21ls23:30:230
1261299181,11cyclictest19292-21tr19:50:160
12612991810,6cyclictest6112-21ntp_states22:05:290
1263999172,10cyclictest0-21swapper/320:10:143
12639991711,4cyclictest0-21swapper/320:50:183
12639991711,4cyclictest0-21swapper/320:40:143
12639991711,4cyclictest0-21swapper/320:30:253
1261299179,6cyclictest17292-21sshd19:12:320
1261299172,6cyclictest25584-21kthreadcore19:55:210
1261299172,11cyclictest22778-21kthreadcore21:00:170
1261299171,13cyclictest801-21kthreadcore00:25:230
1261299171,12cyclictest6475-21kthreadcore23:35:200
12639991610,4cyclictest0-21swapper/319:25:223
1261299162,11cyclictest135-21systemd-journal20:15:010
1261299161,11cyclictest32401-21grep20:00:300
1263999103,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional