You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 00:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 22, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23162996957,8cyclictest35966irq/58-eth011:35:143
2313599561,38cyclictest12769-21sh09:30:132
2313599561,38cyclictest12769-21sh09:30:132
2311099559,7cyclictest131rcu_preempt08:15:121
2311099542,39cyclictest15503-21ssh11:20:121
23162995312,26cyclictest35966irq/58-eth010:55:123
23162995016,24cyclictest35966irq/58-eth012:35:103
2311099495,32cyclictest9953-21ssh11:35:131
23110994926,16cyclictest630-21Async09:36:311
2316299483,25cyclictest35966irq/58-eth007:10:123
2316299483,25cyclictest35966irq/58-eth007:10:123
2311099482,33cyclictest30599-21ssh09:40:131
2309699487,5cyclictest131rcu_preempt11:30:120
23096994419,16cyclictest22964-21ssh10:30:120
2311099421,32cyclictest26554-21ssh10:50:111
2309699422,30cyclictest12102-21ssh11:00:120
23096994216,20cyclictest12-21ksoftirqd/010:05:120
23096994216,20cyclictest12-21ksoftirqd/010:05:120
2316299404,23cyclictest35966irq/58-eth009:30:123
2316299404,23cyclictest35966irq/58-eth009:30:123
2316299403,23cyclictest35966irq/58-eth010:25:143
2311099406,22cyclictest1661-21ssh10:00:131
23096994014,18cyclictest12-21ksoftirqd/007:35:110
23110993925,8cyclictest21-21ksoftirqd/112:34:141
23110993925,8cyclictest21-21ksoftirqd/112:34:141
23135993829,4cyclictest7232-21ssh09:45:122
2309699386,22cyclictest21660-21ssh09:35:120
23096993819,16cyclictest12-21ksoftirqd/007:20:140
2316299371,20cyclictest35966irq/58-eth010:10:123
2313599361,26cyclictest21031-21ssh11:05:122
2311099362,22cyclictest4453-21ssh11:50:131
2309699367,23cyclictest19514-21ssh10:10:120
2316299353,23cyclictest35966irq/58-eth008:15:123
2313599355,20cyclictest10349-21kworker/u8:012:30:122
2313599355,20cyclictest10349-21kworker/u8:012:30:122
2313599351,20cyclictest10579-21ssh10:05:132
2313599351,20cyclictest10579-21ssh10:05:132
2309699356,20cyclictest3845-21ssh09:25:120
23096993518,8cyclictest131rcu_preempt10:20:130
23135993421,8cyclictest27-21ksoftirqd/211:44:142
23135993421,8cyclictest27-21ksoftirqd/211:44:142
2311099342,24cyclictest17552-21ssh10:45:121
23096993422,6cyclictest0-21swapper/007:40:130
2316299332,19cyclictest35966irq/58-eth010:00:143
23135993310,13cyclictest27-21ksoftirqd/210:45:122
2313599325,16cyclictest27346irq/40-dwc2_hso08:30:142
23135993213,16cyclictest0-21swapper/211:45:122
2316299313,15cyclictest35966irq/58-eth010:45:133
2313599313,21cyclictest0-21swapper/207:10:062
2313599313,21cyclictest0-21swapper/207:10:052
2311099314,19cyclictest12965-21kworker/u8:110:25:131
2311099312,22cyclictest3154-21ssh10:55:141
2311099311,22cyclictest12788-21ssh09:30:141
2311099311,22cyclictest12788-21ssh09:30:141
2309699312,21cyclictest22989-21rm10:46:140
23096993120,7cyclictest0-21swapper/012:35:100
2313599309,13cyclictest27-21ksoftirqd/212:20:112
23135993015,9cyclictest20181-21/usr/sbin/munin12:35:142
2313599296,14cyclictest27346irq/40-dwc2_hso07:30:122
2313599295,14cyclictest27346irq/40-dwc2_hso08:10:122
2313599293,18cyclictest27346irq/40-dwc2_hso10:35:132
23135992917,6cyclictest27-21ksoftirqd/210:30:122
2311099292,8cyclictest21-21ksoftirqd/111:25:121
2311099292,21cyclictest20495-21wc12:35:141
2316299283,17cyclictest0-21swapper/310:07:143
2316299283,17cyclictest0-21swapper/310:07:143
2313599289,12cyclictest27-21ksoftirqd/211:50:122
2313599285,13cyclictest27346irq/40-dwc2_hso09:05:132
2313599284,18cyclictest27346irq/40-dwc2_hso07:50:112
2313599282,20cyclictest1513-21ssh10:16:142
2313599282,20cyclictest0-21swapper/209:25:142
2313599282,14cyclictest25124-21ssh09:55:142
2311099282,18cyclictest27962-21ssh11:45:131
2311099281,21cyclictest16911-21ssh12:15:131
2316299278,13cyclictest35966irq/58-eth011:00:123
2316299273,15cyclictest35966irq/58-eth011:25:123
2313599274,13cyclictest27346irq/40-dwc2_hso07:45:142
2313599272,16cyclictest27346irq/40-dwc2_hso08:55:152
2311099274,15cyclictest21-21ksoftirqd/109:00:121
2309699273,17cyclictest8900-21diskmemload12:05:130
2309699271,2cyclictest131rcu_preempt09:50:130
2316299263,17cyclictest0-21swapper/310:40:143
2313599265,13cyclictest27346irq/40-dwc2_hso10:55:142
2313599262,9cyclictest21116-21ssh10:28:142
2313599262,10cyclictest27-21ksoftirqd/210:40:142
2311099262,19cyclictest23126-21ssh10:10:231
2311099262,16cyclictest7233-21sh09:45:121
23110992616,6cyclictest0-21swapper/112:20:121
2316299252,13cyclictest35966irq/58-eth009:40:123
2316299251,15cyclictest35966irq/58-eth011:20:133
2313599254,13cyclictest27346irq/40-dwc2_hso07:25:132
2313599252,7cyclictest131rcu_preempt11:23:512
2313599252,20cyclictest0-21swapper/209:10:132
23135992514,8cyclictest0-21swapper/212:25:142
23110992513,8cyclictest201rcuc/111:55:131
23096992517,4cyclictest8900-21diskmemload11:25:130
23162992410,9cyclictest0-21swapper/310:50:133
23135992412,9cyclictest3377-21ssh11:11:412
23135992412,10cyclictest959-21ssh09:22:002
2313599241,14cyclictest1743-21apt-get07:55:112
2311099242,12cyclictest21-21ksoftirqd/108:35:121
23110992413,9cyclictest9003-21sed10:20:281
23110992412,10cyclictest0-21swapper/108:20:151
2309699249,5cyclictest131rcu_preempt08:45:120
2309699249,5cyclictest131rcu_preempt08:45:120
23162992311,8cyclictest35966irq/58-eth010:30:133
2313599234,4cyclictest27-21ksoftirqd/208:35:132
2313599232,6cyclictest0-21swapper/211:39:052
2313599232,15cyclictest27346irq/40-dwc2_hso10:50:122
23135992312,4cyclictest13647-21idleruntime-cro10:25:012
23135992312,4cyclictest0-21swapper/208:25:252
23135992310,11cyclictest6085-21ssh11:30:492
2311099232,18cyclictest8593-21ssh10:40:141
2311099231,19cyclictest9550-21cron11:35:011
2309699239,11cyclictest10156-21kthreadcore12:10:210
2309699236,9cyclictest0-21swapper/007:55:120
2309699232,14cyclictest28612-21diskstats10:15:140
23096992314,6cyclictest12-21ksoftirqd/012:30:120
23096992314,6cyclictest12-21ksoftirqd/012:30:120
23096992312,8cyclictest15616-21cat10:25:200
2309699231,17cyclictest8900-21diskmemload09:15:140
2313599225,6cyclictest0-21swapper/207:35:142
2313599223,13cyclictest27346irq/40-dwc2_hso12:10:122
2313599222,10cyclictest16562-21irqcore09:50:172
2313599222,10cyclictest13017-21kthreadcore08:05:202
2313599221,7cyclictest0-21swapper/208:40:162
23135992216,4cyclictest27-21ksoftirqd/211:57:272
2313599221,17cyclictest21160-21fschecks_time08:15:162
2313599221,12cyclictest30596-21sh09:40:122
2313599221,11cyclictest4295-21cat09:05:022
2313599221,11cyclictest27-21ksoftirqd/211:25:122
2311099227,3cyclictest21-21ksoftirqd/110:05:311
2311099227,3cyclictest21-21ksoftirqd/110:05:311
2311099222,17cyclictest22415-21ssh12:00:141
2311099222,16cyclictest2371-21ssh12:25:141
23110992216,3cyclictest21-21ksoftirqd/112:11:201
23110992214,6cyclictest32644-21cron07:20:001
23110992214,5cyclictest0-21swapper/110:35:231
2311099221,18cyclictest20767-21ls11:05:011
2313599215,13cyclictest27-21ksoftirqd/210:10:192
2313599212,16cyclictest2040-21sh10:00:182
2313599212,14cyclictest0-21swapper/212:00:062
2313599212,10cyclictest21820-21fschecks_count09:35:162
2313599211,9cyclictest9777-21tr11:15:242
2313599211,9cyclictest22435-21cut08:50:152
2313599211,9cyclictest12680-21sed11:00:182
23135992113,6cyclictest18258-21/usr/sbin/munin09:15:272
2313599211,16cyclictest25999-21cut08:20:162
2313599211,16cyclictest17662-21idleruntime08:45:162
2313599211,16cyclictest17662-21idleruntime08:45:162
2313599211,16cyclictest11206-21latency_hist08:05:012
2313599211,14cyclictest16910-21ssh12:15:142
2313599211,10cyclictest32741-21latency_hist07:20:022
2311099219,3cyclictest21-21ksoftirqd/112:08:061
2311099219,3cyclictest21-21ksoftirqd/109:19:431
2311099218,3cyclictest21-21ksoftirqd/111:19:541
2311099218,3cyclictest21-21ksoftirqd/109:20:191
2311099212,3cyclictest131rcu_preempt08:10:121
23110992115,4cyclictest0-21swapper/108:05:251
2311099211,17cyclictest25300-21fschecks_time09:55:161
2309699214,13cyclictest8900-21diskmemload11:55:140
2309699211,4cyclictest0-21swapper/009:55:370
2309699211,10cyclictest0-21swapper/011:15:020
2316299207,9cyclictest0-21swapper/309:17:443
23162992010,8cyclictest0-21swapper/312:27:073
23162992010,8cyclictest0-21swapper/312:19:103
23162992010,7cyclictest0-21swapper/309:20:223
2313599209,8cyclictest0-21swapper/212:05:132
2313599201,9cyclictest2072-21kthreadcore07:20:202
2311099208,3cyclictest21-21ksoftirqd/111:05:261
23110992014,4cyclictest0-21swapper/108:40:191
23110992013,4cyclictest0-21swapper/109:35:151
23110992012,5cyclictest30717-21kthreadcore11:10:191
23110992011,6cyclictest30874-21kthreadcore10:15:201
2311099201,16cyclictest22615-21sh11:40:261
2311099201,16cyclictest22615-21sh11:40:261
2311099201,14cyclictest16546-21ssh09:50:171
2309699209,8cyclictest6660-21df_inode11:15:160
2309699207,10cyclictest14106-21grep11:35:310
2309699202,14cyclictest27231-21ssh09:20:120
2309699202,10cyclictest0-21swapper/010:51:390
2309699201,14cyclictest8032-21ssh11:50:240
2316299199,8cyclictest0-21swapper/309:59:113
2316299198,8cyclictest0-21swapper/311:15:433
2316299193,11cyclictest0-21swapper/310:38:143
2316299192,11cyclictest0-21swapper/309:50:263
2316299192,11cyclictest0-21swapper/309:00:133
2316299192,11cyclictest0-21swapper/307:55:043
2316299192,10cyclictest0-21swapper/308:25:053
23162991910,6cyclictest35966irq/58-eth011:55:143
2313599191,10cyclictest19664-21sort07:40:102
2311099197,8cyclictest3085-1kworker/u9:009:25:261
2311099197,3cyclictest21-21ksoftirqd/109:11:451
23110991913,4cyclictest0-21swapper/108:00:311
23110991913,4cyclictest0-21swapper/107:50:241
23110991913,3cyclictest21-21ksoftirqd/109:05:151
23110991912,4cyclictest26695-21sh10:30:251
2309699199,7cyclictest3142-21apt-get10:55:130
2309699193,13cyclictest28291-21fschecks_time07:15:150
2309699192,7cyclictest9379-21kthreadcore09:45:200
2309699192,14cyclictest29714-21kthreadcore11:45:190
2309699191,9cyclictest9836-21latency_hist07:30:020
2309699191,4cyclictest8900-21diskmemload12:05:010
2309699191,4cyclictest0-21swapper/010:40:060
2309699191,3cyclictest0-21swapper/011:41:420
2309699191,3cyclictest0-21swapper/011:41:420
23096991913,4cyclictest0-21swapper/008:25:180
23096991911,5cyclictest31336-21kthreadcore09:40:200
2309699191,13cyclictest8900-21diskmemload10:35:110
2309699191,13cyclictest29474-21timerandwakeup08:20:280
23096991910,7cyclictest1640-21apt-get10:00:130
2316299189,6cyclictest35966irq/58-eth012:05:123
2316299189,6cyclictest35966irq/58-eth011:30:123
2316299189,6cyclictest0-21swapper/312:24:103
2316299189,6cyclictest0-21swapper/311:50:233
2316299189,6cyclictest0-21swapper/311:40:163
2316299189,6cyclictest0-21swapper/311:40:163
2316299184,8cyclictest0-21swapper/307:40:113
2316299182,8cyclictest0-21swapper/311:45:123
2316299182,12cyclictest0-21swapper/311:13:143
2316299182,11cyclictest0-21swapper/307:25:143
2316299182,10cyclictest0-21swapper/312:00:053
2316299182,10cyclictest0-21swapper/307:45:143
23162991811,5cyclictest0-21swapper/310:15:243
2316299181,14cyclictest0-21swapper/309:25:223
2316299181,12cyclictest0-21swapper/307:50:203
2316299181,11cyclictest0-21swapper/308:05:013
2316299181,10cyclictest0-21swapper/307:40:013
23162991810,6cyclictest35966irq/58-eth012:30:133
23162991810,6cyclictest35966irq/58-eth012:30:133
23162991810,6cyclictest0-21swapper/312:12:473
2311099184,11cyclictest2833-21cron08:30:001
2311099182,12cyclictest1428-21apt-get07:55:051
2311099181,9cyclictest19912-21kthreadcore08:45:201
2311099181,9cyclictest19912-21kthreadcore08:45:201
23110991812,4cyclictest0-21swapper/108:50:281
23110991811,5cyclictest0-21swapper/107:20:171
23110991811,4cyclictest21-21ksoftirqd/107:30:221
23110991810,6cyclictest0-21swapper/107:10:171
23110991810,6cyclictest0-21swapper/107:10:171
23110991810,3cyclictest21-21ksoftirqd/107:35:291
2309699189,7cyclictest11385-21apt-get08:05:120
2309699189,6cyclictest19795-21sh12:15:210
2309699186,9cyclictest2468-21df_inode12:25:150
2309699182,13cyclictest32115-21fschecks_time09:00:160
2309699182,12cyclictest26574-21irqstats12:20:170
2309699182,11cyclictest0-21swapper/008:40:140
2309699181,9cyclictest19631-21ntp_offset08:10:240
2309699181,14cyclictest7845-21apt-get08:35:120
2309699181,14cyclictest26091-21cut07:10:210
2309699181,14cyclictest26091-21cut07:10:210
2309699181,11cyclictest25091-21timerwakeupswit11:05:300
2316299179,6cyclictest0-21swapper/311:06:103
2316299179,6cyclictest0-21swapper/309:46:193
2316299179,6cyclictest0-21swapper/309:37:123
2316299179,6cyclictest0-21swapper/309:10:393
2316299179,5cyclictest35966irq/58-eth010:20:133
2311099176,3cyclictest21-21ksoftirqd/107:40:161
2311099172,3cyclictest131rcu_preempt07:45:141
2311099171,14cyclictest27297-21fschecks_time08:55:171
23110991711,4cyclictest0-21swapper/107:25:191
2309699173,11cyclictest13747-21kthreadcore09:30:190
2309699173,11cyclictest13747-21kthreadcore09:30:190
2309699171,5cyclictest9399-21sshd09:10:140
2309699171,14cyclictest11149-21cron08:05:010
23096991711,3cyclictest24448-21cut08:50:200
2309699171,13cyclictest1432-21munin-run07:55:010
2309699171,12cyclictest15495-21apt-get11:20:140
2316299168,5cyclictest35966irq/58-eth008:55:143
2316299167,6cyclictest35966irq/58-eth008:40:143
2316299162,9cyclictest0-21swapper/308:50:043
2316299162,9cyclictest0-21swapper/308:30:153
2316299162,10cyclictest0-21swapper/308:20:143
2316299162,10cyclictest0-21swapper/308:10:123
2313599164,9cyclictest23223-21sed07:10:022
2309699164,9cyclictest24678-21fschecks_time07:45:160
2309699162,11cyclictest23296-21kthreadcore08:15:200
2309699161,8cyclictest3155-21df08:30:160
2309699161,12cyclictest5544-21kthreadcore09:05:190
2309699161,12cyclictest30541-21ntp_192.168.11408:55:250
2309699161,10cyclictest10279-21hwlatdetect07:30:160
2316299157,6cyclictest0-21swapper/309:06:253
2316299157,3cyclictest0-21swapper/308:45:183
2316299157,3cyclictest0-21swapper/308:45:173
2316299152,9cyclictest0-21swapper/308:35:133
2316299151,8cyclictest0-21swapper/308:05:273
2316299149,3cyclictest0-21swapper/307:22:153
2316299149,3cyclictest0-21swapper/307:15:273
2311099141,10cyclictest23109-21sort07:10:021
2309699142,8cyclictest23089-21cyclictest07:10:020
2316299122,8cyclictest0-21swapper/307:30:153
2316299101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional