You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-10 - 15:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Jun 10, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
901699592,45cyclictest7424-21ssh12:00:130
901699563,43cyclictest5068-21ssh10:10:110
902799554,3cyclictest0-21swapper/108:30:101
902799531,4cyclictest4783-21ssh12:35:121
901699532,38cyclictest24281-21ssh11:15:120
9035995217,19cyclictest14012-21ssh10:15:092
901699512,41cyclictest1613-21ssh12:15:120
9016995020,22cyclictest12-21ksoftirqd/009:25:120
903599481,30cyclictest28391-21ssh12:30:122
904399474,28cyclictest35966irq/58-eth012:15:123
903599472,19cyclictest31380-21ssh10:40:352
902799462,3cyclictest4099-21ssh09:15:121
902799461,36cyclictest9725-21ssh11:25:151
901699462,34cyclictest16464-21ssh09:40:120
901699460,34cyclictest27585-21ssh09:10:120
901699460,34cyclictest27585-21ssh09:10:110
902799433,22cyclictest13059-21ssh09:20:111
902799411,4cyclictest0-21swapper/107:30:121
902799410,4cyclictest736-21apt-get07:35:131
904399403,21cyclictest35966irq/58-eth009:15:123
903599402,21cyclictest4010-21ssh11:02:352
901699402,29cyclictest3077-21ssh10:45:140
901699401,30cyclictest27519-21ssh11:35:130
904399392,22cyclictest35966irq/58-eth010:05:133
903599392,20cyclictest15655-21sh09:55:352
902799381,6cyclictest27197-21diskmemload09:35:111
903599372,29cyclictest22638-21ssh11:13:352
902799371,4cyclictest1596-21apt-get12:15:121
901699372,28cyclictest10511-21rm12:20:100
904399364,23cyclictest35966irq/58-eth012:30:113
9043993619,11cyclictest35966irq/58-eth011:05:133
9043993619,11cyclictest35966irq/58-eth011:05:123
904399361,23cyclictest35966irq/58-eth008:40:123
904399361,23cyclictest35966irq/58-eth008:40:113
9035993616,15cyclictest27-21ksoftirqd/211:15:122
903599361,14cyclictest630-21Async09:36:312
902799367,25cyclictest22625-21irqrtprio11:50:171
902799361,8cyclictest28837-21ntp_192.168.11412:10:241
902799361,6cyclictest15643-21ntpq10:50:241
9027993611,17cyclictest7442-21diskmemload12:00:131
903599359,9cyclictest23191-21kworker/2:009:45:132
902799358,23cyclictest23990-21cat12:25:291
902799358,23cyclictest23990-21cat12:25:281
902799352,3cyclictest30952-21apt-get09:30:121
902799351,9cyclictest10376-21timerandwakeup11:05:271
902799351,9cyclictest10376-21timerandwakeup11:05:271
902799351,8cyclictest19158-21kthreadcore08:25:211
902799351,8cyclictest19158-21kthreadcore08:25:201
902799351,10cyclictest29659-21sed11:00:011
901699352,23cyclictest30963-21ssh09:30:110
904399345,16cyclictest35966irq/58-eth009:30:133
904399341,21cyclictest35966irq/58-eth009:00:113
903599343,9cyclictest0-21swapper/210:09:092
902799343,23cyclictest10534-21ssh12:20:131
902799341,9cyclictest6674-21tr11:40:211
902799341,8cyclictest17205-21kernelversion09:40:181
902799341,8cyclictest16145-21cut07:15:191
902799341,6cyclictest28474-21kthreadcore09:45:211
902799341,6cyclictest12415-21kthreadcore07:45:211
902799341,5cyclictest15303-21kthreadcore11:45:221
902799341,10cyclictest1398-21ls11:00:271
902799340,5cyclictest24277-21apt-get11:15:131
903599336,17cyclictest27-21ksoftirqd/210:20:132
903599335,16cyclictest27346irq/40-dwc2_hso08:05:132
903599333,13cyclictest261rcuc/209:35:122
902799337,24cyclictest29734-21kthreadcore10:40:211
902799333,4cyclictest0-21swapper/107:55:121
902799331,9cyclictest31646-21grep11:35:291
902799331,9cyclictest22033-21latency07:20:211
902799331,8cyclictest8756-21grep10:30:161
902799331,8cyclictest5213-21kthreadcore10:45:221
902799331,8cyclictest3386-21diskstats08:45:161
902799331,8cyclictest27889-21cut08:35:221
902799331,8cyclictest24964-21kthreadcore09:05:211
902799331,8cyclictest22159-21fschecks_time09:25:141
902799331,8cyclictest22119-21missed_timers11:30:231
902799331,8cyclictest2139-21cpu08:10:131
902799331,8cyclictest21338-21cat10:35:291
902799331,8cyclictest14660-21irqrtprio10:15:161
902799331,6cyclictest31951-21ssh10:25:101
902799331,5cyclictest31485-21cut08:05:201
902799331,5cyclictest31154-21fschecks_time11:55:181
902799331,5cyclictest17436-21users08:55:311
902799331,10cyclictest29737-21grep09:10:211
902799331,10cyclictest29737-21grep09:10:201
904399323,16cyclictest35966irq/58-eth010:35:143
903599321,13cyclictest24944-21kworker/u8:211:39:342
902799321,9cyclictest15527-21fschecks_time11:10:161
902799321,8cyclictest19421-21latency_hist10:00:011
902799321,8cyclictest16876-21kthreadcore07:50:191
902799321,7cyclictest2382-21grep09:50:181
902799321,6cyclictest3260-21latency_hist08:45:011
902799321,6cyclictest25907-21kthreadcore07:25:211
902799321,6cyclictest23606-21cut10:20:161
902799321,12cyclictest5196-21diskstats10:10:131
902799320,7cyclictest22439-21latency_hist09:05:011
904399313,20cyclictest35966irq/58-eth010:15:093
9043993113,12cyclictest35966irq/58-eth012:05:153
903599313,22cyclictest0-21swapper/211:33:352
902799319,19cyclictest0-21swapper/111:20:121
902799313,3cyclictest0-21swapper/108:40:111
902799311,6cyclictest32183-21find12:30:271
902799311,6cyclictest28843-21fschecks_time10:05:171
902799311,5cyclictest13931-21sort07:15:011
902799310,8cyclictest16660-21if_eth012:05:201
901699312,25cyclictest18973-21ssh09:59:340
901699312,23cyclictest24855-21ssh10:38:340
903599304,18cyclictest9012-21cyclictest07:20:122
903599304,15cyclictest27346irq/40-dwc2_hso10:35:132
9035993017,10cyclictest27-21ksoftirqd/211:50:122
902799302,6cyclictest0-21swapper/108:20:161
902799301,8cyclictest6012-21irqcore07:40:161
902799301,7cyclictest7074-21fschecks_time08:15:181
902799301,7cyclictest20360-21irqrtprio10:00:171
902799301,4cyclictest8216-21fschecks_count08:50:131
902799300,6cyclictest25456-21ps08:00:181
901699302,22cyclictest4008-21ssh10:09:090
903599295,10cyclictest27-21ksoftirqd/210:10:112
903599293,21cyclictest0-21swapper/211:27:352
903599292,11cyclictest25427-21ssh10:01:342
901699292,19cyclictest29882-21ssh11:00:130
9016992915,6cyclictest0-21swapper/010:00:140
904399283,18cyclictest0-21swapper/311:04:203
903599287,17cyclictest27346irq/40-dwc2_hso09:50:122
903599284,15cyclictest27346irq/40-dwc2_hso08:20:122
903599282,21cyclictest0-21swapper/211:20:132
903599282,10cyclictest27-21ksoftirqd/208:40:112
903599282,10cyclictest27-21ksoftirqd/208:40:102
903599281,22cyclictest0-21swapper/207:30:132
903599281,20cyclictest27346irq/40-dwc2_hso07:15:102
903599281,18cyclictest27-21ksoftirqd/207:55:122
901699282,19cyclictest5312-21ssh10:26:350
904399273,15cyclictest0-21swapper/309:49:043
904399272,17cyclictest35966irq/58-eth011:50:133
904399272,15cyclictest35966irq/58-eth010:20:113
903599272,19cyclictest0-21swapper/208:55:122
903599272,15cyclictest4105-21ssh09:15:142
903599271,17cyclictest12682-21ls11:45:012
903599271,15cyclictest5792-21cut07:40:152
9016992714,6cyclictest6407-21ssh11:05:130
9016992714,6cyclictest6407-21ssh11:05:120
903599262,9cyclictest863-21ssh12:14:352
903599262,17cyclictest27346irq/40-dwc2_hso08:50:102
903599262,16cyclictest27346irq/40-dwc2_hso07:45:142
9035992612,10cyclictest11378-21kthreadcore12:20:182
901699262,18cyclictest27197-21diskmemload11:47:350
9016992616,6cyclictest8521-21sh10:30:130
9016992614,9cyclictest11449-21cron08:20:020
901699261,20cyclictest15328-21ssh11:10:120
903599251,18cyclictest0-21swapper/209:20:122
9035992510,11cyclictest9012-21cyclictest07:35:112
901699258,10cyclictest9012-21cyclictest08:10:120
901699252,16cyclictest0-21swapper/010:40:130
904399243,14cyclictest0-21swapper/309:55:353
9043992410,9cyclictest35966irq/58-eth011:30:263
903599242,17cyclictest0-21swapper/210:58:342
9035992412,10cyclictest15332-21ssh10:32:542
9035992411,6cyclictest10298-21ssh10:48:222
9035992411,11cyclictest21322-21ssh11:49:422
903599240,19cyclictest0-21swapper/207:25:132
901699247,10cyclictest25538-21kworker/u8:010:50:130
9043992311,8cyclictest35966irq/58-eth012:00:133
904399231,16cyclictest0-21swapper/310:45:153
903599239,5cyclictest0-21swapper/210:25:102
903599238,7cyclictest4215-21kthreadcore08:10:212
903599232,11cyclictest29222-21cut09:10:202
903599232,11cyclictest29222-21cut09:10:192
903599232,10cyclictest21203-21tr12:25:192
903599232,10cyclictest21203-21tr12:25:182
903599232,10cyclictest13710-21kthreadcore10:50:202
9035992312,8cyclictest0-21swapper/208:00:302
903599231,18cyclictest9085-21kthreadcore08:15:232
9035992311,8cyclictest464-21kthreadcore09:30:192
903599231,10cyclictest17086-21sort12:05:212
903599230,12cyclictest8360-21ssh12:00:202
902799231,5cyclictest9050-21latency_hist07:10:011
901699232,16cyclictest27197-21diskmemload09:52:350
9016992314,6cyclictest0-21swapper/010:20:130
904399223,13cyclictest0-21swapper/310:40:343
903599227,7cyclictest27-21ksoftirqd/212:38:592
903599224,9cyclictest17578-21kthreadcore09:40:192
903599223,10cyclictest25815-21cron08:35:012
903599222,7cyclictest6393-21ssh11:05:122
903599222,7cyclictest6393-21ssh11:05:112
903599222,10cyclictest11828-21kthreadcore07:10:192
903599221,17cyclictest21229-21latency_hist08:30:022
903599221,17cyclictest21229-21latency_hist08:30:012
903599221,10cyclictest19199-21head07:50:282
903599220,18cyclictest5364-21kthreadcore08:45:212
9016992213,6cyclictest0-21swapper/008:55:250
904399219,9cyclictest0-21swapper/310:52:363
904399216,10cyclictest0-21swapper/311:45:353
904399213,12cyclictest0-21swapper/311:40:353
904399213,12cyclictest0-21swapper/310:26:353
904399212,13cyclictest0-21swapper/310:01:343
9043992110,8cyclictest0-21swapper/309:38:093
903599211,9cyclictest28983-21kthreadcore08:35:232
903599211,9cyclictest22356-21irqcore09:25:172
903599211,11cyclictest31010-21cpuspeed11:55:152
903599211,11cyclictest18780-21kthreadcore09:00:192
901699219,3cyclictest131rcu_preempt11:30:180
901699211,9cyclictest25561-21ntpq11:50:250
901699211,2cyclictest131rcu_preempt11:55:150
901699211,15cyclictest8888-21latency07:40:200
901699211,15cyclictest20901-21rm10:55:140
901699211,14cyclictest32258-21switchtime12:30:280
904399209,6cyclictest0-21swapper/310:59:103
904399205,9cyclictest35966irq/58-eth010:10:103
904399203,12cyclictest0-21swapper/311:39:343
9043992013,5cyclictest0-21swapper/309:11:383
9043992013,5cyclictest0-21swapper/309:11:373
904399201,13cyclictest0-21swapper/307:55:013
903599202,9cyclictest22735-21fschecks_time09:05:142
903599201,15cyclictest9094-21latency_hist07:10:022
901699202,8cyclictest131rcu_preempt07:25:180
901699201,8cyclictest25901-21irqstats12:10:190
9016992011,6cyclictest18203-21cron11:30:010
9016992010,7cyclictest26873-21sshd12:28:360
9016992010,7cyclictest26873-21sshd12:28:350
901699200,13cyclictest6916-21ls08:45:270
904399198,6cyclictest0-21swapper/309:41:563
904399193,11cyclictest0-21swapper/311:12:363
9043991912,4cyclictest0-21swapper/311:25:143
904399191,14cyclictest0-21swapper/309:50:133
904399191,14cyclictest0-21swapper/307:45:013
904399191,13cyclictest0-21swapper/308:35:313
904399191,12cyclictest0-21swapper/312:15:023
904399191,12cyclictest0-21swapper/307:15:203
904399191,11cyclictest0-21swapper/308:50:213
901699197,9cyclictest25271-21sort09:45:110
901699197,7cyclictest0-21swapper/008:00:130
901699192,9cyclictest1595-21kthreadcore11:20:210
901699192,11cyclictest27197-21diskmemload10:19:110
901699191,3cyclictest0-21swapper/009:21:010
901699191,3cyclictest0-21swapper/009:19:080
901699191,16cyclictest23387-21ls07:25:010
9016991910,6cyclictest18523-21sshd07:18:350
901699190,16cyclictest15608-21ls07:50:150
904399189,6cyclictest35966irq/58-eth012:20:113
904399189,6cyclictest0-21swapper/309:20:263
904399182,11cyclictest0-21swapper/311:15:133
904399182,11cyclictest0-21swapper/310:34:353
904399182,10cyclictest0-21swapper/309:25:123
904399182,10cyclictest0-21swapper/309:05:123
904399182,10cyclictest0-21swapper/308:25:143
904399182,10cyclictest0-21swapper/308:25:133
904399182,10cyclictest0-21swapper/308:10:133
904399181,13cyclictest0-21swapper/308:05:293
904399181,10cyclictest35966irq/58-eth011:55:133
9043991810,5cyclictest35966irq/58-eth011:20:133
901699189,7cyclictest4613-21irqrtprio11:40:190
901699189,6cyclictest31185-21kthreadcore07:30:210
901699188,8cyclictest8309-21fw_packets08:50:150
901699187,8cyclictest22147-21kthreadcore08:30:170
901699187,3cyclictest131rcu_preempt09:35:190
901699183,10cyclictest628-21tcf-agent09:00:130
901699181,7cyclictest12-21ksoftirqd/012:05:330
901699181,15cyclictest7370-21cut12:35:230
901699181,14cyclictest16410-21latency_hist08:25:010
901699181,13cyclictest0-21swapper/007:35:120
904399179,3cyclictest0-21swapper/312:38:573
9043991711,4cyclictest0-21swapper/312:25:163
9043991711,4cyclictest0-21swapper/312:25:163
901699178,6cyclictest28701-21cut08:35:230
901699175,9cyclictest26418-21grep09:05:280
9016991711,4cyclictest0-21swapper/007:45:300
904399162,10cyclictest0-21swapper/308:55:123
904399162,10cyclictest0-21swapper/308:00:133
904399161,8cyclictest0-21swapper/308:35:003
901699168,5cyclictest19459-21kthreadcore08:25:220
901699168,5cyclictest19459-21kthreadcore08:25:210
901699163,11cyclictest10794-21tr07:10:190
901699161,12cyclictest925-21tr08:40:200
901699161,12cyclictest925-21tr08:40:190
901699161,12cyclictest20129-21apt-get07:55:130
901699161,10cyclictest9095-21latency_hist07:10:010
901699160,9cyclictest701-21ntpq08:05:260
904399151,7cyclictest0-21swapper/308:20:133
904399151,11cyclictest0-21swapper/307:20:213
904399151,10cyclictest0-21swapper/307:30:123
904399141,7cyclictest0-21swapper/307:10:113
904399141,10cyclictest321rcuc/307:45:143
904399141,10cyclictest0-21swapper/307:55:163
904399131,6cyclictest0-21swapper/307:35:213
904399128,2cyclictest33-21ksoftirqd/308:49:433
904399123,7cyclictest0-21swapper/307:25:153
904399122,8cyclictest0-21swapper/308:15:193
904399111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional