You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-06 - 10:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Jul 06, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
913399635,51cyclictest27346irq/40-dwc2_hso22:30:132
913399571,7cyclictest0-21swapper/222:15:122
913399571,7cyclictest0-21swapper/222:15:112
912599555,40cyclictest20217-21ssh22:35:111
913399544,27cyclictest4672-21ssh23:20:132
913399543,31cyclictest12232-21ssh21:55:122
912599544,40cyclictest13760-21ssh23:25:141
911799519,9cyclictest12-21ksoftirqd/022:15:120
911799519,9cyclictest12-21ksoftirqd/022:15:110
913399492,22cyclictest24799-21irqrtprio22:55:192
913399481,14cyclictest2086-21ssh22:25:132
9141994511,22cyclictest35966irq/58-eth022:50:153
913399452,4cyclictest0-21swapper/219:45:112
913399442,8cyclictest27346irq/40-dwc2_hso20:00:122
913399436,6cyclictest8528-21apt-get23:40:142
913399433,9cyclictest27346irq/40-dwc2_hso20:40:122
913399433,5cyclictest0-21swapper/220:25:142
913399432,3cyclictest0-21swapper/223:45:132
912599431,32cyclictest26675-21ssh21:45:121
9141994218,12cyclictest35966irq/58-eth000:05:153
913399422,34cyclictest24524-21ssh23:11:182
911799422,31cyclictest25035-21ssh00:25:140
914199412,29cyclictest35966irq/58-eth000:15:123
913399412,22cyclictest16853-21ssh23:07:572
911799416,7cyclictest131rcu_preempt00:20:130
912599403,10cyclictest0-21swapper/100:25:131
9141993923,8cyclictest0-21swapper/321:15:133
913399391,7cyclictest13042-21unixbench_multi19:10:272
913399391,7cyclictest1155366ssh22:46:132
914199381,24cyclictest35966irq/58-eth022:35:123
913399382,23cyclictest26702-21ssh23:50:132
913399382,18cyclictest7146-21sh22:10:152
911799381,25cyclictest4506-21ssh21:15:130
913399372,9cyclictest0-21swapper/223:00:232
913399372,9cyclictest0-21swapper/223:00:232
913399371,8cyclictest10301-21sh00:35:012
913399371,10cyclictest23867-21kthreadcore20:30:202
911799374,26cyclictest875-21ssh23:00:140
911799374,26cyclictest875-21ssh23:00:130
913399364,22cyclictest29424-21ssh22:40:142
9133993628,4cyclictest27-21ksoftirqd/200:10:132
913399362,5cyclictest0-21swapper/223:15:192
913399361,8cyclictest22884-21kthreadcore22:35:192
913399361,8cyclictest21042-21latency_hist00:05:022
913399361,8cyclictest0-21swapper/220:15:122
913399361,7cyclictest19376-21grep19:20:172
913399361,6cyclictest30277-21ntp_192.168.11421:45:252
913399361,6cyclictest17401-21ntp_kernel_pll_23:25:252
913399361,12cyclictest27159-21usage_Anybus23:30:292
913399361,10cyclictest28697-21awk00:25:262
913399360,11cyclictest27441-21latency_hist21:10:012
912599362,27cyclictest3246-21ssh23:55:131
912599361,26cyclictest13614-21ssh21:20:131
911799363,30cyclictest22795-21ssh23:30:140
9117993610,5cyclictest131rcu_preempt23:10:130
913399359,3cyclictest6718-21load23:55:222
913399351,9cyclictest592-21kthreadcore23:35:192
913399351,9cyclictest4093-21cron21:15:002
913399351,9cyclictest26308-21/usr/sbin/munin20:35:332
913399351,9cyclictest12104-21fschecks_count20:20:172
913399351,13cyclictest6844-21sort22:10:012
913399350,9cyclictest20212-21sort19:55:012
913399350,9cyclictest20212-21sort19:55:012
913399350,14cyclictest4630-21grep19:35:302
9141993417,12cyclictest35966irq/58-eth020:00:143
913399349,22cyclictest6656-21ssh21:33:212
913399348,5cyclictest0-21swapper/219:15:222
913399342,8cyclictest24463-21kthreadcore19:25:192
913399341,9cyclictest930-21snmpd00:22:272
913399341,9cyclictest20471-21tune2fs19:55:162
913399341,8cyclictest15444-21kthreadcore21:20:212
913399341,30cyclictest2827866ssh22:02:552
9133993411,21cyclictest930-21snmpd21:17:232
913399341,11cyclictest31520-21grep21:30:022
913399341,10cyclictest6579-21latency_hist00:15:022
9125993412,16cyclictest21-21ksoftirqd/100:35:141
911799346,23cyclictest12-21ksoftirqd/023:05:140
911799341,27cyclictest8539-21sh23:40:140
914199331,23cyclictest35966irq/58-eth023:55:143
913399331,9cyclictest0-21swapper/219:30:302
913399331,10cyclictest13503-21kthreadcore00:35:232
9133993310,20cyclictest24405-21ssh21:42:462
912599335,16cyclictest17595-21ssh21:40:131
912599332,24cyclictest17689-21ssh23:45:141
911799334,24cyclictest7122-21ssh22:10:130
911799332,26cyclictest10675-21ssh00:35:140
9117993316,13cyclictest12-21ksoftirqd/021:30:140
9117993313,5cyclictest131rcu_preempt19:55:130
913399326,21cyclictest27-21ksoftirqd/222:52:532
913399322,10cyclictest17201-21latency_hist21:40:022
913399322,10cyclictest17201-21latency_hist21:40:012
913399321,9cyclictest930-21snmpd21:52:572
913399321,9cyclictest930-21snmpd21:52:562
913399321,7cyclictest6105-21grep20:10:292
913399321,13cyclictest8329-21apt-get20:50:152
913399321,10cyclictest6351-21irqrtprio19:40:192
913399320,9cyclictest27433-21ssh00:06:542
912599322,26cyclictest27817-21ssh21:10:121
9125993220,8cyclictest0-21swapper/119:20:121
911799322,21cyclictest12-21ksoftirqd/020:45:130
9117993219,6cyclictest131rcu_preempt19:30:140
913399313,8cyclictest0-21swapper/222:24:172
913399311,7cyclictest13212-21/usr/sbin/munin20:55:142
912599311,23cyclictest28781-21ssh23:15:181
9125993110,17cyclictest21-21ksoftirqd/123:35:131
911799314,24cyclictest3210-21ssh21:50:130
911799314,24cyclictest3210-21ssh21:50:130
911799313,24cyclictest1609-21ssh00:30:130
9117993117,11cyclictest12-21ksoftirqd/022:25:130
913399303,9cyclictest31778-21kthreadcore20:05:202
911799303,25cyclictest21417-21ssh00:05:140
911799302,21cyclictest11643-21irqcore22:30:180
913399291,8cyclictest19413-21cut21:00:202
913399291,7cyclictest7350-21unixbench_multi20:45:292
912599292,21cyclictest11642-21irqcore22:30:171
912599292,21cyclictest10342-21irqcore23:05:171
912599292,19cyclictest6737-21sed22:45:181
911799293,20cyclictest628-21tcf-agent20:25:140
914199281,16cyclictest35966irq/58-eth023:20:143
911799285,17cyclictest12-21ksoftirqd/019:25:130
9117992811,3cyclictest12-21ksoftirqd/019:45:110
914199273,16cyclictest0-21swapper/323:44:173
914199273,16cyclictest0-21swapper/323:17:243
912599272,19cyclictest30083-21sed22:40:181
912599272,19cyclictest1068-21ssh22:24:171
912599271,19cyclictest24192-21ssh22:55:141
911799275,14cyclictest0-21swapper/023:55:130
911799272,21cyclictest24798-21irqrtprio22:55:180
911799272,19cyclictest6972-21ssh00:15:140
911799272,15cyclictest15148-21ssh22:50:150
9117992712,13cyclictest12-21ksoftirqd/021:05:120
911799271,17cyclictest13613-21sh21:20:140
9117992710,13cyclictest14851-21kworker/u8:322:20:140
912599262,18cyclictest16022-21ssh00:20:151
911799262,20cyclictest13744-21ssh23:25:120
911799262,20cyclictest12-21ksoftirqd/023:50:110
911799262,19cyclictest4876-21tune2fs23:20:180
914199252,17cyclictest0-21swapper/323:38:173
912599252,17cyclictest1625-21ssh00:10:271
912599250,14cyclictest17118-21apt-get21:40:031
912599250,14cyclictest17118-21apt-get21:40:021
911799252,19cyclictest3394-21ssh22:42:190
9117992511,4cyclictest12-21ksoftirqd/022:00:180
9141992414,7cyclictest35966irq/58-eth022:40:133
912599243,17cyclictest30511-21ssh22:05:171
9125992412,9cyclictest6676-21ps23:20:221
914199233,13cyclictest0-21swapper/322:46:173
914199233,12cyclictest0-21swapper/321:40:043
9141992314,5cyclictest0-21swapper/323:45:143
9141992312,4cyclictest0-21swapper/300:35:133
912599231,18cyclictest14172-21cut19:15:151
911799239,5cyclictest0-21swapper/022:37:300
911799238,9cyclictest29915-21kworker/u8:100:00:110
911799232,11cyclictest30509-21sh22:05:160
9117992315,4cyclictest12-21ksoftirqd/022:50:010
914199224,10cyclictest0-21swapper/323:25:133
914199223,13cyclictest0-21swapper/300:10:143
9141992213,6cyclictest0-21swapper/300:25:213
9141992212,6cyclictest35966irq/58-eth000:30:133
914199221,11cyclictest35966irq/58-eth019:20:133
912599224,14cyclictest21-21ksoftirqd/123:40:141
912599222,2cyclictest131rcu_preempt22:25:141
912599222,15cyclictest4519-21diskmemload21:15:151
9125992212,7cyclictest19310-21grep22:50:311
9125992210,3cyclictest131rcu_preempt00:30:181
911799225,10cyclictest12-21ksoftirqd/020:20:140
9117992212,7cyclictest5199-21sshd00:13:210
911799221,12cyclictest12214-21ssh21:55:110
9117992210,3cyclictest12-21ksoftirqd/021:27:300
914199213,13cyclictest0-21swapper/322:24:183
9141992112,6cyclictest0-21swapper/321:52:183
9141992112,6cyclictest0-21swapper/321:52:173
9141992110,7cyclictest0-21swapper/322:05:013
9125992112,6cyclictest0-21swapper/122:15:281
9125992112,6cyclictest0-21swapper/122:15:281
912599211,13cyclictest10775-21cut19:45:151
9125992110,7cyclictest7221-21fw_packets00:15:161
911799219,3cyclictest12-21ksoftirqd/021:44:050
911799212,15cyclictest151rcuc/019:35:140
911799211,16cyclictest8489-21ssh21:35:140
911799211,16cyclictest8489-21ssh21:35:130
911799211,10cyclictest30153-21cut23:15:210
914199209,9cyclictest0-21swapper/322:19:183
914199209,9cyclictest0-21swapper/322:19:173
9141992013,4cyclictest0-21swapper/322:05:223
912599208,9cyclictest7495-21if_eth122:10:171
912599206,3cyclictest8990-21kworker/1:019:10:151
912599205,4cyclictest131rcu_preempt21:55:181
912599203,11cyclictest0-21swapper/123:31:181
912599202,12cyclictest0-21swapper/120:15:131
912599201,9cyclictest21875-21irqcore22:00:181
912599201,16cyclictest471-21kthreadcore20:05:211
912599200,14cyclictest3292-21latency_hist20:45:011
911799202,2cyclictest131rcu_preempt20:10:270
911799202,14cyclictest3055-21ntp_192.168.11423:35:230
9117992013,3cyclictest12-21ksoftirqd/019:40:160
911799200,14cyclictest14149-21cut19:15:140
914199199,7cyclictest0-21swapper/321:30:003
914199199,6cyclictest0-21swapper/323:10:083
914199199,6cyclictest0-21swapper/321:37:323
914199199,6cyclictest0-21swapper/321:37:313
914199199,3cyclictest0-21swapper/321:21:523
914199193,11cyclictest0-21swapper/323:06:193
914199193,11cyclictest0-21swapper/322:26:173
914199192,12cyclictest0-21swapper/323:02:243
914199192,12cyclictest0-21swapper/323:02:233
914199192,11cyclictest0-21swapper/320:10:153
914199192,10cyclictest0-21swapper/320:55:053
914199191,13cyclictest0-21swapper/320:25:283
914199191,13cyclictest0-21swapper/319:40:253
912599194,11cyclictest0-21swapper/121:05:131
912599192,14cyclictest23431-21irqstats21:25:191
912599192,13cyclictest992-21ssh23:00:151
912599192,13cyclictest992-21ssh23:00:141
912599191,7cyclictest19621-21cut21:00:201
9125991913,3cyclictest21-21ksoftirqd/123:10:201
9125991911,6cyclictest26083-21sshd20:32:251
912599191,15cyclictest20624-21cut19:55:171
911799197,3cyclictest12-21ksoftirqd/021:12:320
911799192,5cyclictest29776-21ntp_kernel_pll_20:35:250
911799192,14cyclictest30604-21find21:45:300
9117991912,4cyclictest12-21ksoftirqd/023:50:010
911799190,11cyclictest26096-21latency_hist20:35:020
914199189,7cyclictest0-21swapper/322:10:173
914199189,4cyclictest0-21swapper/300:20:313
914199183,10cyclictest0-21swapper/323:31:193
914199183,10cyclictest0-21swapper/322:32:183
914199182,11cyclictest0-21swapper/322:57:183
914199182,11cyclictest0-21swapper/321:56:193
914199182,10cyclictest0-21swapper/321:45:143
914199182,10cyclictest0-21swapper/321:05:143
914199182,10cyclictest0-21swapper/321:00:153
914199182,10cyclictest0-21swapper/320:15:133
913399181,7cyclictest9152-21latency_hist19:10:022
913399181,7cyclictest9152-21latency_hist19:10:012
912599188,8cyclictest31929-21df_inode21:30:151
912599186,9cyclictest15678-21grep20:20:301
912599182,11cyclictest10396-21ssh21:53:181
912599182,11cyclictest10396-21ssh21:53:171
912599181,8cyclictest25520-21kthreadcore19:25:201
912599181,6cyclictest3882-21irqcore20:45:171
912599181,14cyclictest640-21latency_hist19:35:011
912599181,14cyclictest23305-21kthreadcore00:05:211
912599181,14cyclictest15644-21fschecks_count19:50:171
912599181,14cyclictest15644-21fschecks_count19:50:161
912599181,13cyclictest27071-21cut20:35:191
912599181,13cyclictest13458-21cut00:00:181
912599181,12cyclictest5541-21apt-key19:40:011
912599180,12cyclictest13282-21cpuspeed20:55:141
911799187,8cyclictest29674-21cron20:05:010
9117991812,4cyclictest0-21swapper/021:00:300
911799181,11cyclictest2341-21ls20:40:280
914199179,5cyclictest35966irq/58-eth000:00:253
914199179,3cyclictest0-21swapper/321:33:543
914199172,9cyclictest0-21swapper/321:10:143
914199172,10cyclictest0-21swapper/320:05:143
914199171,9cyclictest0-21swapper/320:45:023
914199171,9cyclictest0-21swapper/319:50:053
914199171,9cyclictest0-21swapper/319:50:043
912599175,10cyclictest18077-21kthreadcore20:25:191
912599174,10cyclictest9142-21kthreadcore20:50:211
912599171,8cyclictest25089-21/usr/sbin/munin20:00:291
912599171,13cyclictest3228-21kthreadcore20:10:211
912599171,13cyclictest26829-21diskstats23:50:151
912599171,11cyclictest9068-21missed_timers19:40:241
911799179,6cyclictest7290-21perl20:15:160
911799172,10cyclictest0-21swapper/019:10:140
911799171,2cyclictest131rcu_preempt21:00:010
911799171,13cyclictest11367-21tr20:50:230
914199168,5cyclictest35966irq/58-eth023:50:143
914199161,9cyclictest0-21swapper/320:35:023
914199161,12cyclictest0-21swapper/320:35:203
914199161,10cyclictest0-21swapper/319:35:233
911799163,10cyclictest857-21netstat20:05:230
911799162,11cyclictest18885-21cpu19:20:140
911799161,12cyclictest15569-21sort19:50:150
911799161,12cyclictest15569-21sort19:50:140
914199153,9cyclictest0-21swapper/319:45:233
914199152,8cyclictest0-21swapper/320:50:143
914199151,9cyclictest35966irq/58-eth019:10:123
912599151,11cyclictest9182-21latency_hist19:10:021
912599151,11cyclictest9182-21latency_hist19:10:011
914199149,3cyclictest0-21swapper/319:30:293
914199144,3cyclictest0-21swapper/319:25:133
914199141,6cyclictest0-21swapper/320:45:163
914199141,11cyclictest0-21swapper/319:15:163
911799141,10cyclictest9007-21latency_hist19:10:020
911799141,10cyclictest9007-21latency_hist19:10:010
914199131,7cyclictest0-21swapper/319:55:183
914199131,10cyclictest0-21swapper/320:20:183
914199111,7cyclictest0-21swapper/319:10:023
914199111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional