You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 14:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Aug 23, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29466995432,12cyclictest12-21ksoftirqd/021:45:120
2947599531,41cyclictest24928-21ssh22:45:121
2947599531,38cyclictest1548-21ssh22:50:131
2946699532,11cyclictest131rcu_preempt00:30:140
2946699503,34cyclictest26733-21ssh23:40:120
2947599461,38cyclictest19195-21ssh21:30:151
2946699463,33cyclictest10422-21ssh00:25:120
29493994525,13cyclictest35966irq/58-eth022:55:153
29493994327,7cyclictest35966irq/58-eth021:40:123
2946699422,5cyclictest131rcu_preempt23:15:120
2946699422,32cyclictest3224-21ssh22:15:130
2946699422,32cyclictest3224-21ssh22:15:120
2946699422,30cyclictest12258-21ssh22:20:140
29466994214,21cyclictest12-21ksoftirqd/020:45:150
2947599390,29cyclictest0-21swapper/120:35:131
2946699391,30cyclictest0-21swapper/000:35:140
2948599372,18cyclictest24578-21ssh23:37:562
2948599362,16cyclictest261rcuc/221:45:112
2947599361,24cyclictest10164-21ssh21:25:131
2946699365,22cyclictest12-21ksoftirqd/021:00:140
2949399352,21cyclictest35966irq/58-eth020:15:123
2949399352,21cyclictest35966irq/58-eth020:15:123
2947599357,17cyclictest15063-21diskmemload23:50:131
2946699354,23cyclictest3377-21ssh23:45:120
2949399342,19cyclictest35966irq/58-eth000:00:143
2949399333,15cyclictest35966irq/58-eth022:25:133
2948599332,24cyclictest0-21swapper/221:30:142
2947599332,23cyclictest10551-21ssh22:55:141
29466993319,12cyclictest12-21ksoftirqd/023:25:120
29493993224,6cyclictest321rcuc/320:55:133
2948599322,24cyclictest0-21swapper/223:33:562
2947599322,20cyclictest15050-21kworker/u8:221:10:131
29466993216,10cyclictest0-21swapper/020:10:120
2948599311,2cyclictest131rcu_preempt21:50:122
2949399305,14cyclictest35966irq/58-eth021:55:123
2949399304,14cyclictest0-21swapper/321:30:143
29493993020,7cyclictest35966irq/58-eth023:40:133
2948599302,23cyclictest27608-1kworker/u9:123:57:432
2948599302,20cyclictest27346irq/40-dwc2_hso22:50:152
2947599303,19cyclictest17691-21ssh23:35:111
29493992919,7cyclictest35966irq/58-eth022:00:143
2948599294,6cyclictest28436-21ssh00:35:142
2948599294,19cyclictest27346irq/40-dwc2_hso22:05:152
29475992917,8cyclictest15063-21diskmemload22:40:171
29475992911,13cyclictest15545-21kworker/u8:223:57:431
29466992914,10cyclictest8591-21sh22:00:130
2949399284,17cyclictest0-21swapper/321:25:133
2949399283,13cyclictest35966irq/58-eth022:45:123
2949399281,16cyclictest35966irq/58-eth023:10:123
2948599283,16cyclictest27346irq/40-dwc2_hso21:20:142
2948599282,17cyclictest27346irq/40-dwc2_hso21:40:122
2948599282,17cyclictest27346irq/40-dwc2_hso20:30:132
2948599282,15cyclictest27346irq/40-dwc2_hso21:15:132
2948599282,11cyclictest30536-21ssh00:00:152
2948599282,11cyclictest0-21swapper/220:10:132
2948599281,21cyclictest0-21swapper/220:25:142
2948599281,20cyclictest27346irq/40-dwc2_hso22:15:132
2948599281,20cyclictest27346irq/40-dwc2_hso22:15:132
2947599281,16cyclictest4814-21ssh21:40:131
2946699282,20cyclictest31593-21ssh23:24:570
2946699281,21cyclictest6930-21ssh22:35:140
2949399275,19cyclictest0-21swapper/300:10:153
2949399275,19cyclictest0-21swapper/300:10:143
2949399274,13cyclictest0-21swapper/300:25:133
29493992718,5cyclictest321rcuc/320:05:143
29493992714,6cyclictest36066irq/59-eth023:55:123
29493992712,6cyclictest0-21swapper/321:05:133
2949399271,18cyclictest0-21swapper/321:20:133
2948599273,16cyclictest27346irq/40-dwc2_hso20:35:132
2948599272,19cyclictest0-21swapper/222:30:142
2948599271,15cyclictest28209-21ssh21:35:142
2947599273,15cyclictest21274-21ssh22:25:131
29475992715,7cyclictest21-21ksoftirqd/121:05:131
2947599271,22cyclictest14124-21ssh23:15:131
2946699272,22cyclictest26642-21ssh22:10:150
2946699272,22cyclictest26642-21ssh22:10:140
2949399263,15cyclictest0-21swapper/323:20:143
2949399262,14cyclictest35966irq/58-eth021:35:133
29493992616,6cyclictest0-21swapper/319:30:133
2948599263,12cyclictest27346irq/40-dwc2_hso20:55:132
2948599262,18cyclictest8309-21ssh00:22:562
2948599261,14cyclictest23906-21ls20:45:182
2948599261,13cyclictest10550-21sh22:55:142
29475992613,10cyclictest7217-21ssh23:45:271
2946699262,19cyclictest1203-21sh21:20:150
2946699262,18cyclictest16429-21ssh23:33:550
29466992613,6cyclictest0-21swapper/021:55:110
2946699261,18cyclictest19510-21sh23:00:130
2949399252,17cyclictest36066irq/59-eth021:10:133
2948599252,8cyclictest15063-21diskmemload23:19:562
2948599251,17cyclictest27346irq/40-dwc2_hso20:20:122
2947599253,19cyclictest7058-21ssh00:05:151
2947599251,18cyclictest23087-21ssh23:20:141
29475992510,7cyclictest48-21kcompactd020:50:121
2949399243,13cyclictest0-21swapper/323:35:123
2949399241,14cyclictest35966irq/58-eth000:35:133
2948599242,17cyclictest27346irq/40-dwc2_hso00:15:142
2948599242,17cyclictest27346irq/40-dwc2_hso00:15:142
2948599242,17cyclictest0-21swapper/223:53:572
29485992412,4cyclictest7034-21ntp_states23:45:252
2948599241,16cyclictest14023-21ntp_192.168.11400:25:232
2947599244,14cyclictest24918-21ssh00:15:141
2947599244,14cyclictest24918-21ssh00:15:141
2947599242,18cyclictest30781-21apt-get19:45:121
29475992418,3cyclictest21-21ksoftirqd/122:10:141
29475992418,3cyclictest21-21ksoftirqd/122:10:141
29475992415,6cyclictest21-21ksoftirqd/121:20:151
2946699242,17cyclictest16016-21sh22:40:170
2946699241,18cyclictest15463-21apt-get21:10:130
2949399233,15cyclictest35966irq/58-eth022:35:143
29493992316,5cyclictest0-21swapper/323:18:363
29493992316,5cyclictest0-21swapper/322:10:153
29493992316,5cyclictest0-21swapper/322:10:143
29493992311,8cyclictest0-21swapper/323:00:133
2949399231,16cyclictest35966irq/58-eth023:45:133
2948599239,11cyclictest0-21swapper/200:35:012
2948599232,6cyclictest0-21swapper/223:14:562
2948599231,14cyclictest19670-21latency19:30:222
2946699239,11cyclictest21291-21sh22:25:140
29466992314,5cyclictest844-21munin-node23:55:100
2949399224,8cyclictest0-21swapper/320:35:143
2948599223,14cyclictest27346irq/40-dwc2_hso20:50:132
2948599222,6cyclictest0-21swapper/223:21:562
2948599222,15cyclictest0-21swapper/223:40:572
2948599222,14cyclictest27346irq/40-dwc2_hso22:45:132
2948599221,8cyclictest1961866ssh21:10:302
2948599221,16cyclictest7321-21grep00:05:182
2948599221,16cyclictest29531-21mailstats19:40:262
2948599221,10cyclictest32534-21timerwakeupswit23:05:302
2948599221,10cyclictest32534-21timerwakeupswit23:05:302
2947599227,9cyclictest24503-21ssh21:15:131
2947599222,15cyclictest30320-21ssh22:30:141
29475992215,5cyclictest0-21swapper/122:01:011
29475992213,6cyclictest3301-21ssh23:25:251
29475992213,6cyclictest11834-21cron22:20:011
29475992213,6cyclictest11834-21cron22:20:011
2947599221,2cyclictest131rcu_preempt00:23:391
2947599221,18cyclictest3386-21cut19:50:171
2947599221,18cyclictest26077-21ls20:10:281
29475992210,10cyclictest18215-21ssh21:45:301
2947599220,4cyclictest131rcu_preempt23:02:081
2946699221,18cyclictest32508-21ls23:05:300
2946699221,18cyclictest32508-21ls23:05:300
2946699221,16cyclictest24514-21ssh21:15:150
2949399218,6cyclictest321rcuc/321:45:113
2949399213,15cyclictest0-21swapper/323:27:563
29493992112,6cyclictest35966irq/58-eth000:20:273
2948599213,15cyclictest0-21swapper/200:10:162
2948599213,15cyclictest0-21swapper/200:10:152
2948599212,12cyclictest26613-21ssh22:10:122
2948599212,12cyclictest26613-21ssh22:10:122
2948599211,4cyclictest21403-21timerwakeupswit20:05:292
2948599211,17cyclictest27346irq/40-dwc2_hso22:40:162
2948599211,17cyclictest19045-21cut20:40:172
2948599211,16cyclictest5897-21sed21:00:182
2948599211,12cyclictest25772-21awk19:40:002
2948599211,11cyclictest23103-21tr22:25:212
2947599214,13cyclictest0-21swapper/121:35:131
2947599214,11cyclictest21-21ksoftirqd/119:20:121
2947599211,17cyclictest32021-21ssh21:55:131
2947599211,12cyclictest5621-21ls21:00:021
29475992110,8cyclictest13003-21cat23:30:291
2947599210,16cyclictest0-21swapper/100:10:141
2947599210,16cyclictest0-21swapper/100:10:141
2946699219,3cyclictest131rcu_preempt00:00:030
2946699213,5cyclictest131rcu_preempt19:20:110
2946699211,7cyclictest5212-21fschecks_count23:10:160
2949399209,6cyclictest0-21swapper/323:50:113
2949399205,13cyclictest35966irq/58-eth020:30:133
2949399205,12cyclictest35966irq/58-eth019:45:133
2949399202,12cyclictest0-21swapper/322:15:153
2949399202,12cyclictest0-21swapper/322:15:143
29493992010,8cyclictest0-21swapper/322:50:383
29493992010,7cyclictest0-21swapper/300:16:083
29493992010,7cyclictest0-21swapper/300:16:083
2948599209,5cyclictest27184-21sshd23:03:552
2948599201,9cyclictest6965-21cpu22:35:142
2948599201,9cyclictest2706-21kthreadcore19:15:192
2948599201,16cyclictest930-21snmpd19:55:282
2948599201,16cyclictest0-21swapper/221:05:162
2948599201,15cyclictest32131-21diskstats21:55:152
2948599201,15cyclictest2069-21kthreadcore23:25:202
2948599201,11cyclictest7033-21sed19:20:162
2948599201,10cyclictest20868-21awk22:25:002
29485992010,7cyclictest0-21swapper/222:00:132
29475992013,4cyclictest23066-21nfs4_client00:30:241
2947599201,2cyclictest131rcu_preempt23:07:251
2947599201,2cyclictest131rcu_preempt23:07:251
29475992012,5cyclictest5020-21/usr/sbin/munin23:10:301
29475992012,5cyclictest30487-21/usr/sbin/munin00:00:161
29475992012,5cyclictest26775-21switchtime21:50:291
2947599201,13cyclictest31978-21ssh00:35:251
2946699206,9cyclictest12-21ksoftirqd/000:20:150
2946699206,10cyclictest28798-21cut21:35:210
2946699205,11cyclictest0-21swapper/022:55:130
29466992012,4cyclictest12653-21hwlatdetect23:50:160
29466992010,7cyclictest21381-21ntpq23:35:240
29466992010,6cyclictest0-21swapper/020:30:120
2946699200,17cyclictest17450-21ls20:05:010
2949399199,7cyclictest35966irq/58-eth000:30:143
2949399192,11cyclictest0-21swapper/320:10:133
29493991913,4cyclictest0-21swapper/323:30:203
29493991913,4cyclictest0-21swapper/322:30:153
29493991913,4cyclictest0-21swapper/321:50:213
29493991913,4cyclictest0-21swapper/321:15:253
29493991912,5cyclictest0-21swapper/323:08:123
29493991912,5cyclictest0-21swapper/323:08:113
2949399191,12cyclictest0-21swapper/319:50:163
2948599198,8cyclictest14757-21kthreadcore20:00:212
2948599193,9cyclictest27346irq/40-dwc2_hso19:25:122
2948599192,11cyclictest27346irq/40-dwc2_hso20:15:122
2948599192,11cyclictest27346irq/40-dwc2_hso20:15:122
2948599191,9cyclictest31220-21irqcore19:45:172
2948599191,8cyclictest29857-21hwlatdetect19:10:172
2948599191,5cyclictest0-21swapper/221:25:122
2948599191,14cyclictest3234-21cpu19:50:142
2947599192,14cyclictest976-21sshd23:42:391
2947599191,11cyclictest10462-21cpu00:25:131
29475991910,7cyclictest26147-21sshd22:09:561
29475991910,7cyclictest17655-21apt-get20:05:131
2946699199,7cyclictest2189-21cat22:30:320
2946699191,9cyclictest17580-21/usr/sbin/munin22:05:250
2946699191,9cyclictest1066-21fschecks_time20:55:170
29466991914,3cyclictest19081-21kworker/0:119:10:120
29466991912,5cyclictest0-21swapper/019:45:230
29466991911,6cyclictest26072-21sshd21:32:560
2946699191,12cyclictest3222-21find19:50:120
2949399186,5cyclictest0-21swapper/319:25:123
2949399183,11cyclictest0-21swapper/322:40:173
2949399182,10cyclictest0-21swapper/320:50:133
2949399182,10cyclictest0-21swapper/319:10:143
29493991813,3cyclictest0-21swapper/322:20:263
29493991812,4cyclictest0-21swapper/300:05:223
2949399181,10cyclictest0-21swapper/319:20:023
29493991810,5cyclictest0-21swapper/322:05:183
2948599181,14cyclictest29511-21latency_hist19:10:012
2947599188,6cyclictest0-21swapper/120:15:131
2947599188,6cyclictest0-21swapper/120:15:121
2947599188,2cyclictest0-21swapper/120:30:241
2947599185,10cyclictest2099-21fschecks_count19:15:161
2947599184,5cyclictest21-21ksoftirqd/120:45:301
2947599184,12cyclictest8013-21apt-get19:55:111
2947599182,13cyclictest10557-21ssh22:35:261
2947599181,9cyclictest732-21mailstats19:10:281
29475991811,4cyclictest13008-21kthreadcore22:20:201
2947599180,2cyclictest131rcu_preempt19:25:291
2947599180,15cyclictest29506-21ls19:10:011
2947599180,12cyclictest32229-21cut20:20:151
2946699189,7cyclictest7156-21df_inode00:05:160
2946699189,6cyclictest16175-21cron19:30:020
2946699185,9cyclictest0-21swapper/020:25:140
2946699181,9cyclictest602-21munin-run20:55:010
2946699181,14cyclictest8496-21sed21:40:250
29466991811,4cyclictest25650-21kthreadcore00:15:200
29466991811,4cyclictest25650-21kthreadcore00:15:200
29466991810,5cyclictest10166-21cpu21:25:140
2949399172,10cyclictest0-21swapper/320:20:133
2949399171,10cyclictest0-21swapper/320:00:293
2947599175,9cyclictest6329-21kernelversion21:00:201
2947599171,13cyclictest22004-21kthreadcore20:40:231
2947599171,13cyclictest19896-21processes19:30:271
2946699178,7cyclictest27545-21if_eth120:15:180
2946699178,7cyclictest27545-21if_eth120:15:170
2946699175,9cyclictest15057-21cron21:10:010
2946699174,10cyclictest22846-21ssh21:50:130
2946699173,11cyclictest5691-21timerandwakeup19:15:280
2946699172,4cyclictest12-21ksoftirqd/000:10:570
2946699172,4cyclictest12-21ksoftirqd/000:10:570
2946699172,11cyclictest0-21swapper/019:35:160
2946699172,10cyclictest0-21swapper/019:40:130
2946699171,9cyclictest0-21swapper/020:35:160
2946699171,12cyclictest1798-21perl22:50:170
2949399163,9cyclictest0-21swapper/319:40:113
2949399162,9cyclictest0-21swapper/320:45:153
2949399162,9cyclictest0-21swapper/319:55:123
2947599169,5cyclictest9024-21sshd20:29:571
2947599164,10cyclictest14589-21kthreadcore20:00:221
29475991611,3cyclictest0-21swapper/119:40:221
2947599161,11cyclictest21332-21fschecks_count19:35:171
2946699169,5cyclictest986-21kthreadcore20:20:200
2946699169,5cyclictest13832-21tr19:25:210
2946699164,9cyclictest22181-21grep20:40:240
2946699162,11cyclictest21487-21sed20:05:300
2946699161,11cyclictest24945-21cpu22:45:130
2949399158,4cyclictest0-21swapper/319:35:133
2949399151,11cyclictest0-21swapper/321:00:293
2946699151,11cyclictest9789-21kthreadcore19:55:200
2949399142,9cyclictest0-21swapper/320:25:143
2949399141,11cyclictest0-21swapper/319:25:013
2946699131,9cyclictest29504-21latency_hist19:10:020
2949399122,6cyclictest0-21swapper/320:40:153
2949399102,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional