You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 19:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Mar 16, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1709699634,47cyclictest28861-21ssh10:00:121
17121995929,19cyclictest28366irq/58-eth009:10:133
17121995929,19cyclictest28366irq/58-eth009:10:123
1712199575,36cyclictest28366irq/58-eth011:50:123
1712199575,36cyclictest28366irq/58-eth011:50:123
1709699552,40cyclictest8569-21ssh11:55:121
1708399523,44cyclictest26608-21ssh12:05:130
17121995126,13cyclictest28366irq/58-eth012:05:113
1708399511,43cyclictest16044-21ssh09:35:120
17121994838,5cyclictest28366irq/58-eth010:25:123
1712199482,29cyclictest28366irq/58-eth007:45:133
1711199413,27cyclictest3293-21ssh09:10:142
1711199413,27cyclictest3293-21ssh09:10:132
17096994114,22cyclictest21-21ksoftirqd/111:15:111
17083994113,22cyclictest12-21ksoftirqd/010:20:130
17111994012,21cyclictest27-21ksoftirqd/210:15:132
1708399402,32cyclictest6999-21ssh09:30:130
1711199392,20cyclictest14987-21rm11:57:232
17111993813,18cyclictest27-21ksoftirqd/209:50:132
1711199373,21cyclictest22346irq/40-dwc2_hso08:35:122
1709699375,7cyclictest131rcu_preempt09:00:121
1708399371,3cyclictest131rcu_preempt12:20:120
17111993611,16cyclictest27-21ksoftirqd/212:25:112
1711199352,21cyclictest27-21ksoftirqd/212:05:122
1711199341,26cyclictest21642-21ssh09:36:232
1708399342,25cyclictest25117-21ssh09:40:140
1712199332,20cyclictest28366irq/58-eth010:15:123
1709699332,18cyclictest21-21ksoftirqd/108:25:131
1708399333,24cyclictest5156-21ssh11:35:130
1709699325,18cyclictest6840-21ssh12:30:111
1712199312,10cyclictest0-21swapper/309:05:133
1711199311,15cyclictest6702-21rm09:45:392
1709699312,21cyclictest30587-21ssh10:55:131
17096993117,8cyclictest21-21ksoftirqd/109:31:221
1712199302,14cyclictest28366irq/58-eth012:30:113
1711199303,18cyclictest22346irq/40-dwc2_hso11:35:142
1709699307,13cyclictest21-21ksoftirqd/107:25:131
1709699302,21cyclictest14374-21/usr/sbin/munin09:50:221
1709699302,19cyclictest32019-21ssh11:50:131
1709699302,19cyclictest32019-21ssh11:50:131
17121992913,10cyclictest28366irq/58-eth009:30:123
1711199292,20cyclictest0-21swapper/207:35:142
1711199292,11cyclictest14917-21ssh12:34:232
1711199291,20cyclictest9189-21ssh10:25:132
1709699291,23cyclictest23093-21ssh11:45:141
1708399292,22cyclictest5330-21ntp_192.168.11409:45:230
1708399292,20cyclictest21359-21ssh09:20:140
1712199283,18cyclictest0-21swapper/311:34:243
1712199283,17cyclictest28366irq/58-eth011:10:143
1711199283,20cyclictest0-21swapper/210:09:222
1711199283,17cyclictest22346irq/40-dwc2_hso11:30:122
1711199283,17cyclictest12254-21ssh09:15:122
1711199281,17cyclictest10571-21apt-get11:20:122
17096992812,8cyclictest17069-21cyclictest08:10:141
1712199273,18cyclictest0-21swapper/309:35:443
1712199273,16cyclictest0-21swapper/311:58:233
1711199272,21cyclictest0-21swapper/211:43:232
1711199272,18cyclictest3165-21grep07:25:292
1709699272,19cyclictest30691-21ssh09:41:231
1709699272,19cyclictest20070-21ssh10:11:241
1709699271,20cyclictest21269-21ssh12:20:131
17096992711,8cyclictest21-21ksoftirqd/110:25:371
17083992716,5cyclictest0-21swapper/010:25:140
1708399271,21cyclictest14177-21ssh11:40:150
1712199262,17cyclictest0-21swapper/309:45:463
17121992615,8cyclictest0-21swapper/309:55:133
17121992612,9cyclictest28366irq/58-eth010:20:043
1712199261,12cyclictest28366irq/58-eth011:05:133
1711199263,15cyclictest22346irq/40-dwc2_hso09:20:122
1711199262,19cyclictest9659-21ssh11:19:242
1711199262,16cyclictest22346irq/40-dwc2_hso07:30:112
1711199261,19cyclictest27100-21ls10:35:142
1709699262,20cyclictest1712-21ssh09:45:131
1709699262,19cyclictest2889-21diskmemload10:05:401
1709699262,19cyclictest27850-21ssh09:59:231
1709699261,23cyclictest15851-21ssh12:35:131
1708399262,18cyclictest12532-21ssh10:45:130
1708399262,16cyclictest30294-21ssh09:25:110
1708399261,19cyclictest14986-21sh11:57:230
1708399261,18cyclictest19526-21ssh12:35:250
1712199253,16cyclictest0-21swapper/310:10:233
1712199253,15cyclictest0-21swapper/309:41:233
1711199252,18cyclictest24216-21ssh11:09:232
1711199252,13cyclictest1815-21switchtime09:25:272
17111992514,5cyclictest29198-21ssh11:47:062
1709699252,6cyclictest131rcu_preempt08:20:131
1709699252,18cyclictest4333-21ssh09:27:231
1709699252,18cyclictest22465-21ssh09:37:241
1709699252,15cyclictest3284-21ssh09:10:121
1709699252,15cyclictest3284-21ssh09:10:121
1708399258,4cyclictest131rcu_preempt11:30:190
1708399251,3cyclictest131rcu_preempt11:45:140
1712199244,12cyclictest28366irq/58-eth011:45:143
1711199246,8cyclictest18599-21grep10:10:292
1711199242,17cyclictest0-21swapper/212:23:242
1711199242,17cyclictest0-21swapper/209:34:232
1711199242,16cyclictest22346irq/40-dwc2_hso08:00:132
17111992416,3cyclictest32597-21sh10:20:122
17111992411,11cyclictest1-21systemd07:20:022
1711199240,8cyclictest0-21swapper/210:56:042
17096992412,10cyclictest21-21ksoftirqd/111:10:141
1708399244,17cyclictest151rcuc/009:50:130
1708399243,16cyclictest3192-21ssh12:10:140
1708399243,16cyclictest3192-21ssh12:10:140
1708399242,16cyclictest23478-21ssh10:15:120
1708399241,7cyclictest12670-21irqcore12:15:180
1708399241,5cyclictest131rcu_preempt12:30:120
1712199233,15cyclictest28466irq/59-eth009:20:143
1712199231,11cyclictest28366irq/58-eth010:55:133
1711199232,6cyclictest0-21swapper/212:18:232
1711199232,6cyclictest0-21swapper/209:44:242
1711199232,16cyclictest0-21swapper/212:38:232
1711199232,16cyclictest0-21swapper/211:04:232
1711199232,16cyclictest0-21swapper/210:34:232
1711199231,9cyclictest28730-21ssh10:00:042
1711199231,5cyclictest0-21swapper/207:40:142
1709699238,10cyclictest21653-21ssh10:30:241
1709699236,10cyclictest21-21ksoftirqd/109:05:131
1709699232,18cyclictest16530-21ls08:50:171
1709699232,17cyclictest3620-21ssh10:40:141
1709699232,17cyclictest3620-21ssh10:40:141
1709699231,14cyclictest16168-21ssh11:05:131
17096992310,3cyclictest131rcu_preempt10:39:551
1708399231,8cyclictest10839-21ntp_states11:00:250
17083992312,8cyclictest30189-21cron10:55:010
1712199223,17cyclictest0-21swapper/310:03:233
1712199223,13cyclictest0-21swapper/311:37:233
17121992212,6cyclictest28366irq/58-eth010:45:133
17121992210,9cyclictest0-21swapper/312:16:553
17121992210,9cyclictest0-21swapper/312:00:313
1711199223,16cyclictest31603-21kthreadcore08:30:192
1711199223,16cyclictest1546-21kthreadcore11:50:202
1711199223,16cyclictest1546-21kthreadcore11:50:202
1711199223,15cyclictest26061-21fschecks_count09:00:162
1711199223,12cyclictest28183-21cron11:30:022
1711199222,6cyclictest0-21swapper/210:45:132
1709699223,13cyclictest19574-21ssh11:25:141
1709699222,14cyclictest1713-21ls08:35:011
17096992214,6cyclictest0-21swapper/111:35:221
17096992214,5cyclictest28528-21/usr/sbin/munin11:30:181
1709699221,17cyclictest3177-21apt-get12:10:131
1709699221,17cyclictest3177-21apt-get12:10:131
17096992210,10cyclictest0-21swapper/110:21:511
1708399221,19cyclictest11647-21cut08:45:160
1708399221,15cyclictest5469-21ls08:05:130
1712199213,12cyclictest28366irq/58-eth010:05:133
17121992112,6cyclictest0-21swapper/310:35:013
17121992112,4cyclictest0-21swapper/309:25:123
17121992111,7cyclictest28366irq/58-eth010:40:133
17121992111,7cyclictest28366irq/58-eth010:40:133
1711199213,5cyclictest0-21swapper/207:55:122
1711199213,11cyclictest3157-21/usr/sbin/munin12:10:282
1711199213,11cyclictest3157-21/usr/sbin/munin12:10:272
1711199212,15cyclictest17237-21latency_hist07:10:022
1711199212,13cyclictest24721-21cpu08:25:142
1711199211,6cyclictest2769166ssh09:59:112
1711199211,6cyclictest21273-21cut08:55:152
1711199211,10cyclictest19753-21latency_hist08:20:012
17096992112,5cyclictest18437-21kthreadcore12:00:211
1709699211,15cyclictest2117-21ls08:35:151
1709699211,12cyclictest2889-21diskmemload10:50:131
1708399214,3cyclictest131rcu_preempt10:30:120
1708399214,3cyclictest0-21swapper/012:29:520
1708399212,16cyclictest24235-21kthreadcore08:55:210
1708399212,14cyclictest23390-21ssh11:08:240
17083992112,7cyclictest14848-21sshd08:14:230
1708399211,16cyclictest27041-21ssh10:35:120
1712199208,8cyclictest28366irq/58-eth009:15:123
1712199201,11cyclictest0-21swapper/307:40:133
17121992010,8cyclictest0-21swapper/312:25:143
1711199207,10cyclictest21706-21cpuspeed10:50:162
1711199202,16cyclictest12034-21kthreadcore08:10:222
1711199201,16cyclictest10193-21latency_hist08:10:012
1711199201,12cyclictest2499266ssh11:10:032
1711199201,11cyclictest924-21snmpd09:05:122
1711199201,11cyclictest22161-21wc07:45:272
1711199201,10cyclictest3736-21entropy10:40:172
1711199201,10cyclictest3736-21entropy10:40:162
1711199201,10cyclictest27583-21irqrtprio07:20:182
1711199201,10cyclictest26941-21grep07:50:262
1711199201,10cyclictest17647-21/usr/sbin/munin12:00:172
1709699201,16cyclictest345-21cron08:00:021
1709699201,11cyclictest12176-21ssh12:15:121
17096992010,7cyclictest27997-21cron07:55:011
1708399207,3cyclictest17166systemd-network08:03:260
1708399202,15cyclictest7701-21grep10:40:300
1708399202,15cyclictest7701-21grep10:40:300
1708399202,15cyclictest25291-21fschecks_count11:10:160
1708399202,15cyclictest1820-21proc_pri09:05:260
1708399202,14cyclictest28976-21ssh10:00:140
1708399202,14cyclictest14079-21memory11:20:220
1708399202,11cyclictest17285-21/usr/sbin/munin07:10:270
1708399201,5cyclictest0-21swapper/011:54:200
1708399201,5cyclictest0-21swapper/011:54:190
1708399201,4cyclictest0-21swapper/010:06:510
1708399201,16cyclictest5912-21kthreadcore07:30:190
1708399201,16cyclictest1796-21ntp_192.168.11410:55:240
1708399201,16cyclictest16500-21grep08:50:160
1708399201,16cyclictest15539-21kthreadcore09:15:210
1708399201,16cyclictest11508-21kthreadcore07:35:210
1712199199,8cyclictest0-21swapper/312:22:223
1712199193,11cyclictest0-21swapper/312:35:233
1712199192,10cyclictest0-21swapper/307:15:133
17121991911,5cyclictest28366irq/58-eth009:50:123
1712199191,13cyclictest0-21swapper/308:00:163
1712199191,13cyclictest0-21swapper/307:35:283
1712199191,12cyclictest0-21swapper/308:45:273
1711199191,9cyclictest23679-21timerwakeupswit08:20:282
1711199191,9cyclictest12506-21kthreadcore08:45:192
1711199191,10cyclictest11347-21grep08:45:012
1711199191,10cyclictest11347-21grep08:45:012
1709699199,2cyclictest21-21ksoftirqd/111:02:251
1709699192,2cyclictest131rcu_preempt11:43:131
1709699191,3cyclictest131rcu_preempt12:25:121
1709699191,16cyclictest29949-21kthreadcore12:05:221
1709699191,15cyclictest15326-21ls08:15:161
1709699191,13cyclictest14202-21ssh11:20:251
1709699191,12cyclictest20647-21missed_timers07:10:231
1709699191,10cyclictest15600-21kthreadcore09:15:211
1709699190,16cyclictest24777-21ls09:20:231
1708399192,15cyclictest27268-21kthreadcore08:25:230
1708399192,14cyclictest26569-21irqrtprio09:00:170
1708399192,10cyclictest3294-21ssh09:10:130
1708399192,10cyclictest3294-21ssh09:10:130
1708399191,16cyclictest5008-21kthreadcore11:15:200
1708399191,16cyclictest10320-21switchtime08:40:290
1708399191,16cyclictest10320-21switchtime08:40:280
1708399191,15cyclictest4327-21kthreadcore08:35:210
1708399191,15cyclictest3026-21switchtime07:25:270
1708399191,15cyclictest29594-21kthreadcore07:55:190
1708399191,15cyclictest22651-21kthreadcore08:20:220
1708399191,14cyclictest21384-21ssh12:00:260
1712199189,6cyclictest0-21swapper/312:10:213
1712199189,6cyclictest0-21swapper/312:10:213
1712199182,11cyclictest0-21swapper/311:43:243
1712199182,11cyclictest0-21swapper/311:15:103
1712199182,10cyclictest0-21swapper/308:30:123
1712199181,12cyclictest0-21swapper/308:10:013
1712199181,11cyclictest0-21swapper/309:05:013
17121991810,5cyclictest28366irq/58-eth011:00:253
1711199181,8cyclictest16801-21irqrtprio08:50:192
1711199181,14cyclictest17429-21fschecks_count07:10:152
1709699181,9cyclictest1881-21kthreadcore08:00:191
1709699181,2cyclictest131rcu_preempt07:40:331
1709699181,13cyclictest12632-21ssh10:45:141
1708399189,6cyclictest22733-21kthreadcore09:55:210
1708399186,9cyclictest1-21systemd07:45:060
17083991812,4cyclictest0-21swapper/008:30:220
1708399181,15cyclictest22347-21fschecks_time07:15:150
1708399181,15cyclictest18234-21kthreadcore08:15:210
1708399181,14cyclictest20036-21ssh11:25:190
1708399181,14cyclictest16975-21apt-get07:10:020
1708399181,14cyclictest13960-21fschecks_time07:40:170
1712199179,6cyclictest0-21swapper/310:52:403
1712199179,6cyclictest0-21swapper/310:35:423
1712199179,3cyclictest0-21swapper/311:26:393
1712199172,10cyclictest0-21swapper/311:20:133
1712199172,10cyclictest0-21swapper/308:35:123
1712199172,10cyclictest0-21swapper/307:10:143
1712199171,11cyclictest0-21swapper/307:30:273
1709699177,8cyclictest11603-21df_inode08:45:161
1709699174,4cyclictest131rcu_preempt10:17:161
1709699173,12cyclictest28508-21kthreadcore07:20:211
1709699173,12cyclictest22222-21kthreadcore08:55:191
1709699172,10cyclictest0-21swapper/108:40:141
1709699172,10cyclictest0-21swapper/108:40:141
1709699171,13cyclictest26893-21cat07:20:011
1708399178,7cyclictest14689-21fw_packets10:10:170
1708399171,14cyclictest25597-21kthreadcore07:50:210
1712199161,12cyclictest0-21swapper/308:25:023
1712199161,10cyclictest0-21swapper/308:10:243
1709699163,10cyclictest6138-21kthreadcore07:30:191
1709699162,10cyclictest20967-21kthreadcore07:45:211
1709699161,13cyclictest5168-21kworker/1:008:05:221
17096991610,4cyclictest0-21swapper/107:35:161
1708399161,12cyclictest29732-21kthreadcore07:20:210
1712199151,8cyclictest0-21swapper/307:55:143
17121991510,3cyclictest0-21swapper/308:52:233
1709699153,9cyclictest17098-21apt-config07:10:011
1712199143,8cyclictest0-21swapper/308:40:283
1712199143,8cyclictest0-21swapper/308:40:283
1712199142,7cyclictest0-21swapper/307:50:123
1712199141,7cyclictest0-21swapper/308:15:223
1712199141,7cyclictest0-21swapper/307:25:173
1712199138,3cyclictest0-21swapper/307:20:153
1712199131,7cyclictest0-21swapper/308:55:153
1712199131,7cyclictest0-21swapper/308:25:243
1712199123,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional