You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 06:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun May 24, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2273199761,58cyclictest24791-21ssh22:30:130
22762996839,18cyclictest35966irq/58-eth023:05:123
2275099624,38cyclictest11755-21ssh22:05:132
22750995914,20cyclictest27-21ksoftirqd/222:40:132
22750995914,20cyclictest27-21ksoftirqd/222:40:132
22731995534,12cyclictest12-21ksoftirqd/021:20:120
22762995313,27cyclictest35966irq/58-eth022:10:133
2273199523,36cyclictest20818-21sh22:10:120
2276299515,29cyclictest35966irq/58-eth023:20:133
2273199517,5cyclictest131rcu_preempt21:05:140
2275099504,16cyclictest27346irq/40-dwc2_hso00:15:152
2273199460,36cyclictest17079-21ssh23:55:140
2273199460,36cyclictest17079-21ssh23:55:140
2273199451,31cyclictest29891-21ssh22:15:130
2276299434,25cyclictest35966irq/58-eth021:30:123
2275099431,20cyclictest30495-21cut22:15:192
2275099421,16cyclictest11635-21latency_hist00:10:022
2273199420,33cyclictest10606-21ssh22:40:120
2273199420,33cyclictest10606-21ssh22:40:110
2275099413,24cyclictest930-21snmpd22:45:152
2275099411,15cyclictest14152-21ps20:40:202
2276299405,23cyclictest35966irq/58-eth020:10:123
2275099405,25cyclictest9341-21ssh23:15:152
2273199402,35cyclictest28776-21ssh22:50:120
2275099391,14cyclictest27627-21grep19:45:232
2276299382,22cyclictest35966irq/58-eth020:25:113
2275099383,15cyclictest8857-21unixbench_singl19:25:292
2275099383,15cyclictest8857-21unixbench_singl19:25:282
2275099382,10cyclictest27450-21latency_hist19:15:012
2275099381,10cyclictest14408-21kthreadcore21:30:182
22731993822,7cyclictest0-21swapper/022:20:140
2273199381,29cyclictest26011-21ssh21:55:120
2276299374,20cyclictest35966irq/58-eth020:45:123
2276299372,10cyclictest0-21swapper/319:25:143
2276299372,10cyclictest0-21swapper/319:25:133
22762993714,16cyclictest35966irq/58-eth022:55:133
2275099371,9cyclictest17223-21fschecks_count23:55:162
2275099371,9cyclictest17223-21fschecks_count23:55:162
2273199373,24cyclictest8482-21diskmemload22:35:130
2276299367,17cyclictest35966irq/58-eth019:45:143
2275099367,6cyclictest0-21swapper/200:31:072
2275099361,9cyclictest29216-21kthreadcore00:00:222
2275099361,16cyclictest25109-21fschecks_count00:35:172
2275099361,16cyclictest22561-21kthreadcore22:10:202
2273199360,31cyclictest27490-21ssh23:25:120
2276299352,19cyclictest35966irq/58-eth023:40:133
2275099351,8cyclictest27122-21/usr/sbin/munin23:05:232
2275099351,22cyclictest18120-21ssh21:15:152
2275099351,15cyclictest30117-21cpu00:20:152
2275099351,15cyclictest17458-21irqcore21:50:182
2275099351,14cyclictest14403-21kthreadcore23:35:202
2274299352,26cyclictest2840-21ssh00:05:141
2273199353,18cyclictest12-21ksoftirqd/023:40:140
2276299342,23cyclictest35966irq/58-eth022:20:143
2275099341,8cyclictest26130-21sh23:40:292
2275099341,16cyclictest14202-21kthreadcore20:05:222
2275099341,15cyclictest18551-21grep19:35:292
2275099336,17cyclictest6792-21ssh00:25:132
2275099331,8cyclictest29278-21kthreadcore20:55:202
2275099331,8cyclictest28430-21kthreadcore21:20:202
2275099331,8cyclictest21912-21memory23:20:222
2275099331,14cyclictest20568-21latency_hist20:15:022
2274299333,18cyclictest14500-21ssh23:00:131
22742993314,11cyclictest131rcu_preempt00:30:131
2276299311,20cyclictest35966irq/58-eth022:30:133
2275099316,15cyclictest27346irq/40-dwc2_hso21:25:142
22762993014,8cyclictest0-21swapper/319:40:143
22762993014,8cyclictest0-21swapper/319:40:143
2275099305,14cyclictest27346irq/40-dwc2_hso21:35:132
2275099303,17cyclictest27346irq/40-dwc2_hso19:20:132
2275099301,19cyclictest5366-21ssh22:55:142
2273199302,4cyclictest131rcu_preempt19:30:130
22762992917,8cyclictest0-21swapper/320:40:123
2274299295,7cyclictest131rcu_preempt00:13:061
2274299291,6cyclictest131rcu_preempt20:10:111
2274299291,25cyclictest5377-21sh22:55:131
22731992911,9cyclictest26557-21kworker/u8:123:05:130
2276299282,14cyclictest35966irq/58-eth021:20:133
2276299282,10cyclictest0-21swapper/300:31:073
2275099282,17cyclictest31185-21ssh21:40:132
2275099282,15cyclictest27346irq/40-dwc2_hso19:15:132
2275099282,11cyclictest0-21swapper/221:10:142
2275099281,16cyclictest11976-21ssh00:10:152
2275099281,11cyclictest24782-21ssh22:30:122
2274299281,21cyclictest24779-21apt-get22:30:131
22731992817,6cyclictest0-21swapper/000:00:140
2276299273,16cyclictest35966irq/58-eth021:25:133
2276299272,14cyclictest35966irq/58-eth021:50:123
2276299271,14cyclictest35966irq/58-eth022:40:123
2276299271,14cyclictest35966irq/58-eth022:40:113
2275099276,11cyclictest27346irq/40-dwc2_hso20:30:132
2275099275,14cyclictest27346irq/40-dwc2_hso21:45:172
2275099275,12cyclictest27346irq/40-dwc2_hso22:25:132
2275099273,14cyclictest27346irq/40-dwc2_hso19:30:132
2275099272,18cyclictest0-21swapper/223:10:142
2274299274,18cyclictest21-21ksoftirqd/119:15:131
2273199272,19cyclictest24970-21ssh00:35:150
2273199271,17cyclictest844-21munin-node20:20:150
2275099265,12cyclictest0-21swapper/223:30:162
2274299268,11cyclictest22723-21cyclictest20:55:131
2274299264,19cyclictest21-21ksoftirqd/119:20:121
2274299262,20cyclictest12912-21ssh21:30:131
2274299262,19cyclictest19693-21ssh22:45:161
22742992615,6cyclictest0-21swapper/122:35:131
2274299261,19cyclictest0-21swapper/121:15:131
22731992614,7cyclictest121ksoftirqd/019:25:130
22731992614,7cyclictest121ksoftirqd/019:25:130
22750992512,10cyclictest25976-21/usr/sbin/munin21:55:122
2274299254,14cyclictest21-21ksoftirqd/120:05:141
2274299252,20cyclictest18409-21ssh23:20:131
2274299251,21cyclictest31160-21ssh21:40:121
2274299251,16cyclictest31113-21ssh23:25:251
2273199251,20cyclictest2719-21ssh22:00:160
2276299244,15cyclictest0-21swapper/300:15:143
2276299243,10cyclictest0-21swapper/322:05:133
2275099249,12cyclictest28401-21kthreadcore23:25:192
2274299243,3cyclictest131rcu_preempt22:20:341
22742992411,11cyclictest21-21ksoftirqd/100:20:131
22742992410,3cyclictest0-21swapper/119:10:161
2273199242,18cyclictest6816-21ssh00:25:140
22731992413,7cyclictest0-21swapper/023:35:130
2276299232,15cyclictest0-21swapper/300:25:143
2275099233,13cyclictest27346irq/40-dwc2_hso20:15:142
2275099231,19cyclictest25677-21grep20:50:282
2275099231,19cyclictest25677-21grep20:50:282
2275099231,13cyclictest6971-21grep22:20:182
2275099231,10cyclictest28969-21fschecks_time22:50:162
22742992312,7cyclictest0-21swapper/119:50:131
2274299231,18cyclictest32683-21ssh23:10:141
2274299231,16cyclictest22061-21ssh21:35:131
2273199232,17cyclictest21050-21ssh00:15:150
2273199231,9cyclictest2212266ssh23:20:260
22762992213,4cyclictest0-21swapper/323:15:153
2276299221,12cyclictest35966irq/58-eth022:25:123
2275099226,13cyclictest6306-21cpu20:00:142
2275099225,14cyclictest32729-21kthreadcore21:00:182
2275099223,9cyclictest1638-21fschecks_time19:55:172
2275099222,6cyclictest0-21swapper/223:45:152
2275099222,6cyclictest0-21swapper/223:45:152
2275099221,17cyclictest31807-21kthreadcore19:50:222
2275099221,11cyclictest18201-21ntpq23:00:252
2275099221,11cyclictest17190-21fschecks_count20:45:162
2275099221,10cyclictest6797-21tail22:00:322
2275099221,10cyclictest10251-21sh22:40:012
2274299225,10cyclictest0-21swapper/100:25:131
22742992214,6cyclictest0-21swapper/123:35:171
2274299221,17cyclictest28796-21ssh22:50:141
2274299221,17cyclictest14956-21ps19:35:181
2274299221,16cyclictest9609-21sh23:15:181
2274299221,13cyclictest22444-21cut23:40:171
2273199222,14cyclictest5648-21ssh00:24:070
2273199221,18cyclictest31337-21cut21:40:160
2273199221,17cyclictest26543-21ls20:55:010
2273199221,17cyclictest26543-21ls20:55:010
2273199221,12cyclictest26926-21users21:35:310
2273199220,15cyclictest21777-21ls21:35:020
22762992110,8cyclictest0-21swapper/321:42:563
2275099217,11cyclictest19819-21irqcore19:40:192
2275099217,11cyclictest19819-21irqcore19:40:192
2275099212,16cyclictest8820-21kthreadcore20:35:192
2275099212,16cyclictest30806-21irqrtprio20:25:182
2275099211,11cyclictest12073-21timerwakeupswit23:50:302
2275099211,11cyclictest12073-21timerwakeupswit23:50:302
2275099211,10cyclictest5157-21kthreadcore21:05:212
2274299212,14cyclictest7834-21apt-get21:45:161
22742992112,5cyclictest21334-21idleruntime00:15:191
2274299211,16cyclictest4183-21ssh23:30:161
2274299211,15cyclictest15713-21ssh22:25:131
2274299210,18cyclictest11571-21ls23:50:241
2274299210,18cyclictest11571-21ls23:50:231
2273199219,3cyclictest131rcu_preempt00:35:010
2273199211,13cyclictest7821-21find21:45:150
2273199210,2cyclictest131rcu_preempt21:25:140
2276299204,7cyclictest35966irq/58-eth000:05:133
2276299203,11cyclictest35966irq/58-eth022:45:163
2276299201,9cyclictest35966irq/58-eth023:00:143
22762992010,8cyclictest0-21swapper/322:16:503
2275099205,12cyclictest27-21ksoftirqd/219:10:012
2275099201,10cyclictest25918-21irqcore20:20:192
2274299209,8cyclictest20602-21cat23:55:241
2274299209,8cyclictest20602-21cat23:55:241
2274299208,10cyclictest0-21swapper/119:25:231
2274299208,10cyclictest0-21swapper/119:25:231
2274299207,10cyclictest0-21swapper/120:15:311
2274299204,4cyclictest131rcu_preempt21:10:131
2274299202,7cyclictest25627-21awk21:55:011
2274299202,14cyclictest24960-21ssh00:35:151
2274299201,15cyclictest29976-21ls22:15:151
2274299201,15cyclictest24283-21apt-get19:45:121
2274299201,13cyclictest29489-21latency_hist22:15:021
2274299201,13cyclictest12406-21uniq20:40:151
2274299200,17cyclictest31664-21ls23:45:191
2274299200,17cyclictest31664-21ls23:45:181
2273199209,8cyclictest30313-21apt-get20:25:130
2273199205,8cyclictest12-21ksoftirqd/019:40:140
2273199205,8cyclictest12-21ksoftirqd/019:40:140
2273199202,14cyclictest26711-21apt-get20:55:120
2273199201,4cyclictest131rcu_preempt22:55:140
2273199201,14cyclictest16988-21cpuspeed21:50:140
2273199201,14cyclictest14506-21ssh23:00:130
2276299198,9cyclictest0-21swapper/321:19:073
2276299198,7cyclictest35966irq/58-eth023:10:133
2276299196,8cyclictest0-21swapper/300:20:153
2276299193,11cyclictest0-21swapper/300:38:173
2276299193,10cyclictest0-21swapper/322:50:063
2276299192,12cyclictest0-21swapper/323:55:133
2276299192,12cyclictest0-21swapper/323:55:123
2276299192,11cyclictest0-21swapper/320:05:133
2276299192,10cyclictest0-21swapper/320:55:133
2276299192,10cyclictest0-21swapper/319:35:143
22762991911,4cyclictest0-21swapper/321:12:453
2274299199,7cyclictest7468-21cat22:00:341
2274299198,8cyclictest28123-21cat23:05:301
2274299198,8cyclictest25990-21cat21:55:121
2274299198,2cyclictest131rcu_preempt21:25:301
2274299194,12cyclictest30045-21cat00:00:281
2274299193,3cyclictest131rcu_preempt19:55:251
2274299192,15cyclictest12065-21kthreadcore19:30:201
2274299192,14cyclictest12318-21sed22:05:191
2274299192,12cyclictest0-21swapper/120:00:131
2274299191,9cyclictest22713-21kthreadcore20:50:191
2274299191,9cyclictest22713-21kthreadcore20:50:191
2274299191,12cyclictest31066-21ls21:20:291
22742991910,6cyclictest14849-21cat22:40:291
22742991910,6cyclictest14849-21cat22:40:291
2273199192,12cyclictest15734-21ssh22:25:130
22731991912,4cyclictest0-21swapper/023:50:190
22731991912,4cyclictest0-21swapper/023:50:190
22731991911,5cyclictest4841-21kthreadcore00:05:220
2273199190,17cyclictest15734-21ls20:10:010
2276299189,6cyclictest35966irq/58-eth023:30:163
2276299189,6cyclictest0-21swapper/300:10:223
2276299186,8cyclictest35966irq/58-eth000:00:143
2276299182,10cyclictest0-21swapper/319:20:133
2276299182,10cyclictest0-21swapper/319:15:143
22762991813,3cyclictest0-21swapper/321:45:323
2276299181,12cyclictest35966irq/58-eth021:35:133
2276299181,12cyclictest0-21swapper/319:50:163
2276299181,10cyclictest0-21swapper/321:05:023
2274299186,4cyclictest19997-21grep19:40:191
2274299186,4cyclictest19997-21grep19:40:191
2274299181,13cyclictest7480-21apt-get20:35:121
2273199185,10cyclictest32339-21cron19:20:010
2273199184,10cyclictest15083-21ssh00:10:230
2273199182,2cyclictest131rcu_preempt19:50:140
2273199181,8cyclictest679-21sed23:10:180
2273199181,8cyclictest5829-21kthreadcore23:30:230
22731991812,4cyclictest0-21swapper/023:15:140
2273199181,14cyclictest18093-21apt-get21:15:130
2276299179,6cyclictest0-21swapper/323:45:123
2276299179,6cyclictest0-21swapper/323:45:123
2276299179,6cyclictest0-21swapper/323:26:313
2276299179,6cyclictest0-21swapper/322:01:483
2276299178,6cyclictest0-21swapper/322:35:383
2276299172,10cyclictest0-21swapper/320:30:133
2276299172,10cyclictest0-21swapper/320:20:153
2276299171,9cyclictest0-21swapper/319:10:123
22762991711,4cyclictest0-21swapper/323:50:193
22762991711,4cyclictest0-21swapper/323:50:193
22762991711,4cyclictest0-21swapper/323:35:303
2276299171,12cyclictest0-21swapper/320:15:133
2274299178,7cyclictest0-21swapper/120:25:281
2274299173,11cyclictest5881-21kthreadcore21:05:221
2274299172,10cyclictest0-21swapper/120:30:131
2274299171,14cyclictest2374-21latency21:00:211
22742991711,3cyclictest27388-21tail20:20:211
2273199178,7cyclictest28865-21sshd19:47:060
2273199178,6cyclictest15028-21irqcore19:35:170
2273199173,12cyclictest131rcu_preempt20:10:190
2273199173,11cyclictest1422-21ssh23:45:230
2273199173,11cyclictest1422-21ssh23:45:220
2273199171,8cyclictest14791-21kthreadcore22:05:220
2273199171,6cyclictest4728-21munin-run19:25:010
2273199171,10cyclictest9213-21ssh21:10:160
22731991710,4cyclictest8152-21tail20:35:180
22731991710,4cyclictest0-21swapper/019:15:020
2276299169,4cyclictest0-21swapper/321:55:233
2276299162,9cyclictest0-21swapper/319:55:143
2274299161,13cyclictest0-21swapper/120:45:141
2274299161,11cyclictest22766-21latency_hist19:10:021
2273199168,6cyclictest401-21cut21:00:170
2273199166,7cyclictest20982-21tail20:15:200
2273199164,3cyclictest131rcu_preempt22:45:150
2273199162,9cyclictest0-21swapper/019:55:140
2273199162,11cyclictest12429-21fschecks_count20:40:150
2276299152,9cyclictest0-21swapper/320:35:123
2276299151,8cyclictest0-21swapper/320:55:013
2276299151,8cyclictest0-21swapper/320:55:003
2273199153,9cyclictest17748-21kthreadcore20:45:190
2273199151,11cyclictest6722-21cut20:00:190
2273199151,11cyclictest2700-21apt-get20:30:130
2276299149,3cyclictest0-21swapper/320:00:063
2276299148,4cyclictest35966irq/58-eth019:30:223
2276299142,10cyclictest0-21swapper/321:10:023
2273199141,11cyclictest22656-21latency_hist19:10:020
2276299111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional