You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 15:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System (updated Thu Mar 26, 2026 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2531199641,42cyclictest15847-21ssh12:10:132
2531199581,3cyclictest0-21swapper/207:50:142
2531199501,2cyclictest14723-21apt-get10:05:122
2529499503,7cyclictest131rcu_preempt09:15:141
25324994942,5cyclictest28366irq/58-eth010:35:123
2531199493,41cyclictest408-21apt-get08:25:132
2529499492,33cyclictest23138-21ssh10:45:141
2528199484,22cyclictest11797-21ssh11:50:120
2531199470,3cyclictest0-21swapper/208:45:122
2531199462,29cyclictest31177-21ssh11:25:132
2531199442,4cyclictest0-21swapper/208:30:132
2529499442,6cyclictest0-21swapper/111:55:121
2531199420,8cyclictest0-21swapper/207:30:122
2529499425,27cyclictest3916-21ssh11:10:131
2531199412,4cyclictest0-21swapper/207:45:122
2531199412,34cyclictest0-21swapper/210:23:002
2532499405,24cyclictest28366irq/58-eth010:00:123
2531199401,7cyclictest8567-21latency_hist10:55:022
2531199401,4cyclictest0-21swapper/212:05:122
2531199400,5cyclictest0-21swapper/212:25:132
2531199392,6cyclictest29372-21kthreadcore10:30:212
2529499390,30cyclictest27237-21ssh11:05:131
2531199382,4cyclictest20592-21ssh09:15:142
2529499381,31cyclictest15571-21ssh09:30:151
2531199371,9cyclictest5068-21ntp_states12:20:252
2531199371,7cyclictest29274-21acpi08:55:122
2531199371,7cyclictest18365-21ssh11:00:172
2531199370,4cyclictest24516-21apt-get08:50:142
2532499368,21cyclictest0-21swapper/310:30:143
25324993611,16cyclictest28366irq/58-eth010:15:133
2531199365,6cyclictest20354-21latency11:35:222
2531199365,4cyclictest27-21ksoftirqd/211:06:332
2531199362,7cyclictest23347-21kthreadcore11:20:192
2531199361,9cyclictest11555-21cpu09:10:142
2531199361,8cyclictest26072-21kthreadcore10:45:232
2531199361,6cyclictest24983-21kthreadcore08:15:202
2531199361,6cyclictest17459-21ssh09:47:492
2531199361,5cyclictest20734-21cat07:35:282
2531199361,32cyclictest24853-21kthreadcore07:40:212
2531199361,15cyclictest10084-21dpkg08:35:052
2531199361,10cyclictest7588-21ssh11:45:312
2531199351,7cyclictest24807-21wc11:55:262
2531199351,6cyclictest24770-21sort09:35:132
2531199351,6cyclictest23295-21ntp_states12:30:262
2531199351,6cyclictest17063-21sendmail_mailqu08:05:252
2531199351,6cyclictest16115-21rm09:30:202
2531199351,3cyclictest8488-21ntp_192.168.11410:35:252
2531199351,31cyclictest21232-21tr10:25:212
2531199350,5cyclictest21711-21netstat08:10:222
2529499359,5cyclictest8351-21kthreadcore10:00:201
2529499357,7cyclictest1589-21cut10:15:191
2529499351,28cyclictest8915-21ssh10:55:121
25294993512,5cyclictest0-21swapper/108:55:271
2529499351,10cyclictest29289-21grep07:10:271
2529499350,11cyclictest21059-21tr12:30:211
2532499344,18cyclictest28366irq/58-eth010:10:123
2531199345,3cyclictest10160-21sh09:25:262
2531199343,4cyclictest11785-21apt-get11:50:122
2531199341,8cyclictest26224-21fschecks_count11:40:152
2531199341,8cyclictest26224-21fschecks_count11:40:152
2531199341,7cyclictest8765-21ssh09:43:192
2531199341,7cyclictest12575-21sed10:55:242
2531199341,6cyclictest12314-21swap08:00:272
2529499341,9cyclictest22131-21grep08:10:271
2531199339,20cyclictest8930-21ssh10:19:312
2531199338,21cyclictest11449-21ntp_192.168.11411:30:252
2531199331,9cyclictest6646-21diskstats09:05:152
2531199331,7cyclictest6589-21kthreadcore07:55:202
2531199331,6cyclictest28032-21find08:20:122
2531199331,6cyclictest1031-21/usr/sbin/munin09:20:282
2531199331,5cyclictest28995-21processes07:10:242
2531199331,5cyclictest17820-21mailstats10:40:282
2531199331,4cyclictest23998-21ssh10:10:132
2529499337,5cyclictest14337-21kthreadcore11:15:211
2529499331,9cyclictest8008-21kthreadcore12:05:191
2529499331,9cyclictest28618-21/usr/sbin/munin12:35:121
2529499331,24cyclictest31196-21ssh11:25:131
2529499331,10cyclictest8378-21ssh09:25:191
2529499331,10cyclictest1222-21missed_timers12:00:241
2529499331,10cyclictest10359-21irqrtprio10:20:181
25281993326,4cyclictest11120-21diskmemload10:20:130
2532499325,19cyclictest28366irq/58-eth008:40:113
2532499323,20cyclictest28366irq/58-eth012:20:113
2532499323,18cyclictest28366irq/58-eth008:50:123
2532499321,17cyclictest28366irq/58-eth012:35:133
2531199327,23cyclictest4242-21sh12:02:332
2531199324,6cyclictest15793-21kthreadcore11:15:222
2531199323,2cyclictest0-21swapper/212:15:362
2531199321,7cyclictest28451-21cron09:55:022
2531199321,6cyclictest10453-21kthreadcore07:25:212
2531199321,5cyclictest28621-21/usr/sbin/munin12:35:112
2531199321,5cyclictest1663-21/usr/sbin/munin09:00:132
2531199321,27cyclictest28834-21ssh09:55:122
25311993211,19cyclictest11918-21ssh11:14:152
2529499321,9cyclictest29732-21ssh11:40:261
2529499321,9cyclictest29732-21ssh11:40:251
2529499321,13cyclictest22159-21ssh11:20:131
2529499321,10cyclictest20024-21irqcore08:45:171
2529499321,10cyclictest11735-21/usr/sbin/munin09:10:161
2529499320,10cyclictest23802-21cut07:40:181
25324993118,9cyclictest0-21swapper/309:30:153
2531199312,5cyclictest2655-21/usr/sbin/munin07:20:302
2531199311,5cyclictest2396-21vmstat07:15:322
2531199311,27cyclictest0-21swapper/210:00:132
2529499317,3cyclictest0-21swapper/112:26:581
2529499311,11cyclictest31818-21ntp_kernel_pll_10:30:251
2529499311,11cyclictest10625-21tr08:00:201
2532499302,18cyclictest28366irq/58-eth012:25:133
25324993012,11cyclictest28366irq/58-eth011:55:113
2531199301,5cyclictest19443-21vmstat08:40:292
2529499307,4cyclictest15160-21cut10:05:171
2529499301,10cyclictest3786-21cut12:20:191
2529499301,10cyclictest11716-21kthreadcore08:35:191
25281993022,6cyclictest10521-21ssh12:25:140
2528199302,25cyclictest14735-21rm10:05:120
2529499291,9cyclictest27952-21ntp_states08:50:251
2529499291,8cyclictest17522-21sed11:35:171
2529499291,10cyclictest9369-21cut11:30:201
2529499291,10cyclictest5604-21kthreadcore07:55:191
2529499291,10cyclictest4478-21grep09:40:221
2529499291,10cyclictest21881-21tr10:25:211
2529499291,10cyclictest11120-21diskmemload10:40:211
2532499283,17cyclictest0-21swapper/311:35:133
25324992813,8cyclictest28366irq/58-eth009:25:123
2529499281,9cyclictest24699-21ssh09:35:121
2529499281,8cyclictest19968-21fschecks_time09:50:161
2529499281,10cyclictest5194-21kthreadcore11:45:221
2529499281,10cyclictest28864-21ssh09:55:141
2529499281,10cyclictest12337-21diskstats07:30:141
2528199283,22cyclictest19592-21ssh12:30:130
2532499272,11cyclictest0-21swapper/311:20:083
2528199273,19cyclictest28666-21ssh12:35:130
2528199272,21cyclictest4881-21ssh10:35:120
2528199271,24cyclictest23894-21ssh10:10:120
2532499264,16cyclictest0-21swapper/310:25:003
25324992613,9cyclictest28366irq/58-eth010:50:153
2529499269,3cyclictest29611-21sh10:11:291
25281992618,5cyclictest12-21ksoftirqd/007:25:120
25281992617,6cyclictest1374-21sh12:20:120
25281992615,8cyclictest2695-21ssh10:33:010
2532499255,14cyclictest0-21swapper/311:25:143
2532499255,14cyclictest0-21swapper/311:05:133
2532499252,17cyclictest28366irq/58-eth011:10:123
25324992516,6cyclictest28366irq/58-eth011:50:123
2529499251,10cyclictest2889-21tr09:00:181
2528199255,10cyclictest131rcu_preempt09:25:100
2528199253,19cyclictest7847-21ssh11:30:130
2528199252,18cyclictest11120-21diskmemload09:50:150
2532499245,15cyclictest28366irq/58-eth009:50:143
2532499245,14cyclictest0-21swapper/310:55:113
2532499243,13cyclictest0-21swapper/311:00:123
25324992414,7cyclictest0-21swapper/309:25:013
25324992411,9cyclictest0-21swapper/312:15:133
2532499241,15cyclictest28366irq/58-eth010:45:143
2529499248,3cyclictest20772-21sh12:10:401
2529499241,9cyclictest32277-21ssh10:50:141
2528199242,18cyclictest31470-21apt-get07:50:120
2532499234,12cyclictest0-21swapper/311:15:133
2532499232,15cyclictest28366irq/58-eth009:40:153
2529499232,14cyclictest839-21munin-node07:35:141
2529499231,10cyclictest30468-21irqstats09:20:181
2528199233,15cyclictest20887-21ls09:15:170
2532499223,12cyclictest28366irq/58-eth009:15:143
2532499222,15cyclictest28366irq/58-eth007:35:133
2532499222,13cyclictest0-21swapper/311:40:133
2532499222,13cyclictest0-21swapper/311:40:133
25324992210,7cyclictest28366irq/58-eth009:55:143
25294992211,8cyclictest24852-21awk12:15:141
2528199229,10cyclictest12-21ksoftirqd/008:40:110
25281992215,5cyclictest0-21swapper/010:15:130
25281992214,6cyclictest32389-21ssh10:50:160
25281992214,6cyclictest11516-21ssh09:10:130
25281992213,6cyclictest16698-21ssh12:10:190
2528199221,16cyclictest25141-21ls07:40:210
2532499218,8cyclictest0-21swapper/310:05:253
2532499214,10cyclictest0-21swapper/307:25:133
2532499212,14cyclictest28366irq/58-eth012:05:123
25324992111,7cyclictest0-21swapper/310:25:273
2532499211,11cyclictest28366irq/58-eth011:45:133
25294992111,7cyclictest27785-21cron08:20:001
2528199219,10cyclictest10053-21ssh12:05:220
2528199215,9cyclictest0-21swapper/010:40:120
2528199213,16cyclictest26652-21usage_EthcTask11:20:290
2528199213,15cyclictest16933-21ssh11:35:130
2528199212,15cyclictest12997-21ssh11:15:130
25281992112,6cyclictest21745-21ssh11:00:260
2528199211,17cyclictest7235-21kthreadcore08:30:190
2528199211,17cyclictest29767-21sh09:20:140
2528199211,16cyclictest10103-21apt-get08:35:120
2528199211,15cyclictest5612-21ssh10:00:130
2532499203,12cyclictest0-21swapper/308:35:023
2532499202,11cyclictest0-21swapper/309:35:123
25324992011,4cyclictest0-21swapper/309:46:103
2532499201,13cyclictest0-21swapper/308:55:163
2532499201,13cyclictest0-21swapper/307:35:023
2532499201,10cyclictest0-21swapper/310:40:133
2528199203,7cyclictest0-21swapper/012:03:090
2528199202,14cyclictest3889-21apt-get11:10:110
25281992014,4cyclictest0-21swapper/007:35:250
25281992013,5cyclictest23289-21fschecks_count10:45:170
25281992012,5cyclictest20963-21cpu11:55:130
25281992011,6cyclictest6778-21ssh09:41:000
2528199201,16cyclictest26040-21ssh11:40:130
2528199201,16cyclictest26040-21ssh11:40:130
2528199201,16cyclictest10615-21sh09:45:150
2532499199,8cyclictest0-21swapper/309:12:443
2532499194,9cyclictest0-21swapper/311:30:123
2532499193,11cyclictest28366irq/58-eth012:10:133
2532499193,10cyclictest0-21swapper/312:00:143
2532499191,12cyclictest0-21swapper/312:30:233
2529499197,3cyclictest131rcu_preempt11:00:141
2528199193,4cyclictest131rcu_preempt10:30:020
25281991912,4cyclictest31577-21irqcore11:25:180
2528199191,15cyclictest30668-21kthreadcore09:55:210
2528199191,15cyclictest27227-21ssh11:05:120
2528199191,15cyclictest24773-21sh12:15:140
2528199191,15cyclictest10375-21kthreadcore08:00:200
2532499183,9cyclictest0-21swapper/307:20:123
2532499182,11cyclictest0-21swapper/308:20:143
2532499182,11cyclictest0-21swapper/307:40:133
2532499182,10cyclictest0-21swapper/308:15:143
2532499181,13cyclictest0-21swapper/308:05:263
2531199181,9cyclictest25334-21latency_hist07:10:022
2529499182,6cyclictest6733-21cut10:35:211
2529499182,12cyclictest15564-21ntp_states11:50:261
2529499180,9cyclictest8543-21cut09:05:201
2528199189,6cyclictest31238-21cron07:50:010
2528199189,6cyclictest12-21ksoftirqd/007:10:110
2528199185,9cyclictest25277-21cyclictest08:10:120
2528199184,10cyclictest0-21swapper/008:50:110
2528199183,12cyclictest0-21swapper/007:55:120
2528199182,12cyclictest12243-21apt-get07:30:120
2528199181,9cyclictest7213-21cron07:25:010
2528199181,9cyclictest17344-21grep08:05:270
25281991812,4cyclictest0-21swapper/009:35:100
25281991810,5cyclictest19183-21ssh09:30:260
2532499172,9cyclictest0-21swapper/308:00:123
2532499172,10cyclictest0-21swapper/309:00:133
2532499172,10cyclictest0-21swapper/307:45:143
2532499172,10cyclictest0-21swapper/307:15:153
2532499172,10cyclictest0-21swapper/307:10:133
25294991711,4cyclictest0-21swapper/108:25:261
2529499171,13cyclictest9295-21kthreadcore07:25:191
2529499171,12cyclictest0-21swapper/107:15:151
2529499170,9cyclictest31141-21kthreadcore08:20:221
2528199178,7cyclictest4975-21memory09:00:220
2528199172,12cyclictest22853-21kthreadcore08:45:210
2528199172,11cyclictest8893-21apt-get10:55:120
2528199172,11cyclictest28135-21diskstats08:20:140
2528199172,11cyclictest0-21swapper/008:55:140
2528199171,13cyclictest23908-21kthreadcore08:15:190
2528199171,12cyclictest3100-21/usr/sbin/munin11:45:180
2532499163,8cyclictest0-21swapper/307:50:123
2532499161,13cyclictest0-21swapper/308:45:213
2529499168,6cyclictest2929-21if_err_eth007:20:161
2529499163,10cyclictest12885-21cut09:45:231
2529499162,10cyclictest0-21swapper/108:30:141
2529499161,12cyclictest30134-21ntp_states07:45:251
2529499161,11cyclictest14898-21apt-get08:40:111
2528199165,9cyclictest6706-21fschecks_time09:05:160
2528199164,7cyclictest0-21swapper/008:25:130
2532499152,9cyclictest0-21swapper/309:05:033
2532499152,10cyclictest0-21swapper/308:10:283
2532499151,8cyclictest0-21swapper/308:35:283
2529499152,9cyclictest0-21swapper/107:50:141
2529499151,11cyclictest13513-21apt-get08:05:111
2528199151,11cyclictest30282-21apt-get07:15:130
2528199151,10cyclictest1-21systemd07:10:020
2532499141,9cyclictest0-21swapper/308:00:013
2529499148,4cyclictest25322-21latency_hist07:10:011
2532499121,6cyclictest0-21swapper/308:25:183
2532499111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional