You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-19 - 06:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Sep 19, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
167682170111,27sleep20-21swapper/219:07:252
167292162107,28sleep30-21swapper/319:06:543
169212161105,27sleep10-21swapper/119:09:261
167362159107,30sleep00-21swapper/019:06:590
2495021560,1sleep30-21swapper/323:25:033
259232730,2sleep00-21swapper/000:37:130
315242650,2sleep10-21swapper/121:30:081
140352630,2sleep30-21swapper/323:16:083
186362620,3sleep10-21swapper/100:31:391
320372540,1sleep30-21swapper/321:30:143
31782540,2sleep10-21swapper/119:50:211
1724199530,3cyclictest22197-21timerwakeupswit22:10:261
85702510,1sleep00-21swapper/020:05:120
320952500,2sleep30-21swapper/323:30:133
290252490,3sleep1211rcuc/123:03:561
26502490,2sleep00-21swapper/000:20:150
152122470,1sleep10-21swapper/121:17:361
143822470,1sleep00-21swapper/022:29:140
80202460,1sleep30-21swapper/323:11:523
284852460,3sleep04087-21diskmemload00:15:170
17522460,1sleep30-21swapper/319:49:543
175872440,1sleep10-21swapper/122:07:201
221962430,1sleep30-21swapper/321:46:283
1724199427,23cyclictest16980-21seq20:24:581
1724199426,22cyclictest32521-21sh23:54:521
187502390,2chrt0-21swapper/223:44:342
325112360,6chrt281ksoftirqd/221:00:112
1724199360,5cyclictest20569-21ls21:45:211
1724199354,4cyclictest1622-21seq21:01:021
17241993510,22cyclictest18609-21expr23:20:141
108802350,3sleep210878-21munin-run21:15:002
1724199349,22cyclictest24948-21sed20:40:231
1724199349,21cyclictest1214-21kernelversion21:55:161
1724199347,24cyclictest29873-21rm23:28:291
17241993410,21cyclictest5256-21cut21:10:171
1724199340,6cyclictest28262-21basename22:15:171
1724199340,4cyclictest8187-21ssh23:12:011
1724199331,29cyclictest13283-21cut21:40:161
17241993312,18cyclictest5766-21seq23:58:471
17241993311,19cyclictest19194-21date20:30:001
1724199330,4cyclictest22944-21grep20:35:251
1724199330,3cyclictest0-21swapper/122:25:171
260852320,2sleep30-21swapper/321:50:093
234162320,9sleep334-21ksoftirqd/322:11:283
1724199327,22cyclictest3724-21cat00:20:231
1724199327,22cyclictest13205-21expr20:15:131
1724199323,24cyclictest31857-21ssh23:30:121
1724199321,8cyclictest5124-21/usr/sbin/munin22:45:241
1724199321,5cyclictest1477-21apt-get22:20:031
1724199321,1cyclictest0-21swapper/119:30:571
76482310,3sleep27651-21fschecks_count22:00:152
1724199319,19cyclictest22322-21cat19:20:171
1724199318,20cyclictest8932-21awk00:25:121
1724199311,4cyclictest27842-21/usr/sbin/munin00:15:271
1724199311,27cyclictest20136-21kworker/u8:200:05:121
1724199310,7cyclictest16108-21ssh23:18:131
1724199310,4cyclictest11019-21rm23:38:341
1724199310,3cyclictest0-21swapper/122:01:371
1724199309,18cyclictest11767-21sh00:03:031
1724199307,21cyclictest22990-21smartctl22:35:181
17241993023,5cyclictest0-21swapper/119:26:221
1724199302,22cyclictest22471-21ssh00:35:151
1724199301,26cyclictest18480-21kworker/u8:021:25:171
1724199300,6cyclictest21971-21ssh23:46:011
1724199300,4cyclictest9468-21ssh21:37:141
1724199300,4cyclictest782-21sh23:06:211
1724199300,3cyclictest21796-21sh22:34:531
1724099301,26cyclictest26572-21latency19:30:160
1724199297,20cyclictest28359-21fschecks_count19:35:171
1724199296,20cyclictest10682-21gltestperf22:50:151
17241992922,5cyclictest0-21swapper/119:49:571
17241992922,5cyclictest0-21swapper/119:42:091
17241992910,17cyclictest22584-21sh22:59:131
1724199290,5cyclictest53050irq/142-enp3s0-22:40:131
1724199290,3cyclictest26998-21memory21:50:171
243412280,2sleep20-21swapper/222:35:422
177342280,2chrt0-21swapper/000:30:460
17243992812,8cyclictest15593-21ssh22:30:113
1724199289,16cyclictest31719-21unixbench_multi20:55:251
1724199288,17cyclictest27138-21timerwakeupswit20:45:251
1724199286,5cyclictest17370-21expr19:10:121
1724199281,6cyclictest29399-21sed20:50:241
1724199281,6cyclictest29399-21sed20:50:241
1724199281,6cyclictest21467-21ssh00:10:131
1724199281,11cyclictest13297-21taskset23:40:141
1724199280,3cyclictest8787-21df_inode20:05:141
1724099281,25cyclictest0-21swapper/019:42:090
1724099281,25cyclictest0-21swapper/019:35:110
1724399271,3cyclictest26509-21irqstats19:30:163
1724299271,24cyclictest0-21swapper/219:56:342
1724299271,24cyclictest0-21swapper/219:44:092
1724299271,1cyclictest0-21swapper/219:37:492
1724199277,17cyclictest11209-21cut20:10:151
1724199271,7cyclictest20820-21grep20:30:251
1724199271,7cyclictest18627-21grep21:20:151
1724199271,6cyclictest4070-21cron21:10:001
1724199271,10cyclictest21302-21cron19:20:001
17240992713,9cyclictest13320-21cat23:40:150
1724099271,24cyclictest0-21swapper/020:01:250
1724099271,24cyclictest0-21swapper/019:50:520
108622270,1sleep00-21swapper/023:38:250
237262260,1sleep30-21swapper/323:24:033
185612260,2sleep00-21swapper/020:25:240
1724399261,24cyclictest0-21swapper/319:42:103
1724399261,23cyclictest0-21swapper/319:26:583
1724399261,23cyclictest0-21swapper/319:24:373
1724299261,2cyclictest0-21swapper/220:06:232
1724299261,23cyclictest0-21swapper/219:30:312
1724299261,23cyclictest0-21swapper/219:25:322
1724299261,23cyclictest0-21swapper/219:20:192
1724199262,6cyclictest6529-21cstates20:00:131
1724199261,6cyclictest4987-21chrt19:55:181
1724099269,8cyclictest141rcu_preempt22:55:120
17240992624,1cyclictest0-21swapper/019:25:410
17240992623,2cyclictest0-21swapper/019:48:290
1724399252,9cyclictest8955-21ssh00:25:133
1724399251,23cyclictest0-21swapper/319:54:323
1724399251,23cyclictest0-21swapper/319:37:433
1724399251,1cyclictest0-21swapper/320:12:203
1724399250,23cyclictest0-21swapper/320:06:073
1724299251,22cyclictest0-21swapper/219:49:132
1724299251,17cyclictest7450-21ssh00:00:122
1724099253,16cyclictest30531-21ssh23:05:110
17240992523,1cyclictest0-21swapper/019:20:250
17240992522,2cyclictest0-21swapper/020:12:180
17240992511,3cyclictest141rcu_preempt21:35:050
107872250,2sleep30-21swapper/323:38:203
17243992422,1cyclictest0-21swapper/320:44:463
17242992422,1cyclictest0-21swapper/220:11:592
1724099247,15cyclictest12-21ksoftirqd/021:45:250
17240992422,1cyclictest0-21swapper/020:45:570
17240992417,5cyclictest12-21ksoftirqd/023:30:000
17240992415,6cyclictest25530-21users23:00:250
17240992410,2cyclictest141rcu_preempt22:36:140
17243992313,7cyclictest20483-21cron00:10:003
1724399230,1cyclictest0-21swapper/320:45:313
1724299231,5cyclictest4087-21diskmemload22:34:432
1724099239,3cyclictest141rcu_preempt00:26:590
1724099238,3cyclictest12-21ksoftirqd/023:58:370
1724099237,4cyclictest12-21ksoftirqd/022:45:180
1724099237,4cyclictest12-21ksoftirqd/021:53:570
1724099232,10cyclictest12-21ksoftirqd/021:17:570
74482220,1sleep00-21swapper/021:11:370
58752220,1sleep30-21swapper/323:58:543
1724399222,14cyclictest640-21snmpd23:00:103
17242992213,7cyclictest21528-21memory22:10:192
1724099229,2cyclictest141rcu_preempt23:18:340
1724099228,7cyclictest141rcu_preempt22:30:080
1724099226,4cyclictest12-21ksoftirqd/021:30:470
17240992220,1cyclictest0-21swapper/020:44:470
17240992214,6cyclictest31139-21ls20:55:200
322222210,18sleep032317-21ssh23:30:150
17243992119,1cyclictest0-21swapper/320:19:103
17243992114,4cyclictest29405-21sh23:51:443
17243992112,7cyclictest560-21dbus-daemon21:40:003
17243992112,6cyclictest4162-21apt-get19:55:103
17243992110,7cyclictest32474-21apt-get21:00:113
17242992115,4cyclictest28-21ksoftirqd/220:55:002
17242992115,4cyclictest28-21ksoftirqd/220:55:002
17242992112,6cyclictest19753-21df_inode19:15:132
17242992111,7cyclictest21010-21unixbench-2d21:45:252
17242992110,6cyclictest52950irq/141-enp3s0-22:20:112
17242992110,2cyclictest0-21swapper/221:56:452
1724099218,2cyclictest141rcu_preempt23:21:560
1724099217,4cyclictest12-21ksoftirqd/022:52:360
1724099217,3cyclictest141rcu_preempt22:06:010
1724099214,8cyclictest141rcu_preempt23:50:540
1724099214,5cyclictest12-21ksoftirqd/021:59:060
17240992114,4cyclictest24362-21sh21:24:580
17240992113,6cyclictest8844-21ls00:00:230
17240992112,7cyclictest7575-21df22:00:140
17240992111,8cyclictest19463-21ssh00:08:590
17240992111,2cyclictest141rcu_preempt22:13:250
1724399209,8cyclictest17450-21cat22:55:143
1724399209,7cyclictest14093-21grep22:05:143
1724399209,6cyclictest27603-21apt-get22:15:133
1724399209,4cyclictest34-21ksoftirqd/321:58:363
1724399207,4cyclictest19438-21switchtime21:20:223
1724399207,10cyclictest0-21swapper/322:40:273
1724399202,12cyclictest75-21kswapd022:45:113
1724399202,12cyclictest24812-21apt-get21:25:103
1724399201,9cyclictest22143-21/usr/sbin/munin00:10:203
17243992013,4cyclictest636-21runrttasks22:20:113
17243992013,4cyclictest52850irq/140-enp3s0-21:10:123
1724299209,9cyclictest14254-21cpuspeed_turbos00:05:142
17242992014,4cyclictest24989-21sh23:25:052
17242992012,6cyclictest9345-21forks00:25:152
17242992012,6cyclictest26434-21if_enp3s020:45:172
17242992012,6cyclictest17590-21cat19:10:142
17242992012,6cyclictest12051-21tune2fs23:15:142
17242992011,7cyclictest2426-21expr21:05:142
1724299201,14cyclictest29163-21ssh22:40:152
17242992010,6cyclictest52950irq/141-enp3s0-23:10:122
17242992010,6cyclictest52950irq/141-enp3s0-21:40:192
1724299200,10cyclictest24115-21taskset23:48:112
1724099206,12cyclictest141rcu_preempt00:12:030
1724099204,5cyclictest20450-1kworker/u9:121:29:440
1724099201,8cyclictest30480-21python322:40:250
17240992014,4cyclictest12-21ksoftirqd/020:35:010
17240992012,4cyclictest131rcuc/023:48:470
1724399199,6cyclictest9967-21cut22:25:213
1724399196,8cyclictest34-21ksoftirqd/322:50:103
17243991913,4cyclictest34-21ksoftirqd/323:50:013
17243991911,6cyclictest29442-21cat00:15:253
17243991911,6cyclictest12835-21expr21:40:133
1724399191,11cyclictest0-21swapper/300:03:233
17243991910,7cyclictest17234-21sort20:25:103
1724299194,13cyclictest28242-21kworker/2:023:55:202
1724299192,12cyclictest4087-21diskmemload21:25:092
1724299192,12cyclictest20972-21apt-get00:10:042
1724299191,9cyclictest23023-21grep00:35:182
1724299191,8cyclictest31877-21python323:05:242
1724299191,5cyclictest21052-21ssh22:57:412
17242991914,3cyclictest13613-21sh22:28:282
17242991914,3cyclictest12358-21sh23:39:542
17242991913,4cyclictest3430-21rt-features00:20:222
17242991913,4cyclictest30408-21sh23:52:452
17242991913,4cyclictest30299-21rm00:16:072
17242991913,3cyclictest25981-21sh21:50:022
17242991912,5cyclictest30486-21cstates20:55:142
17242991911,6cyclictest31938-21cat23:30:142
17242991911,6cyclictest19117-21ntpq21:20:192
17242991911,6cyclictest12711-21threads21:15:232
17242991911,5cyclictest27601-21apt22:15:142
17242991911,5cyclictest24323-21cat20:40:162
17242991910,7cyclictest23804-21apt-get23:00:112
17242991910,6cyclictest20278-21ntp_kernel_pll_20:30:192
1724099199,8cyclictest19356-21cut19:15:000
17240991913,4cyclictest13871-21sh20:15:200
17240991911,5cyclictest21909-21expr20:35:150
17240991910,7cyclictest12-21ksoftirqd/022:20:000
17240991910,6cyclictest15397-21cat20:20:130
1724399181,5cyclictest0-21swapper/300:22:033
1724399181,4cyclictest0-21swapper/322:02:543
1724399181,3cyclictest141rcu_preempt00:33:003
17243991812,4cyclictest8209-21sh20:05:003
17243991812,4cyclictest26267-21sh00:37:343
17243991812,3cyclictest31065-21sh23:05:153
17243991811,5cyclictest30703-21cut20:55:163
17243991811,5cyclictest23520-21ps22:35:223
17243991811,5cyclictest22803-21sed20:35:243
17243991810,6cyclictest20151-21cat19:15:173
17243991810,6cyclictest18039-21memory19:10:183
1724299182,11cyclictest3393-21apt-get22:45:102
1724299182,11cyclictest10101-21apt-get22:50:112
17242991816,1cyclictest0-21swapper/220:19:032
17242991811,5cyclictest8083-21seq20:03:362
17242991811,5cyclictest6238-21expr21:35:152
17242991811,5cyclictest23435-21grep20:40:012
17242991810,5cyclictest18961-21sed23:20:172
17242991810,5cyclictest16884-21rt-features00:30:232
1724099189,7cyclictest3661-21cat22:20:230
1724099188,7cyclictest4262-21date21:10:010
17240991813,3cyclictest19403-21sort19:15:100
17240991812,4cyclictest12-21ksoftirqd/020:00:000
17240991811,5cyclictest13957-21systemctl21:40:220
17240991811,4cyclictest4761-21sh23:10:060
1724399179,5cyclictest4087-21diskmemload21:15:123
1724399179,5cyclictest3403-21switchtime21:05:233
1724399179,5cyclictest28092-21grep20:50:123
1724399179,5cyclictest28092-21grep20:50:123
1724399171,7cyclictest14771-21ssh23:40:333
1724299179,6cyclictest2489-21grep19:50:142
1724299179,6cyclictest17094-21cut20:25:002
17242991712,3cyclictest14310-21sh22:05:152
17242991711,4cyclictest32594-21ntp_kernel_err21:30:192
17242991710,5cyclictest18747-21uptime20:25:242
1724099179,6cyclictest29321-21cat20:50:250
1724099179,6cyclictest29321-21cat20:50:240
1724099179,6cyclictest1426-21date21:00:230
17243991612,3cyclictest16999-21latency_hist20:25:003
1724399161,10cyclictest19391-21apt-get20:30:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional