You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 04:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Tue May 19, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
252292179102,25sleep20-21swapper/219:06:162
2306921770,1sleep10-21swapper/123:41:391
25386214796,25sleep00-21swapper/019:08:180
254962143106,26sleep30-21swapper/319:09:463
254172137102,24sleep10-21swapper/119:08:451
225222670,1sleep10-21swapper/121:40:311
193122650,2sleep10-21swapper/123:39:371
268042640,2sleep20-21swapper/219:10:212
215582630,2sleep10-21swapper/122:29:061
113062580,2sleep00-21swapper/023:10:000
139312570,2sleep10-21swapper/121:35:091
302652550,1sleep30-21swapper/323:47:103
55502520,2sleep012-21ksoftirqd/000:17:080
123042520,1sleep30-21swapper/300:22:143
309962500,1sleep20-21swapper/223:00:132
289962490,2sleep30-21swapper/322:58:383
252162470,1sleep212428-21diskmemload21:19:222
12642460,1sleep30-21swapper/321:25:163
232232440,1sleep30-21swapper/300:05:443
97102430,1sleep30-21swapper/321:55:213
54442430,1sleep10-21swapper/123:05:141
268982420,2sleep10-21swapper/100:09:281
260262420,1chrt0-21swapper/000:08:340
155462420,3chrt0-21swapper/322:00:163
202832370,1sleep10-21swapper/121:15:181
219162360,2sleep00-21swapper/023:40:300
229062350,2sleep30-21swapper/322:05:213
261132320,1sleep20-21swapper/222:31:562
321812300,2sleep012-21ksoftirqd/021:24:360
268082300,2sleep00-21swapper/022:56:270
169122300,2chrt0-21swapper/023:13:400
2579299294,4cyclictest141rcu_preempt22:30:123
25789992916,7cyclictest4366-21rm21:27:290
229562290,1chrt0-21swapper/323:41:333
290892280,2sleep10-21swapper/121:45:281
264282280,2sleep10-21swapper/122:08:201
39812270,2sleep10-21swapper/100:15:341
2579299275,7cyclictest34-21ksoftirqd/323:05:233
2579099271,2cyclictest141rcu_preempt00:00:011
2579099271,11cyclictest1108-21grep21:25:151
2579099271,11cyclictest0-21swapper/123:25:121
25789992712,7cyclictest12-21ksoftirqd/000:00:000
25791992621,3cyclictest28-21ksoftirqd/200:00:012
25790992615,7cyclictest52750irq/139-enp3s0-22:10:121
25789992624,1cyclictest0-21swapper/019:13:040
87592250,2chrt0-21swapper/021:00:170
280332250,12sleep122-21ksoftirqd/122:33:531
2579299259,4cyclictest34-21ksoftirqd/323:15:033
2579299259,3cyclictest141rcu_preempt21:10:253
2579299257,3cyclictest34-21ksoftirqd/323:25:233
2579299255,7cyclictest141rcu_preempt22:44:143
2579299254,11cyclictest27600-1kworker/u9:122:38:263
25792992514,7cyclictest3692-21grep22:15:203
25792992514,7cyclictest3692-21grep22:15:193
25790992523,1cyclictest0-21swapper/119:13:261
25789992515,5cyclictest12-21ksoftirqd/023:18:370
191002250,2sleep10-21swapper/122:50:261
88562240,1sleep20-21swapper/222:19:492
88562240,1sleep20-21swapper/222:19:482
25792992422,1cyclictest0-21swapper/319:12:403
2579299241,19cyclictest53050irq/142-enp3s0-00:15:133
25789992412,5cyclictest12-21ksoftirqd/000:35:040
251632240,2sleep00-21swapper/022:07:020
209352240,2chrt0-21swapper/000:05:010
60372230,3sleep06015-21cron19:39:590
2579299235,14cyclictest141rcu_preempt22:24:473
2579299234,15cyclictest34-21ksoftirqd/321:50:123
25792992317,4cyclictest34-21ksoftirqd/300:00:003
25792992314,7cyclictest13055-21ntp_states23:10:233
25792992314,6cyclictest29243-21diskstats00:35:143
25792992313,7cyclictest21357-21/usr/sbin/munin20:15:113
25792992310,11cyclictest12929-21kworker/u8:200:00:143
2579299230,3cyclictest141rcu_preempt21:36:033
2579199239,11cyclictest18255-21cron23:15:002
25790992310,3cyclictest141rcu_preempt00:32:231
25789992314,6cyclictest3353-21cron22:40:010
25789992310,3cyclictest141rcu_preempt22:25:210
2579299229,3cyclictest34-21ksoftirqd/321:18:453
2579299229,3cyclictest34-21ksoftirqd/321:00:263
2579299229,3cyclictest34-21ksoftirqd/319:35:233
2579299228,2cyclictest34-21ksoftirqd/323:31:203
2579299225,2cyclictest141rcu_preempt22:27:543
25792992213,6cyclictest278-21systemd-journal20:50:013
25792992212,6cyclictest2213-21cut23:50:233
25792992210,3cyclictest34-21ksoftirqd/321:05:113
25792992210,3cyclictest34-21ksoftirqd/300:10:023
25792992210,2cyclictest34-21ksoftirqd/323:36:593
25792992210,2cyclictest34-21ksoftirqd/300:30:183
2579199229,11cyclictest26863-21seq23:45:152
2579099221,9cyclictest30743-21kworker/u8:323:01:391
25790992212,8cyclictest31821-21ssh21:24:161
25790992212,7cyclictest278-21systemd-journal23:34:551
25790992211,9cyclictest21022-21cron20:15:001
2579099221,13cyclictest141rcu_preempt00:37:361
2579099221,13cyclictest141rcu_preempt00:11:511
25789992213,7cyclictest278-21systemd-journal23:34:590
25789992213,7cyclictest27347-21users23:20:260
25789992212,5cyclictest12-21ksoftirqd/022:02:000
2578999221,19cyclictest31556-21ssh21:47:490
25789992211,8cyclictest15432-21apt-get00:25:090
25789992211,8cyclictest15432-21apt-get00:25:080
2579299219,3cyclictest34-21ksoftirqd/323:02:143
2579299219,3cyclictest34-21ksoftirqd/320:20:163
2579299215,2cyclictest141rcu_preempt21:20:043
2579299211,7cyclictest28407-21grep21:45:223
2579299211,10cyclictest261282perl23:20:173
2579199216,12cyclictest13640-21latency21:10:172
25791992116,3cyclictest15053-21cron00:25:002
25791992112,6cyclictest20954-21cut00:05:002
25791992110,3cyclictest191022rm22:50:252
2579099214,5cyclictest22-21ksoftirqd/123:19:411
25790992112,5cyclictest145222sleep100:00:131
2579099211,15cyclictest141rcu_preempt23:50:151
25790992110,3cyclictest22-21ksoftirqd/122:40:201
25789992115,4cyclictest21851-21hwlatdetect20:15:150
25789992113,6cyclictest21608-21ls21:40:180
25789992113,6cyclictest13498-21awk23:35:110
25789992112,6cyclictest11688-21sensors_temp21:05:240
25789992111,6cyclictest52850irq/140-enp3s0-23:45:150
2578999210,9cyclictest88582rm22:19:490
2578999210,9cyclictest88582rm22:19:480
70932200,1sleep30-21swapper/321:30:023
2579299208,3cyclictest34-21ksoftirqd/319:55:133
2579299206,4cyclictest34-21ksoftirqd/300:29:323
2579299206,4cyclictest34-21ksoftirqd/300:29:313
2579299203,7cyclictest34-21ksoftirqd/321:41:123
25792992015,3cyclictest21039-21dump-pmu-power20:15:013
25792992011,6cyclictest23631-21cron22:54:593
25792992010,8cyclictest1879-21ssh22:14:323
25792992010,8cyclictest13120-21ssh22:46:093
2579199202,12cyclictest16200-21ssh22:25:112
2579199201,7cyclictest24079-21apt-get22:55:132
25791992015,3cyclictest23426-21sh21:41:262
2579199201,4cyclictest0-21swapper/222:05:142
25791992013,5cyclictest20036-21ntpq20:10:202
25791992012,6cyclictest3926-21ssh21:50:532
25791992012,6cyclictest3218-21awk00:15:212
25791992012,6cyclictest10434-21sed19:50:002
25791992012,5cyclictest12428-21diskmemload00:37:092
2579199201,16cyclictest5489-21timerwakeupswit19:35:272
25791992011,6cyclictest25623-21sh23:20:122
25791992011,6cyclictest24360-21ntpq20:20:172
2579099209,8cyclictest10790-21grep22:45:131
2579099208,2cyclictest141rcu_preempt22:56:171
2579099205,8cyclictest141rcu_preempt21:30:301
2579099201,6cyclictest11521-21grep21:05:221
2579099201,4cyclictest0-21swapper/122:00:041
25790992010,7cyclictest9232-21sort22:20:001
25790992010,7cyclictest9232-21sort22:20:001
25790992010,7cyclictest5615-21grep19:35:291
2578999208,2cyclictest12557-21ssh21:33:590
2578999207,2cyclictest636-21runrttasks22:46:020
2578999202,4cyclictest12-21ksoftirqd/023:29:500
2578999202,12cyclictest30078-21apt-get20:35:130
2578999202,11cyclictest12-21ksoftirqd/020:06:100
25789992015,3cyclictest3945-21/usr/sbin/munin22:40:210
25789992014,4cyclictest252152sleep022:31:020
25789992014,4cyclictest12-21ksoftirqd/019:34:590
25789992012,6cyclictest32611-21diskstats19:25:140
25789992012,6cyclictest29283-21ls19:15:250
25789992010,7cyclictest8761-21df19:45:140
25789992010,6cyclictest52850irq/140-enp3s0-21:10:120
2578999200,12cyclictest6781-21tail19:40:180
149892200,2sleep20-21swapper/222:48:012
124142201,4sleep112428-21diskmemload00:22:201
25792991912,5cyclictest12063-21awk19:50:263
25792991912,5cyclictest12063-21awk19:50:253
25792991911,6cyclictest5712-21seq20:51:383
25792991911,6cyclictest17299-21expr20:05:133
2579199191,8cyclictest7652-21ssh21:30:132
25791991914,3cyclictest16115-21sh23:36:242
25791991914,3cyclictest14299-21sh22:23:332
25791991913,4cyclictest7710-21sh22:42:262
25791991913,4cyclictest6363-21sh21:29:312
25791991913,4cyclictest5577-21sh23:29:162
25791991912,4cyclictest18698-21sh23:15:022
25791991911,6cyclictest6731-21cat23:30:102
25791991911,6cyclictest5360-21ls23:05:132
25791991911,6cyclictest31127-21seq00:12:002
25791991911,6cyclictest31014-21ntp_kernel_err19:20:192
25791991911,6cyclictest28552-21ls20:30:192
25791991911,6cyclictest23195-21ntp_states00:30:192
25791991911,6cyclictest17924-21ntp_states20:05:212
25791991911,6cyclictest17138-21grep00:25:232
25791991911,6cyclictest17138-21grep00:25:222
25791991911,6cyclictest11550-21which21:05:232
25791991911,5cyclictest8826-21diskstats21:55:152
2579199191,15cyclictest22105-21rm00:05:192
2579199191,13cyclictest26326-21ssh21:20:112
25791991910,4cyclictest27461-21ssh21:45:152
2579099198,2cyclictest22-21ksoftirqd/121:55:131
2579099197,7cyclictest211rcuc/122:20:121
2579099191,5cyclictest15625-21wc00:25:111
2579099191,5cyclictest15625-21wc00:25:101
25790991913,3cyclictest12696-21acpi19:55:121
25790991911,6cyclictest5999-21basename20:55:011
25790991911,6cyclictest32642-21cut19:25:141
25790991911,6cyclictest3071-21gzip19:30:221
25790991911,5cyclictest9137-21sed19:45:191
25790991910,6cyclictest278-21systemd-journal21:00:001
25790991910,6cyclictest22763-21awk20:15:251
2578999191,9cyclictest6568-21df_inode20:55:140
2578999191,8cyclictest27726-21/usr/sbin/munin20:30:110
2578999191,7cyclictest12428-21diskmemload00:22:590
2578999191,6cyclictest32047-21grep23:00:230
25789991913,4cyclictest52850irq/140-enp3s0-00:30:200
25789991912,5cyclictest13865-21sensors_temp19:55:230
25789991912,4cyclictest18808-21timerandwakeup22:50:230
25789991911,6cyclictest31640-21vmstat19:20:250
25789991911,6cyclictest27957-21wc00:10:140
25789991911,6cyclictest24724-21cat20:20:220
25789991911,6cyclictest2315-21awk20:45:160
25789991911,6cyclictest1194-21ls20:40:250
25789991911,6cyclictest1194-21ls20:40:240
25789991911,6cyclictest11398-21ls19:50:190
25789991911,6cyclictest11398-21ls19:50:190
25789991911,6cyclictest10155-21awk21:55:250
25789991911,5cyclictest2381-21idleruntime21:50:160
25789991910,6cyclictest4537-21perl20:50:150
2579299189,6cyclictest25953-21cut20:25:143
2579299189,6cyclictest2176-21seq19:30:133
2579299187,2cyclictest34-21ksoftirqd/320:35:133
2579299186,2cyclictest34-21ksoftirqd/320:00:253
25792991810,6cyclictest6602-21cut20:55:143
25792991810,5cyclictest30031-21cat19:20:013
2579199189,6cyclictest29432-21cat22:10:182
2579199182,11cyclictest32274-21apt-get19:25:122
25791991812,4cyclictest28-21ksoftirqd/220:40:002
25791991811,5cyclictest3644-21seq19:32:062
25791991811,5cyclictest3466-21awk20:45:252
25791991811,5cyclictest26826-21threads20:25:222
25791991811,5cyclictest21482-21expr20:15:122
25791991811,5cyclictest20306-21expr23:40:122
25791991811,5cyclictest16186-21ntp_kernel_err22:00:202
25791991811,5cyclictest15093-21pluginstate21:35:222
25791991810,6cyclictest31316-21vmstat22:35:262
2579099189,7cyclictest26575-21head20:25:211
2579099187,6cyclictest52750irq/139-enp3s0-23:20:131
2579099182,13cyclictest29716-21tr22:35:131
2579099182,12cyclictest11835-21ssh23:10:051
2579099181,9cyclictest0-21swapper/120:05:211
2579099181,8cyclictest0-21swapper/120:40:231
2579099181,8cyclictest0-21swapper/120:40:221
2579099181,5cyclictest0-21swapper/120:45:161
2579099181,2cyclictest141rcu_preempt21:10:251
2579099181,2cyclictest141rcu_preempt19:40:181
25790991811,5cyclictest24775-21sed20:20:211
25790991811,5cyclictest11601-21cut19:50:221
25790991811,5cyclictest11601-21cut19:50:221
2579099181,12cyclictest26871-21apt-get23:45:161
25790991810,6cyclictest29988-21grep19:20:011
25790991810,5cyclictest278-21systemd-journal20:04:551
2578999182,11cyclictest19146-21apt-get20:10:140
2578999181,5cyclictest0-21swapper/022:10:590
25789991812,4cyclictest4483-21sh23:52:070
25789991811,5cyclictest22709-21seq21:16:510
2578999181,15cyclictest13994-21ssh21:35:110
25789991811,5cyclictest11438-21seq22:20:390
25789991810,6cyclictest16789-21ls20:05:010
2579299179,5cyclictest28140-21cut20:30:153
2579299178,6cyclictest550-21systemd-logind20:40:233
2579299178,6cyclictest550-21systemd-logind20:40:223
2579299177,7cyclictest10316-21cron19:50:013
25792991710,5cyclictest996-21grep19:25:233
2579199179,6cyclictest7011-21cat20:55:192
2579199178,6cyclictest32680-21fschecks_time20:40:152
2579199178,6cyclictest32680-21fschecks_time20:40:152
2579199172,11cyclictest12712-21apt-get19:55:142
25791991712,3cyclictest27951-21/usr/sbin/munin19:15:272
25791991712,3cyclictest14844-21/usr/sbin/munin20:00:232
25791991712,3cyclictest1056-21/usr/sbin/munin23:50:172
25791991710,5cyclictest7443-21basename19:40:252
25791991710,5cyclictest11728-21python319:50:242
25791991710,5cyclictest11728-21python319:50:232
2579099176,3cyclictest22-21ksoftirqd/119:25:001
2579099171,2cyclictest141rcu_preempt21:50:011
2579099171,2cyclictest0-21swapper/120:39:491
25790991711,3cyclictest22-21ksoftirqd/121:05:011
25789991712,3cyclictest25558-21/usr/sbin/munin20:25:150
2579199169,5cyclictest8692-21cat21:00:152
2579199162,12cyclictest4242-21cut20:50:132
2579299152,10cyclictest31291-21python319:20:223
25792991511,2cyclictest7222-21ntp_states19:40:233
2579099151,2cyclictest141rcu_preempt20:30:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional