You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 09:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat May 02, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
648999595,42cyclictest9507-21ssh21:55:130
648999588,9cyclictest12-21ksoftirqd/022:35:130
649999561,43cyclictest10764-21ssh23:25:131
648999564,12cyclictest131rcu_preempt23:10:110
6508995521,28cyclictest1636-21ssh21:15:122
648999535,39cyclictest12008-21ssh22:50:110
650899523,30cyclictest1686-21ssh23:20:142
648999494,41cyclictest30200-21ssh23:00:150
648999494,34cyclictest9408-21ssh00:00:130
649999461,30cyclictest28852-21ssh23:35:121
648999434,25cyclictest27541-21ssh00:10:130
6489994330,8cyclictest0-21swapper/000:15:140
652499422,26cyclictest35966irq/58-eth021:25:123
648999421,34cyclictest8122-21ssh22:30:130
6508994022,10cyclictest22467-21ssh22:20:122
652499391,22cyclictest35966irq/58-eth021:50:123
650899398,18cyclictest0-21swapper/200:05:182
6499993927,5cyclictest0-21swapper/121:15:131
652499382,26cyclictest35966irq/58-eth022:05:123
6508993823,10cyclictest2911-21apt-get22:45:122
648999381,29cyclictest32765-21ssh23:55:140
652499373,18cyclictest35966irq/58-eth021:45:123
652499372,22cyclictest35966irq/58-eth021:35:113
650899375,13cyclictest12170-21fschecks_count22:50:142
6508993730,3cyclictest22241-21apt-get20:00:132
6499993718,17cyclictest21-21ksoftirqd/123:05:111
652499362,22cyclictest35966irq/58-eth022:45:123
6508993620,11cyclictest22290-21apt-get00:25:132
650899361,25cyclictest5452-21ssh23:40:122
648999361,31cyclictest1619-21ssh21:15:120
6524993516,11cyclictest35966irq/58-eth022:50:113
649999351,24cyclictest18539-21ssh00:05:181
648999354,20cyclictest0-21swapper/020:50:110
6489993523,7cyclictest121ksoftirqd/023:15:140
6489993523,7cyclictest121ksoftirqd/023:15:140
652499345,16cyclictest35966irq/58-eth023:55:123
650899345,20cyclictest3425-21ssh21:32:582
6508993416,15cyclictest0-21swapper/222:25:112
6508993416,15cyclictest0-21swapper/222:25:112
652499330,20cyclictest35966irq/58-eth022:30:113
6508993325,5cyclictest27346irq/40-dwc2_hso22:55:142
650899332,20cyclictest28850-21rm23:35:122
6508993312,18cyclictest0-21swapper/223:10:112
6508993310,20cyclictest0-21swapper/223:15:132
6508993310,20cyclictest0-21swapper/223:15:132
652499321,20cyclictest35966irq/58-eth000:15:143
649999324,16cyclictest0-21swapper/100:20:131
648999328,18cyclictest12-21ksoftirqd/022:00:120
648999321,21cyclictest387-21ssh21:50:120
652499311,18cyclictest35966irq/58-eth022:10:123
650899317,10cyclictest580-21ssh21:14:172
650899313,21cyclictest0-21swapper/220:20:132
650899312,12cyclictest0-21swapper/219:40:012
650899312,11cyclictest0-21swapper/221:56:422
6508993119,9cyclictest17192-21ssh22:35:122
650899311,6cyclictest131rcu_preempt22:10:142
6508993111,17cyclictest0-21swapper/223:45:132
6508993110,17cyclictest27-21ksoftirqd/223:25:132
6499993112,14cyclictest21-21ksoftirqd/122:20:131
648999314,19cyclictest19802-21ssh21:25:120
648999313,19cyclictest151rcuc/021:35:120
648999313,18cyclictest26304-21ssh22:40:120
6489993118,9cyclictest0-21swapper/021:45:120
6489993110,15cyclictest12-21ksoftirqd/020:05:140
6524993017,10cyclictest35966irq/58-eth022:25:133
6524993017,10cyclictest35966irq/58-eth022:25:133
650899304,17cyclictest27649-21ssh22:05:112
650899304,10cyclictest9538-21rm23:05:192
650899303,16cyclictest0-21swapper/221:52:282
650899303,14cyclictest6475-21ssh00:33:402
650899302,21cyclictest0-21swapper/221:45:052
6508993021,7cyclictest0-21swapper/222:00:122
6508993016,10cyclictest0-21swapper/200:35:112
6508993012,13cyclictest0-21swapper/221:35:112
650899301,17cyclictest14651-21ssh21:40:132
6508993011,14cyclictest27346irq/40-dwc2_hso00:21:162
6489993020,4cyclictest0-21swapper/019:55:120
6489993014,10cyclictest12-21ksoftirqd/022:55:140
652499298,14cyclictest0-21swapper/322:40:123
650899295,16cyclictest19862-21kthreadcore20:30:182
650899292,12cyclictest24476-21diskmemload22:19:482
650899292,11cyclictest27794-21ssh00:10:162
648999293,20cyclictest22322-21ssh00:25:130
650899283,11cyclictest0-21swapper/221:29:582
650899283,10cyclictest0-21swapper/200:04:172
6508992823,2cyclictest30178-21apt-get23:00:142
650899282,13cyclictest10703-21sh21:20:122
6508992812,12cyclictest0-21swapper/219:15:112
649999283,7cyclictest21-21ksoftirqd/123:40:121
6508992719,6cyclictest4156-21apt-get00:15:152
6508992711,12cyclictest9038-21sendmail-msp00:00:022
6508992710,8cyclictest28348-21cut22:40:202
649999276,16cyclictest21-21ksoftirqd/100:25:121
648999272,19cyclictest27679-21ssh22:05:120
652499263,14cyclictest36066irq/59-eth023:00:153
652499261,16cyclictest35966irq/58-eth023:10:113
650899264,8cyclictest0-21swapper/222:30:362
650899264,12cyclictest23325-21tr21:05:232
6508992617,7cyclictest930-21snmpd20:15:112
650899260,13cyclictest24684-21users23:30:312
650899260,11cyclictest0-21swapper/219:30:112
649999269,9cyclictest21-21ksoftirqd/121:58:351
649999267,9cyclictest21-21ksoftirqd/121:50:021
649999265,8cyclictest21-21ksoftirqd/120:05:271
6499992612,7cyclictest6480-21cyclictest23:00:141
6499992610,8cyclictest21-21ksoftirqd/100:02:241
6499992610,10cyclictest14641-21ssh21:40:121
648999264,13cyclictest14631-21ssh21:40:110
648999263,4cyclictest131rcu_preempt20:25:120
652499253,16cyclictest0-21swapper/300:20:153
6524992512,9cyclictest35966irq/58-eth022:55:273
650899259,8cyclictest23439-21acpi20:35:102
650899259,8cyclictest17199-21sshd19:54:572
650899259,13cyclictest16353-21df_inode19:20:112
650899258,9cyclictest28806-21ssh23:50:432
650899255,12cyclictest14609-21cut20:25:172
650899254,13cyclictest27346irq/40-dwc2_hso19:10:122
650899252,16cyclictest27346irq/40-dwc2_hso20:40:122
6508992515,6cyclictest27346irq/40-dwc2_hso19:55:122
6508992510,7cyclictest18617-21find21:00:262
6508992510,12cyclictest2870-21ntp_kernel_pll_20:10:242
6508992510,12cyclictest2870-21ntp_kernel_pll_20:10:242
649999259,4cyclictest21-21ksoftirqd/100:17:271
649999258,7cyclictest21-21ksoftirqd/121:24:291
649999257,6cyclictest21-21ksoftirqd/100:30:121
649999256,4cyclictest131rcu_preempt22:15:131
649999255,3cyclictest131rcu_preempt20:25:021
649999254,11cyclictest21-21ksoftirqd/123:17:361
649999254,11cyclictest21-21ksoftirqd/123:17:361
648999251,18cyclictest28867-21ssh23:35:130
650899249,12cyclictest7602-21sshd19:41:572
650899242,16cyclictest27346irq/40-dwc2_hso19:45:112
650899242,16cyclictest27346irq/40-dwc2_hso19:45:112
649999249,9cyclictest21-21ksoftirqd/122:07:081
649999249,7cyclictest21-21ksoftirqd/121:48:451
649999248,9cyclictest21-21ksoftirqd/123:50:091
649999248,8cyclictest21-21ksoftirqd/123:59:201
649999248,7cyclictest21-21ksoftirqd/122:53:111
649999247,9cyclictest21-21ksoftirqd/122:02:041
649999247,9cyclictest21-21ksoftirqd/120:45:011
649999247,8cyclictest21-21ksoftirqd/119:50:011
649999247,8cyclictest21-21ksoftirqd/119:50:011
649999245,8cyclictest131rcu_preempt21:31:141
6499992414,7cyclictest14234-21cron21:40:011
6499992412,4cyclictest21-21ksoftirqd/100:10:021
648999241,19cyclictest31486-21sh22:25:130
648999241,19cyclictest31486-21sh22:25:130
6524992314,5cyclictest0-21swapper/321:10:123
650899237,11cyclictest27346irq/40-dwc2_hso20:50:112
649999239,8cyclictest21-21ksoftirqd/123:34:541
649999238,4cyclictest21-21ksoftirqd/123:46:061
649999237,4cyclictest21-21ksoftirqd/121:13:591
649999236,8cyclictest21-21ksoftirqd/122:31:541
649999236,7cyclictest21-21ksoftirqd/122:42:061
649999236,7cyclictest21-21ksoftirqd/122:37:261
649999235,8cyclictest21-21ksoftirqd/120:15:311
649999235,4cyclictest131rcu_preempt22:55:191
649999232,15cyclictest1656-21sh23:20:121
6499992312,5cyclictest21-21ksoftirqd/122:12:381
648999234,6cyclictest12-21ksoftirqd/023:45:130
650899229,5cyclictest30741-21grep20:05:282
649999228,4cyclictest21-21ksoftirqd/122:25:471
649999228,4cyclictest21-21ksoftirqd/122:25:471
649999227,7cyclictest21-21ksoftirqd/121:26:511
649999227,3cyclictest21-21ksoftirqd/123:12:411
649999226,4cyclictest21-21ksoftirqd/100:35:021
649999224,3cyclictest131rcu_preempt20:35:131
649999223,8cyclictest21-21ksoftirqd/119:15:151
649999223,5cyclictest21-21ksoftirqd/120:25:151
649999222,4cyclictest21-21ksoftirqd/122:47:301
648999221,16cyclictest5230-21ls23:20:240
648999221,15cyclictest9241-21cut20:20:150
6524992113,6cyclictest0-21swapper/321:57:053
652499211,12cyclictest35966irq/58-eth019:15:113
650899217,11cyclictest0-21swapper/219:25:112
650899217,10cyclictest27-21ksoftirqd/220:55:132
650899213,9cyclictest1914-21kthreadcore20:45:202
649999214,3cyclictest131rcu_preempt19:10:131
649999213,11cyclictest21-21ksoftirqd/121:05:011
649999211,3cyclictest131rcu_preempt20:05:011
648999217,9cyclictest1-21systemd00:23:300
648999211,15cyclictest28658-21rm23:50:320
652499209,8cyclictest0-21swapper/321:40:453
652499203,12cyclictest0-21swapper/300:35:133
652499202,11cyclictest0-21swapper/320:30:133
649999208,4cyclictest21-21ksoftirqd/120:10:221
649999208,4cyclictest21-21ksoftirqd/120:10:221
649999208,3cyclictest131rcu_preempt19:40:001
649999206,3cyclictest21-21ksoftirqd/120:55:251
649999204,8cyclictest21-21ksoftirqd/120:50:011
649999203,3cyclictest131rcu_preempt19:20:231
648999201,7cyclictest15313-21grep19:15:270
648999201,3cyclictest131rcu_preempt19:45:110
648999201,3cyclictest131rcu_preempt19:45:110
6489992013,4cyclictest536-21usb-storage21:10:110
648999201,2cyclictest131rcu_preempt21:10:010
648999201,16cyclictest22351-21uniq20:00:150
648999201,16cyclictest10921-21cut21:20:150
648999201,12cyclictest151rcuc/000:35:120
648999200,3cyclictest27902-21ssh22:21:120
652499194,8cyclictest0-21swapper/320:25:123
652499192,7cyclictest0-21swapper/300:10:133
652499192,13cyclictest0-21swapper/323:40:123
652499192,11cyclictest35966irq/58-eth000:30:113
6524991912,4cyclictest0-21swapper/321:15:203
6524991911,5cyclictest35966irq/58-eth022:20:133
6524991911,5cyclictest35966irq/58-eth021:30:133
6524991911,4cyclictest0-21swapper/323:15:293
6524991911,4cyclictest0-21swapper/323:15:293
6524991910,7cyclictest0-21swapper/323:25:323
6524991910,7cyclictest0-21swapper/322:00:253
6524991910,6cyclictest35966irq/58-eth000:05:193
6524991910,6cyclictest0-21swapper/300:03:143
649999196,4cyclictest21-21ksoftirqd/119:55:241
649999193,9cyclictest21-21ksoftirqd/121:05:161
648999196,4cyclictest131rcu_preempt20:40:180
648999192,15cyclictest21771-21kthreadcore23:30:200
648999191,14cyclictest2930-21ssh22:45:120
652499189,6cyclictest35966irq/58-eth020:15:133
652499182,10cyclictest0-21swapper/321:00:143
652499182,10cyclictest0-21swapper/320:55:133
652499182,10cyclictest0-21swapper/320:50:113
652499181,13cyclictest0-21swapper/323:35:023
652499181,12cyclictest0-21swapper/323:35:133
6524991810,6cyclictest0-21swapper/323:52:073
6524991810,5cyclictest35966irq/58-eth023:20:143
650899182,9cyclictest6553-21latency_hist19:10:022
649999189,3cyclictest21-21ksoftirqd/119:40:121
649999184,3cyclictest131rcu_preempt19:25:251
649999183,3cyclictest131rcu_preempt19:50:281
648999186,9cyclictest13762-21grep22:15:170
648999182,13cyclictest4334-21apt-get22:10:130
648999181,9cyclictest12976-21irqcore19:50:180
648999181,8cyclictest2858-21switchtime00:30:280
648999181,7cyclictest10386-21missed_timers23:05:210
6489991812,4cyclictest0-21swapper/019:20:230
648999181,15cyclictest2998-21kthreadcore20:45:210
6489991811,4cyclictest18512-21/usr/sbin/munin00:05:230
648999181,11cyclictest11225-21latency_hist19:15:020
648999181,10cyclictest5550-21ls23:40:140
652499179,6cyclictest0-21swapper/322:35:433
652499179,6cyclictest0-21swapper/322:17:293
652499179,6cyclictest0-21swapper/321:23:103
652499179,6cyclictest0-21swapper/300:25:103
652499177,6cyclictest0-21swapper/323:06:253
652499172,10cyclictest0-21swapper/323:45:133
6524991712,3cyclictest33-21ksoftirqd/319:35:163
648999178,6cyclictest23774-21ps20:35:160
648999174,7cyclictest131rcu_preempt20:55:180
648999171,9cyclictest28001-21kthreadcore19:30:200
648999171,6cyclictest12561-21kthreadcore23:25:210
648999171,4cyclictest131rcu_preempt21:00:190
6489991711,4cyclictest7776-21sendmail_mailqu20:15:260
648999171,13cyclictest30896-21kthreadcore21:30:200
648999171,13cyclictest24932-21grep19:25:270
648999170,3cyclictest0-21swapper/020:30:230
652499164,10cyclictest35966irq/58-eth020:20:113
652499161,8cyclictest0-21swapper/319:10:023
652499161,10cyclictest0-21swapper/319:20:133
6524991610,3cyclictest33-21ksoftirqd/320:10:283
6524991610,3cyclictest33-21ksoftirqd/320:10:283
649999167,6cyclictest19147-21tr20:30:171
649999167,3cyclictest21-21ksoftirqd/119:30:301
649999161,12cyclictest7973-21cut20:50:191
648999168,5cyclictest526-21kthreadcore20:10:180
648999168,5cyclictest526-21kthreadcore20:10:180
648999167,7cyclictest12-21ksoftirqd/019:40:110
648999161,10cyclictest30863-21fschecks_time19:35:150
652499152,7cyclictest0-21swapper/319:50:133
652499152,10cyclictest0-21swapper/320:05:143
652499151,7cyclictest0-21swapper/320:35:103
6524991511,2cyclictest33-21ksoftirqd/319:30:113
652499149,3cyclictest0-21swapper/319:55:253
652499146,6cyclictest0-21swapper/319:43:073
652499141,10cyclictest0-21swapper/321:10:013
648999141,2cyclictest0-21swapper/019:10:020
652499138,3cyclictest33-21ksoftirqd/320:00:123
652499137,4cyclictest33-21ksoftirqd/319:10:293
652499131,10cyclictest0-21swapper/320:40:283
649999131,8cyclictest6501-21latency_hist19:10:011
652499127,3cyclictest0-21swapper/319:45:133
652499127,3cyclictest0-21swapper/319:45:133
652499121,7cyclictest0-21swapper/320:50:023
652499121,5cyclictest0-21swapper/319:25:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional