You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-28 - 02:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Sep 28, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1533499644,50cyclictest930-21snmpd22:20:121
1533499566,41cyclictest5515-21ssh00:30:111
1533499562,7cyclictest21-21ksoftirqd/121:55:141
1533499562,7cyclictest21-21ksoftirqd/121:55:131
1533499503,39cyclictest8572-21ssh21:50:121
1533499495,37cyclictest2042-21ssh00:10:121
1533499492,33cyclictest20002-21ssh22:50:121
1534299472,35cyclictest26767-21ssh22:00:142
1533499472,33cyclictest21944-21ssh23:45:121
1534299464,32cyclictest0-21swapper/221:30:142
1533499461,37cyclictest12390-21ssh22:10:111
1533499453,32cyclictest0-21swapper/123:30:121
1533499453,32cyclictest0-21swapper/123:30:121
1533499434,4cyclictest21-21ksoftirqd/121:00:141
1534299421,12cyclictest16894-21tail20:15:292
1535399412,22cyclictest35966irq/58-eth021:55:133
1535399412,22cyclictest35966irq/58-eth021:55:133
1534299410,17cyclictest19822-21ls00:20:022
1534299404,29cyclictest21430-21ssh22:15:132
1533499407,5cyclictest131rcu_preempt19:50:141
1534299392,27cyclictest27-21ksoftirqd/223:25:112
1534299391,9cyclictest30077-21cut21:25:202
1532599394,25cyclictest12-21ksoftirqd/021:05:130
1534299386,5cyclictest15936-21latency_hist22:30:012
1534299382,9cyclictest7605-21ssh00:11:222
1534299382,12cyclictest12355-21/usr/sbin/munin22:10:112
1534299381,9cyclictest25995-21cat20:25:232
1534299381,14cyclictest13956-21kthreadcore23:40:202
1533499381,12cyclictest981-21diskmemload22:27:241
1532599384,31cyclictest18321-21ssh23:25:120
1535399374,18cyclictest35966irq/58-eth021:50:123
15353993631,3cyclictest35966irq/58-eth023:00:133
1534299363,7cyclictest15086-21irqcore23:05:172
1534299363,27cyclictest10358-21ssh21:15:132
15342993625,7cyclictest30941-21ssh23:50:122
1534299361,9cyclictest9118-21kthreadcore20:10:192
1534299361,14cyclictest18161-21ssh21:55:182
1534299361,14cyclictest18161-21ssh21:55:182
1534299353,12cyclictest3968-21ssh23:35:142
1534299352,9cyclictest1952-21kthreadcore20:35:202
1534299352,14cyclictest25160-21ssh22:35:142
1534299351,8cyclictest28639-21sh22:55:012
15342993513,3cyclictest10876-21ssh23:20:192
1534299351,14cyclictest31426-21kthreadcore22:20:192
1533499352,6cyclictest21-21ksoftirqd/119:55:141
1532599352,30cyclictest4989-21ssh21:30:140
1534299343,25cyclictest0-21swapper/222:32:212
1534299341,15cyclictest23030-21kthreadcore23:45:202
1533499345,22cyclictest21-21ksoftirqd/121:45:121
1532599344,27cyclictest10367-21ssh21:15:130
15325993417,12cyclictest2273-21apt-get19:30:150
1534299335,6cyclictest30871-21tr20:00:012
1534299332,23cyclictest19399-21ssh21:20:152
1534299331,9cyclictest26712-21ntp_states21:40:272
1534299331,8cyclictest11161-21fschecks_time22:45:162
1534299331,10cyclictest12729-21tail21:50:272
1534299330,8cyclictest14477-21cut19:40:222
1533499337,19cyclictest21-21ksoftirqd/123:55:121
1534299321,13cyclictest20037-21sh00:20:152
1533499323,19cyclictest25166-21ssh22:35:151
1533499322,22cyclictest3827-21sed23:15:211
1532599322,23cyclictest17699-21ssh21:55:150
1532599322,23cyclictest17699-21ssh21:55:140
1532599322,22cyclictest2032-21memory23:50:220
1535399311,23cyclictest35966irq/58-eth021:40:133
1534299313,18cyclictest27346irq/40-dwc2_hso19:10:132
1533499316,17cyclictest21-21ksoftirqd/120:30:141
1533499312,21cyclictest5824-21cut22:05:211
1532599315,20cyclictest12-21ksoftirqd/023:10:140
1532599313,20cyclictest15559-21kworker/u8:322:05:130
15325993124,3cyclictest26280-21apt-get19:55:130
15353993020,7cyclictest35966irq/58-eth023:25:143
1534299307,7cyclictest25956-21irqcore00:05:172
1533499308,4cyclictest131rcu_preempt00:15:131
1533499304,19cyclictest10914-21ssh22:45:131
15353992915,9cyclictest35966irq/58-eth022:40:133
1534299294,13cyclictest1884-21ssh22:40:132
1534299293,21cyclictest0-21swapper/223:32:372
1534299293,21cyclictest0-21swapper/223:32:372
1534299292,20cyclictest0-21swapper/223:55:122
1534299292,18cyclictest28975-21ssh00:25:122
1534299292,14cyclictest23736-21ssh23:10:152
1533499295,5cyclictest131rcu_preempt21:15:131
15353992819,6cyclictest35966irq/58-eth000:20:133
1533499286,16cyclictest131rcu_preempt22:15:131
1533499282,15cyclictest21-21ksoftirqd/119:35:131
1532599286,7cyclictest12-21ksoftirqd/021:45:170
1532599282,22cyclictest27828-21sh22:17:210
1532599282,21cyclictest30403-21ssh22:00:250
15325992819,3cyclictest27481-21apt-get20:30:120
15325992818,5cyclictest19124-21apt-get20:55:120
1532599280,12cyclictest73-21kswapd022:55:150
1535399277,9cyclictest35966irq/58-eth022:55:123
1534299274,14cyclictest27346irq/40-dwc2_hso19:30:152
1534299272,17cyclictest27346irq/40-dwc2_hso20:45:112
1533499277,13cyclictest131rcu_preempt23:50:121
1533499271,16cyclictest13964-21ssh21:35:121
1532599278,5cyclictest12-21ksoftirqd/023:20:010
1532599276,5cyclictest12-21ksoftirqd/021:10:300
1532599275,4cyclictest12-21ksoftirqd/022:35:130
1532599274,6cyclictest12-21ksoftirqd/000:00:230
1532599274,16cyclictest19383-21ssh21:20:130
1532599273,10cyclictest3342-21ntp_192.168.11420:35:240
1532599272,21cyclictest22517-21ssh22:32:210
1532599272,20cyclictest5634-21ssh00:10:220
1532599272,20cyclictest11496-21kthreadcore21:50:200
1535399263,15cyclictest0-21swapper/322:17:213
1535399262,17cyclictest35966irq/58-eth022:20:133
1535399262,12cyclictest35966irq/58-eth022:00:133
1534299263,14cyclictest27346irq/40-dwc2_hso19:15:122
1532599268,4cyclictest12-21ksoftirqd/022:50:290
1532599266,4cyclictest151rcuc/023:39:070
1532599264,5cyclictest12-21ksoftirqd/023:55:230
1532599263,8cyclictest18138-21ssh22:48:160
1532599263,5cyclictest151rcuc/021:28:560
1532599263,5cyclictest12-21ksoftirqd/019:40:010
1532599262,6cyclictest12-21ksoftirqd/000:22:390
1532599262,17cyclictest14714-21ssh23:05:130
1532599261,6cyclictest23105-21df21:40:140
1535399252,19cyclictest35966irq/58-eth020:10:123
1535399252,15cyclictest35966irq/58-eth023:10:133
1534299252,12cyclictest930-21snmpd19:35:132
1533499254,14cyclictest21-21ksoftirqd/120:55:111
15334992516,4cyclictest0-21swapper/100:25:111
1532599257,5cyclictest12-21ksoftirqd/000:05:250
1532599256,6cyclictest12-21ksoftirqd/000:15:250
1532599255,8cyclictest0-21swapper/020:50:150
1532599255,8cyclictest0-21swapper/020:50:150
1532599255,5cyclictest12-21ksoftirqd/000:35:210
1532599254,6cyclictest32543-21cut20:00:190
1532599254,6cyclictest12-21ksoftirqd/023:00:150
1532599254,5cyclictest12-21ksoftirqd/019:40:280
15325992518,3cyclictest9537-21apt-get20:45:110
15325992518,3cyclictest21471-21apt-get19:50:130
15325992512,8cyclictest28949-21apt-get00:25:120
1532599251,19cyclictest12994-21ssh23:40:160
1532599250,8cyclictest22795-21kthreadcore23:45:190
1534299241,12cyclictest16599-21ssh00:00:132
1533499245,4cyclictest131rcu_preempt23:35:151
1533499244,5cyclictest131rcu_preempt21:40:121
1533499242,7cyclictest131rcu_preempt22:03:211
15334992412,5cyclictest19079-21usage_EthcTask00:35:311
1533499241,16cyclictest29129-21ssh22:55:131
1532599247,4cyclictest0-21swapper/020:10:140
1532599245,4cyclictest151rcuc/022:12:180
1532599244,6cyclictest27574-21cut19:20:210
1532599244,5cyclictest12-21ksoftirqd/023:20:200
1532599243,7cyclictest6855-21grep00:30:190
1532599243,6cyclictest12-21ksoftirqd/019:45:250
1532599243,4cyclictest151rcuc/022:42:080
1532599243,14cyclictest844-21munin-node19:25:130
15325992418,2cyclictest23954-21apt-get21:00:140
1532599241,4cyclictest0-21swapper/020:15:130
15325992413,8cyclictest7067-21apt-get22:25:120
1532599240,8cyclictest30682-21/usr/sbin/munin22:20:150
1535399234,15cyclictest35966irq/58-eth020:35:123
1535399232,10cyclictest0-21swapper/323:20:133
15353992318,3cyclictest321rcuc/300:30:133
1535399231,15cyclictest35966irq/58-eth000:05:123
1534299235,7cyclictest2746ksoftirqd/220:00:132
1534299232,6cyclictest0-21swapper/222:08:212
1534299232,6cyclictest0-21swapper/221:49:212
1534299232,16cyclictest0-21swapper/200:33:222
1534299231,9cyclictest5706-21ssh23:00:132
15334992312,9cyclictest0-21swapper/123:26:531
1533499231,14cyclictest4497-21kworker/1:121:10:061
1532599236,5cyclictest12-21ksoftirqd/021:35:220
1532599234,6cyclictest12-21ksoftirqd/019:15:190
1532599234,5cyclictest12-21ksoftirqd/020:40:250
1532599234,4cyclictest12-21ksoftirqd/023:30:180
1532599234,4cyclictest12-21ksoftirqd/023:30:180
1532599233,9cyclictest0-21swapper/020:20:140
1532599233,6cyclictest12-21ksoftirqd/020:05:280
15325992316,2cyclictest0-21swapper/020:25:130
1535399223,13cyclictest0-21swapper/322:32:213
1535399223,13cyclictest0-21swapper/322:26:213
1535399222,9cyclictest0-21swapper/320:15:123
1535399222,17cyclictest35966irq/58-eth019:45:133
1535399222,14cyclictest35966irq/58-eth023:05:123
15353992212,4cyclictest0-21swapper/300:18:393
1534299227,5cyclictest30203-21kthreadcore22:55:192
1534299222,6cyclictest1392-21sh21:10:132
1534299222,15cyclictest0-21swapper/223:18:212
1534299222,14cyclictest0-21swapper/219:50:152
1534299221,11cyclictest18160-21ssh00:35:242
1534299221,10cyclictest25200-21kthreadcore21:00:212
15334992212,6cyclictest131rcu_preempt21:25:111
15334992211,3cyclictest131rcu_preempt20:25:011
15334992210,3cyclictest21-21ksoftirqd/100:05:161
1535399213,13cyclictest0-21swapper/300:10:223
15353992115,4cyclictest321rcuc/323:55:533
15353992115,4cyclictest321rcuc/323:34:313
15353992115,4cyclictest321rcuc/323:34:303
15353992113,5cyclictest0-21swapper/323:48:323
15353992113,5cyclictest0-21swapper/322:11:213
15353992113,5cyclictest0-21swapper/321:49:133
15353992113,5cyclictest0-21swapper/321:16:243
15353992113,5cyclictest0-21swapper/300:25:393
1535399211,16cyclictest35966irq/58-eth022:50:133
1535399211,13cyclictest35966irq/58-eth022:05:143
1534299214,14cyclictest20406-21kthreadcore20:20:222
1534299211,9cyclictest20617-21kthreadcore20:55:182
1534299211,17cyclictest0-21swapper/219:45:282
1534299211,16cyclictest16773-21kthreadcore20:50:232
1534299211,16cyclictest16773-21kthreadcore20:50:232
1534299211,10cyclictest30001-21fschecks_count19:25:152
1534299211,10cyclictest16648-21kthreadcore21:35:222
1533499219,4cyclictest21-21ksoftirqd/119:25:291
1533499219,3cyclictest21-21ksoftirqd/120:40:231
1533499218,3cyclictest21-21ksoftirqd/123:00:181
1533499216,3cyclictest131rcu_preempt22:30:221
1533499215,4cyclictest2268-21ps22:40:181
1533499213,16cyclictest21-21ksoftirqd/119:45:131
1533499211,6cyclictest12942-21ntp_kernel_pll_23:20:241
15334992115,3cyclictest21-21ksoftirqd/100:20:301
15334992114,4cyclictest21-21ksoftirqd/120:05:191
1533499211,17cyclictest16598-21ssh00:00:131
15334992110,9cyclictest0-21swapper/123:07:501
15334992110,9cyclictest0-21swapper/121:33:461
1532599213,4cyclictest12-21ksoftirqd/019:10:280
15353992015,3cyclictest321rcuc/321:36:023
15353992014,4cyclictest321rcuc/320:20:133
15353992014,4cyclictest0-21swapper/323:51:203
15353992014,4cyclictest0-21swapper/321:25:283
1535399201,16cyclictest0-21swapper/323:40:183
1535399201,11cyclictest35966irq/58-eth023:15:123
1534299204,8cyclictest8063-21cron20:10:012
1534299202,15cyclictest27492-21kthreadcore19:20:202
1534299201,5cyclictest0-21swapper/219:10:022
1534299201,11cyclictest28374-21kthreadcore20:30:192
15342992010,4cyclictest0-21swapper/221:05:042
1533499209,3cyclictest131rcu_preempt23:10:231
1533499209,3cyclictest131rcu_preempt21:20:291
15334992014,3cyclictest21-21ksoftirqd/123:40:211
15334992011,7cyclictest19983-21sshd19:13:211
1535399193,12cyclictest36066irq/59-eth022:45:123
1535399192,11cyclictest0-21swapper/319:15:133
15353991913,4cyclictest0-21swapper/323:35:183
15353991913,4cyclictest0-21swapper/322:37:223
15353991913,4cyclictest0-21swapper/321:23:043
15353991913,4cyclictest0-21swapper/321:13:263
15353991913,4cyclictest0-21swapper/321:00:183
15353991913,4cyclictest0-21swapper/300:36:123
15353991913,4cyclictest0-21swapper/300:03:193
1535399191,13cyclictest0-21swapper/321:10:013
15353991910,7cyclictest0-21swapper/320:45:123
1534299191,9cyclictest5090-21grep20:40:182
15334991914,3cyclictest21-21ksoftirqd/120:25:161
15334991913,4cyclictest0-21swapper/121:05:191
1533499191,2cyclictest131rcu_preempt19:30:331
1533499191,2cyclictest131rcu_preempt19:15:251
1533499191,16cyclictest9297-21kworker/1:220:45:181
15353991813,3cyclictest321rcuc/319:45:013
15353991813,3cyclictest321rcuc/319:15:013
15353991812,4cyclictest0-21swapper/321:30:333
15353991812,4cyclictest0-21swapper/319:55:283
1535399181,11cyclictest0-21swapper/319:20:213
1533499188,2cyclictest21-21ksoftirqd/120:15:211
1533499181,3cyclictest131rcu_preempt20:10:121
15334991813,3cyclictest21-21ksoftirqd/119:40:211
15334991811,4cyclictest21-21ksoftirqd/120:50:191
15334991811,4cyclictest21-21ksoftirqd/120:50:181
15334991811,4cyclictest21-21ksoftirqd/120:35:141
15334991810,6cyclictest29627-21sshd19:23:211
1535399175,4cyclictest0-21swapper/320:25:123
15353991712,3cyclictest0-21swapper/320:55:133
15353991712,3cyclictest0-21swapper/320:40:123
15353991712,3cyclictest0-21swapper/319:51:533
15353991711,4cyclictest0-21swapper/320:50:143
15353991711,4cyclictest0-21swapper/320:50:143
15353991711,4cyclictest0-21swapper/320:05:183
15353991711,4cyclictest0-21swapper/319:25:133
1533499172,2cyclictest131rcu_preempt20:00:191
15353991610,4cyclictest0-21swapper/320:30:203
15353991610,4cyclictest0-21swapper/319:30:143
15334991611,3cyclictest21-21ksoftirqd/119:10:031
1532599162,10cyclictest15397-21latency_hist19:10:020
15353991510,3cyclictest0-21swapper/320:00:153
1535399148,4cyclictest0-21swapper/319:39:343
1535399111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional