You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 06:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Jun 03, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1305699640,34cyclictest17849-21ssh21:10:141
1305699640,34cyclictest17849-21ssh21:10:131
13077996338,14cyclictest35966irq/58-eth021:10:133
13077996338,14cyclictest35966irq/58-eth021:10:133
1306799604,15cyclictest131rcu_preempt21:05:132
1306799583,35cyclictest25169-21ssh21:50:132
1305699566,33cyclictest3357-21ssh00:20:161
1306799552,34cyclictest18158-21ssh22:40:122
1306799551,39cyclictest30787-21ssh23:05:162
1305699542,38cyclictest7198-21ssh21:40:131
1305699513,39cyclictest26855-21ssh21:15:141
1305699513,39cyclictest26855-21ssh21:15:131
1306799485,19cyclictest27-21ksoftirqd/200:00:142
1306799481,31cyclictest26784-21ssh00:15:142
1307799463,36cyclictest0-21swapper/322:30:133
1305699454,4cyclictest0-21swapper/123:45:121
1307799447,24cyclictest35966irq/58-eth000:25:153
1307799436,22cyclictest35966irq/58-eth023:10:133
1304899430,34cyclictest23344-21ssh23:55:120
13077994222,10cyclictest35966irq/58-eth023:30:133
1306799418,27cyclictest27-21ksoftirqd/200:30:152
1306799404,16cyclictest261rcuc/220:55:122
1306799402,16cyclictest32567-21ssh22:30:132
1305699404,26cyclictest19815-21ssh23:35:121
1307799383,21cyclictest35966irq/58-eth021:40:133
1305699386,20cyclictest21-21ksoftirqd/120:50:121
13077993721,11cyclictest0-21swapper/323:20:133
1305699371,29cyclictest9126-21ssh22:35:121
1306799361,30cyclictest7286-21ssh23:10:112
1305699351,27cyclictest17440-21diskmemload22:10:141
13077993419,10cyclictest35966irq/58-eth021:20:143
13077993419,10cyclictest35966irq/58-eth021:20:143
13056993420,8cyclictest0-21swapper/122:25:121
1306799335,22cyclictest0-21swapper/221:55:122
1306799335,17cyclictest27-21ksoftirqd/221:00:142
1306799331,25cyclictest19830-21ssh23:35:132
1306799331,14cyclictest3822-21ssh22:50:132
1304899334,5cyclictest131rcu_preempt19:50:120
1306799322,14cyclictest27106-21ssh22:08:282
1306799313,23cyclictest261rcuc/223:45:122
13056993124,4cyclictest0-21swapper/119:40:081
13056993120,7cyclictest0-21swapper/100:30:141
13048993127,2cyclictest0-21swapper/000:35:130
1307799302,21cyclictest0-21swapper/321:55:123
1307799302,16cyclictest35966irq/58-eth000:05:173
1307799302,12cyclictest35966irq/58-eth000:10:143
13077993016,9cyclictest0-21swapper/322:50:133
13077993010,15cyclictest0-21swapper/322:55:143
1306799308,11cyclictest27-21ksoftirqd/200:35:142
1306799302,22cyclictest27238-21ssh22:45:142
1305699304,18cyclictest17440-21diskmemload21:30:211
1304899302,20cyclictest10968-21ssh23:10:250
1307799293,19cyclictest35966irq/58-eth022:35:133
13077992920,4cyclictest0-21swapper/320:25:113
1306799294,16cyclictest27346irq/40-dwc2_hso20:25:122
1306799292,21cyclictest0-21swapper/220:30:152
1305699294,16cyclictest21-21ksoftirqd/122:15:111
13048992922,4cyclictest14437-21sh23:50:150
13077992816,9cyclictest0-21swapper/322:45:133
1306799283,6cyclictest930-21snmpd20:35:132
1306799283,18cyclictest27346irq/40-dwc2_hso20:05:152
1306799282,19cyclictest0-21swapper/219:40:102
1306799282,18cyclictest27346irq/40-dwc2_hso21:40:112
13067992811,14cyclictest5666-21/usr/sbin/munin22:15:172
13048992821,4cyclictest12-21ksoftirqd/021:00:140
1307799275,14cyclictest0-21swapper/323:35:133
1306799273,20cyclictest27346irq/40-dwc2_hso21:15:152
1306799273,20cyclictest27346irq/40-dwc2_hso21:15:152
1306799272,15cyclictest3445-21ssh21:20:142
1306799272,15cyclictest3445-21ssh21:20:142
1305699272,19cyclictest16279-21ssh23:15:131
1305699271,20cyclictest12835-21ssh22:55:141
1304899273,19cyclictest28885-21ssh23:40:140
1307799267,12cyclictest35966irq/58-eth022:05:153
13077992620,2cyclictest0-21swapper/319:50:103
13077992618,5cyclictest0-21swapper/300:15:153
13077992616,6cyclictest0-21swapper/319:40:093
13077992614,5cyclictest0-21swapper/300:35:133
1307799261,17cyclictest35966irq/58-eth000:30:133
1307799261,16cyclictest35966irq/58-eth021:45:143
1306799264,12cyclictest27-21ksoftirqd/219:55:122
1306799263,16cyclictest27346irq/40-dwc2_hso20:10:132
1306799262,16cyclictest27346irq/40-dwc2_hso21:25:142
1305699263,16cyclictest0-21swapper/120:20:141
1305699263,16cyclictest0-21swapper/120:20:141
1305699262,17cyclictest25147-21/usr/sbin/munin21:50:281
1304899262,21cyclictest1682-21ssh21:55:120
1304899262,19cyclictest2151-21ssh22:48:290
1307799253,13cyclictest35966irq/58-eth022:40:113
13077992519,3cyclictest0-21swapper/323:55:133
13077992519,3cyclictest0-21swapper/319:45:123
13077992518,5cyclictest0-21swapper/321:15:153
13077992518,5cyclictest0-21swapper/321:15:153
1306799251,9cyclictest17312-21ssh22:02:342
1306799251,20cyclictest30684-21uniq21:35:162
1306799251,19cyclictest19305-21ssh23:50:342
1305699252,17cyclictest30522-21ssh21:35:131
1307799243,14cyclictest0-21swapper/321:50:283
13077992418,3cyclictest0-21swapper/323:00:133
13077992418,3cyclictest0-21swapper/319:10:113
13077992416,5cyclictest0-21swapper/300:00:133
1306799244,16cyclictest27346irq/40-dwc2_hso21:45:132
1306799242,18cyclictest2931-21ssh23:42:312
1306799241,19cyclictest21800-21ssh23:00:122
1306799241,10cyclictest9211-21fschecks_time00:05:212
13067992410,5cyclictest31092-21hwlatdetect20:50:172
1306799240,13cyclictest27431-21switchtime22:25:282
1305699241,18cyclictest14558-21ssh22:20:151
1307799239,6cyclictest0-21swapper/322:25:243
13077992317,3cyclictest0-21swapper/320:20:133
13077992317,3cyclictest0-21swapper/320:20:133
13077992317,2cyclictest0-21swapper/323:15:123
13077992312,8cyclictest0-21swapper/322:15:273
1306799233,15cyclictest25530-21fschecks_time23:20:172
1306799232,6cyclictest0-21swapper/222:11:282
1306799232,14cyclictest27346irq/40-dwc2_hso23:25:132
1306799231,16cyclictest0-21swapper/220:00:132
1305699232,15cyclictest5782-21ssh21:55:271
13056992315,4cyclictest32479-21ssh00:00:141
1305699231,16cyclictest17865-21ssh00:10:141
1305699231,15cyclictest30762-21ssh23:05:131
1304899237,12cyclictest1002-21powerfaild23:00:120
1304899237,11cyclictest12-21ksoftirqd/023:24:310
1304899237,11cyclictest12-21ksoftirqd/022:18:290
1304899231,19cyclictest10768-21ssh22:00:130
1304899231,18cyclictest10835-21ssh23:30:150
13077992218,2cyclictest0-21swapper/320:05:163
13077992217,2cyclictest0-21swapper/319:20:093
13077992216,2cyclictest0-21swapper/323:25:133
13077992216,2cyclictest0-21swapper/320:45:133
13077992214,4cyclictest0-21swapper/321:05:133
1306799223,4cyclictest16446-21latency19:10:212
1306799221,17cyclictest0-21swapper/222:55:152
1306799221,13cyclictest3329-21apt-get00:20:152
1306799221,11cyclictest29047-21kthreadcore20:45:222
13067992210,5cyclictest24161-21ssh00:12:182
1306799220,11cyclictest14705-21ssh23:30:302
1305699226,8cyclictest21-21ksoftirqd/120:05:161
1305699222,15cyclictest0-21swapper/122:03:281
1305699221,15cyclictest1832-21rm23:25:141
13056992211,3cyclictest131rcu_preempt22:30:301
1304899222,17cyclictest26808-21ssh00:15:160
1304899221,17cyclictest18169-21ssh22:40:120
1304899221,16cyclictest16155-21ssh21:45:140
1304899221,10cyclictest20525-21kthreadcore00:10:230
1307799218,10cyclictest0-21swapper/321:31:543
13077992115,3cyclictest0-21swapper/321:35:143
13077992115,3cyclictest0-21swapper/319:30:093
13077992115,2cyclictest0-21swapper/300:20:143
13077992111,8cyclictest0-21swapper/321:25:033
13077992111,5cyclictest0-21swapper/320:55:123
1307799211,11cyclictest35966irq/58-eth023:05:163
1306799217,11cyclictest0-21swapper/222:20:242
1306799213,9cyclictest19685-21kthreadcore21:10:222
1306799213,9cyclictest19685-21kthreadcore21:10:222
1306799213,10cyclictest18173-21latency_hist19:20:022
1306799211,16cyclictest27346irq/40-dwc2_hso00:25:142
1306799211,11cyclictest28248-21users23:55:332
1306799211,11cyclictest28132-21apt-get19:40:022
1306799211,11cyclictest22845-21apt19:25:012
1306799210,12cyclictest22013-21if_eth121:30:262
1305699212,15cyclictest19794-21rm22:05:151
1305699211,17cyclictest12361-21cut00:25:171
1304899211,17cyclictest7143-21ssh21:20:270
1304899211,17cyclictest7143-21ssh21:20:270
1304899211,16cyclictest12546-21ssh21:25:150
1307799209,8cyclictest0-21swapper/321:00:243
1307799203,11cyclictest0-21swapper/322:00:293
13077992016,2cyclictest0-21swapper/323:45:113
13077992016,2cyclictest0-21swapper/320:15:133
13077992014,3cyclictest0-21swapper/322:20:133
13077992011,6cyclictest0-21swapper/323:40:123
13077992011,6cyclictest0-21swapper/320:37:183
13077992010,7cyclictest0-21swapper/323:50:133
1306799201,9cyclictest8966-21cat19:50:212
1306799201,9cyclictest16724-21cut23:15:172
1306799201,8cyclictest5395-21apt-get19:50:012
1306799201,10cyclictest32542-21tr20:15:212
1306799201,10cyclictest23445-21cut19:30:122
1306799201,10cyclictest10592-21grep22:35:202
1306799200,16cyclictest21477-21cut20:40:172
1305699202,12cyclictest1306-21ssh22:47:291
1305699201,15cyclictest27003-21ntp_192.168.11423:55:261
1305699201,13cyclictest3454-21sh21:20:151
1305699201,13cyclictest3454-21sh21:20:141
1305699200,17cyclictest1455-21ls23:25:011
1304899209,8cyclictest6775-21cron21:40:010
1304899209,8cyclictest21859-21ps00:30:200
1304899208,9cyclictest14089-21cat23:50:010
1304899208,9cyclictest12-21ksoftirqd/022:10:130
1304899207,4cyclictest131rcu_preempt21:10:000
1304899207,10cyclictest1-21systemd20:37:170
13048992012,6cyclictest25498-21sh21:50:190
1304899201,16cyclictest5714-21switchtime23:25:280
13048992011,6cyclictest0-21swapper/019:55:140
1307799199,8cyclictest0-21swapper/320:40:283
1307799198,4cyclictest0-21swapper/322:10:243
13077991910,6cyclictest0-21swapper/320:30:233
13077991910,6cyclictest0-21swapper/320:00:273
1306799192,9cyclictest2993-21kthreadcore20:20:192
1306799192,9cyclictest2993-21kthreadcore20:20:182
1305699196,4cyclictest131rcu_preempt19:45:181
13056991913,4cyclictest0-21swapper/121:25:021
13056991910,6cyclictest15436-21cat20:30:291
1305699190,16cyclictest30738-21kthreadcore00:35:201
1304899199,7cyclictest17955-21df21:10:160
1304899199,7cyclictest17955-21df21:10:150
1304899198,3cyclictest131rcu_preempt19:50:010
1304899195,7cyclictest28649-21kthreadcore21:15:220
1304899195,7cyclictest28649-21kthreadcore21:15:210
1304899193,13cyclictest22893-21latency_hist19:25:010
1304899192,14cyclictest28627-21fschecks_time19:40:120
1304899192,14cyclictest22224-21ps21:30:260
1304899191,9cyclictest15673-21kthreadcore22:20:200
13048991912,5cyclictest13039-21cyclictest22:30:120
1304899191,15cyclictest3474-21ssh00:20:180
1304899191,15cyclictest12282-21cpuspeed00:25:150
13048991911,5cyclictest12-21ksoftirqd/020:30:140
1304899191,14cyclictest29786-21diskstats20:15:160
1304899191,13cyclictest18018-21latency_hist19:20:010
1304899191,13cyclictest11445-21latency_hist20:30:010
1307799182,10cyclictest0-21swapper/320:10:133
13077991813,3cyclictest0-21swapper/319:55:133
1306799181,13cyclictest23078-21cron19:30:012
1305699189,6cyclictest22836-21cron19:25:001
1305699185,4cyclictest17440-21diskmemload00:18:051
1305699182,4cyclictest12827-21apt-get21:05:131
1305699181,7cyclictest14547-21diskstats23:50:161
13056991811,4cyclictest30364-21latency_hist23:05:021
1305699181,14cyclictest13746-21latency19:55:231
1305699181,13cyclictest16308-21tune2fs21:45:151
1305699181,12cyclictest30830-21kthreadcore23:40:191
13056991810,6cyclictest19603-21cut22:40:191
13056991810,6cyclictest13833-21sshd23:12:311
13056991810,6cyclictest0-21swapper/123:35:011
13056991810,5cyclictest23966-21uname19:30:141
1304899188,7cyclictest20403-21cut22:05:200
1304899186,7cyclictest0-21swapper/022:35:140
1304899186,3cyclictest131rcu_preempt21:40:140
1304899184,9cyclictest0-21swapper/020:40:130
1304899181,8cyclictest15781-21irqrtprio20:00:190
1304899181,14cyclictest28014-21kthreadcore20:10:210
1304899181,12cyclictest24651-21latency_hist20:10:020
1304899180,15cyclictest23683-21df_inode22:25:140
1307799178,6cyclictest0-21swapper/320:50:133
1305699179,5cyclictest0-21swapper/120:55:151
1305699179,5cyclictest0-21swapper/120:25:171
1305699173,4cyclictest21-21ksoftirqd/120:00:241
1305699172,11cyclictest0-21swapper/120:10:141
1305699172,10cyclictest0-21swapper/120:45:151
1305699172,10cyclictest0-21swapper/119:10:131
13056991711,4cyclictest0-21swapper/121:00:281
1305699171,13cyclictest9340-21ssh00:05:221
13056991710,5cyclictest0-21swapper/120:35:321
1305699170,14cyclictest3838-21sh22:50:141
1305699170,10cyclictest10400-21ls19:55:021
1304899178,7cyclictest3896-21sort22:50:140
1304899176,8cyclictest12-21ksoftirqd/019:30:090
1304899175,9cyclictest2224-21fschecks_count20:20:160
1304899175,9cyclictest2224-21fschecks_count20:20:160
1304899172,12cyclictest20523-21grep23:35:170
1304899172,12cyclictest13347-21cut19:10:150
1304899171,7cyclictest3031-21awk20:55:020
1304899171,4cyclictest0-21swapper/023:05:170
13048991711,4cyclictest26412-21tail20:45:190
13048991711,4cyclictest16350-21sh23:15:130
1304899171,13cyclictest8993-21ssh00:05:160
1304899171,13cyclictest12923-21df22:55:150
1304899171,12cyclictest28373-21cat19:40:010
1304899170,15cyclictest6507-21latency20:55:200
1305699161,12cyclictest21295-21apt-get20:40:121
1305699161,11cyclictest28440-21latency_hist19:40:021
1305699161,10cyclictest32012-21kthreadcore20:15:211
1304899161,11cyclictest32591-21diskstats00:00:160
1304899161,11cyclictest23077-21cron19:30:000
1306799158,4cyclictest13165-21latency_hist19:10:022
1305699157,6cyclictest23056-21sshd19:25:271
13077991410,2cyclictest0-21swapper/319:40:013
13077991410,2cyclictest0-21swapper/319:28:273
13077991410,2cyclictest0-21swapper/319:20:023
1305699146,6cyclictest17989-21sshd19:15:271
1305699141,9cyclictest13105-21latency_hist19:10:021
1304899141,9cyclictest13146-21latency_hist19:10:010
1307799131,10cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional