You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 10:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon May 18, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31591996724,32cyclictest35966irq/58-eth023:35:133
3156999647,41cyclictest23982-21ssh00:30:151
3158399502,22cyclictest2011-21ssh00:00:132
31569994931,9cyclictest21-21ksoftirqd/100:05:121
3156999484,6cyclictest131rcu_preempt00:15:151
3156399473,32cyclictest16389-21ssh23:50:150
3156399472,34cyclictest17744-21ssh21:10:140
3158399466,25cyclictest27-21ksoftirqd/222:45:132
3156999463,32cyclictest12687-21ssh21:25:121
3156999462,36cyclictest32297-21ssh23:05:141
3156999453,34cyclictest23179-21ssh23:00:131
3156399453,31cyclictest11676-21ssh22:00:140
3156399442,35cyclictest5091-21ssh22:50:130
3159199431,26cyclictest35966irq/58-eth020:15:133
3158399433,24cyclictest26908-21ssh21:15:182
3156999421,32cyclictest3601-21ssh21:20:131
3156999413,8cyclictest131rcu_preempt20:40:121
3156999411,33cyclictest19416-21ssh22:40:121
31591994021,12cyclictest35966irq/58-eth022:10:133
3156999402,25cyclictest12650-21ssh23:30:131
3158399391,25cyclictest2514-21ssh21:55:132
3156999396,27cyclictest20097-21ssh00:10:131
3159199384,15cyclictest35966irq/58-eth021:20:133
3159199383,24cyclictest35966irq/58-eth022:40:143
31591993830,5cyclictest35966irq/58-eth023:05:153
31569993710,21cyclictest21-21ksoftirqd/123:40:131
3156399374,21cyclictest7647-21ssh21:40:130
3159199364,22cyclictest35966irq/58-eth000:15:143
31569993621,10cyclictest21-21ksoftirqd/122:20:151
3156999361,20cyclictest10330-21ssh22:35:111
3156999361,20cyclictest10330-21ssh22:35:101
3158399352,26cyclictest23194-21ssh23:00:152
3158399351,24cyclictest930-21snmpd20:35:132
3158399351,22cyclictest0-21swapper/200:30:152
3159199341,21cyclictest35966irq/58-eth000:20:133
3158399343,22cyclictest27-21ksoftirqd/223:25:132
3159199333,18cyclictest35966irq/58-eth022:35:103
3159199333,18cyclictest35966irq/58-eth022:35:103
31583993310,15cyclictest27-21ksoftirqd/221:25:122
31563993317,13cyclictest12-21ksoftirqd/023:10:150
3158399324,22cyclictest0-21swapper/222:40:142
3156999322,20cyclictest29757-21ssh22:10:131
3158399311,19cyclictest11660-21ssh22:00:142
31591993017,7cyclictest35966irq/58-eth021:30:143
3156399304,22cyclictest3584-21apt-get21:20:130
3156399302,12cyclictest12-21ksoftirqd/023:45:150
3156399301,21cyclictest6523-21ssh22:15:150
3156399301,21cyclictest5913-21ssh00:20:140
3159199291,20cyclictest35966irq/58-eth000:05:133
31591992910,10cyclictest0-21swapper/319:35:133
3158399295,2cyclictest131rcu_preempt19:40:132
3158399292,20cyclictest0-21swapper/200:10:132
31583992918,8cyclictest27-21ksoftirqd/223:50:142
3158399291,21cyclictest21782-21ssh21:30:132
31569992912,9cyclictest0-21swapper/122:15:131
31591992817,5cyclictest35966irq/58-eth000:10:133
3158399282,14cyclictest27083-21ssh23:20:152
3158399281,16cyclictest930-21snmpd22:35:122
3158399281,16cyclictest930-21snmpd22:35:112
3156999288,9cyclictest31553-21cyclictest23:55:131
3156999288,9cyclictest31553-21cyclictest23:55:121
3156999281,20cyclictest0-21swapper/100:35:131
3156999281,18cyclictest5887-21sh00:20:111
31569992810,7cyclictest21-21ksoftirqd/120:25:121
3158399276,18cyclictest27-21ksoftirqd/223:10:142
3158399261,13cyclictest21686-21ssh23:35:142
3158399260,21cyclictest17998-21ssh23:15:142
3158399260,21cyclictest17998-21ssh23:15:142
3156399269,8cyclictest12-21ksoftirqd/023:30:130
3159199252,10cyclictest35966irq/58-eth000:35:123
3158399253,16cyclictest27346irq/40-dwc2_hso21:00:142
3158399252,18cyclictest17336-21diskmemload22:30:142
3158399251,19cyclictest11062-21ssh00:05:142
3158399251,11cyclictest13869-21cut19:25:162
3158399251,10cyclictest5085-21ssh22:50:132
3156399257,12cyclictest12-21ksoftirqd/020:00:120
3156399255,5cyclictest12-21ksoftirqd/000:36:270
3156399255,14cyclictest20696-21ssh22:05:130
3159199242,13cyclictest0-21swapper/300:30:143
3158399247,8cyclictest3657-21ps23:05:272
3158399242,18cyclictest27346irq/40-dwc2_hso19:30:132
3158399242,17cyclictest27346irq/40-dwc2_hso19:15:142
3158399242,14cyclictest27346irq/40-dwc2_hso19:20:142
3158399242,13cyclictest27507-21cut22:25:222
31583992414,7cyclictest31553-21cyclictest19:55:122
31583992412,9cyclictest4119-21ssh21:36:202
3158399241,17cyclictest449-21diskmemload00:35:132
3158399241,11cyclictest3611-21ssh21:20:152
3158399240,21cyclictest6481-21latency_hist20:25:012
3156999247,8cyclictest131rcu_preempt22:10:011
3156999240,3cyclictest131rcu_preempt21:55:131
3156399248,6cyclictest12-21ksoftirqd/022:37:420
3156399248,6cyclictest12-21ksoftirqd/022:37:420
3156399246,9cyclictest24652-21ssh22:25:130
3156399246,10cyclictest12-21ksoftirqd/021:47:010
3156399244,3cyclictest131rcu_preempt22:24:080
3159199233,14cyclictest35966irq/58-eth023:55:133
3159199233,14cyclictest35966irq/58-eth023:55:133
3159199233,11cyclictest0-21swapper/322:30:143
3158399239,11cyclictest15381-21sed00:25:182
3158399239,11cyclictest15381-21sed00:25:182
3158399232,14cyclictest27346irq/40-dwc2_hso19:35:132
3158399231,7cyclictest6966-21apt23:45:012
3158399231,12cyclictest9286-21kthreadcore00:20:232
3156999232,13cyclictest21-21ksoftirqd/122:55:121
3156999232,11cyclictest0-21swapper/120:45:141
3156999231,14cyclictest30479-21ls23:40:021
3156399239,7cyclictest12-21ksoftirqd/022:41:090
3156399239,4cyclictest12-21ksoftirqd/023:23:130
3156399239,4cyclictest12-21ksoftirqd/021:59:180
3156399238,5cyclictest12-21ksoftirqd/021:00:140
3156399237,9cyclictest12-21ksoftirqd/022:47:020
3156399237,7cyclictest12-21ksoftirqd/021:52:350
3156399237,6cyclictest12-21ksoftirqd/023:06:560
3156399235,7cyclictest12-21ksoftirqd/021:18:130
3156399235,5cyclictest12-21ksoftirqd/020:25:270
3156399233,4cyclictest12-21ksoftirqd/000:29:410
3156399233,4cyclictest12-21ksoftirqd/000:29:400
3156399231,19cyclictest31150-21cut21:35:170
3156399231,18cyclictest25612-21uniq23:55:160
3156399231,18cyclictest25612-21uniq23:55:150
3156399231,17cyclictest4064-21find19:15:140
31563992310,7cyclictest0-21swapper/021:25:100
31563992310,6cyclictest12-21ksoftirqd/022:59:330
31563992310,5cyclictest12-21ksoftirqd/000:00:450
3159199222,13cyclictest36066irq/59-eth023:40:123
31591992213,6cyclictest0-21swapper/322:55:193
3159199221,13cyclictest0-21swapper/300:00:133
31591992210,9cyclictest0-21swapper/322:29:343
3158399228,10cyclictest14518-21kthreadcore23:30:202
3158399222,16cyclictest0-21swapper/223:58:002
3158399222,16cyclictest0-21swapper/223:58:002
3158399221,13cyclictest29999-21cut22:10:172
31583992210,5cyclictest19609-21ssh22:56:172
3156999223,11cyclictest0-21swapper/120:15:141
3156999221,15cyclictest25926-21ssh21:50:151
3156999221,14cyclictest844-21munin-node21:15:181
31569992210,3cyclictest131rcu_preempt23:10:181
3156399229,5cyclictest0-21swapper/023:27:380
3156399229,4cyclictest12-21ksoftirqd/021:32:310
3156399228,4cyclictest12-21ksoftirqd/000:11:050
3156399226,4cyclictest12-21ksoftirqd/000:08:210
3156399225,7cyclictest12-21ksoftirqd/023:41:010
3156399222,16cyclictest29786-21ssh22:10:130
31563992212,4cyclictest12-21ksoftirqd/000:16:240
31591992114,3cyclictest35966irq/58-eth023:20:133
31591992112,7cyclictest0-21swapper/321:50:023
3158399213,15cyclictest747-21fschecks_time19:45:172
3158399213,13cyclictest27346irq/40-dwc2_hso20:45:132
3158399212,15cyclictest13428-21kernelversion21:05:202
3158399212,15cyclictest11634-21fschecks_count20:30:162
3158399212,10cyclictest17278-21sed21:45:192
3158399211,9cyclictest472-21ps00:15:292
3158399211,16cyclictest20866-21fschecks_count22:05:162
3158399211,12cyclictest24592-21apt-get20:10:112
3158399211,11cyclictest9733-21kthreadcore20:25:212
3158399211,11cyclictest18518-21kthreadcore21:10:202
3158399211,11cyclictest10132-21ntp_192.168.11422:15:262
3158399211,10cyclictest21274-21fschecks_count20:40:162
31583992110,8cyclictest14991-21/usr/sbin/munin20:00:142
3156999215,13cyclictest28135-21latency_hist19:40:021
3156999212,14cyclictest21642-21ssh21:45:351
3156999211,17cyclictest17764-21ssh21:10:141
3156999211,15cyclictest4398-21cut19:15:201
3156999211,15cyclictest15185-21idleruntime00:25:171
3156999211,15cyclictest15185-21idleruntime00:25:171
3156999211,15cyclictest1283-21sh22:30:151
3156999211,11cyclictest0-21swapper/121:40:391
3156399219,4cyclictest12-21ksoftirqd/023:36:440
3156399218,5cyclictest0-21swapper/022:31:210
3156399217,3cyclictest131rcu_preempt20:10:150
3156399216,8cyclictest12-21ksoftirqd/023:15:100
3156399216,8cyclictest12-21ksoftirqd/023:15:090
3156399215,9cyclictest12-21ksoftirqd/023:04:360
31563992113,5cyclictest0-21swapper/020:55:110
31563992113,4cyclictest0-21swapper/019:20:130
31563992112,5cyclictest0-21swapper/020:50:140
3156399211,16cyclictest5353-21find19:50:120
31563992110,3cyclictest12-21ksoftirqd/020:40:280
3156399210,18cyclictest18548-21cpuspeed19:30:130
3156399210,17cyclictest11654-21cut20:30:160
3159199204,11cyclictest35966irq/58-eth022:20:153
31591992011,6cyclictest0-21swapper/321:15:263
31591992010,8cyclictest0-21swapper/321:55:483
31591992010,4cyclictest0-21swapper/323:45:233
3158399202,9cyclictest26097-21fschecks_count21:50:172
3158399201,9cyclictest19911-21fschecks_count20:05:172
3158399201,9cyclictest16962-21kthreadcore22:20:202
3158399201,9cyclictest10799-21tr23:45:252
3158399201,9cyclictest10111-21latency_hist19:55:022
3158399201,10cyclictest30920-21fschecks_time20:50:162
3156999201,16cyclictest8688-21missed_timers19:50:241
3156999201,13cyclictest370-21rm22:45:311
3156999200,17cyclictest31545-21ls19:40:231
3156999200,16cyclictest20048-21timerwakeupswit20:35:291
3156399208,4cyclictest12-21ksoftirqd/000:30:400
3156399207,3cyclictest131rcu_preempt20:05:230
3156399203,12cyclictest0-21swapper/020:15:140
3156399201,12cyclictest14759-21latency_hist20:00:020
3159199192,11cyclictest0-21swapper/320:05:153
3159199192,10cyclictest0-21swapper/320:35:153
31591991911,6cyclictest0-21swapper/321:35:183
31591991911,5cyclictest35966irq/58-eth023:50:153
31591991910,6cyclictest35966irq/58-eth023:00:153
31591991910,6cyclictest0-21swapper/321:44:433
3158399191,9cyclictest29587-21fschecks_time20:15:172
3158399191,15cyclictest31850-21fschecks_count19:10:162
3158399191,10cyclictest7956-21basename21:40:172
3158399190,9cyclictest3254-21fschecks_count20:55:152
3156999195,10cyclictest0-21swapper/121:30:131
31569991912,4cyclictest5832-21kthreadcore22:50:181
31569991912,4cyclictest537-21kthreadcore21:35:221
3156999191,15cyclictest5320-21ntp_kernel_err20:20:261
3156999191,15cyclictest11659-21ls21:00:291
3156999191,13cyclictest11787-21diskstats22:00:161
3156999190,13cyclictest10648-21latency_hist00:05:011
3156399196,2cyclictest131rcu_preempt19:30:010
3156399194,5cyclictest12-21ksoftirqd/020:45:280
31563991911,6cyclictest27816-21sshd19:36:590
3156399191,10cyclictest13007-21idleruntime21:05:170
31563991910,6cyclictest0-21swapper/019:10:160
3159199189,6cyclictest0-21swapper/322:15:153
3159199188,8cyclictest0-21swapper/321:27:133
3159199182,10cyclictest0-21swapper/320:25:123
31591991811,5cyclictest35966irq/58-eth022:00:143
3159199181,13cyclictest0-21swapper/319:40:153
3159199181,11cyclictest35966irq/58-eth021:10:143
31591991810,6cyclictest0-21swapper/321:52:283
31591991810,5cyclictest0-21swapper/322:45:143
3156999188,8cyclictest20346-21find23:50:291
3156999185,3cyclictest131rcu_preempt23:45:171
3156999183,12cyclictest857-21tr20:50:211
31569991812,4cyclictest0-21swapper/123:25:121
3156999181,13cyclictest18249-21munin-run19:30:011
31569991810,5cyclictest3836-21cron19:15:011
3156399186,4cyclictest12-21ksoftirqd/019:45:000
3156399185,3cyclictest131rcu_preempt19:45:140
3156399181,9cyclictest12-21ksoftirqd/020:21:360
3159199179,6cyclictest0-21swapper/323:28:163
3159199179,6cyclictest0-21swapper/323:15:363
3159199179,6cyclictest0-21swapper/323:15:353
3159199179,6cyclictest0-21swapper/323:10:503
3159199179,3cyclictest0-21swapper/300:28:483
3159199179,3cyclictest0-21swapper/300:28:483
3159199172,10cyclictest0-21swapper/322:05:143
3159199172,10cyclictest0-21swapper/320:40:143
3159199172,10cyclictest0-21swapper/319:50:153
3159199171,13cyclictest0-21swapper/322:50:193
31591991711,3cyclictest0-21swapper/320:30:133
3159199171,10cyclictest0-21swapper/320:00:283
3159199171,10cyclictest0-21swapper/319:55:143
3156999175,10cyclictest4664-21tr20:55:191
3156999171,14cyclictest22172-21timerandwakeup19:30:271
3156999171,14cyclictest16432-21threads21:05:291
3156999171,13cyclictest9557-21uname19:20:191
31569991711,3cyclictest27800-21kernelversion23:20:201
3156999171,13cyclictest20105-21grep20:05:181
3156999171,11cyclictest4249-21cat19:45:291
3156999171,11cyclictest18391-21grep23:15:191
3156999171,11cyclictest18391-21grep23:15:191
3156999171,11cyclictest0-21swapper/122:25:131
3156399171,3cyclictest131rcu_preempt20:35:150
3159199168,6cyclictest0-21swapper/323:31:193
3159199162,10cyclictest0-21swapper/319:45:143
3159199161,12cyclictest0-21swapper/319:15:313
3158399161,13cyclictest31472-21latency_hist19:10:012
3156999161,12cyclictest15377-21irqcore20:00:171
3156999161,12cyclictest11508-21find20:30:131
3156999161,11cyclictest10235-21ls19:55:131
31569991610,4cyclictest0-21swapper/120:15:011
3159199152,7cyclictest0-21swapper/320:45:133
3159199152,10cyclictest0-21swapper/320:55:123
31591991512,1cyclictest0-21swapper/321:05:143
3156999151,10cyclictest31644-21latency_hist19:10:021
3159199141,11cyclictest0-21swapper/320:55:013
3159199141,10cyclictest0-21swapper/320:20:313
3156399141,9cyclictest31661-21latency_hist19:10:020
3159199139,3cyclictest35966irq/58-eth021:00:133
3159199138,3cyclictest0-21swapper/319:20:273
3159199136,4cyclictest0-21swapper/319:25:123
3159199124,6cyclictest41-21kworker/3:119:10:283
3159199121,9cyclictest0-21swapper/320:10:293
3159199111,7cyclictest0-21swapper/319:30:113
3159199101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional