You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 12:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Jul 23, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1150999531,4cyclictest131rcu_preempt21:25:141
1152199471,28cyclictest22430-21ssh23:35:132
1152199443,16cyclictest131rcu_preempt23:40:152
1152199437,26cyclictest261rcuc/219:40:142
1152199432,29cyclictest4859-21rm21:50:132
1153099421,28cyclictest35966irq/58-eth021:30:143
1152199424,29cyclictest261rcuc/200:25:122
1152199424,29cyclictest261rcuc/200:25:122
1152199397,27cyclictest15690-21ssh21:20:142
1153099373,27cyclictest35966irq/58-eth000:05:143
11530993726,7cyclictest35966irq/58-eth022:00:123
11521993715,19cyclictest27-21ksoftirqd/222:05:122
11521993710,19cyclictest27-21ksoftirqd/220:05:132
1150999361,2cyclictest131rcu_preempt00:30:111
1152199358,11cyclictest27-21ksoftirqd/222:50:132
1153099346,19cyclictest35966irq/58-eth023:10:133
1153099346,19cyclictest35966irq/58-eth023:10:123
1153099343,22cyclictest35966irq/58-eth021:45:143
1152199342,26cyclictest27-21ksoftirqd/222:15:122
1152199332,24cyclictest135-21systemd-journal21:00:242
1150199333,26cyclictest15675-21ssh21:20:120
1153099321,20cyclictest35966irq/58-eth023:55:143
11530993211,16cyclictest0-21swapper/321:35:143
1152199322,24cyclictest1014-21ssh23:05:142
1150199321,23cyclictest10630-21ssh00:25:110
1150199321,23cyclictest10630-21ssh00:25:100
1153099313,16cyclictest35966irq/58-eth019:15:133
1153099311,22cyclictest35966irq/58-eth000:25:113
1153099311,22cyclictest35966irq/58-eth000:25:113
1152199313,16cyclictest27346irq/40-dwc2_hso19:10:132
1152199312,18cyclictest18991-21ssh23:15:162
1153099305,15cyclictest35966irq/58-eth023:00:143
1152199307,13cyclictest27346irq/40-dwc2_hso22:45:142
1152199303,18cyclictest27346irq/40-dwc2_hso21:10:132
1152199302,11cyclictest10312-21ssh21:35:152
1150999302,22cyclictest27356-21rm23:35:381
1153099293,14cyclictest35966irq/58-eth019:40:143
1152199294,15cyclictest29669-21diskmemload23:25:142
1152199293,20cyclictest0-21swapper/219:15:142
1152199292,22cyclictest930-21snmpd19:25:132
1152199292,20cyclictest27346irq/40-dwc2_hso21:55:142
1152199292,20cyclictest27346irq/40-dwc2_hso21:55:142
1152199292,14cyclictest19359-21ssh21:40:142
1150999293,13cyclictest27957-21ssh23:20:141
1150999292,21cyclictest15433-21ssh22:55:131
1153099285,11cyclictest35966irq/58-eth023:35:133
1153099283,19cyclictest0-21swapper/323:25:153
1153099282,13cyclictest35966irq/58-eth021:20:133
1152199286,3cyclictest27-21ksoftirqd/223:10:122
1152199286,3cyclictest27-21ksoftirqd/223:10:122
1152199283,17cyclictest27346irq/40-dwc2_hso00:35:132
1152199283,16cyclictest27346irq/40-dwc2_hso19:55:132
11521992817,5cyclictest27-21ksoftirqd/220:55:142
1152199281,18cyclictest0-21swapper/220:40:132
1150199280,18cyclictest31797-21ssh22:05:140
1153099273,16cyclictest0-21swapper/322:40:113
1152199273,21cyclictest25045-21ls21:25:182
1152199273,14cyclictest27346irq/40-dwc2_hso22:00:122
1152199272,12cyclictest0-21swapper/200:10:062
1152199272,10cyclictest0-21swapper/219:20:132
1150999272,20cyclictest1329-21ssh21:30:151
1150999272,19cyclictest7266-21ssh23:44:371
1150999272,18cyclictest16893-21ssh22:35:441
1150199272,19cyclictest6439-21sh23:43:370
11501992711,11cyclictest12-21ksoftirqd/022:50:130
1150199270,18cyclictest12-21ksoftirqd/021:35:140
1153099261,15cyclictest35966irq/58-eth023:30:143
1152199261,14cyclictest844-21munin-node23:30:152
1150999262,18cyclictest21719-21ssh23:34:381
1150199264,12cyclictest26541-21ssh22:20:130
1150199262,18cyclictest26493-21ssh00:32:580
11530992516,5cyclictest35966irq/58-eth000:35:153
1152199253,16cyclictest27346irq/40-dwc2_hso20:25:142
1152199252,15cyclictest27346irq/40-dwc2_hso20:00:132
1152199251,8cyclictest9135-21latency_hist20:15:022
11521992513,9cyclictest0-21swapper/219:45:172
1152199251,20cyclictest24050-21sendmail23:00:012
11509992518,3cyclictest0-21swapper/122:10:151
1150999251,19cyclictest29669-21diskmemload00:08:431
1150199251,18cyclictest29669-21diskmemload00:15:120
1152199249,12cyclictest8483-21sh22:10:162
1152199247,15cyclictest24746-21sh22:40:252
1152199244,15cyclictest27346irq/40-dwc2_hso23:00:132
1152199242,17cyclictest0-21swapper/222:39:382
11521992413,9cyclictest15259-21ssh23:48:262
11521992411,11cyclictest19061-21ssh00:08:462
11521992410,11cyclictest2629-21ssh23:23:142
1150999242,17cyclictest3024-21ssh00:37:591
1150999242,17cyclictest29669-21diskmemload00:23:351
1150199243,9cyclictest12-21ksoftirqd/020:35:150
1150199241,15cyclictest986-21ssh23:05:120
11501992411,4cyclictest12-21ksoftirqd/020:50:150
1153099235,8cyclictest0-21swapper/321:50:123
11530992313,7cyclictest0-21swapper/300:20:143
1152199239,11cyclictest25644-21kthreadcore20:30:212
1152199233,17cyclictest4875-21kthreadcore19:35:212
1152199232,7cyclictest0-21swapper/220:45:102
1152199232,17cyclictest30990-21fschecks_time19:30:162
1152199232,16cyclictest27346irq/40-dwc2_hso00:00:112
1152199232,16cyclictest0-21swapper/200:33:462
11521992313,3cyclictest0-21swapper/221:05:252
1152199231,20cyclictest10558-21ls21:15:292
1152199231,15cyclictest25905-21ssh23:55:142
1152199231,13cyclictest3029-21kthreadcore00:20:202
11521992311,10cyclictest438-21ssh22:22:202
1150999232,7cyclictest131rcu_preempt23:55:141
1150999231,17cyclictest4662-21ls20:10:141
1150199237,3cyclictest131rcu_preempt23:15:310
1150199231,5cyclictest131rcu_preempt21:50:120
11501992312,3cyclictest29669-21diskmemload22:55:280
1150199231,17cyclictest17409-21diskmemload22:15:120
11530992212,7cyclictest0-21swapper/322:55:013
1152199222,15cyclictest16943-21ssh23:50:132
1152199222,15cyclictest16943-21ssh23:50:132
1152199222,15cyclictest0-21swapper/220:35:142
1152199221,18cyclictest17672-21cpu19:50:132
1152199221,17cyclictest28273-21tr00:15:212
1152199221,12cyclictest7047-21find22:25:282
1152199221,12cyclictest13984-21sendmail-msp20:20:012
1152199221,11cyclictest30991-21kthreadcore21:45:222
1152199221,11cyclictest16059-21cron22:35:012
11521992210,9cyclictest4843-21missed_timers21:30:252
1150999221,18cyclictest8280-21ls19:40:171
11501992216,4cyclictest0-21swapper/021:15:000
1152199211,9cyclictest14375-21fschecks_time20:20:142
1152199211,10cyclictest10630-21cpu20:50:172
1150999219,3cyclictest131rcu_preempt21:50:221
1150999213,15cyclictest7066-21grep19:35:291
1150999211,8cyclictest26641-21df_inode22:20:151
11509992112,6cyclictest10651-21ssh00:25:121
11509992112,6cyclictest10651-21ssh00:25:121
1150999211,18cyclictest25268-21grep00:15:161
1150999211,10cyclictest18072-21irqrtprio22:15:171
1150999210,17cyclictest17899-21ls19:50:151
1150199219,4cyclictest1-21systemd00:13:330
1150199211,17cyclictest25551-21ls23:55:010
1150199211,17cyclictest25551-21ls23:55:010
1150199211,16cyclictest24490-21ssh23:00:140
1150199211,14cyclictest16203-21latency_hist19:15:020
1153099204,11cyclictest0-21swapper/321:55:153
1153099204,11cyclictest0-21swapper/321:55:153
1153099203,13cyclictest0-21swapper/322:05:143
1153099203,11cyclictest0-21swapper/323:41:363
1153099202,11cyclictest0-21swapper/320:00:143
11530992012,5cyclictest0-21swapper/323:24:533
11530992011,6cyclictest0-21swapper/322:25:213
11530992011,5cyclictest0-21swapper/323:15:243
1150999207,8cyclictest0-21swapper/121:05:181
1150999202,15cyclictest22853-21ssh22:00:131
1150999201,14cyclictest16624-21ls23:50:011
1150999200,17cyclictest18906-21ls20:25:011
1150199203,15cyclictest23256-21irqcore22:00:180
1150199201,4cyclictest0-21swapper/019:30:190
1150199201,17cyclictest875-21ls22:45:230
1150199201,14cyclictest9538-21ls20:45:250
1150199200,9cyclictest3143-21ntp_kernel_pll_20:05:250
1153099199,8cyclictest0-21swapper/300:16:403
1153099192,12cyclictest0-21swapper/321:40:143
1153099192,11cyclictest0-21swapper/319:55:143
1153099192,11cyclictest0-21swapper/319:10:143
1153099192,10cyclictest0-21swapper/319:45:153
11530991913,4cyclictest33-21ksoftirqd/321:25:183
11530991913,4cyclictest33-21ksoftirqd/320:55:283
1153099191,13cyclictest0-21swapper/322:55:303
1153099191,12cyclictest0-21swapper/321:15:293
11530991910,6cyclictest35966irq/58-eth022:45:143
11530991910,6cyclictest35966irq/58-eth022:15:123
1150999192,14cyclictest20834-21ssh22:40:111
11509991912,4cyclictest0-21swapper/121:25:021
1150999191,14cyclictest28412-21ssh21:45:131
11509991910,7cyclictest3097-21ssh22:25:121
11509991910,6cyclictest29860-21ssh22:45:121
1150199196,3cyclictest3817-21kthreadcore00:00:190
1150199192,11cyclictest27355-21sh23:35:380
1150199192,11cyclictest11907-21ssh23:28:370
1150199191,9cyclictest14743-21kthreadcore23:30:210
1150199191,9cyclictest13608-21kthreadcore00:05:230
1150199191,8cyclictest5631-21ls20:45:010
1150199191,5cyclictest0-21swapper/021:05:270
1150199191,14cyclictest24764-21ssh21:25:150
1150199190,14cyclictest8308-21timerwakeupswit20:10:280
1153099189,6cyclictest0-21swapper/323:07:523
1153099185,10cyclictest35966irq/58-eth022:20:243
1153099183,7cyclictest0-21swapper/319:50:133
1153099182,10cyclictest0-21swapper/322:10:133
1153099182,10cyclictest0-21swapper/321:10:143
1153099182,10cyclictest0-21swapper/320:25:153
1153099182,10cyclictest0-21swapper/320:10:043
1153099182,10cyclictest0-21swapper/319:20:133
1153099181,12cyclictest0-21swapper/300:30:233
11530991810,6cyclictest0-21swapper/322:36:303
1150999189,7cyclictest2943-21cron19:35:021
1150999188,7cyclictest21011-21cron19:20:011
1150999186,9cyclictest21821-21tr23:15:241
1150999186,2cyclictest131rcu_preempt21:40:271
1150999182,13cyclictest13861-21ssh21:55:151
1150999182,13cyclictest13861-21ssh21:55:151
1150999181,8cyclictest0-21swapper/122:30:271
1150999181,2cyclictest0-21swapper/123:25:151
11509991812,4cyclictest630-21Async09:36:311
1150999181,15cyclictest2431-21ssh00:00:101
1150199185,4cyclictest131rcu_preempt00:35:210
1150199183,12cyclictest7993-21ssh23:45:130
1150199181,6cyclictest131rcu_preempt21:40:200
1150199181,14cyclictest27462-21cut20:00:160
1150199181,11cyclictest15627-21cut20:55:160
11501991810,5cyclictest24807-21timerandwakeup22:40:270
1150199180,13cyclictest10619-21timerandwakeup21:15:300
1153099179,6cyclictest0-21swapper/323:49:083
1153099179,6cyclictest0-21swapper/300:00:573
1153099172,10cyclictest0-21swapper/320:50:173
1153099172,10cyclictest0-21swapper/320:40:143
1153099172,10cyclictest0-21swapper/320:05:153
1153099172,10cyclictest0-21swapper/319:35:133
1153099171,9cyclictest0-21swapper/319:25:123
11530991711,4cyclictest0-21swapper/323:50:283
11530991711,4cyclictest0-21swapper/323:50:273
11530991711,4cyclictest0-21swapper/322:30:373
1150999179,6cyclictest1015-21cat22:05:211
1150999178,7cyclictest21406-21cat23:50:311
1150999178,7cyclictest21406-21cat23:50:311
1150999178,7cyclictest19133-21forks20:25:161
1150999178,6cyclictest10426-21irqcore23:10:181
1150999178,6cyclictest10426-21irqcore23:10:181
1150999173,12cyclictest11474-21kthreadcore20:50:221
1150999171,13cyclictest30073-21ssh21:10:131
1150999171,12cyclictest27226-21netstat20:30:231
11509991710,4cyclictest4292-21apt-key20:10:011
11509991710,4cyclictest2348-21cut20:40:201
1150999170,9cyclictest9885-21tr22:50:231
1150999170,9cyclictest9502-21timerandwakeup20:45:251
1150199179,6cyclictest31564-21cut21:45:220
1150199179,6cyclictest10051-21diskstats23:10:150
1150199179,6cyclictest10051-21diskstats23:10:140
1150199177,8cyclictest18785-21tr19:50:190
1150199174,8cyclictest0-21swapper/021:30:140
1150199171,7cyclictest29669-21diskmemload23:24:190
11501991710,5cyclictest7787-21cron19:40:000
1150199170,12cyclictest20546-21ls22:40:020
1150199170,12cyclictest13939-21ls21:55:150
1150199170,12cyclictest13939-21ls21:55:140
1153099167,7cyclictest0-21swapper/300:10:033
1153099162,10cyclictest0-21swapper/320:45:213
1153099161,12cyclictest0-21swapper/320:15:163
1152199166,7cyclictest11541-21latency_hist19:10:022
1150999168,5cyclictest23116-21uname19:55:191
1150999167,7cyclictest30023-21cut20:35:211
1150999167,6cyclictest0-21swapper/119:25:291
1150999166,7cyclictest15268-21cron21:20:011
1150999164,9cyclictest22159-21ps21:00:201
1150999162,11cyclictest14208-21kthreadcore19:10:211
1150999162,11cyclictest11155-21kthreadcore20:15:201
1150999161,8cyclictest24556-21latency_hist00:15:021
1150999161,7cyclictest18899-21ntp_kernel_pll_20:55:251
11509991611,3cyclictest27277-21sshd20:00:121
1150999161,12cyclictest21271-21apt-get19:20:121
1150999161,11cyclictest24440-21/usr/sbin/munin23:00:161
1150999161,11cyclictest12859-21apt-get19:45:151
1150199169,5cyclictest17984-21cut19:15:190
1150199168,6cyclictest25860-21nfs_client19:55:230
1150199168,6cyclictest18797-21cron20:25:010
1150199167,7cyclictest25893-21diskmemload23:55:130
1150199167,7cyclictest1571-21apt00:20:120
1150199161,6cyclictest29669-21diskmemload22:30:190
1150199161,12cyclictest3319-21fschecks_time22:25:160
1150199161,12cyclictest28398-21munin-run20:35:020
1150199161,12cyclictest24907-21ls19:20:270
1150199161,11cyclictest8366-21apt-get22:10:140
1150199161,11cyclictest0-21swapper/020:15:130
11530991510,3cyclictest0-21swapper/320:20:163
1150999157,5cyclictest11624-21cut19:10:021
1150999151,11cyclictest10298-21ssh21:35:141
1150199158,5cyclictest19327-21cat20:25:180
1150199153,9cyclictest13354-21sed19:45:180
1150199151,11cyclictest8041-21apt-get19:40:140
1150199151,11cyclictest26258-21ls19:25:150
1150199151,11cyclictest20387-21grep21:00:140
1153099141,8cyclictest0-21swapper/321:00:123
1153099141,10cyclictest0-21swapper/320:30:173
1150199147,5cyclictest11492-21cyclictest19:10:020
1153099138,3cyclictest33-21ksoftirqd/320:35:193
1153099137,4cyclictest35966irq/58-eth021:05:283
1153099131,10cyclictest0-21swapper/319:30:203
1153099111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional