You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-12 - 04:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Fri Dec 12, 2025 00:45:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
746499736,20cyclictest21-21ksoftirqd/122:45:131
747399514,42cyclictest0-21swapper/221:10:142
747399502,15cyclictest0-21swapper/221:40:122
745399505,41cyclictest1801-21ssh21:35:100
746499491,14cyclictest21-21ksoftirqd/100:35:171
747399482,41cyclictest24613-21ssh22:30:162
747399482,13cyclictest9496-21ssh22:00:132
745399484,7cyclictest131rcu_preempt22:45:120
7453994810,24cyclictest131rcu_preempt19:25:120
745399472,35cyclictest26358-21ssh21:10:130
7453994522,15cyclictest629-21Async14:00:480
745399442,6cyclictest131rcu_preempt00:15:160
746499431,12cyclictest21-21ksoftirqd/122:25:141
745399422,30cyclictest0-21swapper/020:20:150
745399421,28cyclictest14073-21ssh00:25:180
746499411,30cyclictest8003-21ssh22:40:131
745399412,29cyclictest21895-21ssh00:30:150
745399411,24cyclictest15457-21ssh00:05:150
747399392,6cyclictest0-21swapper/221:47:542
747399392,29cyclictest16029-21ssh23:05:142
747399383,9cyclictest23654-21kthreadcore00:30:242
747399383,28cyclictest25357-21ssh22:10:142
745399385,19cyclictest12-21ksoftirqd/022:05:160
747399379,12cyclictest131rcu_preempt22:05:172
747399372,10cyclictest32534-21kthreadcore23:55:232
747399371,9cyclictest4456-21if_eth119:40:182
747399361,8cyclictest27883-21netstat22:50:282
7473993614,5cyclictest932-21snmpd19:50:162
747399360,11cyclictest17982-21latency_hist21:25:022
746499361,29cyclictest9929-21sh21:40:161
747399351,8cyclictest10965-21sendmail_mailtr23:40:272
747399351,15cyclictest11085-21cut23:00:272
7473993425,6cyclictest26346irq/40-dwc2_hso23:15:132
7473993422,5cyclictest27-21ksoftirqd/219:45:122
7473993319,6cyclictest0-21swapper/221:30:142
747399331,10cyclictest16197-21apt-get22:45:122
746499332,28cyclictest0-21swapper/122:35:121
746499331,29cyclictest25933-21ssh21:50:171
745399334,17cyclictest7160-21ssh23:40:120
745399333,21cyclictest15174-21diskmemload23:45:130
7481993218,12cyclictest0-21swapper/322:30:173
747399328,7cyclictest27-21ksoftirqd/200:35:162
746499327,6cyclictest21-21ksoftirqd/121:10:131
746499323,25cyclictest1535-21ssh21:55:191
746499322,18cyclictest22305-21kworker/u8:123:45:131
745399323,7cyclictest131rcu_preempt19:55:120
748199313,17cyclictest29566irq/58-eth021:40:153
7481993119,7cyclictest29566irq/58-eth021:55:163
747399315,16cyclictest0-21swapper/219:30:142
747399313,17cyclictest26346irq/40-dwc2_hso19:35:122
747399313,16cyclictest26346irq/40-dwc2_hso23:10:162
746499312,22cyclictest21-21ksoftirqd/120:40:121
745399312,20cyclictest874-21ssh22:15:170
746499302,23cyclictest8724-21ssh22:20:151
7464993017,10cyclictest21-21ksoftirqd/120:35:171
748199292,18cyclictest29566irq/58-eth022:00:143
747399293,21cyclictest0-21swapper/221:53:112
747399293,14cyclictest16710-21ssh22:25:152
746499294,17cyclictest31869-21ssh23:15:131
746499291,23cyclictest31612-21ssh23:35:161
746499291,23cyclictest31612-21ssh23:35:151
748199284,15cyclictest29566irq/58-eth000:00:163
748199283,18cyclictest0-21swapper/321:51:123
747399286,10cyclictest26346irq/40-dwc2_hso21:35:102
747399283,18cyclictest26346irq/40-dwc2_hso20:20:142
747399282,20cyclictest0-21swapper/222:55:152
745399282,17cyclictest23327-21ssh23:50:150
747399273,14cyclictest26346irq/40-dwc2_hso20:25:172
747399272,19cyclictest0-21swapper/200:15:172
747399272,16cyclictest23410-21apt-get00:10:152
7453992717,5cyclictest26006-21diskmemload23:05:140
745399271,22cyclictest8007-21ssh22:40:130
745399271,20cyclictest8059-21ssh23:00:170
748199263,14cyclictest0-21swapper/322:16:153
748199262,17cyclictest0-21swapper/322:40:133
747399264,15cyclictest26346irq/40-dwc2_hso20:10:132
747399264,13cyclictest26346irq/40-dwc2_hso20:05:152
747399263,13cyclictest26346irq/40-dwc2_hso20:50:162
747399262,16cyclictest26346irq/40-dwc2_hso21:00:172
747399261,16cyclictest23309-21rm23:50:132
745399265,4cyclictest131rcu_preempt21:15:150
7453992619,2cyclictest12-21ksoftirqd/021:20:160
748199259,7cyclictest0-21swapper/323:55:153
748199252,17cyclictest29566irq/58-eth021:15:163
747399253,16cyclictest26346irq/40-dwc2_hso23:25:142
747399251,20cyclictest3521-21missed_timers22:35:242
746499251,19cyclictest25350-21ls19:25:231
746499251,18cyclictest32362-21ssh22:55:161
745399256,4cyclictest131rcu_preempt19:45:200
748199242,13cyclictest29566irq/58-eth000:30:153
747399247,14cyclictest26666-21cron19:30:002
747399245,9cyclictest26346irq/40-dwc2_hso23:30:132
747399242,10cyclictest0-21swapper/222:20:142
747399240,21cyclictest5008-21missed_timers21:55:262
746499248,11cyclictest21-21ksoftirqd/119:15:151
746499247,3cyclictest21-21ksoftirqd/100:27:531
746499242,17cyclictest10181-21ssh21:20:171
746499242,16cyclictest21-21ksoftirqd/100:05:141
7464992416,6cyclictest0-21swapper/122:10:271
7464992412,3cyclictest21-21ksoftirqd/100:25:021
745399248,5cyclictest131rcu_preempt22:10:140
745399243,8cyclictest12-21ksoftirqd/022:25:150
745399242,15cyclictest19166-21ssh23:25:270
745399241,17cyclictest9930-21ssh21:40:170
748199234,15cyclictest29566irq/58-eth023:15:143
748199233,13cyclictest0-21swapper/323:43:203
7481992311,8cyclictest29566irq/58-eth022:45:143
748199231,14cyclictest0-21swapper/300:05:143
747399232,9cyclictest26006-21diskmemload00:20:152
747399232,16cyclictest0-21swapper/222:19:152
747399231,12cyclictest22967-21latency_hist23:50:022
746499238,2cyclictest131rcu_preempt00:10:451
746499233,13cyclictest21-21ksoftirqd/120:00:121
7464992312,3cyclictest0-21swapper/122:01:571
7464992312,3cyclictest0-21swapper/121:34:571
746499231,16cyclictest7310-21ssh00:00:161
746499231,16cyclictest23582-21rm23:30:151
745399232,17cyclictest151rcuc/023:10:150
745399232,13cyclictest12-21ksoftirqd/021:00:160
745399231,17cyclictest32296-21ls22:35:020
745399231,15cyclictest751-21sendmail-mta20:45:160
745399231,10cyclictest131rcu_preempt22:35:230
748199226,10cyclictest0-21swapper/300:25:173
747399229,4cyclictest9748-21latency_hist21:20:012
747399229,10cyclictest31903-21kthreadcore20:40:222
747399227,5cyclictest13241-21kthreadcore20:55:202
747399222,6cyclictest0-21swapper/221:25:152
747399222,16cyclictest26346irq/40-dwc2_hso19:20:152
747399221,11cyclictest8207-21kthreadcore00:00:222
746499228,3cyclictest131rcu_preempt21:45:221
746499224,15cyclictest21-21ksoftirqd/100:15:441
7464992216,3cyclictest0-21swapper/123:55:151
7464992214,4cyclictest0-21swapper/121:15:161
7464992210,3cyclictest21-21ksoftirqd/123:40:121
7464992210,3cyclictest21-21ksoftirqd/100:30:251
7464992210,3cyclictest0-21swapper/122:54:291
745399221,16cyclictest31575-21find23:35:140
745399221,16cyclictest31575-21find23:35:130
748199213,15cyclictest29566irq/58-eth021:20:163
748199213,13cyclictest0-21swapper/322:10:153
748199213,12cyclictest0-21swapper/323:30:133
747399213,9cyclictest23474-21cpu20:00:132
747399211,17cyclictest19552-21grep00:05:322
747399211,16cyclictest26977-21kthreadcore20:35:242
747399211,16cyclictest12336-21latency_hist19:15:022
747399211,10cyclictest24678-21cat21:05:272
7473992110,8cyclictest2675-21missed_timers23:35:252
7473992110,8cyclictest2675-21missed_timers23:35:242
746499219,3cyclictest21-21ksoftirqd/123:20:171
746499214,11cyclictest21-21ksoftirqd/120:15:111
7464992114,4cyclictest21-21ksoftirqd/120:20:211
7464992113,5cyclictest21-21ksoftirqd/121:00:221
746499211,16cyclictest201rcuc/121:25:161
7464992111,3cyclictest21-21ksoftirqd/119:10:141
7464992110,3cyclictest0-21swapper/123:02:321
745399219,7cyclictest131rcu_preempt21:57:570
745399219,7cyclictest131rcu_preempt21:47:540
745399216,4cyclictest131rcu_preempt21:50:290
745399215,6cyclictest131rcu_preempt22:22:020
7453992114,5cyclictest3507-21unixbench_multi23:15:320
745399211,10cyclictest10982-21cut23:20:260
748199205,12cyclictest321rcuc/300:20:143
748199203,11cyclictest0-21swapper/323:50:203
748199203,11cyclictest0-21swapper/322:06:113
7481992012,5cyclictest0-21swapper/321:10:263
747399203,8cyclictest5619-21fschecks_count20:15:162
747399202,9cyclictest22333-21kthreadcore20:30:222
747399202,10cyclictest26346irq/40-dwc2_hso00:25:162
747399201,9cyclictest4774-21kthreadcore20:45:242
747399201,14cyclictest7586-21ssh23:20:172
747399201,10cyclictest20265-21kthreadcore19:55:212
747399201,10cyclictest13095-21kernelversion19:15:192
746499209,3cyclictest21-21ksoftirqd/120:10:251
746499208,3cyclictest21-21ksoftirqd/123:25:111
746499205,4cyclictest21-21ksoftirqd/119:20:281
7464992012,6cyclictest380-21sshd20:08:111
7464992011,6cyclictest18889-21sh22:05:241
7464992011,6cyclictest0-21swapper/121:35:221
7464992011,3cyclictest0-21swapper/122:30:091
746499200,5cyclictest131rcu_preempt19:55:001
745399207,5cyclictest0-21swapper/021:05:300
745399201,3cyclictest131rcu_preempt20:10:250
745399201,3cyclictest131rcu_preempt19:20:230
745399200,3cyclictest131rcu_preempt22:01:120
748199198,9cyclictest0-21swapper/300:13:033
748199193,12cyclictest29566irq/58-eth021:30:133
748199193,11cyclictest0-21swapper/322:25:143
748199193,11cyclictest0-21swapper/321:47:113
748199192,11cyclictest0-21swapper/323:35:163
748199192,11cyclictest0-21swapper/323:35:153
7481991914,4cyclictest321rcuc/319:55:123
7481991913,4cyclictest29566irq/58-eth023:00:173
7481991913,4cyclictest0-21swapper/322:55:203
7481991913,4cyclictest0-21swapper/322:22:393
7481991913,4cyclictest0-21swapper/321:35:273
7481991911,5cyclictest29566irq/58-eth021:25:153
748199191,12cyclictest29566irq/58-eth023:45:143
748199191,12cyclictest0-21swapper/320:45:323
748199191,12cyclictest0-21swapper/320:40:223
7481991910,6cyclictest0-21swapper/323:28:183
747399191,10cyclictest8690-21kernelversion22:40:212
746499199,2cyclictest21-21ksoftirqd/119:40:281
746499197,5cyclictest131rcu_preempt23:13:261
746499195,4cyclictest131rcu_preempt23:05:201
746499194,12cyclictest12330-21grep19:45:221
746499193,14cyclictest27046-21grep23:50:281
746499192,12cyclictest850-21apt-get22:15:151
746499191,7cyclictest10716-21timerwakeupswit20:50:301
746499191,4cyclictest131rcu_preempt19:10:011
7464991912,4cyclictest21-21ksoftirqd/120:25:231
7464991912,3cyclictest21-21ksoftirqd/120:45:251
746499190,16cyclictest21386-21/usr/sbin/munin21:05:281
745399199,8cyclictest7020-21df_inode20:50:170
745399195,4cyclictest131rcu_preempt20:05:270
745399191,9cyclictest0-21swapper/023:58:190
745399191,7cyclictest842-21munin-node23:30:120
745399191,7cyclictest6799-21tr20:15:200
745399191,3cyclictest131rcu_preempt20:25:300
7453991913,3cyclictest12-21ksoftirqd/022:50:170
7453991912,5cyclictest12-21ksoftirqd/020:05:020
7453991912,5cyclictest0-21swapper/000:14:430
7453991911,5cyclictest131rcu_preempt22:55:140
7453991910,7cyclictest24731-21sshd21:28:100
748199188,6cyclictest29566irq/58-eth022:50:163
748199183,10cyclictest0-21swapper/322:37:153
748199182,11cyclictest0-21swapper/319:35:123
748199181,9cyclictest0-21swapper/321:00:033
7481991812,4cyclictest0-21swapper/319:30:253
7481991812,4cyclictest0-21swapper/300:15:263
7481991810,5cyclictest29566irq/58-eth023:20:163
746499186,9cyclictest11882-21sort20:55:171
7464991813,3cyclictest21-21ksoftirqd/119:35:301
7464991813,3cyclictest0-21swapper/119:55:321
746499181,14cyclictest28107-21kthreadcore19:30:191
745399184,5cyclictest12-21ksoftirqd/000:20:320
745399182,2cyclictest131rcu_preempt00:35:260
7453991812,4cyclictest0-21swapper/000:03:000
7453991811,3cyclictest530-21usb-storage20:40:040
748199179,6cyclictest0-21swapper/323:10:233
748199179,5cyclictest321rcuc/320:20:143
748199172,11cyclictest0-21swapper/323:05:223
748199172,10cyclictest0-21swapper/319:50:163
7481991711,4cyclictest0-21swapper/300:35:233
748199171,13cyclictest0-21swapper/320:10:173
748199171,12cyclictest0-21swapper/320:15:173
7481991710,4cyclictest0-21swapper/319:10:143
747399176,9cyclictest7502-21wc19:10:012
746499171,13cyclictest21161-21kthreadcore20:30:211
745399178,7cyclictest13912-21diskstats19:50:170
745399178,7cyclictest12524-21diskstats19:15:150
745399175,5cyclictest20187-21tune2fs20:30:160
745399171,14cyclictest7958-21timerwakeupswit19:40:300
745399171,13cyclictest28160-21kthreadcore21:30:220
748199162,9cyclictest0-21swapper/320:05:153
748199162,8cyclictest0-21swapper/321:05:183
748199161,12cyclictest0-21swapper/320:35:273
745399162,9cyclictest0-21swapper/019:30:140
7453991611,3cyclictest0-21swapper/020:55:290
745399161,12cyclictest31763-21apt-get19:35:130
745399161,11cyclictest25359-21irqrtprio20:35:220
748199152,9cyclictest0-21swapper/319:25:143
748199152,9cyclictest0-21swapper/319:15:143
748199151,8cyclictest0-21swapper/320:35:013
748199142,9cyclictest0-21swapper/319:20:183
748199141,11cyclictest0-21swapper/320:25:213
745399142,9cyclictest7454-21latency_hist19:10:000
748199138,3cyclictest0-21swapper/320:54:043
748199138,3cyclictest0-21swapper/319:40:203
748199133,8cyclictest0-21swapper/321:00:193
748199131,10cyclictest0-21swapper/320:00:203
748199123,7cyclictest0-21swapper/319:45:253
748199121,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional