You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 06:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Sep 01, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1517899643,49cyclictest8048-21ssh23:20:121
15203995839,11cyclictest35966irq/58-eth023:40:123
1519399581,47cyclictest26186-21ssh22:00:122
1519399551,6cyclictest0-21swapper/220:40:132
1517399544,40cyclictest15389-21ssh22:30:130
1517399543,43cyclictest9934-21ssh22:45:130
15173995434,14cyclictest0-21swapper/022:55:120
1519399501,41cyclictest24932-21apt-get19:20:122
1519399501,3cyclictest6352-21apt-get22:25:132
1519399500,2cyclictest0-21swapper/223:50:132
1517399503,5cyclictest131rcu_preempt22:10:130
1517399502,39cyclictest17165-21ssh21:55:140
1517399502,37cyclictest28492-21ssh00:10:100
1517399502,37cyclictest28492-21ssh00:10:090
1517899495,34cyclictest2653-21ssh23:35:141
1519399482,41cyclictest6869-21apt-get19:35:132
1519399471,41cyclictest22582-21ssh21:40:112
1519399471,3cyclictest0-21swapper/223:15:142
1519399471,3cyclictest0-21swapper/220:30:112
1517899473,32cyclictest9996-21ssh21:15:141
1517399471,40cyclictest2879-21ssh22:05:150
1520399464,27cyclictest35966irq/58-eth021:40:123
1519399461,40cyclictest27346irq/40-dwc2_hso22:10:142
1519399461,3cyclictest0-21swapper/223:30:132
1519399451,5cyclictest0-21swapper/222:55:122
1519399451,2cyclictest917-21apt-get22:40:122
1519399441,4cyclictest24299-21apt-get22:35:142
1517399442,15cyclictest8677-21apt-get00:35:130
1519399431,4cyclictest0-21swapper/219:55:142
15203994226,7cyclictest0-21swapper/319:55:133
1519399426,29cyclictest27346irq/40-dwc2_hso23:05:132
1519399421,3cyclictest0-21swapper/221:00:132
1517399424,32cyclictest23144-21ssh00:25:130
15203994123,10cyclictest35966irq/58-eth023:00:133
1519399411,3cyclictest2083-21apt-get19:30:142
1519399411,2cyclictest0-21swapper/221:15:132
1519399411,2cyclictest0-21swapper/221:05:142
1519399411,2cyclictest0-21swapper/221:05:132
1519399410,36cyclictest0-21swapper/221:50:122
1517399412,5cyclictest0-21swapper/019:45:120
1519399393,2cyclictest0-21swapper/220:15:142
1519399391,3cyclictest0-21swapper/220:10:142
1519399391,3cyclictest0-21swapper/220:05:142
1519399390,2cyclictest0-21swapper/219:10:132
1517399391,11cyclictest12-21ksoftirqd/023:50:120
1520399382,20cyclictest35966irq/58-eth022:55:123
1519399381,35cyclictest0-21swapper/200:10:092
1519399381,35cyclictest0-21swapper/200:10:092
1519399381,2cyclictest0-21swapper/221:55:132
1519399381,2cyclictest0-21swapper/200:20:112
1519399380,3cyclictest26973-21ssh00:08:352
1519399380,2cyclictest20129-21apt-get19:15:142
15178993813,18cyclictest21-21ksoftirqd/123:30:131
1520399372,19cyclictest35966irq/58-eth021:15:143
1519399362,2cyclictest27346irq/40-dwc2_hso23:20:122
1519399361,5cyclictest0-21swapper/220:50:122
1519399361,29cyclictest0-21swapper/221:35:122
1519399360,2cyclictest0-21swapper/222:50:122
1519399358,24cyclictest20555-21cut21:20:202
1519399353,28cyclictest930-21snmpd21:25:162
1519399352,30cyclictest3587-21cat23:35:202
1519399352,30cyclictest21253-21sed22:15:182
1519399351,2cyclictest17787-21apt-get20:20:132
1519399351,10cyclictest4682-21diskstats21:30:162
1517399352,6cyclictest0-21swapper/021:33:160
1517399351,9cyclictest26151-21ntp_192.168.11423:10:250
1520399342,19cyclictest35966irq/58-eth000:25:143
1519399341,2cyclictest18947-21apt-get20:55:142
1519399341,29cyclictest3439-21ntp_states00:30:272
1519399341,29cyclictest26340-21sh23:10:282
1519399340,2cyclictest0-21swapper/223:00:112
1517399343,10cyclictest2040-21tr23:15:220
1517399342,11cyclictest22020-21kthreadcore20:55:220
1520399336,15cyclictest35966irq/58-eth021:25:143
1519399331,3cyclictest930-21snmpd22:45:332
1519399331,2cyclictest29740-21apt-get19:25:132
1519399331,29cyclictest31087-21ssh00:29:172
1519399330,2cyclictest29843-21apt-get22:20:152
1517899336,19cyclictest201rcuc/123:15:131
1517399334,22cyclictest31571-21ssh21:45:120
1517399332,24cyclictest22597-21ssh21:40:140
1517399332,13cyclictest29875-21ssh22:20:150
1517399331,11cyclictest6777-21kthreadcore23:00:200
1520399322,18cyclictest35966irq/58-eth023:45:133
15203993218,9cyclictest35966irq/58-eth023:25:253
1519399327,22cyclictest32504-21ps20:35:172
1519399322,26cyclictest27346irq/40-dwc2_hso23:40:132
1519399321,5cyclictest7639-21ssh21:12:392
1519399321,5cyclictest2069-21/usr/sbin/munin20:00:312
1519399321,2cyclictest0-21swapper/220:45:132
1519399321,27cyclictest23779-21kthreadcore23:45:222
1519399320,5cyclictest21879-21vmstat23:25:332
1519399320,5cyclictest16715-21ssh00:39:182
1519399320,3cyclictest3192-21if_err_eth122:05:182
1519399320,2cyclictest5111-21apt-get00:15:112
1519399320,2cyclictest5111-21apt-get00:15:112
1517399328,6cyclictest20039-21cut21:20:190
1517399323,9cyclictest8077-21latency_hist20:10:020
1517399321,9cyclictest13520-21ssh21:35:120
1517399321,10cyclictest2746-21df23:35:150
1520399313,18cyclictest35966irq/58-eth019:10:123
1520399312,15cyclictest35966irq/58-eth023:15:123
1519399312,2cyclictest27346irq/40-dwc2_hso00:00:122
1519399311,5cyclictest4579-21ssh21:46:132
1519399310,8cyclictest21254-21ssh22:31:482
1519399310,27cyclictest0-21swapper/219:40:132
1517899312,10cyclictest21-21ksoftirqd/120:40:141
1517399311,11cyclictest15833-21kthreadcore20:50:200
1517399311,11cyclictest13900-21latency_hist20:50:020
1519399301,6cyclictest24996-21swap19:50:282
1519399301,5cyclictest11188-21ssh23:55:432
1519399300,6cyclictest19009-21tr19:45:212
1519399300,27cyclictest0-21swapper/220:25:132
1517399301,10cyclictest25317-21cat22:15:310
1517899292,21cyclictest32544-21sh22:02:141
1517899291,22cyclictest602-21diskmemload23:10:131
1520399283,15cyclictest35966irq/58-eth020:05:143
15203992815,9cyclictest35966irq/58-eth022:45:143
1519399281,6cyclictest15286-21latency_hist19:10:012
15178992810,14cyclictest21-21ksoftirqd/122:15:141
1520399272,14cyclictest35966irq/58-eth022:05:153
1520399271,23cyclictest35966irq/58-eth019:20:123
1520399271,19cyclictest35966irq/58-eth023:10:133
1517899273,17cyclictest27973-21ssh21:25:151
1517399271,18cyclictest26202-21ssh22:00:120
1517399271,10cyclictest8737-21irqstats23:20:170
1517399271,10cyclictest17881-21ntp_states00:20:260
1517899265,14cyclictest201rcuc/120:05:121
1517899261,20cyclictest8139-21ssh21:50:121
1517399262,7cyclictest17546-21munin-run20:20:020
1517399262,18cyclictest10023-21ssh22:25:260
1520399251,18cyclictest0-21swapper/320:45:133
1520399251,14cyclictest35966irq/58-eth019:40:123
1517899253,18cyclictest4566-21ssh21:30:141
1517899252,18cyclictest32141-21ssh00:30:151
1517899252,18cyclictest29594-21ssh23:50:131
15178992513,10cyclictest0-21swapper/123:04:011
1520399249,10cyclictest0-21swapper/322:40:123
1520399243,13cyclictest0-21swapper/322:02:143
1520399243,12cyclictest0-21swapper/322:30:133
15203992412,6cyclictest0-21swapper/300:15:133
15203992412,6cyclictest0-21swapper/300:15:133
15178992411,3cyclictest21-21ksoftirqd/121:43:331
1517399243,10cyclictest0-21swapper/021:05:120
1517399243,10cyclictest0-21swapper/021:05:120
1517399242,7cyclictest29714-21ntp_192.168.11423:30:260
1517399242,5cyclictest0-21swapper/021:27:160
1517399241,18cyclictest11594-21latency21:50:240
1520399232,18cyclictest35966irq/58-eth023:50:123
1520399231,15cyclictest35966irq/58-eth022:10:143
1517899232,2cyclictest131rcu_preempt22:07:091
1517899232,17cyclictest28480-21apt-get00:10:091
1517899232,17cyclictest28480-21apt-get00:10:091
1517899231,17cyclictest14166-21ssh00:20:121
1517399233,16cyclictest6169-21ssh23:55:130
1517399232,14cyclictest11592-21ssh23:40:130
1517399231,8cyclictest942-21ssh22:40:130
1520399223,13cyclictest0-21swapper/321:39:163
1517899227,5cyclictest131rcu_preempt23:05:221
1517899222,16cyclictest20484-21munin-run22:15:011
15178992213,7cyclictest0-21swapper/122:35:141
1517899221,19cyclictest0-21swapper/119:20:121
1517899221,16cyclictest20627-21sh23:45:141
1517899220,18cyclictest9650-21ls22:45:021
15173992214,4cyclictest32374-21hwlatdetect00:30:180
15203992111,5cyclictest0-21swapper/322:25:323
1520399211,13cyclictest0-21swapper/300:10:093
1520399211,13cyclictest0-21swapper/300:10:083
1517899219,3cyclictest21-21ksoftirqd/100:15:111
1517899219,3cyclictest21-21ksoftirqd/100:15:111
1517899212,2cyclictest131rcu_preempt00:39:591
15178992114,4cyclictest21-21ksoftirqd/121:57:441
15178992113,5cyclictest15098-21latency23:40:231
1517899211,16cyclictest32316-21cut20:35:151
15178992111,6cyclictest12731-1kworker/u9:200:00:131
15178992110,3cyclictest21-21ksoftirqd/123:27:111
1517399211,16cyclictest23751-21apt-get21:00:130
1517399210,16cyclictest20626-21ssh23:45:130
1520399208,9cyclictest0-21swapper/323:30:583
1520399202,11cyclictest0-21swapper/320:25:133
15203992013,4cyclictest0-21swapper/322:15:183
15203992011,6cyclictest0-21swapper/323:20:303
1520399201,12cyclictest35966irq/58-eth000:00:113
15203992010,8cyclictest0-21swapper/323:55:073
1517899209,3cyclictest21-21ksoftirqd/121:48:161
1517899209,3cyclictest21-21ksoftirqd/121:24:011
1517899207,4cyclictest21-21ksoftirqd/121:10:461
1517899203,13cyclictest21-21ksoftirqd/122:25:121
15178992014,4cyclictest201rcuc/121:05:131
15178992014,4cyclictest201rcuc/121:05:121
15178992013,4cyclictest0-21swapper/120:25:171
15178992012,6cyclictest31950-21cron20:35:021
15178992012,6cyclictest0-21swapper/123:55:281
1517899201,17cyclictest29821-21ls19:55:281
1517899200,17cyclictest10226-21ls19:35:231
1517399209,7cyclictest3144-21cron20:05:010
1517399202,14cyclictest1017-21ssh21:10:110
15173992013,4cyclictest630-21Async09:36:310
1517399201,15cyclictest0-21swapper/020:30:120
1520399199,7cyclictest35966irq/58-eth000:35:133
1520399199,7cyclictest0-21swapper/300:30:253
1520399197,9cyclictest0-21swapper/321:45:193
1520399193,11cyclictest0-21swapper/322:22:243
1520399193,11cyclictest0-21swapper/321:32:163
1520399191,13cyclictest0-21swapper/320:20:233
1520399191,12cyclictest0-21swapper/319:55:023
1520399191,11cyclictest0-21swapper/320:15:173
15203991910,6cyclictest0-21swapper/321:20:303
1517899199,8cyclictest21-21ksoftirqd/120:10:261
1517899199,2cyclictest21-21ksoftirqd/121:39:121
1517899198,3cyclictest21-21ksoftirqd/100:29:351
1517899193,11cyclictest0-21swapper/100:08:161
1517899193,10cyclictest0-21swapper/119:31:401
1517899192,2cyclictest131rcu_preempt22:33:091
1517899191,5cyclictest0-21swapper/122:20:151
15178991913,4cyclictest0-21swapper/120:55:301
15178991913,3cyclictest21-21ksoftirqd/120:50:011
15178991912,4cyclictest21-21ksoftirqd/119:15:211
15178991910,6cyclictest10635-21cut22:45:181
1517399192,15cyclictest18947-21proc_pri00:00:270
1517399191,15cyclictest15635-21grep19:40:290
1517399191,14cyclictest26868-21sed19:55:200
1517399191,10cyclictest22911-21irqcore20:25:180
1517399190,12cyclictest25794-21gpgv23:30:020
1520399189,6cyclictest0-21swapper/323:05:073
1520399189,6cyclictest0-21swapper/321:11:423
1520399182,9cyclictest0-21swapper/320:10:143
1520399182,11cyclictest0-21swapper/300:08:173
15203991810,6cyclictest0-21swapper/321:50:353
15203991810,6cyclictest0-21swapper/300:20:133
1517899189,7cyclictest18712-21cron20:55:021
1517899189,6cyclictest29173-21cat22:55:191
1517899188,8cyclictest11784-21apt-get19:40:131
1517899181,15cyclictest18460-21kthreadcore20:20:211
15178991811,5cyclictest0-21swapper/119:25:291
1517899181,14cyclictest13106-21grep20:15:161
1517899181,12cyclictest18905-21ssh22:50:131
1517399189,7cyclictest29821-21df19:25:150
1517399189,6cyclictest7073-21perl19:35:170
1517399183,10cyclictest0-21swapper/020:20:150
1517399182,12cyclictest8837-21ntp_states00:15:260
1517399182,12cyclictest8837-21ntp_states00:15:250
1517399182,10cyclictest0-21swapper/019:50:130
1517399181,14cyclictest2238-21cut19:30:150
15173991810,6cyclictest12704-21sshd20:11:580
1520399179,6cyclictest0-21swapper/322:35:103
1520399178,6cyclictest35966irq/58-eth023:35:133
1520399172,9cyclictest0-21swapper/320:40:143
1520399172,9cyclictest0-21swapper/320:35:133
15203991712,3cyclictest0-21swapper/321:55:193
1520399171,10cyclictest0-21swapper/322:50:173
1517899177,3cyclictest131rcu_preempt20:00:201
1517899175,9cyclictest16958-21kthreadcore19:10:201
1517899173,9cyclictest0-21swapper/119:50:121
15178991711,3cyclictest21-21ksoftirqd/121:00:201
1517399176,6cyclictest0-21swapper/019:10:120
1517399172,11cyclictest28073-21tr19:20:210
1517399172,11cyclictest19017-21diskstats22:50:150
1517399172,11cyclictest10094-21awk21:15:150
1517399171,12cyclictest13479-21ssh23:05:130
1520399163,11cyclictest0-21swapper/319:30:113
1520399162,9cyclictest0-21swapper/320:30:123
1520399161,10cyclictest0-21swapper/320:50:163
1517899161,12cyclictest20429-21unixbench_singl19:45:281
1517399161,3cyclictest0-21swapper/000:07:580
1517399161,12cyclictest3131-21ntp_192.168.11420:35:250
1517399161,12cyclictest24305-21ssh22:35:140
1517399161,11cyclictest9297-21latency_hist20:45:020
1520399152,9cyclictest0-21swapper/321:05:133
1520399152,9cyclictest0-21swapper/321:05:133
1520399151,11cyclictest0-21swapper/320:05:023
1517399151,10cyclictest15234-21latency_hist19:10:010
1520399142,9cyclictest0-21swapper/319:25:153
1520399142,8cyclictest0-21swapper/319:45:143
1520399141,7cyclictest0-21swapper/321:00:153
1520399141,10cyclictest0-21swapper/319:15:163
1517899142,9cyclictest15165-21cyclictest19:10:011
1520399137,4cyclictest33-21ksoftirqd/320:55:173
1520399135,5cyclictest0-21swapper/319:35:253
1520399111,8cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional