You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 18:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri May 08, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28241995537,7cyclictest35966irq/58-eth010:35:123
2822699546,34cyclictest18944-21ssh11:55:112
28226995310,18cyclictest27-21ksoftirqd/207:10:112
2820699502,5cyclictest0-21swapper/008:05:130
2821799480,40cyclictest27858-21apt-get12:00:131
2821799470,42cyclictest26178-21apt-get08:15:131
2821799461,38cyclictest10327-21apt-get07:25:121
2821799452,35cyclictest16585-21apt-get08:05:131
28241994426,13cyclictest35966irq/58-eth010:15:113
2821799441,38cyclictest32339-21apt-get08:55:121
2821799441,37cyclictest28941-21apt-get10:30:131
2821799441,36cyclictest15135-21apt-get07:30:121
28226994316,19cyclictest930-21snmpd08:20:122
2821799432,36cyclictest5519-21apt-get07:20:121
2821799431,37cyclictest32349-21apt-get10:50:131
2821799431,33cyclictest30327-21ssh11:25:131
2821799431,33cyclictest30327-21ssh11:25:121
2821799430,37cyclictest8207-21apt-get08:30:121
2821799430,35cyclictest26852-21ssh11:05:121
2821799424,34cyclictest19882-21apt-get10:25:111
2821799421,35cyclictest22599-21apt-get08:45:131
2821799411,35cyclictest6948-21apt-get07:55:121
2821799401,23cyclictest9003-21/usr/sbin/munin10:35:231
2821799400,36cyclictest31292-21ssh12:20:111
2821799400,36cyclictest31292-21ssh12:20:111
2821799400,33cyclictest668-21apt-get07:15:121
2820699406,3cyclictest23367-21diskmemload10:45:110
2821799392,34cyclictest14493-21apt-get09:10:121
2821799392,34cyclictest12361-21apt-get11:15:121
2821799392,32cyclictest18257-21apt-get09:30:121
2821799391,34cyclictest30836-21apt-get09:55:111
2821799390,30cyclictest27312-21ssh11:21:551
2821799390,25cyclictest31715-21ntp_192.168.11407:10:221
2821799381,34cyclictest0-21swapper/111:10:111
2821799381,21cyclictest9640-21df09:05:131
2821799380,34cyclictest29760-21apt-get07:45:131
2821799380,33cyclictest21396-21apt-get08:10:121
2821799371,33cyclictest14098-21diskmemload09:25:121
2821799371,31cyclictest0-21swapper/109:15:131
2821799371,21cyclictest7791-21apt-get12:25:121
2821799371,21cyclictest7791-21apt-get12:25:111
2821799370,25cyclictest22816-21ssh11:00:381
2821799370,25cyclictest22816-21ssh11:00:381
2820699371,6cyclictest19908-21ssh10:25:120
2822699362,26cyclictest14367-21ssh10:40:122
2821799361,21cyclictest0-21swapper/110:09:591
2821799361,21cyclictest0-21swapper/110:09:591
2821799360,22cyclictest882-21ssh12:38:441
2820699361,8cyclictest17001-21irqrtprio10:05:190
2820699361,8cyclictest17001-21irqrtprio10:05:190
28206993617,13cyclictest26849-21sh11:05:120
2820699361,28cyclictest25769-21ssh12:35:120
2824199353,18cyclictest35966irq/58-eth009:20:143
2822699352,26cyclictest10955-21ssh10:20:122
2821799351,31cyclictest14098-21diskmemload10:55:121
2821799350,33cyclictest0-21swapper/110:20:121
2821799350,31cyclictest17814-21apt-get08:40:131
2821799350,29cyclictest20055-21apt-get07:35:131
2821799350,27cyclictest24647-21ssh11:40:131
2820699356,5cyclictest10935-21irqrtprio07:25:170
2820699351,8cyclictest9674-21sed10:55:200
2820699351,7cyclictest17539-21cut12:30:180
2824199343,22cyclictest35966irq/58-eth011:00:123
2824199343,22cyclictest35966irq/58-eth011:00:123
2824199342,25cyclictest35966irq/58-eth009:35:123
28241993418,10cyclictest35966irq/58-eth009:55:113
2821799341,30cyclictest3399-21apt-get08:25:121
2821799341,29cyclictest4405-21kthreadcore11:45:231
2821799341,27cyclictest0-21swapper/108:00:121
2821799341,20cyclictest5549-21ntp_192.168.11407:50:231
2821799340,21cyclictest24501-21ssh12:33:561
2821799340,21cyclictest2033-21ssh09:38:191
2820699341,9cyclictest1310-21grep11:45:170
2820699341,9cyclictest12743-21kthreadcore11:50:220
2820699341,9cyclictest12518-21uniq11:15:150
2820699341,8cyclictest32453-21kthreadcore11:25:200
2820699341,8cyclictest32453-21kthreadcore11:25:200
2820699341,8cyclictest2810-21kthreadcore07:50:170
2820699341,8cyclictest2512-21/usr/sbin/munin12:20:250
2820699341,8cyclictest2512-21/usr/sbin/munin12:20:250
2820699341,8cyclictest21842-21apt-get09:50:110
2820699341,6cyclictest28807-21/usr/sbin/munin07:40:280
2820699341,5cyclictest22291-21ssh12:15:120
2820699341,10cyclictest2174-21fschecks_time10:15:150
2820699340,9cyclictest8057-21tr07:55:180
2822699331,20cyclictest6803-21ssh11:30:122
2821799332,27cyclictest4711-21apt-get09:00:111
2821799332,27cyclictest4711-21apt-get09:00:101
2821799331,30cyclictest0-21swapper/110:15:111
2821799331,29cyclictest0-21swapper/109:40:111
2821799330,23cyclictest6953-21ssh09:22:551
2820699338,22cyclictest8217-21cut12:25:170
2820699338,22cyclictest8217-21cut12:25:160
2820699331,9cyclictest984-21if_eth107:15:160
2820699331,9cyclictest30903-21grep08:20:010
2820699331,9cyclictest24039-21kthreadcore08:45:190
2820699331,9cyclictest18436-21fschecks_count09:30:150
2820699331,8cyclictest9451-21cut08:30:180
2820699331,8cyclictest6928-21fschecks_count11:30:150
2820699331,8cyclictest24630-21apt-get11:40:120
2820699331,7cyclictest9266-21mailstats10:35:270
2820699331,7cyclictest3824-21rm09:40:110
2820699331,7cyclictest26476-21cut09:15:230
2820699331,7cyclictest16109-21latency_hist10:05:010
2820699330,8cyclictest4677-21kthreadcore08:25:190
2824199324,18cyclictest0-21swapper/310:40:133
2824199322,19cyclictest35966irq/58-eth010:45:123
2822699326,18cyclictest27346irq/40-dwc2_hso08:00:112
28226993216,10cyclictest536-21usb-storage07:40:122
2821799321,28cyclictest6782-21apt-get11:30:121
2821799321,27cyclictest8469-21cut10:00:191
2821799321,27cyclictest19693-21kthreadcore11:55:181
2821799321,27cyclictest13585-21uname09:45:181
2821799321,25cyclictest0-21swapper/107:40:131
2821799320,30cyclictest14984-21ssh12:10:181
2821799320,29cyclictest10061-21ssh11:50:141
2821799320,22cyclictest22678-21ssh10:44:391
2820699322,25cyclictest13332-21ssh12:10:100
2820699321,8cyclictest29472-21irqcore10:30:180
2820699321,8cyclictest18124-21interrupts08:40:170
2820699321,7cyclictest20436-21sed07:35:170
2820699321,7cyclictest18183-21hwlatdetect11:00:160
2820699321,7cyclictest18183-21hwlatdetect11:00:160
2820699321,7cyclictest1314-21kthreadcore09:20:200
2820699321,6cyclictest9583-21ps09:25:180
2820699321,6cyclictest27535-21hwlatdetect09:35:160
2820699321,6cyclictest17415-21cut07:30:200
2820699321,6cyclictest11647-21cut10:20:170
2820699321,10cyclictest6229-21kthreadcore07:20:180
2820699320,8cyclictest4679-21latency_hist09:00:020
2824199314,12cyclictest35966irq/58-eth010:25:113
2824199312,22cyclictest0-21swapper/311:40:143
28241993115,9cyclictest35966irq/58-eth010:50:123
2822699317,16cyclictest27346irq/40-dwc2_hso09:10:132
2821799314,24cyclictest23492-21cut12:15:201
2821799311,27cyclictest29852-21cat10:10:301
2821799311,26cyclictest28304-21sed07:10:021
2821799311,26cyclictest16426-21irqstats11:35:161
2820699315,4cyclictest4700-21acpi09:00:100
2820699315,4cyclictest4700-21acpi09:00:100
2820699312,7cyclictest7127-21ntp_states11:10:260
2820699311,9cyclictest8130-21ssh12:05:250
2820699311,8cyclictest9784-21sed09:05:160
2820699311,8cyclictest28791-21irqcore07:10:160
2820699311,8cyclictest21702-21if_eth108:10:170
2820699311,8cyclictest13020-21ls08:35:130
2820699311,7cyclictest32710-21ssh12:00:320
2820699311,7cyclictest16936-21sed09:45:290
2820699311,6cyclictest31151-21fschecks_count08:20:150
2820699310,9cyclictest30916-21cpuspeed09:55:130
2824199301,21cyclictest35966irq/58-eth009:30:113
2821799301,27cyclictest23364-21apt-get10:45:121
2821799301,26cyclictest0-21swapper/112:05:121
2821799301,25cyclictest21977-21diskstats09:50:141
2821799301,25cyclictest1220-21cut08:20:211
2820699305,4cyclictest0-21swapper/007:45:270
2820699305,3cyclictest0-21swapper/011:56:550
2820699301,9cyclictest17090-21cut11:35:190
2820699301,8cyclictest27583-21df_inode08:50:130
2820699301,6cyclictest469-21cut10:50:180
2820699301,6cyclictest14524-21grep10:40:160
2820699301,5cyclictest1592-21cat10:15:010
2820699301,10cyclictest17598-21ssh09:10:220
2824199295,19cyclictest35966irq/58-eth012:10:103
2822699292,20cyclictest0-21swapper/210:30:132
2822699292,17cyclictest27346irq/40-dwc2_hso08:55:122
2821799293,23cyclictest17033-21cat08:35:291
2820699290,8cyclictest21227-21ssh11:20:100
28241992815,9cyclictest35966irq/58-eth009:50:123
28241992814,9cyclictest35966irq/58-eth011:35:133
2822699288,4cyclictest27-21ksoftirqd/211:09:462
2822699286,9cyclictest27-21ksoftirqd/212:09:232
2822699285,8cyclictest7443-21ssh10:00:122
28226992814,12cyclictest0-21swapper/212:15:272
2821799280,24cyclictest28221-21latency_hist07:10:011
2820699281,6cyclictest15110-21cat08:00:220
2820699280,6cyclictest28208-21latency_hist07:10:010
2820699280,6cyclictest28208-21latency_hist07:10:010
2824199274,16cyclictest0-21swapper/311:05:123
2824199273,20cyclictest35966irq/58-eth010:10:123
2824199273,19cyclictest35966irq/58-eth011:55:113
28241992712,11cyclictest35966irq/58-eth012:05:133
28241992711,10cyclictest35966irq/58-eth009:25:143
2822699272,19cyclictest15730-21ssh11:35:132
2822699272,19cyclictest0-21swapper/207:20:132
2822699272,10cyclictest0-21swapper/207:45:132
2822699271,4cyclictest271ksoftirqd/209:30:132
28226992714,9cyclictest0-21swapper/212:00:142
2822699271,16cyclictest10060-21sh11:50:142
2824199264,13cyclictest35966irq/58-eth011:10:123
2824199263,12cyclictest35966irq/58-eth012:25:123
2824199263,12cyclictest35966irq/58-eth012:25:123
2824199261,18cyclictest35966irq/58-eth008:00:113
2822699269,7cyclictest27-21ksoftirqd/210:08:302
2822699269,7cyclictest27-21ksoftirqd/210:08:292
2822699265,11cyclictest27-21ksoftirqd/210:14:422
2822699262,21cyclictest0-21swapper/208:50:122
2822699262,16cyclictest27346irq/40-dwc2_hso09:45:122
2822699262,13cyclictest27346irq/40-dwc2_hso08:35:122
2822699261,6cyclictest27-21ksoftirqd/210:37:542
28226992615,8cyclictest261rcuc/207:35:172
2822699261,14cyclictest24657-21ssh11:40:132
2821799261,22cyclictest27495-21/usr/sbin/munin08:50:111
2824199252,13cyclictest35966irq/58-eth007:10:123
2822699257,3cyclictest27-21ksoftirqd/210:47:452
2822699256,7cyclictest27-21ksoftirqd/209:21:442
2822699255,6cyclictest27-21ksoftirqd/207:30:172
2822699255,2cyclictest131rcu_preempt10:15:132
2822699252,15cyclictest27346irq/40-dwc2_hso08:15:132
28226992512,9cyclictest3417-21apt-get11:10:132
2822699251,21cyclictest21914-21latency_hist12:15:022
2824199243,13cyclictest35966irq/58-eth012:00:133
2824199243,13cyclictest0-21swapper/311:25:053
2824199243,13cyclictest0-21swapper/311:25:053
2822699249,10cyclictest27-21ksoftirqd/210:53:532
2822699246,4cyclictest27-21ksoftirqd/209:35:432
2822699246,12cyclictest27-21ksoftirqd/212:22:192
2822699246,12cyclictest27-21ksoftirqd/212:22:192
2822699246,11cyclictest27-21ksoftirqd/212:36:502
2822699246,11cyclictest27-21ksoftirqd/210:58:042
2822699245,13cyclictest27-21ksoftirqd/210:26:152
2822699243,3cyclictest27-21ksoftirqd/212:25:222
2822699243,3cyclictest27-21ksoftirqd/212:25:222
2822699242,14cyclictest27346irq/40-dwc2_hso09:15:132
28226992412,9cyclictest19950-21ssh11:18:522
2822699241,12cyclictest2185666ssh09:50:112
2822699240,6cyclictest27-21ksoftirqd/211:25:122
2822699240,6cyclictest27-21ksoftirqd/211:25:112
2822699240,21cyclictest7279-21latency09:40:212
2824199233,15cyclictest0-21swapper/312:30:133
2824199231,16cyclictest35966irq/58-eth009:10:133
28241992310,9cyclictest35966irq/58-eth010:20:123
2822699239,4cyclictest27-21ksoftirqd/209:57:322
2822699235,7cyclictest27-21ksoftirqd/211:02:152
2822699235,7cyclictest27-21ksoftirqd/211:02:142
2822699235,3cyclictest131rcu_preempt09:25:232
2822699232,6cyclictest0-21swapper/212:30:042
28226992316,4cyclictest0-21swapper/207:25:122
2822699230,8cyclictest21306-21ls08:10:022
2824199222,12cyclictest35966irq/58-eth010:00:123
2824199221,13cyclictest35966irq/58-eth012:20:133
2824199221,13cyclictest35966irq/58-eth012:20:133
28241992210,9cyclictest0-21swapper/310:55:323
2822699225,7cyclictest131rcu_preempt07:15:162
2822699224,11cyclictest27-21ksoftirqd/211:21:012
28226992212,8cyclictest261rcuc/211:47:192
2822699221,13cyclictest27346irq/40-dwc2_hso08:40:122
2824199212,12cyclictest35966irq/58-eth009:15:143
2824199211,12cyclictest0-21swapper/309:45:133
2822699212,15cyclictest22069-21kthreadcore08:10:182
2822699211,12cyclictest24646-21kthreadcore08:45:212
28226992110,9cyclictest5581-21ntp_kernel_pll_07:50:232
2824199204,12cyclictest0-21swapper/311:20:143
2824199203,14cyclictest0-21swapper/309:40:113
28241992013,4cyclictest36066irq/59-eth011:45:223
2824199201,13cyclictest0-21swapper/309:05:133
2822699206,11cyclictest3619-21hwlatdetect08:25:162
2822699203,8cyclictest7059-21fschecks_count07:55:152
2822699202,9cyclictest12787-21cat08:35:022
2822699202,10cyclictest0-21swapper/209:00:122
2822699202,10cyclictest0-21swapper/209:00:122
2822699201,15cyclictest10738-21kthreadcore09:05:192
2824199191,13cyclictest0-21swapper/311:15:123
2824199191,13cyclictest0-21swapper/308:05:233
28241991910,6cyclictest0-21swapper/312:17:373
2824199189,6cyclictest35966irq/58-eth011:50:143
2824199189,6cyclictest0-21swapper/310:32:193
2824199183,11cyclictest0-21swapper/311:30:133
2824199182,9cyclictest35966irq/58-eth007:55:103
2824199182,10cyclictest0-21swapper/308:50:043
2824199182,10cyclictest0-21swapper/308:25:123
2824199182,10cyclictest0-21swapper/307:40:053
2824199182,10cyclictest0-21swapper/307:35:033
2824199181,11cyclictest0-21swapper/308:20:183
2822699181,7cyclictest28190-21cyclictest07:10:012
2822699181,7cyclictest28190-21cyclictest07:10:002
2824199179,6cyclictest0-21swapper/312:35:113
2824199172,10cyclictest0-21swapper/308:40:133
2824199172,10cyclictest0-21swapper/308:35:133
2824199172,10cyclictest0-21swapper/307:25:133
2824199171,9cyclictest0-21swapper/308:10:143
2824199168,6cyclictest0-21swapper/310:05:463
2824199168,6cyclictest0-21swapper/310:05:453
2824199162,10cyclictest0-21swapper/308:55:133
2824199161,9cyclictest0-21swapper/308:35:023
2824199161,12cyclictest0-21swapper/307:15:183
2824199151,11cyclictest0-21swapper/308:45:183
2824199142,10cyclictest0-21swapper/307:20:133
2824199141,8cyclictest0-21swapper/307:45:143
2824199132,8cyclictest0-21swapper/309:00:123
2824199132,8cyclictest0-21swapper/309:00:123
2824199131,9cyclictest0-21swapper/307:35:023
2824199127,3cyclictest0-21swapper/308:15:123
2824199121,8cyclictest0-21swapper/307:50:143
2824199101,6cyclictest0-21swapper/307:10:023
2824199101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional