You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 15:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Mar 31, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16485997557,15cyclictest28366irq/58-eth011:20:123
16485996935,24cyclictest28366irq/58-eth012:25:123
16485996442,13cyclictest28366irq/58-eth011:35:153
1646699511,37cyclictest12879-21ssh12:15:131
1646699452,35cyclictest19130-21ssh09:55:141
1648599423,23cyclictest28366irq/58-eth010:20:123
1648599422,23cyclictest28366irq/58-eth009:55:143
1648599365,18cyclictest28366irq/58-eth012:00:123
1648599361,19cyclictest28366irq/58-eth010:45:143
1648599345,22cyclictest28366irq/58-eth010:15:143
1646699341,2cyclictest131rcu_preempt12:20:121
1645499344,2cyclictest131rcu_preempt08:40:130
1645499342,24cyclictest20695-21sh09:20:120
1648599332,17cyclictest28366irq/58-eth010:55:143
1646699331,2cyclictest131rcu_preempt08:20:121
1648599314,9cyclictest28366irq/58-eth008:45:133
1648599302,19cyclictest28366irq/58-eth008:05:133
1648599301,16cyclictest28366irq/58-eth011:05:143
1646699307,13cyclictest21-21ksoftirqd/110:10:121
1648599296,16cyclictest28366irq/58-eth009:35:133
1648599291,13cyclictest28366irq/58-eth008:55:113
1647599291,21cyclictest5640-21ssh11:35:152
16485992823,3cyclictest28366irq/58-eth010:50:123
1648599281,17cyclictest28366irq/58-eth011:00:123
1648599281,14cyclictest28366irq/58-eth011:50:133
1647599273,20cyclictest23618-21ssh11:45:122
1648599265,12cyclictest0-21swapper/311:25:143
1648599262,18cyclictest0-21swapper/312:19:193
1647599262,15cyclictest3825-21ssh12:10:162
16475992516,7cyclictest0-21swapper/209:05:142
16475992513,9cyclictest5560-21ssh11:51:092
1647599251,19cyclictest10079-21ssh09:50:142
1645499259,3cyclictest131rcu_preempt12:05:180
1645499259,3cyclictest131rcu_preempt12:05:180
1645499252,15cyclictest12-21ksoftirqd/011:15:120
1648599242,11cyclictest0-21swapper/310:30:143
1648599241,16cyclictest0-21swapper/309:00:113
1647599248,6cyclictest839-21munin-node08:10:142
1647599244,12cyclictest22346irq/40-dwc2_hso07:15:132
1647599242,13cyclictest924-21snmpd12:15:142
16475992416,5cyclictest24601-21ssh12:20:212
1647599241,5cyclictest0-21swapper/207:20:122
16475992412,9cyclictest1745-21ssh10:01:512
16475992412,9cyclictest1419-21ssh11:30:352
16475992411,10cyclictest19726-21kthreadcore12:35:202
16475992411,10cyclictest19726-21kthreadcore12:35:202
16475992411,10cyclictest11547-21ssh09:15:082
1645499249,4cyclictest131rcu_preempt10:05:230
1645499249,4cyclictest131rcu_preempt10:05:230
1645499244,17cyclictest10917-21ssh11:20:120
1647599236,9cyclictest19520-21munin-plugin-st11:25:022
1647599235,15cyclictest22346irq/40-dwc2_hso10:40:152
1647599235,14cyclictest21249-21cron07:15:012
1647599235,14cyclictest21249-21cron07:15:012
1647599235,13cyclictest22346irq/40-dwc2_hso08:40:142
1647599235,12cyclictest22346irq/40-dwc2_hso08:50:112
1647599232,16cyclictest14598-21sh11:40:132
16475992312,8cyclictest20142-21threads11:05:292
1647599231,19cyclictest17683-21kworker/u8:210:55:132
1647599231,12cyclictest23027-21diskstats10:15:172
1647599231,12cyclictest18818-21cut12:00:182
16475992310,5cyclictest4070-21ssh09:27:442
1645499233,17cyclictest32708-21diskstats11:50:150
1645499231,19cyclictest0-21swapper/010:45:140
1648599229,9cyclictest28366irq/58-eth011:40:133
16485992210,6cyclictest0-21swapper/309:10:153
1647599229,10cyclictest24256-21latency_hist09:40:022
1647599229,10cyclictest12259-21sh12:30:302
1647599223,10cyclictest27230-21sort12:05:112
1647599223,10cyclictest27230-21sort12:05:112
1647599222,16cyclictest28174-21sh09:40:252
1647599222,10cyclictest2609-21kernelversion11:15:192
16475992216,3cyclictest2646rcuc/211:00:142
16475992211,9cyclictest27928-21ssh10:52:212
1647599221,12cyclictest25968-21irqstats11:10:192
1647599221,12cyclictest18162-21if_eth110:30:182
1647599221,12cyclictest16120-21ntp_192.168.11410:45:252
1647599221,11cyclictest26935-21diskstats10:35:142
1647599221,11cyclictest134-21systemd-journal09:05:012
16475992210,4cyclictest22573-21kthreadcore09:20:222
1647599220,19cyclictest0-21swapper/211:25:142
1646699221,6cyclictest31325-21uname10:55:191
16466992213,7cyclictest28889-21apt-get11:30:121
1646699221,18cyclictest2091-21diskmemload10:35:111
1646699221,16cyclictest12500-21ssh10:45:141
16466992210,9cyclictest21-21ksoftirqd/108:05:141
16454992213,6cyclictest26058-21cron07:20:000
1648599219,8cyclictest28366irq/58-eth010:05:143
1648599219,8cyclictest28366irq/58-eth010:05:143
1648599216,10cyclictest28366irq/58-eth012:10:143
1648599214,11cyclictest0-21swapper/309:50:143
16485992111,7cyclictest28366irq/58-eth012:35:113
16485992111,7cyclictest28366irq/58-eth012:35:113
16485992111,7cyclictest28366irq/58-eth009:25:143
16485992111,7cyclictest0-21swapper/309:30:123
1647599216,8cyclictest19231-21df_inode09:55:152
1647599216,11cyclictest22346irq/40-dwc2_hso07:40:122
1647599214,9cyclictest9155-21irqcore10:25:172
1647599214,13cyclictest22346irq/40-dwc2_hso07:50:132
1647599213,15cyclictest6575-21kthreadcore08:05:202
1647599213,13cyclictest22346irq/40-dwc2_hso09:10:122
1647599213,13cyclictest22346irq/40-dwc2_hso07:55:122
1647599211,16cyclictest924-21snmpd10:05:212
1647599211,16cyclictest924-21snmpd10:05:212
1647599211,12cyclictest21350-21ntp_states07:45:252
1647599211,11cyclictest6376-21ssh09:30:122
1647599211,11cyclictest2251-21ssh10:20:212
1647599211,11cyclictest1309-21fschecks_time09:45:162
1647599211,10cyclictest31575-21cut12:25:182
1647599211,10cyclictest14539-21uname10:10:182
16466992113,6cyclictest27212-21ntp_192.168.11411:45:241
16466992112,7cyclictest23722-21sshd12:01:181
16466992112,6cyclictest25225-21ntp_states10:50:261
16466992112,6cyclictest0-21swapper/109:00:131
1646699211,18cyclictest16830-21cut12:35:151
1646699211,18cyclictest16830-21cut12:35:141
16466992111,7cyclictest0-21swapper/107:35:281
16466992111,7cyclictest0-21swapper/107:35:281
16466992110,8cyclictest0-21swapper/111:40:121
1645499216,3cyclictest2091-21diskmemload09:37:110
1645499212,15cyclictest13798-21apt-get10:10:120
16454992112,6cyclictest12-21ksoftirqd/008:50:090
1645499211,16cyclictest7494-21ssh12:30:120
1645499211,10cyclictest0-21swapper/011:00:150
1648599202,11cyclictest28366irq/58-eth011:10:133
16485992013,4cyclictest0-21swapper/310:35:173
16485992012,5cyclictest0-21swapper/309:20:213
16485992011,6cyclictest0-21swapper/310:02:093
16485992010,4cyclictest0-21swapper/312:31:193
1647599205,12cyclictest27-21ksoftirqd/208:00:172
1647599205,12cyclictest27-21ksoftirqd/208:00:172
1647599204,8cyclictest29097-21irqcore08:30:192
1647599204,13cyclictest22346irq/40-dwc2_hso11:55:122
1647599204,12cyclictest22346irq/40-dwc2_hso08:55:122
1647599202,14cyclictest0-21swapper/208:25:152
1647599202,13cyclictest32173-21kthreadcore07:25:192
1647599201,9cyclictest8974-21ps07:35:202
1647599201,9cyclictest8974-21ps07:35:202
1647599201,8cyclictest3003-21kthreadcore08:35:202
1647599201,15cyclictest22346irq/40-dwc2_hso07:30:142
1647599201,10cyclictest14297-21sort08:15:152
1646699207,3cyclictest2091-21diskmemload12:07:091
1646699207,3cyclictest2091-21diskmemload12:07:091
1646699206,3cyclictest2091-21diskmemload11:14:281
1646699202,15cyclictest26510-21awk10:15:251
1646699201,7cyclictest17166systemd-network09:35:161
1646699201,5cyclictest10052-21apt-get09:50:131
16466992012,6cyclictest6443-21sshd09:46:101
1646699201,17cyclictest31149-21fschecks_time12:25:161
1646699201,15cyclictest23352-21kthreadcore07:50:191
1645499203,14cyclictest26805-21diskmemload10:35:110
1645499202,11cyclictest8633-21ssh10:25:130
1645499201,6cyclictest16625-21latency_hist07:10:010
16454992013,5cyclictest0-21swapper/009:30:210
16454992012,6cyclictest25610-21cut11:45:200
16454992011,6cyclictest0-21swapper/011:55:320
16454992011,6cyclictest0-21swapper/008:20:170
1645499201,14cyclictest16829-21grep12:35:140
1645499201,14cyclictest16829-21grep12:35:140
1645499201,10cyclictest0-21swapper/009:12:590
1648599193,11cyclictest0-21swapper/312:20:213
1648599191,9cyclictest28366irq/58-eth008:35:123
16485991912,4cyclictest0-21swapper/309:40:153
16485991911,6cyclictest0-21swapper/312:08:033
16485991911,6cyclictest0-21swapper/312:08:033
16485991910,4cyclictest0-21swapper/311:30:443
1647599193,7cyclictest19277-21fschecks_time08:20:162
1647599191,9cyclictest10744-21sort08:45:162
1646699199,7cyclictest27505-21cut09:40:231
1646699199,2cyclictest21-21ksoftirqd/109:17:421
1646699196,3cyclictest2091-21diskmemload11:07:351
1646699194,3cyclictest131rcu_preempt09:05:251
1646699193,11cyclictest20696-21sh09:20:121
1646699192,2cyclictest0-21swapper/111:59:471
16466991912,5cyclictest0-21swapper/107:30:281
16466991912,4cyclictest0-21swapper/111:35:201
1646699191,15cyclictest31916-21ssh10:20:111
1646699191,14cyclictest7493-21sh12:30:131
1646699191,13cyclictest17350-21apt-get10:30:031
16466991910,6cyclictest12462-21cron12:15:001
16466991910,6cyclictest0-21swapper/107:55:171
1645499199,7cyclictest29771-21apt-get09:25:130
1645499199,7cyclictest18233-21apt-get12:00:130
1645499195,3cyclictest131rcu_preempt09:57:400
1645499194,12cyclictest10333-21usage_EthcTask11:35:330
1645499193,14cyclictest12240-21kthreadcore09:50:210
16454991913,4cyclictest12-21ksoftirqd/010:00:140
16454991911,6cyclictest22520-21sshd11:44:100
16454991911,5cyclictest29976-21kthreadcore11:30:180
1645499191,10cyclictest24537-21ssh09:40:130
1645499191,10cyclictest0-21swapper/009:19:450
1648599188,7cyclictest28366irq/58-eth010:40:143
1648599185,10cyclictest28366irq/58-eth009:15:123
1648599182,10cyclictest0-21swapper/308:50:113
1648599182,10cyclictest0-21swapper/307:15:153
1648599181,13cyclictest0-21swapper/310:10:293
1648599181,13cyclictest0-21swapper/308:20:273
1648599181,13cyclictest0-21swapper/308:10:293
1648599181,12cyclictest0-21swapper/311:55:153
1648599181,12cyclictest0-21swapper/310:25:283
1648599181,12cyclictest0-21swapper/308:05:013
1648599181,12cyclictest0-21swapper/308:05:013
1648599181,10cyclictest0-21swapper/307:45:163
16485991810,6cyclictest0-21swapper/311:49:053
16485991810,6cyclictest0-21swapper/311:15:103
16485991810,6cyclictest0-21swapper/307:40:273
1647599181,14cyclictest16618-21cat07:10:012
1646699186,9cyclictest17549-21sed10:30:031
1646699182,2cyclictest131rcu_preempt11:01:381
1646699181,3cyclictest0-21swapper/111:20:041
1646699181,3cyclictest0-21swapper/111:20:001
1646699181,15cyclictest4415-21cron10:05:011
1646699181,15cyclictest4415-21cron10:05:011
16466991811,5cyclictest18815-21sshd08:18:101
1646699181,14cyclictest28417-21fschecks_time10:00:181
16466991811,4cyclictest2091-21diskmemload09:30:401
1646699181,11cyclictest2649-21ls09:10:131
1645499189,6cyclictest7911-21cat10:40:310
1645499188,7cyclictest8390-21df_inode07:35:160
1645499188,7cyclictest8390-21df_inode07:35:160
1645499183,13cyclictest28460-21kthreadcore11:10:220
1645499183,13cyclictest2137-21ls10:55:290
1645499182,7cyclictest2073366ssh12:19:050
16454991811,5cyclictest0-21swapper/008:55:160
16454991811,4cyclictest25023-21latency_hist11:10:020
1645499181,14cyclictest21538-21sh10:50:130
16454991811,4cyclictest0-21swapper/007:55:110
1648599179,6cyclictest0-21swapper/309:45:233
1648599172,10cyclictest0-21swapper/307:55:133
1648599172,10cyclictest0-21swapper/307:35:153
1648599172,10cyclictest0-21swapper/307:35:153
1648599172,10cyclictest0-21swapper/307:30:143
1648599171,13cyclictest0-21swapper/308:30:233
1646699179,6cyclictest24068-21fschecks_time08:25:181
1646699178,6cyclictest22445-21cron07:50:011
1646699174,2cyclictest131rcu_preempt11:54:061
1646699171,3cyclictest0-21swapper/109:26:351
1646699171,14cyclictest29047-21irqcore08:30:201
16466991711,4cyclictest0-21swapper/108:35:111
1646699171,13cyclictest24994-21wc07:15:281
1646699171,12cyclictest3599-21fschecks_count10:40:161
16466991710,5cyclictest0-21swapper/108:00:061
16466991710,5cyclictest0-21swapper/108:00:061
1646699170,14cyclictest9456-21apt-get08:10:121
1645499179,5cyclictest0-21swapper/007:45:260
1645499178,7cyclictest2719-21cat12:25:300
1645499177,8cyclictest22888-21apt-get10:15:140
1645499173,12cyclictest4651-21ntp_192.168.11409:45:260
1645499173,12cyclictest22964-21kthreadcore11:25:220
1645499173,11cyclictest2098-21latency_hist09:10:020
1645499171,8cyclictest0-21swapper/012:13:030
16454991712,3cyclictest25177-21cpu09:00:110
16454991711,4cyclictest0-21swapper/008:10:120
1645499171,13cyclictest3231-21kthreadcore08:35:210
16454991710,5cyclictest29098-21sshd12:23:180
16454991710,5cyclictest0-21swapper/007:20:250
16454991710,4cyclictest0-21swapper/008:05:270
1648599168,6cyclictest0-21swapper/308:27:563
1648599162,6cyclictest0-21swapper/309:05:273
1648599161,9cyclictest0-21swapper/307:20:263
1646699169,5cyclictest19976-21sed07:10:231
1646699169,5cyclictest19976-21sed07:10:221
1646699161,13cyclictest8961-21kthreadcore08:40:231
1646699161,13cyclictest29994-21timerandwakeup07:20:261
1646699161,13cyclictest22636-21kthreadcore08:55:201
1646699161,13cyclictest20841-21kthreadcore11:25:191
1646699161,12cyclictest1840-21kthreadcore07:25:221
1646699161,12cyclictest10691-21apt-get08:45:131
1646699161,11cyclictest16995-21kthreadcore08:50:171
1645499169,5cyclictest3349-21sed07:30:020
1645499169,5cyclictest18330-21cut07:10:190
1645499169,5cyclictest18330-21cut07:10:190
1645499169,5cyclictest0-21swapper/007:50:280
1645499168,6cyclictest23869-21sshd08:25:110
1645499163,8cyclictest31931-21ssh10:20:130
1645499163,11cyclictest17834-21latency_hist07:45:010
1645499163,10cyclictest29785-21kthreadcore08:30:210
1645499161,12cyclictest5842-21kthreadcore07:30:210
1645499161,12cyclictest17833-21ssh10:30:140
1648599151,9cyclictest0-21swapper/307:25:163
1646699151,6cyclictest13723-21cut07:40:191
1645499158,5cyclictest32304-21sshd08:00:100
1645499158,5cyclictest32304-21sshd08:00:100
1645499153,8cyclictest0-21swapper/008:15:130
1645499152,11cyclictest11889-21kthreadcore08:45:210
1648599149,3cyclictest0-21swapper/307:50:143
1648599141,7cyclictest0-21swapper/308:40:283
1646699141,11cyclictest16453-21cyclictest07:10:011
1648599138,3cyclictest0-21swapper/307:10:233
1648599138,3cyclictest0-21swapper/307:10:233
1648599132,9cyclictest0-21swapper/308:15:153
1648599101,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional