You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 00:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Nov 20, 2025 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2102699623,50cyclictest12716-21ssh09:35:101
2102699586,18cyclictest131rcu_preempt12:25:101
21034995611,38cyclictest0-21swapper/211:10:102
21034995611,38cyclictest0-21swapper/211:10:102
21034995132,13cyclictest2485-21apt-get10:10:152
21034994815,22cyclictest0-21swapper/209:20:142
21034994638,5cyclictest13977-21kworker/u8:010:55:132
21018994633,8cyclictest26014-21apt-get07:15:150
2101899463,36cyclictest10156-21ssh10:15:150
2102699441,34cyclictest2501-21ssh10:10:161
2102699433,8cyclictest131rcu_preempt07:25:121
2101899439,25cyclictest0-21swapper/010:20:120
21018994335,4cyclictest6056-21apt-get08:35:160
21018994329,9cyclictest20537-21apt-get12:25:110
21034994230,8cyclictest19987-21apt-get11:45:172
21018994233,5cyclictest27120-21apt-get10:05:130
2104699412,24cyclictest29566irq/58-eth011:55:133
21034994135,3cyclictest3224-21apt-get07:25:122
21018994112,18cyclictest0-21swapper/009:20:140
21034994029,7cyclictest971-21apt-get10:30:132
21034994021,14cyclictest0-21swapper/210:45:132
21034993919,15cyclictest0-21swapper/207:50:122
21046993821,10cyclictest29566irq/58-eth010:15:143
2103499381,3cyclictest0-21swapper/209:30:142
21018993832,4cyclictest15699-21apt-get08:45:120
21018993832,4cyclictest15699-21apt-get08:45:110
21018993823,10cyclictest12-21ksoftirqd/010:55:130
21034993726,8cyclictest3851-21apt-get09:50:142
21034993722,10cyclictest5600-21apt-get11:15:122
21034993718,9cyclictest0-21swapper/210:50:132
21034993716,18cyclictest0-21swapper/210:15:132
21034993716,15cyclictest0-21swapper/208:45:122
21034993716,15cyclictest0-21swapper/208:45:112
2102699375,22cyclictest21-21ksoftirqd/108:55:131
2102699373,24cyclictest20006-21ssh11:45:171
2102699372,28cyclictest25731-21ssh10:25:131
21034993621,11cyclictest19328-21apt-get10:00:122
21026993622,7cyclictest7263-21diskmemload11:05:111
21018993630,3cyclictest4635-21apt-get08:00:120
2103499358,12cyclictest1526-21ssh12:12:142
21034993528,3cyclictest22334-21apt-get11:05:112
2103499352,28cyclictest32656-21ssh10:06:392
21034993517,14cyclictest0-21swapper/212:30:122
21018993518,13cyclictest10910-21apt-get08:40:090
21034993415,16cyclictest0-21swapper/209:55:132
21034993412,8cyclictest25215-21rm12:06:402
21034993410,7cyclictest0-21swapper/211:29:242
2101899349,10cyclictest5398-21ssh10:53:000
2101899347,10cyclictest9399-21ssh12:16:220
21018993415,17cyclictest0-21swapper/009:30:130
2103499339,7cyclictest22690-21ssh10:43:462
2103499339,7cyclictest0-21swapper/211:24:392
21034993324,6cyclictest12870-21apt-get07:35:122
21034993318,12cyclictest3405-21ssh11:55:052
21034993310,9cyclictest19452-21ssh09:38:462
2101899338,16cyclictest12013-1kworker/0:2H09:05:110
2101899336,16cyclictest0-21swapper/009:12:480
21018993323,6cyclictest11580-21apt-get09:55:140
2103499327,9cyclictest0-21swapper/210:25:132
2103499325,16cyclictest26346irq/40-dwc2_hso09:15:132
2103499324,13cyclictest0-21swapper/211:00:482
2103499322,20cyclictest12270-21ssh12:20:122
21034993211,7cyclictest12464-21ssh12:37:522
2101899329,10cyclictest27737-21ssh11:06:310
2101899327,9cyclictest25153-21ssh12:06:340
2101899326,10cyclictest29351-21diskstats11:30:180
21018993226,2cyclictest25354-21apt-get08:55:140
21018993222,7cyclictest12705-21apt-get09:35:120
21018993220,5cyclictest7263-21diskmemload11:10:110
21018993220,5cyclictest7263-21diskmemload11:10:100
21018993216,12cyclictest0-21swapper/012:00:120
2103499317,6cyclictest93266snmpd11:38:482
2103499317,6cyclictest93266snmpd11:38:472
21034993123,5cyclictest20593-21ssh09:40:152
21034993116,12cyclictest26089-21proc_pri07:45:282
21034993115,13cyclictest4635-21apt-get08:00:122
21034993112,16cyclictest2664-21/usr/sbin/munin09:05:252
2102699316,15cyclictest21-21ksoftirqd/108:40:111
2101899319,10cyclictest26969-21ssh11:27:040
2101899315,15cyclictest0-21swapper/011:22:480
21018993126,3cyclictest29300-21apt-get12:30:120
2101899311,24cyclictest27828-21ssh11:50:150
21046993018,6cyclictest29566irq/58-eth010:00:133
21046993012,9cyclictest0-21swapper/311:20:143
2103499309,7cyclictest14682-21ssh10:38:352
2103499306,7cyclictest0-21swapper/211:30:572
21034993017,10cyclictest0-21swapper/208:35:162
21034993017,10cyclictest0-21swapper/207:55:102
21034993016,11cyclictest15475-21munin-run08:45:012
21034993015,12cyclictest7457-21wc12:15:232
21034993014,13cyclictest25302-21grep08:55:022
21034993014,12cyclictest29596-21mailstats07:15:292
21034993013,14cyclictest25471-21ssh12:25:332
2102699304,19cyclictest21249-21ssh11:25:161
2102699301,22cyclictest5719-21ssh12:35:151
2101899308,10cyclictest14723-21sh10:38:390
2101899307,8cyclictest27798-21ssh12:10:100
2101899307,10cyclictest3483-21if_err_eth007:25:170
2101899304,11cyclictest953-21/usr/sbin/munin10:30:120
2101899302,22cyclictest17650-21ssh12:21:160
2101899302,21cyclictest12172-21ssh11:40:120
21046992915,10cyclictest29566irq/58-eth012:25:113
2103499293,9cyclictest27822-21ssh11:50:152
2103499293,11cyclictest0-21swapper/209:47:382
21034992915,12cyclictest18928-21sshd08:12:382
21034992915,11cyclictest21272-21diskstats07:10:152
21034992914,12cyclictest9534-21munin-node08:05:112
21034992912,14cyclictest25556-21latency_hist10:25:022
21034992912,14cyclictest11192-21ls09:10:242
21034992911,11cyclictest28608-21cron08:25:002
21034992910,7cyclictest19167-21ssh12:04:552
2101899297,7cyclictest11380-21ssh11:39:170
2101899297,7cyclictest11380-21ssh11:39:160
21018992923,4cyclictest0-21swapper/009:15:120
21018992920,7cyclictest25713-21apt-get10:25:140
21046992819,5cyclictest29566irq/58-eth010:55:123
21034992818,7cyclictest0-21swapper/208:15:142
21034992813,8cyclictest1070-21/usr/sbin/munin09:25:222
2102699283,9cyclictest0-21swapper/110:01:401
2102699282,11cyclictest4532-21ssh11:13:401
2102699282,11cyclictest4532-21ssh11:13:401
21026992815,7cyclictest0-21swapper/111:34:411
2101899288,8cyclictest9221-21rm10:13:480
2101899285,10cyclictest0-21swapper/012:36:520
2101899284,10cyclictest26551-21ssh10:04:380
21018992818,6cyclictest28513-21apt-get09:45:170
21018992815,9cyclictest1339-21/usr/sbin/munin08:30:130
2104699272,18cyclictest0-21swapper/311:51:403
2104699272,18cyclictest0-21swapper/311:34:413
2104699272,18cyclictest0-21swapper/311:13:413
2104699272,18cyclictest0-21swapper/311:13:403
2103499278,16cyclictest1382-21mailstats09:00:272
21034992716,8cyclictest0-21swapper/207:40:132
2103499271,11cyclictest0-21swapper/211:42:402
21034992711,12cyclictest5937-21latency_hist08:35:012
2101899276,8cyclictest27661-21ssh09:44:140
2101899272,10cyclictest3848-21ssh09:50:130
21018992716,8cyclictest24509-21ps08:20:190
21018992715,10cyclictest8734-21cut07:30:180
21018992711,13cyclictest0-21swapper/007:50:110
21046992615,5cyclictest0-21swapper/311:00:223
2104699261,17cyclictest29566irq/58-eth012:15:143
21034992617,7cyclictest26346irq/40-dwc2_hso07:30:132
21034992613,10cyclictest26647-21kthreadcore08:55:202
21034992611,12cyclictest31075-21sed07:20:162
21034992611,12cyclictest31075-21sed07:20:152
2102699262,19cyclictest19592-21ssh12:05:131
21018992615,8cyclictest28936-21entropy08:25:130
21018992613,9cyclictest12969-21diskstats07:35:160
21018992612,11cyclictest23887-21missed_timers08:50:230
2102699256,12cyclictest201rcuc/112:30:131
2102699253,17cyclictest0-21swapper/111:42:401
2102699252,15cyclictest7263-21diskmemload12:15:141
2102699251,18cyclictest986-21rm10:30:151
2102699251,17cyclictest8510-21ssh10:35:141
21026992510,9cyclictest21-21ksoftirqd/108:45:121
21026992510,9cyclictest21-21ksoftirqd/108:45:111
2101899252,9cyclictest0-21swapper/010:40:140
2101899252,8cyclictest0-21swapper/011:15:390
21018992519,3cyclictest30847-21apt-get07:20:140
21018992519,3cyclictest30847-21apt-get07:20:130
21018992515,8cyclictest0-21swapper/007:40:120
2104699243,11cyclictest0-21swapper/309:55:153
21046992414,6cyclictest0-21swapper/310:05:193
21034992411,10cyclictest932-21snmpd08:25:342
21018992419,3cyclictest29938-21apt-get09:25:130
21018992413,9cyclictest21303-21tr11:45:230
21018992413,8cyclictest17479-21cut11:00:240
21018992412,9cyclictest21160-21acpi07:10:130
21018992412,8cyclictest30278-21/usr/sbin/munin09:00:150
21018992411,10cyclictest0-21swapper/010:45:110
21034992312,9cyclictest21043-21latency_hist07:10:012
2102699232,16cyclictest13285-21ssh11:20:161
21026992314,5cyclictest0-21swapper/108:30:121
2102699231,16cyclictest30654-21ls07:50:211
2101899234,9cyclictest662-21ntpd08:10:130
2101899233,10cyclictest12-21ksoftirqd/011:55:570
21018992313,8cyclictest4474-21sed08:00:020
21018992313,7cyclictest22497-21memory08:15:260
21018992311,9cyclictest11730-21kthreadcore08:05:200
2104699223,13cyclictest0-21swapper/310:30:143
2102699222,15cyclictest19174-21apt-get08:15:141
21026992213,6cyclictest0-21swapper/107:15:181
21026992212,7cyclictest13959-21cron11:00:021
2102699221,18cyclictest3459-21ls07:55:251
2102699221,18cyclictest22062-21ssh11:05:011
2104699211,16cyclictest29566irq/58-eth010:25:013
2102699212,14cyclictest874-21ssh11:51:401
2102699212,14cyclictest10664-21ssh11:16:041
2102699211,16cyclictest0-21swapper/108:35:161
2101899215,13cyclictest23599-21cut07:45:210
2104699203,12cyclictest0-21swapper/311:43:403
2104699203,10cyclictest0-21swapper/307:10:143
21046992011,5cyclictest0-21swapper/312:20:123
21046992010,7cyclictest0-21swapper/310:13:433
2102699209,8cyclictest18112-21apt-get10:20:131
2102699207,10cyclictest32188-21irqstats10:50:191
2102699204,12cyclictest2406-21ls09:05:011
2102699201,9cyclictest11755-21grep09:10:291
21026992013,5cyclictest32370-21usage_EthcTask12:10:301
2102699201,16cyclictest27210-21ls10:45:221
2102699201,12cyclictest4766-21cut08:00:151
21026992010,7cyclictest17873-21cat10:20:021
2104699198,7cyclictest0-21swapper/310:35:213
2104699193,11cyclictest0-21swapper/312:05:133
2104699193,11cyclictest0-21swapper/311:46:053
2104699193,11cyclictest0-21swapper/311:35:413
2104699193,11cyclictest0-21swapper/311:35:403
2104699193,11cyclictest0-21swapper/311:29:393
2104699193,11cyclictest0-21swapper/311:15:413
2104699193,11cyclictest0-21swapper/310:48:403
2104699193,11cyclictest0-21swapper/309:40:393
2104699192,13cyclictest0-21swapper/312:35:173
2104699192,13cyclictest0-21swapper/309:45:173
21046991913,3cyclictest29666irq/59-eth011:08:433
21046991911,5cyclictest29566irq/58-eth009:35:113
21046991911,5cyclictest29566irq/58-eth009:30:123
2104699191,14cyclictest0-21swapper/312:00:303
2104699191,13cyclictest0-21swapper/308:15:253
2104699191,12cyclictest0-21swapper/310:55:033
2104699191,12cyclictest0-21swapper/309:15:153
2104699191,11cyclictest0-21swapper/307:55:023
2102699195,5cyclictest14835-21sed08:10:181
2102699191,8cyclictest12369-21df12:20:141
2102699191,2cyclictest131rcu_preempt09:50:141
2102699191,13cyclictest24463-21cut09:20:191
21026991910,6cyclictest7475-21cat09:10:031
2101899194,3cyclictest0-21swapper/007:10:010
2104699189,6cyclictest0-21swapper/312:32:003
2104699183,10cyclictest0-21swapper/312:14:403
2104699182,10cyclictest0-21swapper/308:05:053
2104699182,10cyclictest0-21swapper/307:25:143
2104699182,10cyclictest0-21swapper/307:20:143
2104699182,10cyclictest0-21swapper/307:20:133
21046991812,3cyclictest0-21swapper/309:25:143
2104699181,13cyclictest0-21swapper/307:55:123
2104699181,12cyclictest0-21swapper/309:25:013
2104699181,11cyclictest0-21swapper/307:30:273
21046991810,5cyclictest0-21swapper/310:40:223
2102699185,10cyclictest27682-21irqrtprio10:05:171
2102699184,11cyclictest30711-21kthreadcore09:45:231
2102699183,9cyclictest0-21swapper/112:01:411
2102699183,5cyclictest131rcu_preempt09:55:281
2102699181,9cyclictest19993-21kthreadcore07:40:211
21026991810,6cyclictest8316-21sshd11:35:401
21026991810,6cyclictest8316-21sshd11:35:401
2104699172,10cyclictest0-21swapper/307:35:153
2104699171,12cyclictest0-21swapper/307:45:193
21046991710,5cyclictest29566irq/58-eth010:25:123
21046991710,4cyclictest29566irq/58-eth009:10:133
21046991710,4cyclictest0-21swapper/308:00:233
2102699179,5cyclictest11666-21latency_hist12:00:011
2102699178,7cyclictest30320-21sed09:25:171
2102699178,7cyclictest21752-21cut09:40:211
2102699178,7cyclictest16054-21diskstats10:40:171
2102699176,8cyclictest13468-21sed07:35:181
2102699175,10cyclictest25785-21idleruntime-cro07:15:011
2102699173,11cyclictest10518-21kthreadcore08:05:181
2102699172,12cyclictest31451-21irqrtprio07:20:171
2102699172,12cyclictest31451-21irqrtprio07:20:171
2102699171,9cyclictest23954-21ntp_192.168.11408:50:251
2102699171,5cyclictest4737-21apt-get09:30:141
2102699171,13cyclictest29740-21tr08:25:161
2102699171,13cyclictest24526-21kthreadcore07:45:211
21026991711,3cyclictest10681-21cut07:30:211
2102699171,11cyclictest22577-21latency_hist09:20:021
2104699164,8cyclictest0-21swapper/308:55:143
2104699163,7cyclictest0-21swapper/309:05:013
2104699163,11cyclictest0-21swapper/308:30:103
2104699162,10cyclictest0-21swapper/308:45:133
2104699162,10cyclictest0-21swapper/308:45:123
2104699161,12cyclictest0-21swapper/308:25:403
21046991610,4cyclictest0-21swapper/309:54:203
21026991610,3cyclictest24842-21kthreadcore08:20:211
2104699151,8cyclictest0-21swapper/307:15:293
2104699141,10cyclictest0-21swapper/308:20:233
2102699141,10cyclictest21041-21grep07:10:021
2104699133,7cyclictest0-21swapper/308:35:173
2104699132,9cyclictest0-21swapper/309:10:023
2104699132,8cyclictest29566irq/58-eth008:10:123
2104699132,8cyclictest0-21swapper/308:50:293
2104699132,8cyclictest0-21swapper/307:45:013
2104699132,7cyclictest0-21swapper/308:40:093
2104699111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional