You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 09:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Tue Dec 02, 2025 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1484799603,45cyclictest14512-21ssh22:55:152
1484799571,3cyclictest0-21swapper/219:25:162
1483899548,41cyclictest11696-21sshd21:55:151
1484799532,7cyclictest0-21swapper/221:45:162
1484799521,31cyclictest6272-21ssh22:50:142
1484799511,45cyclictest30375-21sshd22:45:162
1484799511,45cyclictest30375-21sshd22:45:152
1484799511,3cyclictest0-21swapper/223:10:142
1484799502,3cyclictest856-21sshd19:35:132
1484799501,44cyclictest0-21swapper/222:30:132
1484799491,8cyclictest0-21swapper/221:10:132
1484799491,40cyclictest0-21swapper/223:30:152
1484799481,3cyclictest0-21swapper/220:10:152
1482499482,9cyclictest131rcu_preempt23:05:160
1484799472,24cyclictest438-21ssh00:05:132
1484799463,38cyclictest7549-21apt-get00:30:112
1484799460,3cyclictest31153-21ssh23:05:162
1484799451,37cyclictest26346irq/40-dwc2_hso21:25:152
1484799450,4cyclictest11688-21apt-get21:55:162
1484799441,37cyclictest26346irq/40-dwc2_hso19:20:152
1484799441,2cyclictest8506-21apt-get00:10:172
1484799441,2cyclictest1973-21apt-get19:30:142
1484799441,2cyclictest0-21swapper/200:20:112
1482499443,7cyclictest131rcu_preempt20:05:170
1484799431,33cyclictest16107-21ssh23:35:142
14856994227,8cyclictest29566irq/58-eth022:50:133
1484799423,21cyclictest19147-21ssh21:40:152
1484799422,36cyclictest0-21swapper/219:45:122
1484799422,2cyclictest11611-21apt-get19:40:142
1484799421,4cyclictest0-21swapper/223:20:132
1484799421,4cyclictest0-21swapper/223:20:132
1484799411,3cyclictest0-21swapper/222:35:152
1482499418,7cyclictest131rcu_preempt00:30:110
1484799402,31cyclictest0-21swapper/221:50:142
1484799401,5cyclictest24053-21grep22:00:292
1484799401,2cyclictest28850-21apt-get21:05:132
1484799400,25cyclictest0-21swapper/200:25:132
1484799392,7cyclictest0-21swapper/223:15:142
1484799392,3cyclictest16759-21apt-get00:15:132
1484799391,7cyclictest26346irq/40-dwc2_hso21:15:122
1484799391,3cyclictest0-21swapper/223:25:142
1484799391,2cyclictest14389-21apt-get20:50:152
1483899391,29cyclictest5376-21ssh22:30:141
1485699381,24cyclictest29566irq/58-eth000:30:123
1484799381,5cyclictest0-21swapper/220:20:152
1484799381,5cyclictest0-21swapper/220:20:152
1484799381,3cyclictest4683-21apt-get20:40:142
1484799381,33cyclictest8540-21unixbench_singl22:10:292
1483899381,32cyclictest24794-21sshd19:20:131
1484799372,25cyclictest24078-21cut22:40:222
1484799371,7cyclictest24439-21cron00:00:002
1484799371,3cyclictest0-21swapper/200:00:132
1484799371,2cyclictest0-21swapper/220:25:152
1484799361,4cyclictest21110-21sort20:55:232
1484799361,3cyclictest0-21swapper/220:00:162
1484799361,2cyclictest0-21swapper/220:35:142
1484799361,2cyclictest0-21swapper/219:10:152
1482499363,30cyclictest444-21sh00:05:130
14824993618,13cyclictest21166-21sshd22:20:150
1484799351,3cyclictest0-21swapper/219:50:142
1484799351,29cyclictest28775-21irqcore22:05:192
1484799350,5cyclictest18769-21ssh22:17:122
1484799350,31cyclictest24488-21apt-get23:40:132
1484799350,2cyclictest18010-21apt-get21:20:142
1483899354,23cyclictest21-21ksoftirqd/100:30:111
1483899351,17cyclictest131rcu_preempt20:20:151
1483899351,17cyclictest131rcu_preempt20:20:151
14824993527,6cyclictest12-21ksoftirqd/021:20:120
1484799341,2cyclictest0-21swapper/222:25:152
1484799341,25cyclictest8225-21apt23:50:012
1485699331,19cyclictest29566irq/58-eth023:25:143
1484799335,25cyclictest2810-21irqcore21:30:182
1484799335,25cyclictest2810-21irqcore21:30:182
1484799334,25cyclictest8793-21fschecks_time23:50:162
1484799331,3cyclictest26312-21memory23:00:252
1484799331,28cyclictest10121-21uname20:45:202
1484799330,7cyclictest27582-21cut21:00:282
1484799330,3cyclictest21150-21apt-get22:20:142
1484799330,29cyclictest16554-21switchtime20:15:292
1482499332,5cyclictest131rcu_preempt20:55:170
1484799321,6cyclictest22287-21ssh00:40:022
1484799321,5cyclictest26054-21/usr/sbin/munin19:55:312
1484799320,2cyclictest10728-21apt-get21:35:142
1483899322,22cyclictest30380-21ssh22:45:151
1483899322,22cyclictest30380-21ssh22:45:151
14838993212,11cyclictest21-21ksoftirqd/123:55:151
14824993221,7cyclictest12-21ksoftirqd/020:00:160
14856993125,4cyclictest0-21swapper/321:40:163
1484799311,6cyclictest30244-21cut20:30:232
1484799311,5cyclictest5438-21tr20:05:262
1484799310,5cyclictest19918-21/usr/sbin/munin19:15:162
1483899315,16cyclictest21-21ksoftirqd/120:50:141
1483899313,19cyclictest8580-21ssh23:50:121
1482499314,21cyclictest29453-21ssh22:25:150
1482499313,21cyclictest24515-21ssh23:40:150
1482499312,25cyclictest932-21snmpd19:55:150
1483899302,22cyclictest29470-21ssh22:25:151
1482499303,24cyclictest7627-21ssh23:30:140
1482499302,12cyclictest20111-21ssh22:00:140
14824993019,6cyclictest131rcu_preempt23:10:140
1483899291,9cyclictest530-21usb-storage21:10:041
1482499294,19cyclictest1000-21diskmemload23:50:120
1482499293,18cyclictest16777-21ssh00:15:140
14824992915,11cyclictest11465-21ssh21:35:190
1482499291,21cyclictest14510-21rm22:55:140
1485699285,19cyclictest29566irq/58-eth023:55:153
1485699284,16cyclictest29566irq/58-eth022:25:143
1485699283,21cyclictest29566irq/58-eth021:55:163
1483899284,13cyclictest15219-21ssh23:15:131
1483899282,19cyclictest18032-21ssh21:20:141
1483899281,23cyclictest8532-21ssh00:10:161
1482499289,9cyclictest131rcu_preempt21:55:150
1482499282,19cyclictest9722-21ssh21:15:140
14824992819,7cyclictest12-21ksoftirqd/022:15:140
14824992818,4cyclictest24735-21ssh00:20:110
14856992711,9cyclictest29566irq/58-eth023:05:173
1483899272,3cyclictest131rcu_preempt21:35:131
1483899272,20cyclictest14523-21ssh22:55:151
1483899270,19cyclictest20116-21ssh22:00:141
1485699263,14cyclictest0-21swapper/322:30:143
1485699262,18cyclictest0-21swapper/300:05:133
14856992620,4cyclictest29566irq/58-eth000:35:143
1483899262,19cyclictest28386-21ssh22:05:141
1483899262,17cyclictest26365-21ssh21:25:161
1482499265,12cyclictest0-21swapper/020:40:140
14824992618,4cyclictest0-21swapper/022:35:160
1485699255,14cyclictest29566irq/58-eth021:50:133
1485699254,13cyclictest0-21swapper/322:00:143
14856992511,11cyclictest29566irq/58-eth022:05:133
14838992518,4cyclictest0-21swapper/122:15:131
1483899251,20cyclictest4410-21ssh22:10:141
1483899251,19cyclictest16102-21ssh23:35:141
1482499251,20cyclictest24906-21ssh00:00:140
1482499243,18cyclictest31733-21ssh23:25:140
14824992417,3cyclictest0-21swapper/019:30:150
14824992412,6cyclictest6761-21sshd21:11:150
1482499241,17cyclictest32340-21ls20:35:150
1485699234,15cyclictest0-21swapper/323:45:133
1482499231,16cyclictest8517-21ssh00:10:150
1485699228,10cyclictest29566irq/58-eth023:10:153
1485699222,15cyclictest29566irq/58-eth021:10:133
1483899225,12cyclictest15452-21ssh00:35:131
1483899222,17cyclictest21-21ksoftirqd/120:15:141
1483899221,18cyclictest3432-21cut21:50:171
1483899221,14cyclictest32751-21ssh23:45:131
1483899221,14cyclictest31198-21sed20:00:201
14838992210,8cyclictest0-21swapper/119:25:161
1482499228,4cyclictest1000-21diskmemload22:34:290
1482499222,15cyclictest32155-21ssh00:25:130
1482499222,15cyclictest15445-21apt-get00:35:130
1482499221,18cyclictest32060-21ls20:35:010
1482499221,17cyclictest6330-21ls22:50:140
1482499220,11cyclictest656-21ssh21:46:490
1485699214,12cyclictest321rcuc/322:35:153
1485699213,9cyclictest29566irq/58-eth020:10:143
1485699213,14cyclictest29566irq/58-eth023:35:143
1485699213,13cyclictest0-21swapper/321:30:153
1485699213,13cyclictest0-21swapper/321:30:143
1485699212,13cyclictest0-21swapper/322:15:133
14856992113,6cyclictest0-21swapper/322:45:293
14856992113,6cyclictest0-21swapper/322:45:293
14856992112,3cyclictest0-21swapper/321:05:133
1484799212,2cyclictest14862-21sed19:10:012
1483899211,14cyclictest3506-21ls20:35:281
1483899211,13cyclictest21988-21ssh22:40:141
14838992110,8cyclictest6953-21cpu23:10:141
1483899210,17cyclictest31200-21timerwakeupswit20:30:301
1482499215,4cyclictest131rcu_preempt23:35:210
1482499212,15cyclictest1000-21diskmemload21:30:140
1482499212,15cyclictest1000-21diskmemload21:30:140
1482499211,4cyclictest131rcu_preempt22:05:300
1485699204,12cyclictest29566irq/58-eth019:35:133
1485699203,13cyclictest0-21swapper/323:30:163
14856992011,4cyclictest0-21swapper/321:25:173
1485699201,13cyclictest0-21swapper/320:25:153
1483899203,14cyclictest0-21swapper/119:40:151
1483899202,6cyclictest3280-21apt-get20:05:171
1483899201,16cyclictest25080-21irqcore00:20:161
1483899201,14cyclictest7619-21apt-get23:30:141
1483899200,6cyclictest131rcu_preempt23:20:041
1483899200,6cyclictest131rcu_preempt23:20:041
1483899200,16cyclictest17770-21ls21:20:021
1483899200,15cyclictest2592-21ls23:05:311
1482499209,5cyclictest17830-21apt-get20:20:150
1482499209,5cyclictest17830-21apt-get20:20:150
1482499203,13cyclictest16455-21sshd19:45:130
1482499202,4cyclictest131rcu_preempt20:25:290
1482499201,4cyclictest131rcu_preempt19:20:260
14824992012,6cyclictest22671-21ssh22:40:190
14824992012,6cyclictest16810-21ssh23:55:160
1485699193,9cyclictest0-21swapper/320:05:193
1485699193,15cyclictest0-21swapper/300:20:113
1485699193,11cyclictest0-21swapper/322:55:143
1485699192,12cyclictest0-21swapper/322:20:153
1485699192,11cyclictest0-21swapper/320:00:173
14856991912,4cyclictest0-21swapper/321:15:143
14856991911,5cyclictest29566irq/58-eth019:45:133
1485699191,13cyclictest0-21swapper/323:20:143
1485699191,13cyclictest0-21swapper/323:20:133
1485699191,13cyclictest0-21swapper/321:35:233
1485699191,11cyclictest0-21swapper/319:25:303
1483899197,9cyclictest2801-21cat23:25:241
1483899195,4cyclictest131rcu_preempt20:00:011
1483899192,14cyclictest6918-21fschecks_count19:35:171
1483899192,13cyclictest1-21systemd22:20:151
1483899191,15cyclictest3716-21grep00:25:291
1483899191,15cyclictest14167-21ls20:50:011
1483899191,13cyclictest27404-21ssh21:45:141
1483899191,12cyclictest6541-21idleruntime22:50:171
1482499199,8cyclictest0-21swapper/019:10:010
1482499199,2cyclictest12-21ksoftirqd/021:25:200
1482499195,7cyclictest131rcu_preempt22:46:370
1482499195,7cyclictest131rcu_preempt22:46:370
1482499193,3cyclictest131rcu_preempt20:10:150
1482499192,14cyclictest151rcuc/023:20:130
1482499192,14cyclictest151rcuc/023:20:130
14824991911,6cyclictest19155-21ssh21:40:170
1482499191,15cyclictest3271-21ssh21:50:140
14824991910,6cyclictest1016-21ssh23:45:180
1485699189,6cyclictest29566irq/58-eth023:40:133
1485699189,6cyclictest0-21swapper/323:15:173
1485699187,9cyclictest33-21ksoftirqd/323:53:003
1485699182,8cyclictest0-21swapper/322:40:213
1485699182,11cyclictest0-21swapper/319:50:163
1485699182,10cyclictest0-21swapper/322:10:043
1485699182,10cyclictest0-21swapper/321:00:153
1485699182,10cyclictest0-21swapper/320:45:153
1485699182,10cyclictest0-21swapper/320:20:063
1485699182,10cyclictest0-21swapper/320:20:053
1485699182,10cyclictest0-21swapper/319:15:153
1485699182,10cyclictest0-21swapper/319:10:143
1485699182,10cyclictest0-21swapper/300:00:043
1485699181,12cyclictest0-21swapper/300:25:143
1485699181,11cyclictest0-21swapper/320:15:243
14856991810,6cyclictest0-21swapper/323:00:163
14856991810,6cyclictest0-21swapper/321:47:453
1483899181,15cyclictest25188-21kthreadcore21:00:211
1483899181,14cyclictest425-21apt-get00:05:121
14838991810,6cyclictest19515-21/usr/sbin/munin21:40:201
1482499189,6cyclictest12-21ksoftirqd/019:25:150
1482499183,5cyclictest167-21systemd-network20:50:160
1482499181,3cyclictest131rcu_preempt23:15:150
1482499181,14cyclictest13376-21head20:45:300
1482499180,11cyclictest4384-21find22:10:110
1485699179,6cyclictest0-21swapper/300:16:203
1485699173,11cyclictest0-21swapper/319:20:133
1485699172,10cyclictest0-21swapper/320:35:153
14856991711,4cyclictest0-21swapper/321:20:213
1485699171,10cyclictest0-21swapper/320:45:013
1485699171,10cyclictest0-21swapper/319:55:153
14856991710,4cyclictest0-21swapper/300:10:223
1483899178,6cyclictest24730-21kthreadcore20:25:231
1483899177,8cyclictest21071-21sshd22:39:151
1483899177,7cyclictest21240-21apt-get19:50:141
1483899173,11cyclictest24443-21apt-get00:00:031
1483899172,10cyclictest0-21swapper/119:10:141
1483899171,6cyclictest2054-21df19:30:161
1483899171,12cyclictest26507-21ntp_states23:00:291
1482499179,6cyclictest25830-21sshd19:54:140
1482499176,6cyclictest0-21swapper/023:00:150
1482499172,11cyclictest0-21swapper/021:00:150
1482499171,8cyclictest12934-21/usr/sbin/munin20:15:260
1482499171,13cyclictest7514-21cut19:35:190
1485699162,10cyclictest0-21swapper/320:30:163
1485699161,10cyclictest0-21swapper/319:30:163
1483899166,7cyclictest30341-21kthreadcore21:05:201
1483899166,7cyclictest25824-21kthreadcore23:40:221
1483899163,10cyclictest17370-21cut00:15:191
1483899162,9cyclictest2538-21fschecks_time21:30:161
1483899162,9cyclictest2538-21fschecks_time21:30:161
1483899161,6cyclictest20529-21cat19:45:291
1483899161,11cyclictest20031-21diskstats19:15:151
1482499164,9cyclictest21676-21kthreadcore19:15:200
1482499162,9cyclictest0-21swapper/019:40:160
1482499162,11cyclictest31984-21kthreadcore21:05:220
1483899153,10cyclictest9999-21sed20:10:211
1483899151,8cyclictest0-21swapper/120:40:151
1483899151,10cyclictest19331-21diskstats20:55:191
1485699142,4cyclictest0-21swapper/320:55:163
1485699141,10cyclictest0-21swapper/319:40:233
1483899141,10cyclictest14892-21latency_hist19:10:011
1482499143,9cyclictest14823-21cyclictest19:10:010
1485699132,9cyclictest0-21swapper/320:50:153
1485699113,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional