You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-22 - 03:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Jun 22, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266799512,27cyclictest30359-21ssh00:15:102
266799441,28cyclictest6890-21ssh00:20:122
266799441,28cyclictest6890-21ssh00:20:122
2667994123,13cyclictest0-21swapper/223:00:122
266799392,30cyclictest16085-21ssh22:55:142
266799380,18cyclictest23361-21ssh22:05:132
2667993711,9cyclictest10512-21ssh21:40:042
265499375,4cyclictest131rcu_preempt22:35:140
2667993611,7cyclictest0-21swapper/200:30:132
267899352,24cyclictest35966irq/58-eth021:10:123
266799352,24cyclictest8740-21ssh23:45:122
267899343,20cyclictest35966irq/58-eth000:30:123
267899341,23cyclictest35966irq/58-eth023:20:143
267899331,21cyclictest35966irq/58-eth022:55:123
2654993315,3cyclictest12-21ksoftirqd/023:05:120
267899324,18cyclictest35966irq/58-eth022:20:123
267899324,17cyclictest35966irq/58-eth022:05:123
267899323,18cyclictest35966irq/58-eth023:00:143
2678993210,16cyclictest35966irq/58-eth019:40:123
266799323,17cyclictest28710-21sh21:50:162
266799323,17cyclictest28710-21sh21:50:162
266799322,22cyclictest27-21ksoftirqd/200:25:152
2667993211,9cyclictest17475-21ssh00:05:482
267899315,21cyclictest35966irq/58-eth000:25:143
267899313,18cyclictest35966irq/58-eth000:05:153
267899306,21cyclictest35966irq/58-eth023:45:133
267899297,7cyclictest0-21swapper/321:45:113
267899293,21cyclictest35966irq/58-eth000:00:123
267899293,18cyclictest33-21ksoftirqd/322:35:133
2678992914,9cyclictest35966irq/58-eth023:25:113
266799292,17cyclictest27346irq/40-dwc2_hso21:15:142
267899288,13cyclictest35966irq/58-eth000:15:133
267899285,14cyclictest0-21swapper/321:35:143
265599283,22cyclictest26916-21ssh22:25:121
265599283,20cyclictest26794-21ssh23:55:141
266799271,20cyclictest7685-21ls19:15:142
266799271,12cyclictest5241-21ssh21:55:122
267899264,18cyclictest35966irq/58-eth022:15:133
267899263,15cyclictest0-21swapper/321:20:163
266799262,16cyclictest27346irq/40-dwc2_hso23:40:132
2667992616,3cyclictest27-21ksoftirqd/222:15:132
266799261,19cyclictest14284-21ls23:10:242
266799261,15cyclictest17931-21ssh22:20:112
265599269,3cyclictest21-21ksoftirqd/119:40:111
267899256,8cyclictest0-21swapper/320:25:143
266799253,16cyclictest27346irq/40-dwc2_hso23:50:132
2655992513,8cyclictest21352-21ssh21:10:141
265499254,16cyclictest12-21ksoftirqd/019:20:130
267899245,12cyclictest35966irq/58-eth021:50:153
267899245,12cyclictest35966irq/58-eth021:50:143
267899243,15cyclictest36066irq/59-eth000:35:123
267899243,15cyclictest0-21swapper/323:40:143
2678992414,4cyclictest0-21swapper/323:35:153
266799243,13cyclictest27346irq/40-dwc2_hso19:10:132
266799242,13cyclictest21369-21ssh00:10:132
266799240,15cyclictest28771-21grep23:55:202
265599243,18cyclictest1646-21ssh21:35:151
265599241,17cyclictest25017-21ssh21:30:131
265599241,17cyclictest10804-21uniq21:40:161
265499245,13cyclictest12-21ksoftirqd/020:15:130
265499243,18cyclictest9043-21sh22:50:220
266799235,8cyclictest671-21uname22:10:192
266799235,5cyclictest87766lldpd22:25:132
266799233,14cyclictest27346irq/40-dwc2_hso00:00:132
266799231,19cyclictest5745-21kthreadcore19:45:212
2667992311,9cyclictest10785-21ssh22:33:252
266799231,13cyclictest12748-21fschecks_time22:35:172
266799230,7cyclictest13734-21cpuspeed19:55:152
266799230,20cyclictest28408-21timerandwakeup20:40:272
265599235,16cyclictest21-21ksoftirqd/121:00:121
265599235,15cyclictest30388-21kworker/u8:022:00:141
265599232,18cyclictest19897-21fschecks_time23:15:151
265499236,3cyclictest131rcu_preempt21:40:010
265499235,3cyclictest131rcu_preempt23:55:240
265499235,11cyclictest12-21ksoftirqd/020:10:140
265499231,7cyclictest27925-21usage_EthcTask22:05:310
267899223,15cyclictest0-21swapper/323:05:143
2678992210,9cyclictest0-21swapper/300:20:193
2678992210,9cyclictest0-21swapper/300:20:193
266799228,11cyclictest14970-21apt-get20:30:132
266799223,14cyclictest27346irq/40-dwc2_hso22:45:132
2667992216,3cyclictest20943-21diskmemload23:05:242
2667992210,9cyclictest21719-21fschecks_count22:40:162
2667992210,4cyclictest17658-21latency23:30:242
265599227,7cyclictest131rcu_preempt00:15:141
265599227,12cyclictest16074-21diskmemload21:25:141
265599223,17cyclictest14174-21ssh23:30:121
265599222,2cyclictest0-21swapper/122:54:531
265599222,13cyclictest21-21ksoftirqd/120:35:091
265599221,17cyclictest28975-21sed21:50:181
265599221,17cyclictest28975-21sed21:50:181
265599221,17cyclictest26844-21apt-get19:35:131
265499223,16cyclictest25546-21grep19:30:260
265499223,14cyclictest3403-21ssh00:00:130
265499222,17cyclictest3538-21ssh22:30:130
265499222,17cyclictest23145-21ssh23:35:140
265499221,7cyclictest32161-21cut22:45:200
2654992213,6cyclictest27879-21ssh23:00:230
2654992213,6cyclictest19707-21ssh23:15:110
265499221,18cyclictest24901-21ssh00:30:150
265499221,16cyclictest25016-21sh21:30:130
265499221,15cyclictest1406-21ssh00:35:120
265499221,14cyclictest17474-21ls23:50:010
265499220,17cyclictest24667-21ls00:30:010
267899215,11cyclictest0-21swapper/322:30:143
267899213,11cyclictest0-21swapper/322:25:133
2678992111,8cyclictest0-21swapper/323:30:143
267899211,14cyclictest0-21swapper/320:20:163
267899211,14cyclictest0-21swapper/320:20:163
266799219,9cyclictest9405-21ssh21:39:022
266799219,9cyclictest13641-21ssh21:22:352
266799212,10cyclictest29010-21kthreadcore20:10:192
266799211,9cyclictest31700-21cut23:20:232
266799211,9cyclictest16512-21irqcore21:25:182
266799211,17cyclictest22099-21kthreadcore20:35:192
266799211,17cyclictest0-21swapper/221:10:132
266799211,10cyclictest27346irq/40-dwc2_hso20:50:132
2667992110,9cyclictest28275-21cron21:50:012
2667992110,3cyclictest9968-21munin-plugin-st00:40:012
265599217,11cyclictest11439-21unixbench_multi19:15:291
265599213,15cyclictest12020-21awk00:05:021
265599212,17cyclictest31534-21kthreadcore21:15:211
265599212,16cyclictest8365-21kthreadcore20:20:221
265599212,16cyclictest8365-21kthreadcore20:20:211
265599212,16cyclictest422-21kernelversion23:40:181
265599212,16cyclictest24735-21ssh23:00:011
265599211,4cyclictest0-21swapper/100:31:161
265599211,17cyclictest23703-21kthreadcore00:10:221
265599211,17cyclictest10825-21kthreadcore20:25:201
265499212,15cyclictest14344-21sh22:00:130
265499211,16cyclictest24695-21apt-get20:40:110
265499211,16cyclictest16077-21ssh21:25:140
267899203,12cyclictest0-21swapper/321:30:143
267899202,12cyclictest0-21swapper/323:15:133
267899201,13cyclictest0-21swapper/320:00:173
2678992010,7cyclictest35966irq/58-eth022:45:133
266799209,8cyclictest0-21swapper/223:35:132
266799206,12cyclictest9123-21wc19:50:172
266799202,15cyclictest8568-21latency20:20:222
266799202,15cyclictest8568-21latency20:20:222
266799201,9cyclictest27042-21fschecks_time19:35:172
266799201,9cyclictest17350-21fschecks_count19:25:152
266799201,16cyclictest5604-21ssh23:25:172
266799201,13cyclictest28862-21grep21:30:282
266799201,11cyclictest27165-21grep20:05:282
266799201,11cyclictest20507-21kthreadcore23:15:182
266799201,10cyclictest16184-21timerwakeupswit19:20:292
266799200,14cyclictest22042-21find19:30:112
266799200,11cyclictest15328-21tr22:00:202
265599209,8cyclictest21669-21df22:40:151
265599205,3cyclictest131rcu_preempt20:50:001
2655992013,5cyclictest0-21swapper/119:20:181
2655992012,5cyclictest25130-21ssh23:00:101
265599201,16cyclictest888-21kthreadcore22:10:201
265599201,16cyclictest8732-21apt-get23:45:131
265599201,16cyclictest7494-21tr21:55:211
2655992011,6cyclictest7421-21interrupts21:20:191
265599201,16cyclictest31694-21switchtime20:10:261
265599201,16cyclictest22147-21switchtime20:00:271
265599201,16cyclictest2130-21fschecks_time20:50:161
265599201,15cyclictest23146-21sh23:35:151
265599200,14cyclictest21042-21ls00:10:021
265499203,15cyclictest12989-21cut20:25:230
265499203,14cyclictest2947-21cut20:50:180
265499203,14cyclictest21677-21kthreadcore20:35:190
265499203,14cyclictest18230-21ssh23:30:300
265499201,5cyclictest53646usb-storage22:55:120
2654992012,6cyclictest0-21swapper/019:45:270
2654992010,7cyclictest5354-21forks23:25:140
267899198,5cyclictest0-21swapper/323:55:213
267899193,12cyclictest0-21swapper/300:10:273
267899192,8cyclictest0-21swapper/319:35:143
267899192,11cyclictest0-21swapper/319:50:143
267899192,11cyclictest0-21swapper/319:45:153
2678991912,4cyclictest0-21swapper/323:10:253
2678991911,6cyclictest0-21swapper/321:57:153
267899191,13cyclictest0-21swapper/320:05:163
267899191,12cyclictest0-21swapper/321:15:163
267899191,11cyclictest0-21swapper/320:35:153
266799195,6cyclictest0-21swapper/220:45:132
266799192,14cyclictest11149-21kthreadcore20:25:212
266799191,9cyclictest907-21irqcore20:15:182
266799191,8cyclictest20187-21grep21:05:312
266799191,15cyclictest7204-21irqcore20:55:182
266799191,15cyclictest18833-21if_eth120:00:182
266799191,14cyclictest7797-21kernelversion22:50:192
266799191,14cyclictest11735-21fschecks_count21:00:142
266799191,10cyclictest32391-21kthreadcore19:40:192
265599197,3cyclictest21-21ksoftirqd/122:09:051
265599196,9cyclictest8951-21ssh22:15:141
265599194,3cyclictest131rcu_preempt23:50:251
265599193,3cyclictest131rcu_preempt23:13:461
265599193,14cyclictest13835-21fschecks_time19:55:161
265599192,3cyclictest131rcu_preempt00:27:521
265599192,2cyclictest0-21swapper/122:45:111
265599192,15cyclictest522-21fschecks_count20:15:161
265599191,3cyclictest13952-21ssh23:30:021
265599191,3cyclictest0-21swapper/122:38:551
265599191,16cyclictest3779-21kthreadcore23:05:211
265599191,16cyclictest30681-21kthreadcore23:20:201
265599191,16cyclictest26058-21kthreadcore20:40:181
265599191,16cyclictest24586-21kthreadcore20:05:201
2655991911,6cyclictest19551-21sshd20:31:251
265599191,15cyclictest6913-21grep00:20:141
265599191,15cyclictest6913-21grep00:20:141
265599191,15cyclictest3101-21ls19:10:181
265599191,15cyclictest10376-21kthreadcore19:50:191
265499194,12cyclictest2348-21grep23:40:220
265499193,14cyclictest21962-21tr22:40:180
265499193,14cyclictest20088-21irqcore21:45:180
265499193,13cyclictest27173-21ps19:35:180
265499193,13cyclictest21561-21grep00:10:170
265499193,13cyclictest135-21systemd-journal21:00:010
265499193,12cyclictest753-21sendmail-mta21:00:130
265499193,12cyclictest27973-21cat20:10:010
265499191,9cyclictest29491-21/usr/sbin/munin20:45:130
265499191,4cyclictest53646usb-storage20:30:040
2654991912,5cyclictest28896-21fschecks_count23:20:170
2654991911,5cyclictest2013-21ssh21:15:300
265499191,10cyclictest0-21swapper/021:59:430
267899188,6cyclictest35966irq/58-eth022:50:143
267899187,6cyclictest0-21swapper/321:25:423
267899186,9cyclictest0-21swapper/321:40:213
267899182,10cyclictest0-21swapper/320:45:123
267899182,10cyclictest0-21swapper/320:40:133
267899182,10cyclictest0-21swapper/320:30:043
2678991810,6cyclictest0-21swapper/322:41:093
266799181,13cyclictest2821-21latency_hist19:10:022
265599182,14cyclictest8186-21kthreadcore20:55:191
265599181,15cyclictest6269-21kthreadcore19:45:221
265599181,15cyclictest18373-21irqcore22:20:181
265599181,15cyclictest17829-21irqrtprio19:25:181
265599181,15cyclictest1681-21idleruntime00:35:161
265599181,15cyclictest16512-21fschecks_count21:05:171
265599181,14cyclictest25586-21grep19:30:251
265599181,14cyclictest21820-21kthreadcore21:45:211
265499188,7cyclictest22054-21irqstats21:10:180
265499188,7cyclictest19327-21cron23:15:020
265499184,12cyclictest7678-21kthreadcore00:20:190
265499184,12cyclictest7678-21kthreadcore00:20:190
265499183,13cyclictest5726-21irqcore20:20:180
265499183,13cyclictest5726-21irqcore20:20:180
265499183,13cyclictest17369-21fschecks_count19:25:140
265499183,13cyclictest1053-21kthreadcore00:15:210
265499181,9cyclictest12209-21latency_hist19:20:010
265499181,3cyclictest131rcu_preempt00:05:030
2654991811,5cyclictest0-21swapper/022:20:170
265499181,13cyclictest1230-21kthreadcore22:10:210
265499181,13cyclictest0-21swapper/021:40:150
265499181,10cyclictest20119-21timerwakeupswit21:05:300
2654991810,6cyclictest10600-21cat21:20:260
267899179,6cyclictest0-21swapper/322:10:043
267899179,6cyclictest0-21swapper/322:01:153
267899172,10cyclictest0-21swapper/323:50:143
267899172,10cyclictest0-21swapper/320:50:053
267899172,10cyclictest0-21swapper/319:55:143
267899172,10cyclictest0-21swapper/319:15:133
267899172,10cyclictest0-21swapper/319:10:153
265599172,11cyclictest3547-21ssh22:30:141
265499179,6cyclictest21896-21nfs_client20:00:240
265499178,7cyclictest707-21cat21:50:330
265499178,7cyclictest707-21cat21:50:320
265499174,11cyclictest3974-21tr19:10:210
265499171,13cyclictest2744-21ls19:10:020
265499171,11cyclictest9145-21ssh22:15:170
2654991710,5cyclictest18129-21ssh23:50:180
265499170,14cyclictest3312-21latency_hist22:30:020
267899161,8cyclictest35966irq/58-eth020:15:113
265599161,13cyclictest2820-21latency_hist19:10:021
265499163,10cyclictest1191-21kthreadcore19:40:200
265499161,11cyclictest13789-21grep19:55:150
267899159,4cyclictest33-21ksoftirqd/319:10:013
267899151,8cyclictest0-21swapper/319:30:273
265499151,11cyclictest9024-21tune2fs19:50:160
267899138,3cyclictest0-21swapper/321:05:043
267899138,3cyclictest0-21swapper/320:12:253
267899131,9cyclictest0-21swapper/321:05:013
267899122,7cyclictest0-21swapper/319:20:213
267899121,8cyclictest0-21swapper/320:55:143
267899121,7cyclictest0-21swapper/319:25:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional