You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-17 - 12:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Jul 17, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1877199525,10cyclictest131rcu_preempt23:20:111
1877999511,34cyclictest4711-21ssh21:10:112
1877199513,7cyclictest131rcu_preempt21:05:131
1875999512,41cyclictest13137-21ssh21:50:120
1875999503,34cyclictest18114-21ssh00:15:130
1875999491,40cyclictest21556-21ssh22:30:130
18771994827,17cyclictest131rcu_preempt19:55:121
1875999461,28cyclictest30635-21ssh22:35:130
1877199454,35cyclictest8207-21ssh22:05:141
1875999453,5cyclictest131rcu_preempt22:25:130
1875999441,34cyclictest4007-21ssh00:25:130
1878799432,23cyclictest35966irq/58-eth023:50:133
1877199431,38cyclictest19244-21ssh23:40:141
1877199424,33cyclictest24309-21ssh23:25:141
1877199424,33cyclictest24309-21ssh23:25:141
1877199422,32cyclictest29491-21ssh23:10:131
1875999426,23cyclictest6201-21ssh23:15:130
1877199413,11cyclictest131rcu_preempt20:35:131
1877199411,34cyclictest8876-21ssh21:30:131
1877199411,27cyclictest13852-21ssh21:15:141
1878799402,23cyclictest35966irq/58-eth023:30:133
1878799402,22cyclictest35966irq/58-eth021:10:103
1878799394,21cyclictest35966irq/58-eth022:25:133
18779993921,14cyclictest27396-21apt-get21:00:132
18759993933,4cyclictest0-21swapper/021:40:120
1875999391,29cyclictest28297-21ssh23:45:130
18779993820,13cyclictest0-21swapper/219:50:122
1877199382,29cyclictest13093-21ssh00:30:121
18787993712,14cyclictest35966irq/58-eth022:45:153
18779993725,8cyclictest13125-21apt-get21:50:142
18779993721,10cyclictest27193-21apt-get21:40:122
1877199372,29cyclictest25622-21ssh22:50:131
1877199364,22cyclictest23025-21ssh21:20:141
1875999362,24cyclictest23305-21ssh00:00:130
18779993515,16cyclictest0-21swapper/223:00:142
1878799342,21cyclictest35966irq/58-eth023:45:133
1877999345,12cyclictest0-21swapper/223:40:142
18779993427,3cyclictest26542-21apt-get22:15:132
18779993423,7cyclictest32193-21apt-get21:05:142
1877999342,25cyclictest22216-21ssh00:35:142
18779993316,12cyclictest27346irq/40-dwc2_hso20:30:132
1877199332,22cyclictest7347-21ssh22:40:131
1875999332,25cyclictest23035-21ssh21:20:140
1877999325,14cyclictest21196-21awk20:50:242
1877199325,21cyclictest21-21ksoftirqd/122:45:151
1877199322,19cyclictest4702-21ssh21:10:111
18771993217,11cyclictest21-21ksoftirqd/120:40:121
1875999324,21cyclictest3997-21ssh21:45:150
1878799313,18cyclictest35966irq/58-eth000:15:143
1877999312,18cyclictest7366-21ssh22:40:142
18779993119,8cyclictest27-21ksoftirqd/221:25:122
18779993112,14cyclictest0-21swapper/200:15:132
1875999315,19cyclictest12-21ksoftirqd/019:40:120
18779993012,11cyclictest27-21ksoftirqd/222:25:122
18779993011,15cyclictest0-21swapper/222:05:132
1875999302,19cyclictest4692-21ssh21:10:090
1877999298,14cyclictest0-21swapper/222:14:162
1877999294,9cyclictest13082-21apt00:30:112
1877999294,10cyclictest22327-21ssh21:55:132
1877999293,21cyclictest23327-21rm00:00:152
1877999293,16cyclictest27346irq/40-dwc2_hso21:35:132
1877999292,16cyclictest2243-21ssh22:55:132
18779992919,7cyclictest14075-21apt-get23:55:142
1877199292,17cyclictest21-21ksoftirqd/119:20:111
18771992914,4cyclictest131rcu_preempt22:00:131
1875999293,21cyclictest3307-21ssh22:20:140
1875999292,20cyclictest32482-21ssh00:05:170
1878799281,11cyclictest35966irq/58-eth020:10:123
1877999288,13cyclictest25636-21ssh22:50:152
1877999287,18cyclictest6064-21ps19:30:162
1877999287,18cyclictest6064-21ps19:30:162
1877999283,19cyclictest27346irq/40-dwc2_hso20:25:122
1877999283,18cyclictest27346irq/40-dwc2_hso22:20:142
1877999283,15cyclictest27346irq/40-dwc2_hso20:05:132
1877999283,15cyclictest27346irq/40-dwc2_hso20:05:132
1877999282,13cyclictest32232-21kthreadcore22:35:202
1877199283,20cyclictest27334-21ssh00:20:141
1877999276,13cyclictest27346irq/40-dwc2_hso20:40:132
1877999276,13cyclictest27346irq/40-dwc2_hso20:15:152
1877999275,7cyclictest21943-21ssh21:19:152
1877999272,10cyclictest0-21swapper/220:00:142
1877999271,18cyclictest21546-21ssh22:30:122
1875999273,19cyclictest12-21ksoftirqd/019:45:150
1877999267,14cyclictest0-21swapper/219:25:112
1877999266,13cyclictest0-21swapper/221:24:492
1877999262,15cyclictest27346irq/40-dwc2_hso21:45:142
18779992616,7cyclictest20426-21apt-get23:05:132
18779992610,8cyclictest13311-21cat00:10:312
18771992614,6cyclictest131rcu_preempt00:25:141
1877199261,20cyclictest27222-21sh21:40:131
1877199261,17cyclictest23319-21ssh00:00:131
1875999261,8cyclictest0-21swapper/021:05:130
1875999261,19cyclictest17389-21ssh22:10:140
1877999256,4cyclictest0-21swapper/223:46:092
1877999256,12cyclictest2024-21kthreadcore00:05:252
1877999254,17cyclictest23995-21sort19:45:302
1877999252,19cyclictest31502-21rm22:00:142
18779992519,2cyclictest3985-21apt-get00:25:142
1877999251,16cyclictest27346irq/40-dwc2_hso23:10:132
1877199251,4cyclictest131rcu_preempt21:30:001
18771992513,7cyclictest0-21swapper/123:50:131
1877199251,19cyclictest6218-21rm23:15:141
1875999252,15cyclictest2244-21ssh22:55:130
1875999251,20cyclictest31500-21ssh22:00:140
1878799242,13cyclictest35966irq/58-eth000:10:133
18787992412,9cyclictest0-21swapper/300:00:233
1877999248,4cyclictest0-21swapper/223:17:202
1877999248,12cyclictest18995-21sendmail_mailtr19:40:272
1877999246,12cyclictest0-21swapper/223:35:122
1877999240,11cyclictest0-21swapper/222:47:042
1877199245,4cyclictest131rcu_preempt22:15:141
1875999249,7cyclictest131rcu_preempt19:15:130
1878799234,16cyclictest35966irq/58-eth021:05:133
1878799233,15cyclictest0-21swapper/322:00:133
1877999239,11cyclictest0-21swapper/219:20:102
1877999235,13cyclictest0-21swapper/220:20:122
1877999233,16cyclictest31187-21mailstats00:20:292
1877999232,13cyclictest27346irq/40-dwc2_hso20:10:132
1877999231,8cyclictest27-21ksoftirqd/223:22:342
18779992311,9cyclictest30861-21ssh23:27:312
18779992311,9cyclictest30861-21ssh23:27:312
1877999231,14cyclictest0-21swapper/223:34:312
18779992310,10cyclictest0-21swapper/219:10:112
1877199236,14cyclictest20458-21ssh23:05:141
1877199231,16cyclictest14106-21ssh23:55:141
1875999231,18cyclictest9136-21ls00:10:140
1875999231,18cyclictest31258-21ls20:30:170
1875999231,15cyclictest28573-21find19:20:100
1878799229,8cyclictest35966irq/58-eth022:10:143
1878799221,12cyclictest0-21swapper/300:35:133
18787992210,6cyclictest0-21swapper/322:19:453
1877999222,14cyclictest0-21swapper/223:54:372
1877999221,13cyclictest8882-21ssh21:30:132
1877199222,17cyclictest11339-21apt-get23:00:131
1877199221,19cyclictest22029-21ls21:35:311
1877199221,17cyclictest23596-21ls19:45:251
1877199221,17cyclictest21391-21ls20:50:271
1875999228,11cyclictest5509-21cron19:30:010
1875999223,13cyclictest4955-21ssh23:50:130
18759992215,5cyclictest20438-21ssh23:05:130
1875999221,15cyclictest17935-21cut20:50:160
1875999220,18cyclictest26571-21ssh22:15:150
1878799219,8cyclictest35966irq/58-eth021:30:143
1878799211,14cyclictest35966irq/58-eth019:15:133
18787992110,7cyclictest0-21swapper/322:50:333
1877999217,11cyclictest17529-21sshd20:47:272
1877999212,14cyclictest27346irq/40-dwc2_hso20:55:132
1877999211,16cyclictest7980-21sendmail20:40:012
1877999211,13cyclictest0-21swapper/219:35:122
18779992110,9cyclictest28342-21munin-plugin-st19:20:012
1877199215,4cyclictest131rcu_preempt00:05:271
1877199212,14cyclictest23812-21ssh00:16:301
1877199211,17cyclictest16475-21timerwakeupswit22:25:301
1877199211,10cyclictest131rcu_preempt22:20:261
1875999217,11cyclictest31263-21grep21:00:290
1875999215,11cyclictest0-21swapper/023:30:130
1875999213,12cyclictest0-21swapper/020:45:140
18759992112,7cyclictest601-21systemd-resolve21:59:390
1875999211,16cyclictest8158-21ls20:40:010
1875999211,10cyclictest16234-21tr21:15:230
1875999210,17cyclictest21606-21cut20:20:160
1875999210,15cyclictest22975-21ssh00:00:020
1878799208,9cyclictest0-21swapper/300:25:163
18787992011,7cyclictest0-21swapper/321:48:283
18787992011,6cyclictest35966irq/58-eth021:50:133
1877999208,8cyclictest29929-21df_inode19:55:142
1877199209,8cyclictest28673-21interrupts23:45:191
1877199204,3cyclictest131rcu_preempt00:12:531
1877199202,15cyclictest17492-21df_inode22:10:141
1877199200,15cyclictest6925-21latency_hist22:40:021
1877199200,15cyclictest10286-21uniq23:35:161
1875999207,9cyclictest12-21ksoftirqd/019:35:130
1875999207,10cyclictest6101-21grep20:00:290
1875999205,4cyclictest131rcu_preempt22:50:010
1875999201,13cyclictest9420-21timerandwakeup19:30:270
1875999201,13cyclictest9420-21timerandwakeup19:30:270
1875999201,13cyclictest18726-21missed_timers23:20:220
1875999201,10cyclictest25790-21fschecks_count22:50:170
1878799199,6cyclictest35966irq/58-eth021:35:123
1878799193,9cyclictest0-21swapper/319:25:103
1878799193,13cyclictest35966irq/58-eth023:05:133
18787991914,3cyclictest33-21ksoftirqd/322:40:233
1878799191,12cyclictest0-21swapper/323:25:173
1878799191,12cyclictest0-21swapper/323:25:173
1878799191,12cyclictest0-21swapper/319:40:163
18787991910,6cyclictest35966irq/58-eth021:55:123
18787991910,6cyclictest0-21swapper/323:23:473
18787991910,6cyclictest0-21swapper/322:35:203
1877199198,7cyclictest6790-21cron20:05:011
1877199194,3cyclictest0-21swapper/100:37:511
1877199192,15cyclictest1712-21kthreadcore19:25:181
1877199191,8cyclictest988-21ssh23:30:131
18771991913,4cyclictest0-21swapper/120:55:271
18771991913,4cyclictest0-21swapper/119:40:271
18771991912,4cyclictest21-21ksoftirqd/122:35:011
1877199191,15cyclictest26233-21apt-get20:25:131
1877199191,15cyclictest22325-21rm21:55:131
1877199191,14cyclictest13146-21ssh21:50:131
1875999198,3cyclictest131rcu_preempt23:00:320
1875999192,14cyclictest29722-21latency_hist19:55:020
1875999191,8cyclictest18852-21kthreadcore20:15:220
1875999191,3cyclictest0-21swapper/023:11:060
1875999191,16cyclictest1091-21xargs19:55:270
1875999191,15cyclictest25925-21ntp_states00:35:270
18759991910,6cyclictest8940-21ssh22:05:200
18759991910,6cyclictest29988-21wc20:25:290
1875999190,16cyclictest19198-21find23:40:120
1875999190,13cyclictest3765-21ls21:25:290
1875999190,11cyclictest14821-21tr00:30:200
1878799189,6cyclictest0-21swapper/322:31:053
1878799189,6cyclictest0-21swapper/321:40:393
1878799182,11cyclictest0-21swapper/320:55:133
1878799182,10cyclictest0-21swapper/320:30:133
1878799182,10cyclictest0-21swapper/320:00:143
1878799181,13cyclictest0-21swapper/319:10:233
1878799181,12cyclictest0-21swapper/321:20:303
1878799181,12cyclictest0-21swapper/319:35:213
1878799181,11cyclictest0-21swapper/323:00:243
18787991810,6cyclictest0-21swapper/323:41:213
18787991810,6cyclictest0-21swapper/323:35:293
18787991810,3cyclictest0-21swapper/322:56:073
1877999181,14cyclictest18934-21latency_hist19:10:022
1877199189,6cyclictest1-21systemd19:37:131
1877199184,4cyclictest21-21ksoftirqd/122:55:121
1875999184,9cyclictest0-21swapper/020:55:130
1875999182,13cyclictest550-21cron23:30:010
1875999182,13cyclictest550-21cron23:30:000
1875999181,9cyclictest16397-21awk20:15:010
1875999181,6cyclictest26821-21dpkg21:40:010
1875999181,6cyclictest12-21ksoftirqd/000:20:190
1875999181,13cyclictest7316-21/usr/sbin/munin22:40:180
1875999181,12cyclictest11584-21cut21:30:380
1878799179,6cyclictest0-21swapper/323:56:473
1878799179,6cyclictest0-21swapper/323:14:033
1878799179,6cyclictest0-21swapper/322:25:013
1878799179,6cyclictest0-21swapper/321:17:153
1878799179,6cyclictest0-21swapper/300:31:013
1878799179,6cyclictest0-21swapper/300:10:023
1878799178,6cyclictest35966irq/58-eth023:15:133
1878799173,6cyclictest35966irq/58-eth000:20:143
18787991711,4cyclictest0-21swapper/321:29:193
1878799171,13cyclictest0-21swapper/322:05:293
1878799171,12cyclictest0-21swapper/320:55:023
1878799171,10cyclictest35966irq/58-eth019:55:123
1877199178,7cyclictest21-21ksoftirqd/121:45:121
1877199172,10cyclictest0-21swapper/119:50:131
18771991712,3cyclictest9265-21sendmail_mailst19:30:261
18771991712,3cyclictest9265-21sendmail_mailst19:30:261
18771991712,3cyclictest0-21swapper/120:20:161
1877199171,13cyclictest2403-21grep20:30:291
18771991710,5cyclictest0-21swapper/121:00:161
1877199170,14cyclictest21150-21cut19:10:211
1875999179,6cyclictest11596-21cron20:10:010
1875999179,6cyclictest11596-21cron20:10:010
1875999175,8cyclictest10281-21ssh23:35:160
1875999171,13cyclictest22667-21ls19:10:280
1875999171,12cyclictest8540-21sed20:40:170
1877199166,7cyclictest23992-21idleruntime19:15:171
1877199162,11cyclictest16479-21proc_pri20:45:271
1877199162,10cyclictest0-21swapper/120:05:141
1877199162,10cyclictest0-21swapper/120:05:131
1877199161,12cyclictest16612-21apt-get20:15:141
1877199160,10cyclictest11985-21grep20:10:151
1878799152,9cyclictest0-21swapper/319:45:163
1878799151,11cyclictest0-21swapper/321:00:183
1878799151,11cyclictest0-21swapper/320:40:183
1877199151,10cyclictest18842-21cat19:10:021
1875999151,10cyclictest18844-21latency_hist19:10:020
1878799149,3cyclictest0-21swapper/320:15:183
1878799143,8cyclictest0-21swapper/320:20:163
1878799141,7cyclictest0-21swapper/319:30:273
1878799141,7cyclictest0-21swapper/319:30:273
1878799141,10cyclictest0-21swapper/320:05:223
1878799141,10cyclictest0-21swapper/320:05:223
1878799132,6cyclictest0-21swapper/320:45:143
1878799131,8cyclictest0-21swapper/319:20:103
1878799131,6cyclictest0-21swapper/320:25:193
1878799127,3cyclictest0-21swapper/320:35:243
1878799127,3cyclictest0-21swapper/319:54:493
1878799111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional