You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 06:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Wed Dec 10, 2025 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2482899702,50cyclictest4647-21ssh21:25:161
24813995626,4cyclictest12-21ksoftirqd/022:05:140
2483999481,3cyclictest0-21swapper/220:50:162
2482899472,35cyclictest16520-21ssh22:10:131
24839994535,4cyclictest26346irq/40-dwc2_hso23:30:132
2483999452,18cyclictest2560266ssh22:33:412
2483999441,4cyclictest0-21swapper/219:35:132
2483999381,8cyclictest20460-21latency_hist00:05:022
2482899381,28cyclictest24868-21ssh23:30:141
24852993718,11cyclictest29566irq/58-eth022:40:143
2483999375,5cyclictest12304-21irqrtprio22:45:192
2483999372,19cyclictest17841-21rm22:29:422
2483999371,9cyclictest18924-21kthreadcore23:25:232
2483999371,8cyclictest5842-21kthreadcore19:55:202
2483999371,8cyclictest305-21kthreadcore19:15:232
2483999371,8cyclictest11270-21kthreadcore20:00:242
2483999371,7cyclictest6001-21kthreadcore20:30:202
2483999371,4cyclictest31164-21apt-get23:15:132
2483999371,17cyclictest4110-21ssh00:15:132
2483999370,10cyclictest13353-21cut21:10:232
2485299365,21cyclictest29566irq/58-eth023:45:143
2483999361,9cyclictest17947-21sendmail_mailqu23:05:282
2483999361,9cyclictest1549-21fschecks_time23:35:182
2483999361,9cyclictest13256-21fschecks_count21:30:162
2483999361,9cyclictest13256-21fschecks_count21:30:162
2483999361,8cyclictest23091-21timerandwakeup20:45:302
2483999361,7cyclictest0-21swapper/221:05:132
2483999361,31cyclictest921-21kthreadcore20:25:212
2485299353,21cyclictest0-21swapper/300:20:153
2485299353,21cyclictest0-21swapper/300:20:153
2483999356,5cyclictest1725-21cut19:50:222
2483999352,8cyclictest7977-21kthreadcore21:25:252
2483999351,8cyclictest7092-21fschecks_count19:25:162
2483999351,8cyclictest7092-21fschecks_count19:25:162
2483999351,7cyclictest29846-21cat22:15:322
2483999351,7cyclictest15038-21cut20:40:202
2483999351,6cyclictest32242-21sort20:55:242
2483999351,31cyclictest707-21rs:main2
2483999351,30cyclictest0-21swapper/222:50:182
2483999350,8cyclictest23047-21sort21:55:012
2483999350,7cyclictest28759-21fschecks_time00:10:182
2485299342,23cyclictest29566irq/58-eth023:40:133
2483999347,4cyclictest10803-21diskmemload22:44:142
2483999345,5cyclictest5454-21rm23:00:032
2483999343,4cyclictest27907-21fschecks_time20:20:162
2483999343,4cyclictest27907-21fschecks_time20:20:162
2483999341,9cyclictest31734-21cut21:40:192
2483999341,9cyclictest11312-21timerwakeupswit23:20:312
2483999341,9cyclictest10564-21kthreadcore20:35:182
2483999341,8cyclictest7936-21/usr/sbin/munin22:05:192
2483999341,8cyclictest15578-21timerwakeupswit19:30:282
2483999341,8cyclictest1259-21cpu22:20:152
2483999341,6cyclictest27732-21ssh21:55:382
2483999341,29cyclictest5854-21timerandwakeup19:20:282
2483999340,6cyclictest28485-21find00:30:122
2481399341,24cyclictest0-21swapper/022:25:140
24852993315,14cyclictest0-21swapper/322:05:143
2483999331,9cyclictest0-21swapper/221:20:172
2483999331,8cyclictest5232-21grep21:00:302
2483999331,6cyclictest30106-21ssh23:14:062
2483999331,6cyclictest25239-21timerwakeupswit19:40:272
2483999331,5cyclictest4721-21ssh00:35:192
2483999330,8cyclictest6947-21irqrtprio21:45:192
2483999330,8cyclictest6947-21irqrtprio21:45:182
2483999330,7cyclictest32627-21ssh23:50:442
2483999330,6cyclictest16391-21sh00:20:322
2483999330,6cyclictest16391-21sh00:20:312
2483999322,8cyclictest22738-21kthreadcore21:15:232
2483999321,8cyclictest28095-21kthreadcore19:10:222
2483999321,8cyclictest22603-21processes23:45:282
2483999321,8cyclictest19296-21cut22:10:212
2483999321,7cyclictest26576-21ssh21:35:332
2483999321,6cyclictest3185166ssh22:35:372
2483999320,4cyclictest10803-21diskmemload00:05:432
2485299312,22cyclictest29566irq/58-eth000:10:133
2483999312,5cyclictest10077-21ssh23:40:142
2483999311,8cyclictest24736-21kthreadcore20:15:262
2483999311,8cyclictest19784-21kthreadcore20:10:232
2483999311,8cyclictest15635-21kthreadcore20:05:222
2483999311,8cyclictest15635-21kthreadcore20:05:222
2483999311,8cyclictest1033-21kthreadcore22:00:212
2483999311,8cyclictest0-21swapper/200:25:122
2483999311,7cyclictest0-21swapper/223:04:582
2483999310,5cyclictest30542-21cat19:45:302
2482899312,22cyclictest20704-21ssh00:25:161
2481399312,20cyclictest2890-21ssh22:40:140
24813993113,15cyclictest12-21ksoftirqd/023:55:160
2485299303,19cyclictest29566irq/58-eth021:15:153
2485299285,17cyclictest0-21swapper/300:35:143
24852992814,8cyclictest29566irq/58-eth000:30:143
24852992813,11cyclictest29566irq/58-eth023:55:173
2481399282,18cyclictest12380-21ssh00:00:150
24813992821,4cyclictest626-21tcf-agent19:25:140
24813992821,4cyclictest626-21tcf-agent19:25:130
2485299274,15cyclictest0-21swapper/322:20:143
2485299273,16cyclictest0-21swapper/322:27:133
2481399271,23cyclictest18506-21rm22:30:150
2481399271,20cyclictest22728-21ssh23:10:160
24852992616,6cyclictest0-21swapper/322:35:283
2482899261,19cyclictest17840-21sh22:29:411
24813992616,7cyclictest18407-21ssh21:50:250
2485299253,16cyclictest0-21swapper/322:50:133
2481399253,17cyclictest1356-21ssh23:35:150
2481399252,19cyclictest4031-21ssh00:35:150
2481399251,21cyclictest20525-21ssh22:50:130
2483999242,3cyclictest26346irq/40-dwc2_hso19:10:012
2482899241,19cyclictest25075-21ssh22:15:131
2482899241,17cyclictest16048-21ssh23:25:151
2481399244,5cyclictest12-21ksoftirqd/021:15:320
2481399242,3cyclictest131rcu_preempt22:55:140
24813992414,6cyclictest7323-21ssh23:20:150
2485299233,14cyclictest0-21swapper/322:30:413
24852992314,5cyclictest0-21swapper/323:30:013
24852992310,5cyclictest0-21swapper/321:25:353
2482899233,7cyclictest131rcu_preempt22:50:131
2481399233,16cyclictest28526-21ssh00:30:140
2481399231,17cyclictest12530-21ssh00:20:150
2481399231,17cyclictest12530-21ssh00:20:150
2485299229,10cyclictest0-21swapper/323:35:173
2485299223,13cyclictest0-21swapper/321:45:143
2485299223,13cyclictest0-21swapper/321:45:143
2485299223,13cyclictest0-21swapper/300:00:143
2485299222,10cyclictest29566irq/58-eth021:10:143
24852992215,4cyclictest0-21swapper/322:15:133
24852992213,6cyclictest29566irq/58-eth022:10:133
24852992212,7cyclictest29566irq/58-eth021:35:153
24852992210,7cyclictest29566irq/58-eth022:55:153
2482899229,9cyclictest1-21systemd19:11:361
2482899225,12cyclictest0-21swapper/123:45:131
24828992216,3cyclictest0-21swapper/123:55:181
2482899221,17cyclictest21435-21apt-get19:40:111
2482899221,16cyclictest22866-21apt-get20:15:161
2481399227,3cyclictest131rcu_preempt22:45:130
24813992214,6cyclictest10066-21ssh23:40:140
2481399221,18cyclictest28567-21ssh21:20:130
2481399221,18cyclictest25604-21ls21:35:300
2485299214,14cyclictest0-21swapper/300:15:133
2485299212,12cyclictest29566irq/58-eth000:05:193
24828992114,5cyclictest0-21swapper/121:53:421
2482899211,11cyclictest12145-21sendmail00:20:011
2481399212,16cyclictest26963-21ssh22:35:150
24813992113,5cyclictest24973-21usage_Anybus00:05:350
24813992112,6cyclictest3288-21sshd19:52:420
24813992111,7cyclictest27707-21ssh23:50:160
24813992111,6cyclictest131rcu_preempt21:10:150
24813992110,7cyclictest12-21ksoftirqd/021:00:140
2485299204,4cyclictest0-21swapper/300:25:153
2485299202,12cyclictest0-21swapper/323:50:173
2482899209,8cyclictest3565-21grep22:00:301
2482899202,15cyclictest1257-21ssh22:20:151
2482899201,8cyclictest31398-21munin-run22:00:011
24828992014,4cyclictest0-21swapper/100:20:351
24828992014,4cyclictest0-21swapper/100:20:341
2482899201,10cyclictest3832-21grep00:35:021
2482899200,15cyclictest5430-21ls23:00:011
2481399203,13cyclictest31888-21ssh22:00:150
2481399202,2cyclictest91-21pr/ttyS000:15:010
2481399201,16cyclictest9901-21ssh21:45:240
2481399201,16cyclictest9901-21ssh21:45:240
2481399201,16cyclictest4636-21apt-get21:25:150
2481399200,13cyclictest8121-21latency_hist20:00:010
2485299199,7cyclictest29566irq/58-eth021:55:153
2485299194,12cyclictest0-21swapper/323:00:163
2485299193,9cyclictest0-21swapper/321:20:143
2485299193,10cyclictest0-21swapper/321:30:143
2485299193,10cyclictest0-21swapper/321:30:143
2485299193,10cyclictest0-21swapper/320:15:053
2485299192,13cyclictest0-21swapper/323:15:213
2485299192,10cyclictest0-21swapper/320:45:163
2485299191,14cyclictest0-21swapper/319:35:173
2485299191,14cyclictest0-21swapper/319:25:013
2485299191,14cyclictest0-21swapper/319:10:233
2485299191,13cyclictest0-21swapper/320:55:013
2485299191,12cyclictest0-21swapper/323:30:133
2485299191,10cyclictest0-21swapper/320:55:183
24852991910,6cyclictest29566irq/58-eth023:20:163
24852991910,6cyclictest29566irq/58-eth021:50:143
2482899196,4cyclictest21-21ksoftirqd/100:13:551
2482899192,15cyclictest14235-21usage_Anybus23:40:311
2482899192,10cyclictest20694-21ssh00:05:191
2482899191,9cyclictest0-21swapper/121:15:321
2482899191,8cyclictest27129-21kthreadcore19:45:201
24828991912,5cyclictest27567-21/usr/sbin/munin23:50:291
24828991911,5cyclictest7915-21threads00:35:301
2482899191,15cyclictest11225-21ssh21:10:151
2482899191,10cyclictest0-21swapper/121:48:511
2482899191,10cyclictest0-21swapper/121:48:501
24828991910,7cyclictest1352-21ssh23:35:151
24828991910,6cyclictest30265-21ssh21:40:121
2481399198,8cyclictest3919-21cut22:20:220
2481399197,9cyclictest14301-21cron20:40:010
2481399193,14cyclictest30099-21ps19:15:200
24813991912,5cyclictest25436-21ssh23:47:450
24813991912,5cyclictest18169-21ssh21:30:490
24813991912,5cyclictest18169-21ssh21:30:480
24813991912,5cyclictest10438-21ssh00:17:410
24813991912,4cyclictest12931-21munin-plugin-st20:05:010
2481399191,14cyclictest31183-21ssh23:15:150
24813991910,6cyclictest2641-21cat21:40:310
2481399190,16cyclictest15079-21ls19:30:230
2485299186,8cyclictest0-21swapper/322:45:183
2485299184,9cyclictest0-21swapper/323:05:143
2485299182,13cyclictest29566irq/58-eth019:55:123
2485299182,11cyclictest0-21swapper/319:50:143
2485299182,10cyclictest0-21swapper/320:35:133
2485299182,10cyclictest0-21swapper/319:45:043
2485299181,13cyclictest0-21swapper/321:40:203
2485299181,13cyclictest0-21swapper/320:40:173
2485299181,11cyclictest0-21swapper/321:00:153
24852991810,4cyclictest0-21swapper/323:10:293
2482899189,7cyclictest31175-21ntp_states20:20:271
2482899189,7cyclictest31175-21ntp_states20:20:261
2482899189,6cyclictest3717-21cut21:00:221
2482899186,6cyclictest21-21ksoftirqd/120:10:171
2482899182,12cyclictest6507-21ntp_192.168.11422:40:251
2482899182,12cyclictest28679-21diskstats21:20:151
2482899181,9cyclictest18468-21cut19:35:221
2482899181,8cyclictest708-21cron20:45:011
2482899181,8cyclictest1172-21awk21:00:021
24828991813,3cyclictest0-21swapper/119:25:161
24828991813,3cyclictest0-21swapper/119:25:161
24828991812,3cyclictest0-21swapper/120:30:111
24828991811,5cyclictest0-21swapper/121:05:301
2482899181,14cyclictest22727-21ssh23:10:161
24828991811,4cyclictest0-21swapper/119:50:281
2482899181,13cyclictest9738-21apt-get20:35:111
24828991810,6cyclictest20306-21cat22:30:241
24828991810,6cyclictest0-21swapper/122:45:271
2481399189,7cyclictest32282-21cron20:25:010
2481399189,7cyclictest32282-21cron20:25:000
2481399189,6cyclictest10891-21latency_hist21:10:010
2481399186,3cyclictest131rcu_preempt22:10:300
2481399185,4cyclictest131rcu_preempt19:45:180
2481399185,10cyclictest17375-21kthreadcore23:25:210
2481399184,12cyclictest20681-21diskmemload00:25:150
24813991811,5cyclictest23202-21ssh21:55:050
24813991811,5cyclictest14023-21ssh23:05:020
2481399181,13cyclictest25081-21grep22:15:130
2481399181,10cyclictest24847-21ssh23:30:140
24813991810,4cyclictest9763-21usage_Anybus23:00:290
2481399180,15cyclictest19451-21fschecks_count20:45:170
2481399180,15cyclictest14713-21fschecks_time20:40:170
2481399180,14cyclictest4922-21find20:30:110
2481399180,12cyclictest29538-21latency_hist19:15:010
2485299172,11cyclictest0-21swapper/322:00:213
2482899179,5cyclictest10440-21kthreadcore22:05:221
2482899178,7cyclictest11793-21apt-get19:30:131
2482899177,8cyclictest29851-21df19:15:161
2482899172,12cyclictest5630-21ssh23:00:141
2482899172,11cyclictest14297-21df_inode23:05:151
2482899171,9cyclictest4167-21irqstats19:55:191
2482899171,5cyclictest0-21swapper/121:30:111
2482899171,5cyclictest0-21swapper/121:30:111
24828991712,3cyclictest0-21swapper/120:25:261
24828991711,4cyclictest0-21swapper/120:05:311
24828991711,4cyclictest0-21swapper/120:05:301
2482899171,13cyclictest12371-21sh00:00:131
2482899171,12cyclictest30468-21memory22:35:261
2481399176,8cyclictest17111-21tr19:35:190
2481399174,10cyclictest26028-21munin-run19:45:020
2481399173,10cyclictest0-21swapper/020:50:160
2481399172,11cyclictest4839-21latency_hist20:30:010
2481399171,8cyclictest24004-21kthreadcore20:15:240
2481399171,13cyclictest17969-21apt-get20:10:170
2481399171,12cyclictest2367-21uniq19:20:160
24813991710,4cyclictest1298-21cat21:00:020
2485299162,10cyclictest0-21swapper/319:15:163
2485299161,7cyclictest0-21swapper/321:05:153
2485299161,12cyclictest0-21swapper/320:10:173
2485299161,10cyclictest0-21swapper/319:40:203
2482899169,4cyclictest0-21swapper/119:20:301
2482899168,6cyclictest12907-21sshd20:01:411
2482899167,3cyclictest131rcu_preempt21:35:191
2482899161,12cyclictest24168-21apt-get20:50:151
2482899161,11cyclictest7434-21diskstats23:20:171
2482899161,11cyclictest19753-21cut20:45:191
24828991610,3cyclictest0-21swapper/123:15:141
2481399161,11cyclictest13254-21diskstats20:05:180
2481399161,11cyclictest13254-21diskstats20:05:170
2485299151,6cyclictest0-21swapper/320:25:313
2485299151,11cyclictest0-21swapper/320:05:173
2485299151,11cyclictest0-21swapper/320:05:173
2485299142,10cyclictest0-21swapper/320:20:143
2485299142,10cyclictest0-21swapper/320:20:143
2485299141,7cyclictest0-21swapper/320:00:253
2481399141,10cyclictest24856-21latency_hist19:10:020
2485299132,8cyclictest0-21swapper/320:31:523
2485299132,8cyclictest0-21swapper/319:30:023
2485299132,8cyclictest0-21swapper/319:30:013
2485299131,7cyclictest0-21swapper/319:30:133
2482899131,9cyclictest24890-21latency_hist19:10:011
2485299101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional