You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 12:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Apr 02, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
753199532,38cyclictest0-21swapper/300:20:113
7531995129,14cyclictest28366irq/58-eth023:10:133
7531995129,14cyclictest28366irq/58-eth023:10:123
752399501,31cyclictest924-21snmpd23:45:112
751199502,2cyclictest131rcu_preempt21:45:131
753199465,28cyclictest28366irq/58-eth021:10:133
750399464,5cyclictest131rcu_preempt23:20:130
752399451,30cyclictest29654-21ssh21:30:142
753199444,28cyclictest28366irq/58-eth019:10:133
752399443,31cyclictest15026-21ssh23:10:132
752399443,31cyclictest15026-21ssh23:10:132
752399422,23cyclictest28216-21/usr/sbin/munin21:45:272
7523994122,5cyclictest0-21swapper/223:00:122
751199416,6cyclictest131rcu_preempt22:15:141
7503994113,20cyclictest131rcu_preempt23:10:120
7503994113,20cyclictest131rcu_preempt23:10:110
752399402,30cyclictest4381-21ssh22:10:132
752399402,16cyclictest31016-21ssh23:55:142
750399400,28cyclictest8046-21rm22:30:120
753199393,19cyclictest28366irq/58-eth023:45:113
753199391,25cyclictest28366irq/58-eth020:55:133
752399392,29cyclictest10618-21cat21:35:292
752399392,18cyclictest13990-21cat21:55:282
753199381,32cyclictest0-21swapper/321:05:123
753199381,28cyclictest0-21swapper/323:00:123
751199362,26cyclictest499-21ssh23:20:131
753199354,24cyclictest0-21swapper/323:20:133
7531993515,15cyclictest28366irq/58-eth000:30:133
753199348,21cyclictest28366irq/58-eth023:55:123
753199342,27cyclictest0-21swapper/323:40:103
753199341,2cyclictest0-21swapper/323:05:123
753199341,2cyclictest0-21swapper/321:30:153
753199320,2cyclictest0-21swapper/322:05:123
753199320,2cyclictest0-21swapper/321:15:163
753199320,1cyclictest0-21swapper/321:50:123
752399324,18cyclictest22346irq/40-dwc2_hso20:00:142
7523993215,13cyclictest8033-21apt-get22:30:112
7503993214,3cyclictest12-21ksoftirqd/021:05:120
753199311,3cyclictest0-21swapper/300:15:123
752399318,15cyclictest11548-21ssh22:50:122
752399315,18cyclictest26047-21ssh22:40:132
750399314,18cyclictest13373-21ssh22:15:150
753199303,22cyclictest0-21swapper/300:10:103
753199301,2cyclictest0-21swapper/322:30:123
753199301,2cyclictest0-21swapper/321:40:123
753199301,2cyclictest0-21swapper/321:40:123
752399308,11cyclictest475-21apt-get23:20:142
752399306,10cyclictest27-21ksoftirqd/222:35:122
752399303,17cyclictest20033-21ssh00:25:102
750399302,22cyclictest27786-21/usr/sbin/munin22:05:310
753199295,21cyclictest0-21swapper/323:25:123
753199292,22cyclictest0-21swapper/322:40:133
753199291,9cyclictest0-21swapper/321:35:123
753199291,2cyclictest0-21swapper/321:45:143
753199291,24cyclictest0-21swapper/320:35:153
753199291,24cyclictest0-21swapper/319:55:133
753199290,2cyclictest0-21swapper/323:50:133
753199290,2cyclictest0-21swapper/323:50:123
752399296,11cyclictest27-21ksoftirqd/223:34:392
752399294,4cyclictest27-21ksoftirqd/223:35:172
752399292,22cyclictest4943-21unixbench_singl21:50:292
752399292,20cyclictest0-21swapper/219:55:142
751199292,21cyclictest31879-21cat22:05:301
750399292,21cyclictest24297-21ssh21:45:140
753199281,6cyclictest0-21swapper/321:25:123
753199281,4cyclictest0-21swapper/322:10:133
753199281,3cyclictest0-21swapper/323:15:133
752399288,13cyclictest0-21swapper/223:25:112
752399285,13cyclictest22346irq/40-dwc2_hso20:25:142
752399283,6cyclictest27-21ksoftirqd/223:19:402
752399282,19cyclictest0-21swapper/220:20:142
7523992819,8cyclictest261rcuc/200:01:582
7523992818,5cyclictest0-21swapper/200:15:142
752399281,19cyclictest0-21swapper/221:25:132
751199282,20cyclictest15486-21ssh21:56:291
753199278,16cyclictest28366irq/58-eth022:55:123
753199271,5cyclictest0-21swapper/322:15:153
753199271,2cyclictest0-21swapper/322:25:123
753199271,2cyclictest0-21swapper/319:50:143
752399277,3cyclictest131rcu_preempt00:22:552
7511992715,9cyclictest1-21systemd19:40:041
7511992714,5cyclictest25784-21diskmemload22:35:281
750399272,19cyclictest8404-21/usr/sbin/munin22:10:280
750399272,15cyclictest2684-21ssh21:15:140
753199262,2cyclictest0-21swapper/322:00:123
753199261,3cyclictest0-21swapper/320:25:133
753199261,3cyclictest0-21swapper/300:07:053
753199261,2cyclictest0-21swapper/320:10:103
753199261,22cyclictest0-21swapper/322:35:213
753199260,2cyclictest0-21swapper/319:35:093
752399265,13cyclictest22346irq/40-dwc2_hso20:40:122
752399262,9cyclictest0-21swapper/220:05:142
752399262,21cyclictest0-21swapper/221:05:142
7523992616,8cyclictest261rcuc/200:37:022
7523992616,4cyclictest261rcuc/200:12:262
7523992616,3cyclictest531-21usb-storage20:10:112
752399261,17cyclictest6126-21ls23:05:152
752399260,20cyclictest9172-21cut19:45:162
751199261,19cyclictest4563-21ssh21:33:291
7503992618,3cyclictest12-21ksoftirqd/020:05:130
7503992613,10cyclictest18440-21cron22:00:010
753199252,3cyclictest0-21swapper/300:40:023
753199252,18cyclictest0-21swapper/320:40:113
753199251,3cyclictest0-21swapper/322:45:273
7531992513,7cyclictest0-21swapper/322:20:153
753199251,2cyclictest0-21swapper/321:55:133
753199251,2cyclictest0-21swapper/320:05:143
753199251,2cyclictest0-21swapper/300:25:123
753199251,2cyclictest0-21swapper/300:00:123
753199251,21cyclictest0-21swapper/323:40:023
753199251,21cyclictest0-21swapper/321:20:163
752399259,4cyclictest27-21ksoftirqd/222:08:372
752399257,12cyclictest0-21swapper/220:15:282
752399255,8cyclictest27-21ksoftirqd/223:51:142
752399255,8cyclictest27-21ksoftirqd/223:51:132
752399253,15cyclictest22346irq/40-dwc2_hso20:45:122
752399253,14cyclictest22346irq/40-dwc2_hso21:00:142
752399252,19cyclictest11641-21apt-get21:20:112
7523992514,9cyclictest261rcuc/222:48:362
7523992512,9cyclictest25784-21diskmemload22:00:142
7523992512,8cyclictest15250-21apt-get21:40:132
7523992512,8cyclictest15250-21apt-get21:40:132
752399251,14cyclictest29043-21sh00:30:152
750399252,18cyclictest29474-21apt-get23:00:120
753199249,11cyclictest0-21swapper/323:30:203
753199241,3cyclictest0-21swapper/319:45:283
753199241,2cyclictest0-21swapper/322:50:133
753199241,2cyclictest0-21swapper/320:45:123
753199240,2cyclictest0-21swapper/320:30:123
752399244,12cyclictest0-21swapper/200:05:122
752399243,14cyclictest22346irq/40-dwc2_hso20:30:112
7523992415,7cyclictest261rcuc/221:10:302
7523992411,10cyclictest0-21swapper/219:25:132
751199247,8cyclictest131rcu_preempt22:30:001
7511992417,4cyclictest0-21swapper/100:00:111
7511992412,7cyclictest21-21ksoftirqd/119:15:121
7511992410,10cyclictest0-21swapper/120:35:131
753199231,2cyclictest0-21swapper/319:25:103
753199231,2cyclictest0-21swapper/319:15:123
753199231,19cyclictest0-21swapper/320:50:273
753199231,18cyclictest0-21swapper/320:00:283
752399235,3cyclictest131rcu_preempt21:00:022
7523992317,4cyclictest261rcuc/222:26:082
7523992314,3cyclictest261rcuc/222:21:512
752399231,3cyclictest20501-21apt-get22:55:132
752399231,18cyclictest8315-21sed23:40:292
751199233,17cyclictest0-21swapper/121:28:271
750399232,6cyclictest11094-21apt-get00:20:110
750399231,18cyclictest22377-21ssh22:20:140
750399231,17cyclictest804-21ssh21:50:130
750399231,17cyclictest22078-21ssh23:50:140
750399231,17cyclictest22078-21ssh23:50:130
753199221,2cyclictest0-21swapper/320:20:123
753199221,2cyclictest0-21swapper/319:40:123
753199221,2cyclictest0-21swapper/319:30:283
753199221,2cyclictest0-21swapper/319:20:243
752399227,8cyclictest27-21ksoftirqd/222:15:442
752399226,7cyclictest27-21ksoftirqd/221:15:052
752399223,13cyclictest0-21swapper/219:35:102
752399221,5cyclictest0-21swapper/219:30:112
7523992212,4cyclictest24682-21apt-get20:35:152
751199229,4cyclictest4676-21sh22:45:201
751199226,6cyclictest21-21ksoftirqd/119:55:141
7511992216,3cyclictest21-21ksoftirqd/122:30:131
7511992212,4cyclictest91-21pr/ttyS023:40:021
7503992215,3cyclictest12-21ksoftirqd/021:10:130
7503992213,7cyclictest12-21ksoftirqd/019:40:130
7503992211,9cyclictest17742-21kthreadcore22:35:200
750399221,17cyclictest24689-21sshd20:35:140
750399221,15cyclictest31208-21grep23:55:150
753199211,17cyclictest0-21swapper/320:15:163
753199210,18cyclictest0-21swapper/321:00:133
752399216,11cyclictest0-21swapper/219:50:142
7523992114,3cyclictest12464-21apt-get19:15:132
7523992110,5cyclictest7260-21sed20:50:172
751199218,3cyclictest25784-21diskmemload23:28:361
751199211,8cyclictest15007-21sed21:40:011
751199211,7cyclictest23966-21sed21:45:021
751199211,7cyclictest23966-21sed21:45:011
751199211,14cyclictest11120-21ssh00:20:121
7511992110,8cyclictest13130-21sshd00:38:481
752399201,9cyclictest18182-21kthreadcore19:20:172
752399201,15cyclictest7834-21fschecks_time19:10:162
751199201,3cyclictest0-21swapper/122:00:241
751199201,16cyclictest26193-21ssh21:10:151
751199201,13cyclictest4942-21grep21:50:291
751199201,10cyclictest24460-21idleruntime-cro20:35:011
750399203,2cyclictest131rcu_preempt20:25:000
750399202,4cyclictest53146usb-storage20:40:040
750399202,13cyclictest20517-21ssh22:55:140
7503992013,4cyclictest14856-21latency_hist21:40:010
7503992012,5cyclictest0-21swapper/021:35:000
7503992011,7cyclictest23621-21fw_packets20:00:170
7503992011,7cyclictest14627-21cron23:10:000
750399201,16cyclictest3398-21cat22:25:310
7503992010,7cyclictest18622-21apt-get19:55:130
752399192,9cyclictest4225-21/usr/sbin/munin19:40:112
752399191,10cyclictest7563-21latency_hist19:10:022
751199199,7cyclictest7200-21cron00:00:011
751199198,9cyclictest6550-21cron20:50:001
751199192,15cyclictest18888-21ssh21:23:281
751199192,12cyclictest22376-21ssh22:20:151
7511991911,6cyclictest7059-21if_eth020:50:161
751199191,14cyclictest22185-21apt-get19:25:121
751199191,13cyclictest29491-21ssh23:00:121
751199191,12cyclictest2710-21sh21:15:151
751199191,12cyclictest16299-21cat22:50:311
751199190,13cyclictest30117-21timerwakeupswit22:40:301
750399195,11cyclictest2135-21ssh00:15:150
750399194,12cyclictest0-21swapper/000:35:130
750399193,14cyclictest27147-21cut19:30:150
750399191,8cyclictest6432-21mailstats22:45:280
7503991912,4cyclictest25784-21diskmemload23:25:130
7503991911,5cyclictest0-21swapper/020:10:270
750399191,14cyclictest14964-21latency22:50:220
7503991910,6cyclictest9040-21apt-get19:45:120
753199181,14cyclictest0-21swapper/319:10:013
751199188,7cyclictest18379-21cron19:55:011
751199188,3cyclictest25784-21diskmemload00:05:121
751199187,3cyclictest25784-21diskmemload22:57:041
751199186,4cyclictest25784-21diskmemload00:26:561
751199185,3cyclictest25784-21diskmemload00:18:071
751199185,10cyclictest27983-21cron20:05:001
751199181,9cyclictest25784-21diskmemload23:30:591
751199181,9cyclictest0-21swapper/123:40:171
751199181,5cyclictest1680-21apt-config00:15:011
751199181,13cyclictest18675-21ssh23:10:261
751199181,13cyclictest18675-21ssh23:10:251
7511991810,6cyclictest5264-21cron20:15:011
7511991810,6cyclictest32107-21interrupts19:35:151
7511991810,5cyclictest23989-21cpu23:15:151
750399189,6cyclictest30730-21cat22:40:310
750399188,7cyclictest7146-21irqcore20:50:160
750399188,7cyclictest15293-21ntp_kernel_pll_21:20:250
750399185,9cyclictest616-21dbus-daemon20:30:120
750399185,9cyclictest28206-21kworker/0:221:40:130
750399185,9cyclictest28206-21kworker/0:221:40:120
750399184,11cyclictest31443-21kthreadcore00:30:230
750399182,13cyclictest26989-21kthreadcore00:10:180
750399182,13cyclictest23988-21ssh23:15:150
750399182,12cyclictest13120-21ssh23:45:120
750399181,9cyclictest0-21swapper/023:32:150
750399181,9cyclictest0-21swapper/019:15:130
750399181,9cyclictest0-21swapper/000:27:270
750399181,6cyclictest12-21ksoftirqd/023:40:180
7503991812,4cyclictest0-21swapper/020:25:240
750399181,14cyclictest22335-21ssh22:00:210
7503991811,4cyclictest0-21swapper/021:25:140
751199179,6cyclictest9933-21sshd23:05:291
751199179,6cyclictest8032-21sed19:10:191
751199178,6cyclictest22777-21ssh23:50:191
751199178,6cyclictest22777-21ssh23:50:191
751199178,6cyclictest16171-21basename21:00:011
751199177,3cyclictest25784-21diskmemload00:30:211
751199171,8cyclictest19491-21kthreadcore21:00:221
751199171,7cyclictest13581-21cron19:50:011
751199171,14cyclictest17765-21cut19:20:171
751199171,13cyclictest4482-21df22:10:151
751199171,12cyclictest13237-21diskstats23:45:141
750399179,6cyclictest7878-21wc19:10:160
750399178,6cyclictest28124-21sed23:35:180
750399171,13cyclictest17281-21ntp_states19:50:270
750399171,12cyclictest8366-21cut00:00:190
750399171,12cyclictest16705-21fschecks_count00:05:140
750399171,11cyclictest15261-21awk20:55:290
750399170,6cyclictest21081-21sort21:05:010
751199169,5cyclictest27264-21if_eth119:30:161
751199168,5cyclictest0-21swapper/120:05:321
751199167,7cyclictest21208-21apt-get21:05:121
751199164,7cyclictest131rcu_preempt20:20:311
751199161,11cyclictest7971-21kthreadcore20:15:211
750399169,5cyclictest3687-21cut20:45:190
750399169,5cyclictest23676-21cut19:25:180
750399161,13cyclictest7507-21kthreadcore20:15:210
7503991611,3cyclictest31909-21uniq19:35:130
750399161,11cyclictest17447-21diskstats19:20:140
751199152,3cyclictest131rcu_preempt20:44:281
751199151,11cyclictest18500-21ntp_192.168.11420:25:231
751199131,8cyclictest7562-21grep19:10:011
750399131,9cyclictest7517-21latency_hist19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional