You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-12 - 03:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Jan 12, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
907799576,40cyclictest10794-21ssh22:30:130
907799576,40cyclictest10794-21ssh22:30:130
9077995710,10cyclictest12-21ksoftirqd/023:30:120
9088995417,10cyclictest131rcu_preempt21:05:121
911299487,25cyclictest28366irq/58-eth023:25:143
911299487,25cyclictest28366irq/58-eth023:25:133
9112994735,7cyclictest28366irq/58-eth022:10:123
9112994735,7cyclictest28366irq/58-eth022:10:123
9098994611,12cyclictest630-21tcf-agent20:10:142
908899463,36cyclictest12287-21rm21:55:111
9077994628,12cyclictest0-21swapper/023:10:110
909899452,27cyclictest7062-21ssh23:40:122
9112994326,12cyclictest28366irq/58-eth023:05:133
909899435,18cyclictest261rcuc/221:35:122
907799432,7cyclictest131rcu_preempt19:20:140
9077994225,9cyclictest28089-21sh21:10:120
908899412,33cyclictest0-21swapper/120:00:151
908899411,29cyclictest24967-21ssh23:50:121
909899402,18cyclictest7034-21ssh22:10:132
909899402,18cyclictest7034-21ssh22:10:132
907799391,25cyclictest32417-21sh23:00:130
909899381,10cyclictest27702-21diskmemload22:55:112
908899381,26cyclictest17617-21ssh21:40:131
908899363,21cyclictest25058-21ssh22:20:131
911299352,20cyclictest28366irq/58-eth000:15:123
9098993511,18cyclictest27-21ksoftirqd/222:30:132
9098993511,18cyclictest27-21ksoftirqd/222:30:122
9088993520,10cyclictest13828-21diskmemload21:20:141
908899344,22cyclictest12524-21ssh23:25:141
908899344,22cyclictest12524-21ssh23:25:131
9088993420,7cyclictest0-21swapper/122:00:141
908899341,23cyclictest3181-21ssh21:50:101
911299332,18cyclictest28366irq/58-eth022:00:133
907799332,25cyclictest1519-21ssh23:55:120
909899326,18cyclictest17611-21ssh21:40:122
908899321,23cyclictest3511-21ssh23:20:131
911299311,21cyclictest28366irq/58-eth020:15:123
909899312,12cyclictest28076-21ssh21:10:122
908899311,23cyclictest14414-21ssh22:50:131
909899303,19cyclictest22346irq/40-dwc2_hso20:55:132
909899303,19cyclictest22346irq/40-dwc2_hso20:55:132
909899302,20cyclictest27-21ksoftirqd/200:20:132
908899303,8cyclictest131rcu_preempt23:55:121
908899303,18cyclictest21-21ksoftirqd/121:25:111
9088993022,3cyclictest28808-21ssh22:40:121
907799301,11cyclictest0-21swapper/021:45:120
909899297,12cyclictest0-21swapper/223:35:132
909899292,19cyclictest18603-21kworker/2:019:20:142
909899292,18cyclictest22346irq/40-dwc2_hso20:20:122
908899292,21cyclictest16918-21ssh00:39:471
908899292,18cyclictest4742-21ssh21:15:141
907799292,19cyclictest22907-21ssh21:25:130
911299283,18cyclictest28366irq/58-eth023:45:143
911299283,16cyclictest28366irq/58-eth021:20:133
909899283,14cyclictest22346irq/40-dwc2_hso20:25:142
908899282,21cyclictest8999-21ssh23:05:151
907799282,19cyclictest4738-21ssh21:15:130
907799281,19cyclictest26911-21ssh23:15:130
909899273,16cyclictest22346irq/40-dwc2_hso19:15:132
909899271,19cyclictest32105-21ssh00:10:252
909899271,16cyclictest3495-21apt-get23:20:132
908899272,20cyclictest1666-21ssh22:25:141
908899272,18cyclictest4651-21sh00:30:451
907799273,18cyclictest1676-21ssh22:25:140
911299264,14cyclictest28366irq/58-eth021:00:123
911299263,15cyclictest0-21swapper/300:30:463
9112992611,8cyclictest321rcuc/321:05:123
909899264,13cyclictest22346irq/40-dwc2_hso19:50:132
909899264,13cyclictest22346irq/40-dwc2_hso19:50:132
9088992612,5cyclictest21-21ksoftirqd/100:23:321
908899261,18cyclictest23007-21rm00:25:121
907799263,16cyclictest12-21ksoftirqd/021:30:130
9077992618,4cyclictest0-21swapper/023:35:120
911299252,13cyclictest28366irq/58-eth023:35:123
9112992517,5cyclictest28366irq/58-eth023:40:123
909899257,10cyclictest16227-21/usr/sbin/munin23:25:282
909899257,10cyclictest16227-21/usr/sbin/munin23:25:272
909899251,18cyclictest32418-21ssh23:00:122
908899253,16cyclictest5411-21ssh22:45:141
908899253,15cyclictest8584-21ssh21:35:121
908899251,19cyclictest30599-21ls23:35:141
9088992511,6cyclictest131rcu_preempt23:42:591
911299244,14cyclictest0-21swapper/300:00:143
911299243,15cyclictest0-21swapper/300:39:463
911299243,13cyclictest28366irq/58-eth000:10:113
9112992417,5cyclictest28366irq/58-eth021:45:253
9112992414,7cyclictest28366irq/58-eth023:30:143
909899241,19cyclictest14380-21apt-get22:50:122
9098992411,6cyclictest11844-21cron20:20:022
908899245,9cyclictest9068-21cyclictest20:05:141
908899242,17cyclictest5019-21ssh00:15:131
9088992410,11cyclictest18608-21cron19:20:011
907799242,18cyclictest1259-1kworker/u9:200:20:130
907799241,3cyclictest131rcu_preempt23:20:110
907799241,17cyclictest21357-21ssh21:40:270
9112992311,7cyclictest28366irq/58-eth021:15:153
909899232,9cyclictest22346irq/40-dwc2_hso21:30:042
9098992311,9cyclictest17536-21cron23:10:012
909899231,10cyclictest3026066ssh23:35:012
908899233,12cyclictest26669-21ssh21:45:131
908899231,17cyclictest15955-21ssh23:45:131
908899231,15cyclictest10588-21ssh00:00:141
907799232,14cyclictest24922-21ssh22:20:040
907799232,13cyclictest19854-21ssh22:35:140
911299223,12cyclictest0-21swapper/322:05:133
911299222,14cyclictest321rcuc/321:55:113
9112992216,4cyclictest28366irq/58-eth023:00:123
9112992216,4cyclictest0-21swapper/322:55:123
9112992212,7cyclictest0-21swapper/322:35:183
909899229,10cyclictest5401-21grep00:15:172
909899228,7cyclictest29532-21cut22:40:182
909899227,5cyclictest3091-21cron23:20:012
909899227,12cyclictest1914-21irqcore23:55:192
909899224,10cyclictest16483-21irqcore23:45:182
909899223,14cyclictest22346irq/40-dwc2_hso20:30:122
909899223,14cyclictest22346irq/40-dwc2_hso20:30:112
909899222,15cyclictest0-21swapper/223:10:132
909899222,15cyclictest0-21swapper/200:38:492
909899221,12cyclictest881-21lldpd23:50:052
909899221,11cyclictest15263-21usage_EthcTask00:00:332
9088992213,7cyclictest23481-21ssh22:35:261
9088992213,5cyclictest0-21swapper/100:10:131
908899221,18cyclictest28745-21ls19:30:161
907799222,15cyclictest12540-21ssh23:25:140
907799222,15cyclictest12540-21ssh23:25:140
9077992213,6cyclictest0-21swapper/021:05:130
9077992211,8cyclictest618-21systemd-logind00:10:390
907799221,15cyclictest14088-21ls19:15:140
911299216,8cyclictest0-21swapper/322:45:143
911299212,15cyclictest28366irq/58-eth023:15:133
9112992113,4cyclictest0-21swapper/322:25:153
9112992112,6cyclictest28366irq/58-eth023:50:133
9112992112,6cyclictest0-21swapper/323:20:213
909899219,9cyclictest25478-21ssh22:36:282
909899219,8cyclictest16200-21fw_packets22:15:172
909899218,5cyclictest26463-21missed_timers21:25:232
909899214,13cyclictest22346irq/40-dwc2_hso20:00:122
909899213,15cyclictest22346irq/40-dwc2_hso21:15:122
909899212,15cyclictest23232-21fschecks_time21:05:152
909899212,13cyclictest22346irq/40-dwc2_hso21:45:132
909899211,9cyclictest19883-21irqcore00:05:172
909899211,11cyclictest15831-21kthreadcore21:20:212
909899211,10cyclictest9046-21netstat22:45:252
909899211,10cyclictest3097-21cat00:30:232
909899211,10cyclictest25011-21/usr/sbin/munin22:20:192
909899211,10cyclictest16434-21unixbench_singl21:55:282
9098992110,9cyclictest15177-21sshd19:46:192
909899210,11cyclictest10373-21apt19:45:012
9088992111,7cyclictest363-21sshd23:16:461
908899211,16cyclictest21518-21apt-get23:30:121
908899211,16cyclictest16910-21apt-get20:25:131
908899211,11cyclictest4741-21kworker/1:222:59:001
908899210,18cyclictest24009-21ps19:25:171
907799213,11cyclictest151rcuc/021:35:130
907799213,11cyclictest0-21swapper/020:25:140
907799211,7cyclictest14427-21sed21:20:180
907799211,13cyclictest9334-21ls22:45:300
911299207,10cyclictest0-21swapper/300:20:133
911299206,11cyclictest33-21ksoftirqd/322:50:123
911299203,12cyclictest0-21swapper/300:28:463
9112992014,4cyclictest0-21swapper/323:10:203
9112992011,6cyclictest0-21swapper/323:55:193
909899209,9cyclictest3529-21latency_hist20:45:012
909899208,4cyclictest1010-1kworker/2:2H21:50:132
909899208,10cyclictest28982-21irqcore19:30:172
909899202,8cyclictest1480-21unixbench_singl20:05:302
909899202,10cyclictest10527-21kthreadcore19:10:192
909899201,9cyclictest5479-21kthreadcore20:45:192
9098992014,4cyclictest0-21swapper/220:50:152
909899201,17cyclictest1673-21sh22:25:142
909899201,16cyclictest22414-21kthreadcore22:00:202
909899201,15cyclictest31621-21kthreadcore22:05:192
909899201,11cyclictest20478-21fschecks_time19:55:162
909899201,11cyclictest20060-21kthreadcore21:00:192
909899201,10cyclictest23690-21ssh00:25:192
909899200,10cyclictest1096-21fschecks_count19:35:162
908899201,7cyclictest19445-21idleruntime-cro22:35:001
908899201,7cyclictest19445-21idleruntime-cro22:35:001
9088992011,6cyclictest333-21tail23:00:171
9088992011,6cyclictest32397-21ssh21:30:181
907799209,8cyclictest15777-21load21:55:230
907799207,7cyclictest0-21swapper/000:30:450
907799203,11cyclictest0-21swapper/020:30:130
907799203,11cyclictest0-21swapper/020:30:130
907799202,13cyclictest839-21munin-node20:50:130
9077992012,6cyclictest28669-21awk20:00:270
9077992011,6cyclictest0-21swapper/000:10:010
907799201,15cyclictest27362-21timerandwakeup22:55:270
907799201,11cyclictest7161-21df23:40:140
9077992010,7cyclictest20515-21cat22:15:320
9077992010,7cyclictest13697-21cat23:05:330
907799200,3cyclictest0-21swapper/000:36:130
911299198,8cyclictest0-21swapper/322:41:533
911299194,12cyclictest321rcuc/319:20:153
911299194,12cyclictest0-21swapper/322:20:253
911299193,8cyclictest28366irq/58-eth021:35:123
911299192,13cyclictest0-21swapper/321:25:133
9112991911,6cyclictest0-21swapper/321:10:043
911299191,12cyclictest0-21swapper/320:50:133
911299191,12cyclictest0-21swapper/320:30:123
911299191,12cyclictest0-21swapper/320:30:123
911299191,12cyclictest0-21swapper/319:45:023
911299191,11cyclictest0-21swapper/319:45:153
911299191,10cyclictest0-21swapper/320:10:013
9112991910,6cyclictest28366irq/58-eth020:25:133
909899191,9cyclictest27027-21wc19:25:232
909899191,9cyclictest26773-21grep20:35:172
908899199,7cyclictest17982-21cpuspeed23:10:121
908899196,2cyclictest131rcu_preempt22:06:161
908899193,13cyclictest27702-21diskmemload00:05:141
908899192,14cyclictest12726-21ps19:10:261
908899192,11cyclictest0-21swapper/120:45:131
9088991912,5cyclictest24279-21ssh22:19:411
9088991912,5cyclictest0-21swapper/119:40:161
908899191,15cyclictest7308-21ls22:10:171
908899191,15cyclictest7308-21ls22:10:161
908899191,14cyclictest13407-21apt-get20:55:141
908899191,14cyclictest13407-21apt-get20:55:131
907799192,12cyclictest10577-21apt-get00:00:120
907799191,9cyclictest7184-21fschecks_count22:10:160
907799191,9cyclictest7184-21fschecks_count22:10:150
9077991913,4cyclictest0-21swapper/019:55:240
907799191,15cyclictest24257-21irqrtprio19:25:170
907799191,14cyclictest12961-21timerwakeupswit19:10:280
907799191,13cyclictest5262-21fw_packets00:15:160
907799191,10cyclictest23811-21tr22:00:220
911299189,6cyclictest0-21swapper/321:30:103
911299185,9cyclictest0-21swapper/300:05:143
911299182,10cyclictest0-21swapper/320:45:143
911299182,10cyclictest0-21swapper/320:00:143
911299182,10cyclictest0-21swapper/319:30:143
9112991812,4cyclictest0-21swapper/321:54:563
9112991810,6cyclictest28366irq/58-eth022:30:143
9112991810,6cyclictest28366irq/58-eth022:30:133
908899183,12cyclictest0-21swapper/121:00:111
908899182,11cyclictest0-21swapper/120:40:151
908899181,3cyclictest131rcu_preempt19:50:301
908899181,3cyclictest131rcu_preempt19:50:291
9088991811,5cyclictest1510-21ssh21:11:391
908899181,11cyclictest10915-21ls20:15:271
907799189,7cyclictest916-21memory20:05:230
907799182,12cyclictest26538-21memory00:25:250
907799181,6cyclictest16765-21kthreadcore22:50:220
9077991811,5cyclictest0-21swapper/019:50:180
9077991811,5cyclictest0-21swapper/019:50:170
907799181,14cyclictest30463-21ssh22:05:140
907799181,14cyclictest20201-21cat23:45:300
907799181,10cyclictest27863-21kthreadcore20:35:210
907799180,12cyclictest29021-21ssh23:50:290
911299178,6cyclictest28366irq/58-eth021:40:123
911299172,10cyclictest0-21swapper/322:15:153
911299172,10cyclictest0-21swapper/319:25:133
909899171,14cyclictest9068-21cyclictest19:10:022
908899174,11cyclictest21851-21grep20:30:151
908899174,11cyclictest21851-21grep20:30:151
908899173,11cyclictest0-21swapper/120:35:141
908899173,10cyclictest0-21swapper/119:45:111
908899172,12cyclictest19771-21kthreadcore19:20:201
908899171,8cyclictest23969-21sort19:55:271
908899171,14cyclictest12015-21ntp_kernel_err20:50:241
908899171,13cyclictest4895-21kthreadcore20:10:221
908899171,13cyclictest3036-21kthreadcore19:35:211
907799173,10cyclictest0-21swapper/020:55:150
907799173,10cyclictest0-21swapper/020:55:140
907799172,10cyclictest0-21swapper/019:35:140
907799171,3cyclictest131rcu_preempt22:40:160
9077991711,4cyclictest0-21swapper/020:20:290
9077991710,5cyclictest8346-21sshd20:48:470
9077991710,4cyclictest12007-21wc21:55:010
907799170,15cyclictest6070-21ls20:10:290
911299161,12cyclictest0-21swapper/320:55:153
911299161,12cyclictest0-21swapper/320:55:153
911299161,11cyclictest0-21swapper/320:20:203
908899161,12cyclictest9147-21wc19:10:011
908899161,12cyclictest12156-21ls20:20:131
907799169,5cyclictest746-21wc19:35:010
907799163,10cyclictest9265-21kthreadcore20:15:200
907799162,10cyclictest0-21swapper/021:00:130
907799161,12cyclictest12-21ksoftirqd/019:45:190
911299151,9cyclictest0-21swapper/319:50:143
911299151,9cyclictest0-21swapper/319:50:133
907799151,13cyclictest0-21swapper/020:40:140
907799151,11cyclictest5781-21apt-get19:40:100
911299149,3cyclictest0-21swapper/320:10:193
911299143,7cyclictest0-21swapper/319:15:123
911299141,10cyclictest0-21swapper/320:00:013
911299141,10cyclictest0-21swapper/319:35:173
907799141,10cyclictest9102-21latency_hist19:10:010
911299122,6cyclictest0-21swapper/320:40:153
911299113,5cyclictest0-21swapper/319:10:143
911299111,7cyclictest0-21swapper/320:35:183
911299101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional