You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 06:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed May 06, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2979199664,54cyclictest15303-21ssh23:15:131
29799996010,7cyclictest131rcu_preempt00:25:152
29779996011,7cyclictest131rcu_preempt21:55:140
29779995820,9cyclictest3575-21apt-get22:15:140
2979999571,32cyclictest28487-21ssh21:35:152
2979999551,32cyclictest0-21swapper/219:40:102
2977999513,41cyclictest15993-21ssh21:10:110
29799995029,12cyclictest27-21ksoftirqd/222:30:132
2979199491,38cyclictest6191-21ssh23:10:121
29779994814,25cyclictest0-21swapper/020:55:110
2977999481,36cyclictest7499-21ssh22:35:150
2979999477,24cyclictest131rcu_preempt23:26:412
29779994715,13cyclictest28378-21grep23:20:270
29799994515,7cyclictest29771-21cyclictest22:25:152
2977999452,34cyclictest30788-21ssh22:30:120
2979999442,30cyclictest930-21snmpd20:35:122
2977999431,33cyclictest6198-21rm23:10:110
2979999422,25cyclictest131rcu_preempt23:24:402
29779994214,11cyclictest12491-21ssh21:43:360
2979199412,9cyclictest21-21ksoftirqd/119:20:131
29779994111,12cyclictest16110-21ssh22:55:320
2979999401,22cyclictest19759-21ssh23:51:392
29779994028,9cyclictest28469-21apt-get21:35:140
29779994027,8cyclictest20447-21apt-get23:00:130
29779994024,12cyclictest23066-21ssh23:55:120
29779994014,11cyclictest15156-21kthreadcore23:50:180
29779994013,10cyclictest32334-21fschecks_count00:00:160
29779994010,10cyclictest21791-21ssh22:05:270
2980799393,24cyclictest35966irq/58-eth023:50:133
29807993929,7cyclictest35966irq/58-eth000:20:143
29779993932,4cyclictest31189-21apt-get19:45:120
29779993930,3cyclictest16479-21apt-get22:40:120
29779993914,8cyclictest12280-21sed19:55:280
29779993911,11cyclictest20472-21kthreadcore23:35:190
2980799381,21cyclictest35966irq/58-eth021:40:153
2979999384,4cyclictest131rcu_preempt20:05:142
29799993825,6cyclictest15609-21diskmemload21:40:132
29791993814,8cyclictest131rcu_preempt21:35:131
2979199381,32cyclictest19134-21ssh23:35:131
2977999389,12cyclictest29858-21ntp_states19:40:240
29779993831,3cyclictest18057-21apt-get20:05:130
29779993829,5cyclictest7203-21apt-get19:20:130
29779993814,8cyclictest27657-21ntp_192.168.11420:45:240
29779993814,8cyclictest27657-21ntp_192.168.11420:45:240
29779993812,8cyclictest0-21swapper/000:17:530
29807993718,8cyclictest35966irq/58-eth000:30:143
2979999376,21cyclictest27-21ksoftirqd/221:55:132
2977999379,12cyclictest4496-21latency_hist23:45:020
2977999379,12cyclictest4496-21latency_hist23:45:010
29779993715,8cyclictest19231-21ssh21:45:550
29779993712,11cyclictest10141-21diskstats23:30:140
29779993711,9cyclictest18214-21/usr/sbin/munin00:10:180
29779993711,9cyclictest14125-21ntp_states21:25:260
29779993710,11cyclictest16795-21awk22:20:270
29779993630,2cyclictest24925-21apt-get21:15:140
29779993630,2cyclictest24925-21apt-get21:15:130
29779993626,5cyclictest21744-21apt-get22:25:130
29779993612,14cyclictest31817-21usage_Anybus20:15:270
29779993612,11cyclictest5160-21ssh23:45:180
29779993611,9cyclictest5687-21memory22:50:240
29779993611,8cyclictest23145-21ntp_states21:30:250
29779993611,11cyclictest9549-21apt-config20:30:010
2979999352,27cyclictest0-21swapper/223:31:512
2979999351,28cyclictest2216-21ssh22:50:122
2977999359,9cyclictest27603-21usage_Anybus21:50:310
2977999359,9cyclictest27603-21usage_Anybus21:50:300
2977999359,9cyclictest15754-21sed19:25:280
2977999359,8cyclictest12419-21ntp_192.168.11422:00:250
2977999359,10cyclictest29822-21date19:10:020
2977999359,10cyclictest15110-21irqrtprio20:35:190
29779993513,8cyclictest32210-21ssh22:47:380
2977999351,29cyclictest1472-21ssh21:20:120
29779993511,8cyclictest8823-21ssh23:29:040
29779993510,11cyclictest28127-21ssh00:32:290
2980799343,21cyclictest35966irq/58-eth021:30:123
2977999349,8cyclictest10438-21irqstats20:30:190
2977999347,12cyclictest962-21/usr/sbin/munin23:05:310
2977999344,18cyclictest12-21ksoftirqd/019:10:140
29779993414,6cyclictest15609-21diskmemload00:36:160
29779993411,10cyclictest10730-21rm00:23:120
29779993410,8cyclictest32476-21ntp_192.168.11420:50:250
29779993410,11cyclictest10143-21grep21:00:280
29779993410,10cyclictest2368-21awk19:15:130
2979999338,19cyclictest27-21ksoftirqd/219:15:132
29779993328,2cyclictest16797-21apt-get19:30:130
29779993313,7cyclictest12112-21cut00:05:240
29779993312,8cyclictest18030-21ssh00:26:120
29779993312,8cyclictest17180-21cat20:00:290
29779993312,7cyclictest15609-21diskmemload23:15:360
29779993310,9cyclictest30656-21ntp_states22:10:250
2979999321,25cyclictest0-21swapper/223:10:122
29779993211,8cyclictest22919-21df20:10:150
29779993210,10cyclictest19709-21cat20:40:190
29807993114,10cyclictest35966irq/58-eth022:25:153
2979999317,19cyclictest27-21ksoftirqd/222:15:142
2979999312,4cyclictest131rcu_preempt19:45:132
29791993121,8cyclictest21-21ksoftirqd/121:55:131
29791993121,5cyclictest201rcuc/123:00:141
29791993115,9cyclictest21-21ksoftirqd/100:25:151
2979199311,24cyclictest16504-21sh22:40:131
2977999318,8cyclictest3727-21diskstats19:50:160
2977999318,10cyclictest24830-21wc19:35:220
2977999318,10cyclictest24830-21wc19:35:220
2980799303,12cyclictest35966irq/58-eth022:20:113
29807993021,7cyclictest0-21swapper/322:35:143
2979999303,17cyclictest27346irq/40-dwc2_hso23:15:142
2980799292,16cyclictest35966irq/58-eth020:15:113
2979999293,20cyclictest0-21swapper/220:25:142
2979999291,21cyclictest0-21swapper/219:50:142
29791992914,9cyclictest21-21ksoftirqd/122:16:351
2979199291,23cyclictest948-21ssh23:25:141
2977999298,8cyclictest2082-21cut20:20:210
29807992810,9cyclictest35966irq/58-eth023:05:143
2979999283,6cyclictest0-21swapper/223:55:122
2979999283,15cyclictest27346irq/40-dwc2_hso20:20:132
2979999282,20cyclictest0-21swapper/220:45:142
2979999282,20cyclictest0-21swapper/220:45:132
2979999282,20cyclictest0-21swapper/219:55:142
2979999281,21cyclictest25595-21ssh22:45:152
2980799273,17cyclictest0-21swapper/323:29:403
2980799273,16cyclictest0-21swapper/321:56:393
2979999272,15cyclictest27346irq/40-dwc2_hso21:15:132
2979999272,15cyclictest27346irq/40-dwc2_hso21:15:132
2979999272,15cyclictest27346irq/40-dwc2_hso00:00:132
2979999272,10cyclictest0-21swapper/222:14:372
2979199272,16cyclictest2222-21ssh22:50:141
2979199271,23cyclictest8841-21ssh22:00:141
2980799263,16cyclictest35966irq/58-eth020:55:113
2979999263,18cyclictest27346irq/40-dwc2_hso22:40:112
2979999262,20cyclictest0-21swapper/223:40:132
2979999262,20cyclictest0-21swapper/223:40:132
2979999262,19cyclictest27459-21ssh00:31:422
2979199266,13cyclictest21-21ksoftirqd/119:30:121
2980799253,15cyclictest0-21swapper/322:04:383
29807992519,4cyclictest0-21swapper/322:05:133
2979999254,16cyclictest0-21swapper/222:02:392
2979999253,15cyclictest27346irq/40-dwc2_hso19:30:132
2979999251,18cyclictest10443-21ssh21:25:142
2979999251,14cyclictest14076-21ssh21:45:122
2979199254,3cyclictest131rcu_preempt22:22:291
2979199252,16cyclictest15609-21diskmemload21:15:131
2979199252,16cyclictest15609-21diskmemload21:15:121
29791992513,4cyclictest0-21swapper/123:52:171
29779992514,7cyclictest14502-21ntp_states21:05:260
2980799243,13cyclictest35966irq/58-eth023:35:113
29807992413,8cyclictest0-21swapper/322:33:423
2980799241,14cyclictest35966irq/58-eth000:35:133
29807992410,8cyclictest35966irq/58-eth023:15:133
29807992410,8cyclictest35966irq/58-eth021:35:143
2979999248,7cyclictest131rcu_preempt00:10:022
2979999242,8cyclictest20200-21ssh22:23:392
2979999242,15cyclictest27346irq/40-dwc2_hso20:40:152
2979199247,7cyclictest21-21ksoftirqd/122:05:371
2979199243,13cyclictest0-21swapper/119:45:131
29791992412,9cyclictest18742-21cron23:35:011
29791992410,7cyclictest29597-21ssh23:05:151
2980799233,13cyclictest0-21swapper/300:12:053
2980799233,12cyclictest35966irq/58-eth021:45:113
29807992313,6cyclictest0-21swapper/323:30:193
2979999233,16cyclictest27346irq/40-dwc2_hso00:35:132
2979999232,6cyclictest0-21swapper/221:30:132
29799992312,8cyclictest0-21swapper/222:35:032
2979999231,10cyclictest2156066ssh23:00:202
2979199232,17cyclictest11347-21ssh22:55:131
2979199232,14cyclictest28239-21ssh23:40:121
2979199232,14cyclictest28239-21ssh23:40:111
2979199232,14cyclictest0-21swapper/121:40:141
29791992314,6cyclictest22349-21ssh00:30:181
2979199231,13cyclictest8293-21tr21:00:201
2980799224,12cyclictest0-21swapper/321:25:143
2980799223,9cyclictest0-21swapper/321:20:133
2979999226,6cyclictest28054-21irqcore20:15:172
2979999223,10cyclictest29817-21fschecks_time23:05:182
2979999222,15cyclictest0-21swapper/223:46:532
2979999222,15cyclictest0-21swapper/222:05:382
2979999221,7cyclictest2770166ssh21:50:322
2979999221,7cyclictest2770166ssh21:50:312
2979999221,18cyclictest22103-21irqrtprio19:35:182
2979999221,18cyclictest22103-21irqrtprio19:35:172
29799992211,3cyclictest20270-21kthreadcore00:10:202
2979199227,4cyclictest131rcu_preempt23:20:191
2979199226,6cyclictest131rcu_preempt20:15:121
29791992211,9cyclictest0-21swapper/123:56:061
29791992211,6cyclictest0-21swapper/100:10:141
29791992211,3cyclictest0-21swapper/120:25:291
2980799216,9cyclictest35966irq/58-eth023:55:133
2980799211,16cyclictest0-21swapper/322:50:143
2980799211,14cyclictest35966irq/58-eth022:45:133
2979999219,5cyclictest24075-21ssh21:14:282
2979999216,12cyclictest0-21swapper/220:30:162
2979999212,13cyclictest27346irq/40-dwc2_hso19:25:122
2979999211,13cyclictest13355-21cut20:00:162
2979199216,3cyclictest21-21ksoftirqd/122:30:291
29791992112,6cyclictest7297-21ssh22:35:021
29791992112,6cyclictest2986-21ssh00:00:241
29791992110,4cyclictest131rcu_preempt23:45:141
2980799208,7cyclictest35966irq/58-eth022:40:133
2980799203,12cyclictest0-21swapper/323:46:403
2980799203,12cyclictest0-21swapper/323:10:143
2980799203,11cyclictest0-21swapper/323:40:133
2980799203,11cyclictest0-21swapper/323:40:123
29807992014,4cyclictest0-21swapper/300:28:323
29807992013,5cyclictest0-21swapper/322:11:503
29807992013,5cyclictest0-21swapper/300:15:163
2980799201,14cyclictest35966irq/58-eth023:20:123
2979999201,9cyclictest444-21unixbench_multi20:50:282
2979999201,14cyclictest7873-21kthreadcore19:20:192
2979999201,11cyclictest19150-21ssh23:35:142
2979999201,11cyclictest1481-21ssh21:20:142
2979999201,10cyclictest4774-21cat20:55:232
2979999201,10cyclictest26953-21/usr/sbin/munin00:15:292
2979999201,10cyclictest12234-21cron00:25:002
2979199205,13cyclictest201rcuc/100:15:131
2979199204,3cyclictest131rcu_preempt21:30:191
2979199202,4cyclictest131rcu_preempt20:50:141
29791992011,7cyclictest31360-21ssh22:46:381
29791992011,6cyclictest1812-21ssh21:20:181
2979199200,3cyclictest131rcu_preempt00:20:301
2980799193,11cyclictest0-21swapper/322:15:383
29807991913,4cyclictest0-21swapper/321:51:133
29807991913,4cyclictest0-21swapper/321:51:123
29807991913,4cyclictest0-21swapper/321:10:113
29807991913,4cyclictest0-21swapper/300:05:193
29807991913,4cyclictest0-21swapper/300:02:403
2980799191,12cyclictest0-21swapper/320:35:153
2980799191,12cyclictest0-21swapper/320:10:163
2980799191,11cyclictest0-21swapper/321:05:023
2979999197,9cyclictest31295-21kthreadcore19:10:212
2979999192,14cyclictest0-21swapper/221:00:142
2979999191,9cyclictest22993-21fschecks_time20:10:162
2979999191,9cyclictest14965-21ntp_192.168.11422:55:252
2979999190,9cyclictest11208-21fschecks_count21:05:172
2979199198,3cyclictest0-21swapper/119:35:271
2979199198,3cyclictest0-21swapper/119:35:271
2979199196,7cyclictest0-21swapper/121:45:181
2979199196,5cyclictest0-21swapper/119:40:111
2979199196,2cyclictest131rcu_preempt21:10:311
2979199191,9cyclictest24262-21tr22:25:211
2979199191,3cyclictest131rcu_preempt20:55:131
29791991913,4cyclictest21-21ksoftirqd/121:50:141
29791991913,4cyclictest21-21ksoftirqd/121:50:141
29791991913,4cyclictest0-21swapper/100:08:531
29791991912,4cyclictest27436-21idleruntime-cro20:15:001
29791991911,6cyclictest29625-21ssh22:10:221
2979199191,12cyclictest1200-21switchtime19:10:291
29791991910,6cyclictest10030-21cron21:25:011
2980799183,9cyclictest35966irq/58-eth023:00:143
2980799182,10cyclictest0-21swapper/320:40:163
2980799182,10cyclictest0-21swapper/319:30:133
29807991812,4cyclictest0-21swapper/322:55:143
2980799181,13cyclictest0-21swapper/319:20:213
2980799181,11cyclictest0-21swapper/321:15:183
2980799181,11cyclictest0-21swapper/321:15:173
2980799181,11cyclictest0-21swapper/320:05:023
2979199183,4cyclictest131rcu_preempt21:05:211
2979199181,8cyclictest31253-21grep00:35:191
29791991810,5cyclictest9722-21kthreadcore19:55:201
2979199180,2cyclictest131rcu_preempt20:35:331
2980799179,5cyclictest321rcuc/320:50:143
2980799172,9cyclictest0-21swapper/319:45:133
2980799172,10cyclictest0-21swapper/320:45:143
2980799172,10cyclictest0-21swapper/320:45:133
2980799172,10cyclictest0-21swapper/320:30:043
2980799172,10cyclictest0-21swapper/319:50:053
2980799172,10cyclictest0-21swapper/319:40:123
29807991711,4cyclictest0-21swapper/320:00:013
2980799171,13cyclictest0-21swapper/319:25:273
2979199178,7cyclictest21499-21ntp_states20:05:271
2979199177,7cyclictest14236-21kthreadcore20:00:181
2979199176,4cyclictest25712-21kthreadcore20:45:201
2979199176,4cyclictest25712-21kthreadcore20:45:191
2979199173,3cyclictest131rcu_preempt19:15:131
2979199170,14cyclictest5204-21kthreadcore19:50:201
2980799162,9cyclictest0-21swapper/320:25:143
2980799162,9cyclictest0-21swapper/319:35:143
2980799162,9cyclictest0-21swapper/319:35:133
2980799162,9cyclictest0-21swapper/319:15:153
2980799162,10cyclictest0-21swapper/320:05:143
2979999161,12cyclictest29829-21latency_hist19:10:012
2979199163,5cyclictest3908-21cat20:20:281
2979199162,11cyclictest14718-21kthreadcore19:25:211
2979199161,12cyclictest9485-21apt-get20:30:041
2980799152,9cyclictest0-21swapper/321:05:143
2980799152,9cyclictest0-21swapper/319:10:143
2979199155,2cyclictest131rcu_preempt20:40:181
2979199142,10cyclictest29716-21latency_hist19:10:011
2980799137,4cyclictest0-21swapper/320:20:193
298079993,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional