You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 18:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Wed Jun 03, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
658499705,53cyclictest19327-21ssh12:25:131
6608996959,6cyclictest35966irq/58-eth011:40:133
6608996745,18cyclictest35966irq/58-eth010:40:133
6608996745,18cyclictest35966irq/58-eth010:40:133
6608995836,17cyclictest35966irq/58-eth011:45:133
6573995714,14cyclictest131rcu_preempt11:40:130
659999551,9cyclictest0-21swapper/207:25:132
658499522,39cyclictest26762-21ssh11:35:111
658499522,39cyclictest26762-21ssh11:35:101
657399526,7cyclictest131rcu_preempt09:05:130
657399515,41cyclictest16010-21ssh10:35:120
659999501,29cyclictest1823-21ssh09:15:152
659999501,29cyclictest1823-21ssh09:15:142
657399506,39cyclictest5249-21ssh11:05:140
6608994925,17cyclictest35966irq/58-eth011:50:123
658499498,6cyclictest131rcu_preempt08:05:131
657399491,43cyclictest17732-21ssh11:30:120
657399485,6cyclictest131rcu_preempt07:30:150
657399485,6cyclictest131rcu_preempt07:30:150
657399477,35cyclictest10273-21ssh12:20:120
657399468,34cyclictest24694-21ssh12:10:140
6573994630,10cyclictest0-21swapper/009:10:130
6573994630,10cyclictest0-21swapper/009:10:120
659999454,25cyclictest32124-21ssh11:20:122
659999443,20cyclictest21580-21ssh10:20:122
658499446,6cyclictest131rcu_preempt07:20:131
657399441,33cyclictest21266-21ssh11:50:120
658499431,35cyclictest28811-21ssh09:30:131
658499419,22cyclictest21-21ksoftirqd/109:55:131
658499419,22cyclictest21-21ksoftirqd/109:55:121
658499412,30cyclictest10505-21ssh10:50:121
660899402,22cyclictest35966irq/58-eth010:55:133
660899402,22cyclictest35966irq/58-eth010:55:123
657399401,26cyclictest28298-21ssh12:30:130
659999394,22cyclictest0-21swapper/211:30:132
6599993924,7cyclictest24844-21diskmemload10:30:132
6599993922,11cyclictest18055-21sh10:00:142
658499393,26cyclictest18056-21ssh10:00:141
658499371,29cyclictest28590-21ssh11:00:141
657399377,23cyclictest12-21ksoftirqd/009:20:110
657399362,25cyclictest12-21ksoftirqd/007:50:130
657399361,27cyclictest24988-21ssh10:40:140
657399361,27cyclictest24988-21ssh10:40:130
6608993518,11cyclictest35966irq/58-eth009:35:123
659999352,25cyclictest15719-21ssh12:05:132
6584993518,12cyclictest21-21ksoftirqd/111:55:121
660899347,18cyclictest35966irq/58-eth011:00:133
658499343,23cyclictest21585-21ssh10:20:131
657399341,26cyclictest1296-21ssh12:15:130
660899334,22cyclictest35966irq/58-eth009:05:123
6608993312,12cyclictest0-21swapper/310:00:123
6599993315,13cyclictest27346irq/40-dwc2_hso09:05:132
6599993312,14cyclictest27-21ksoftirqd/211:15:142
658499331,20cyclictest3649-21ssh10:10:131
658499330,20cyclictest27053-21ssh10:05:131
6608993222,7cyclictest0-21swapper/312:00:123
659999321,12cyclictest261rcuc/212:00:122
659999313,18cyclictest27346irq/40-dwc2_hso08:15:132
657399313,19cyclictest12600-21ssh10:15:130
6608993017,8cyclictest0-21swapper/312:30:133
659999304,11cyclictest0-21swapper/212:35:132
659999302,20cyclictest3393-21sh11:40:132
659999301,17cyclictest3664-21ssh10:10:142
658499301,21cyclictest30599-21ssh10:25:121
657399302,3cyclictest131rcu_preempt08:35:150
657399301,24cyclictest28579-21ssh11:00:120
657399301,22cyclictest4848-21ssh12:35:130
659999299,9cyclictest0-21swapper/210:50:112
659999293,17cyclictest27346irq/40-dwc2_hso09:20:122
659999292,16cyclictest930-21snmpd07:20:142
659999292,16cyclictest5303-21ssh09:35:132
6599992915,7cyclictest5233-21ssh11:05:122
659999291,17cyclictest27346irq/40-dwc2_hso09:25:142
658499292,20cyclictest23123-21ssh11:15:131
657399292,21cyclictest1543-21ssh10:45:140
657399292,21cyclictest1543-21ssh10:45:130
657399291,22cyclictest32387-21ssh09:50:120
659999284,16cyclictest27346irq/40-dwc2_hso08:40:122
659999282,7cyclictest16019-21ssh10:35:132
659999282,20cyclictest23379-21ssh09:45:132
659999282,20cyclictest0-21swapper/208:35:152
659999282,20cyclictest0-21swapper/207:55:152
659999282,11cyclictest0-21swapper/208:55:132
657399285,13cyclictest12336-21ssh11:45:130
660899272,20cyclictest33-21ksoftirqd/311:05:133
659999273,17cyclictest27346irq/40-dwc2_hso11:45:132
659999271,9cyclictest317066ssh10:45:212
659999271,9cyclictest317066ssh10:45:202
6599992717,7cyclictest261rcuc/212:25:142
658499272,19cyclictest17850-21ssh11:10:281
657399271,20cyclictest32139-21ssh11:20:120
6608992613,6cyclictest0-21swapper/309:25:203
660899261,14cyclictest36066irq/59-eth009:10:133
660899261,14cyclictest36066irq/59-eth009:10:123
659999264,7cyclictest6560-21cyclictest10:40:132
659999264,7cyclictest6560-21cyclictest10:40:122
659999264,16cyclictest27346irq/40-dwc2_hso11:50:112
659999263,15cyclictest32358-21ssh09:50:102
6599992614,5cyclictest0-21swapper/208:30:282
658499262,19cyclictest14362-21ssh09:40:141
657399263,14cyclictest0-21swapper/008:30:140
657399262,20cyclictest6769-21ssh12:00:140
6573992613,8cyclictest12-21ksoftirqd/010:00:120
657399261,17cyclictest15692-21ssh12:05:110
660899251,11cyclictest35966irq/58-eth011:25:153
6599992514,8cyclictest0-21swapper/212:23:252
659999251,19cyclictest0-21swapper/211:35:132
659999251,19cyclictest0-21swapper/211:35:122
659999251,12cyclictest19535-21ssh10:55:132
659999251,12cyclictest19535-21ssh10:55:122
658499255,12cyclictest6560-21cyclictest10:55:121
658499255,12cyclictest6560-21cyclictest10:55:111
657399259,5cyclictest12-21ksoftirqd/010:50:130
657399253,18cyclictest12-21ksoftirqd/009:25:140
659999246,9cyclictest8995-21apt-get09:55:142
659999246,9cyclictest8995-21apt-get09:55:132
659999242,13cyclictest27346irq/40-dwc2_hso07:50:132
6599992413,8cyclictest0-21swapper/211:57:142
6599992413,4cyclictest0-21swapper/208:05:282
6599992410,6cyclictest0-21swapper/207:40:262
6584992415,6cyclictest0-21swapper/108:05:021
660899234,8cyclictest35966irq/58-eth012:20:113
660899233,14cyclictest0-21swapper/312:10:163
660899232,17cyclictest35966irq/58-eth008:55:133
659999239,2cyclictest131rcu_preempt11:10:592
659999237,2cyclictest131rcu_preempt10:16:362
659999232,15cyclictest0-21swapper/207:45:052
659999232,15cyclictest0-21swapper/207:45:042
6599992312,9cyclictest2286-21ssh10:08:522
659999231,17cyclictest2879166ssh09:10:232
659999231,17cyclictest2879166ssh09:10:222
659999230,5cyclictest28276-21find12:30:112
658499235,12cyclictest0-21swapper/109:15:151
658499235,12cyclictest0-21swapper/109:15:141
658499231,19cyclictest19398-21cron09:25:011
657399239,11cyclictest30247-21ssh11:55:130
657399236,11cyclictest0-21swapper/008:40:140
657399235,11cyclictest0-21swapper/007:40:140
657399233,14cyclictest12-21ksoftirqd/008:15:130
660899224,11cyclictest35966irq/58-eth010:10:123
6608992214,5cyclictest35966irq/58-eth012:25:143
660899221,13cyclictest35966irq/58-eth009:55:153
660899221,13cyclictest35966irq/58-eth009:55:143
659999226,3cyclictest10378-21timerandwakeup07:10:292
659999224,10cyclictest27346irq/40-dwc2_hso09:00:142
659999222,17cyclictest14310-21/usr/sbin/munin09:40:292
659999222,16cyclictest12814-21ntp_states08:20:272
6599992215,4cyclictest1046-21ssh12:15:022
6599992212,8cyclictest0-21swapper/208:45:162
659999221,11cyclictest3662-21kthreadcore12:15:212
6599992210,3cyclictest131rcu_preempt11:02:322
658499220,18cyclictest5463-21grep11:05:161
658499220,17cyclictest28317-21ssh12:30:141
6573992213,6cyclictest11479-21sshd10:30:290
6573992212,8cyclictest23050-21ssh09:45:020
6608992111,8cyclictest0-21swapper/310:45:313
6608992111,8cyclictest0-21swapper/310:45:303
659999212,15cyclictest28907-21awk09:30:132
659999211,5cyclictest11664-21fschecks_time07:15:152
6599992114,4cyclictest310-21ssh10:25:222
659999211,17cyclictest30663-21grep07:35:022
659999211,17cyclictest30663-21grep07:35:012
659999211,16cyclictest32335-21kthreadcore07:35:212
6599992111,4cyclictest0-21swapper/208:25:172
659999211,12cyclictest1680-21kthreadcore08:10:202
659999211,10cyclictest5851-21fschecks_count08:50:172
659999211,10cyclictest24260-21kthreadcore08:00:192
658499212,14cyclictest13041-21rm11:45:191
657399215,9cyclictest0-21swapper/009:35:130
657399213,10cyclictest0-21swapper/008:55:120
657399211,7cyclictest1745-21ntp_192.168.11410:25:260
657399211,17cyclictest22958-21latency10:55:220
657399211,17cyclictest22958-21latency10:55:220
657399211,15cyclictest27073-21ssh10:05:150
657399211,11cyclictest21574-21apt-get10:20:110
660899206,6cyclictest0-21swapper/308:25:133
660899205,13cyclictest35966irq/58-eth007:10:133
660899201,13cyclictest35966irq/58-eth010:50:133
6608992010,7cyclictest35966irq/58-eth011:35:113
6608992010,7cyclictest35966irq/58-eth011:35:103
660899200,13cyclictest35966irq/58-eth011:55:133
659999204,7cyclictest11429-21kthreadcore11:25:232
658499205,2cyclictest20278-21apt-get09:05:141
658499203,11cyclictest0-21swapper/109:45:121
6584992013,4cyclictest5558-21hwlatdetect09:35:151
6584992010,8cyclictest8064-21apt-get07:45:131
6584992010,8cyclictest8064-21apt-get07:45:121
657399207,9cyclictest12-21ksoftirqd/009:00:130
657399207,3cyclictest131rcu_preempt08:05:250
657399206,4cyclictest12-21ksoftirqd/009:50:000
657399204,5cyclictest12-21ksoftirqd/011:20:020
6573992012,6cyclictest9020-21ssh09:55:140
6573992012,6cyclictest9020-21ssh09:55:130
660899198,8cyclictest0-21swapper/309:41:393
660899197,9cyclictest0-21swapper/310:05:253
660899193,12cyclictest35966irq/58-eth010:20:123
660899193,11cyclictest0-21swapper/312:36:323
660899193,10cyclictest0-21swapper/309:45:123
660899193,10cyclictest0-21swapper/308:05:053
660899192,11cyclictest0-21swapper/307:15:143
660899191,12cyclictest0-21swapper/307:50:023
660899191,12cyclictest0-21swapper/307:50:023
660899191,10cyclictest35966irq/58-eth009:50:123
6608991910,6cyclictest0-21swapper/311:20:153
6608991910,4cyclictest0-21swapper/309:15:133
6608991910,4cyclictest0-21swapper/309:15:123
658499193,9cyclictest0-21swapper/112:00:141
658499192,11cyclictest1492-21find10:45:111
658499192,11cyclictest1492-21find10:45:101
658499191,7cyclictest12907-21kthreadcore12:20:211
6584991912,5cyclictest15251-21apt-get12:05:041
6584991912,4cyclictest21240-21/usr/sbin/munin11:50:311
658499191,13cyclictest25941-21apt-get07:30:131
658499191,13cyclictest25941-21apt-get07:30:121
657399197,3cyclictest131rcu_preempt11:25:230
657399193,3cyclictest131rcu_preempt08:50:150
657399191,13cyclictest2967-21munin-run11:40:010
657399191,13cyclictest2967-21munin-run11:40:010
657399191,12cyclictest22290-21ls08:00:020
660899189,6cyclictest0-21swapper/309:32:143
660899182,10cyclictest35966irq/58-eth008:15:123
660899182,10cyclictest0-21swapper/308:45:143
6608991811,5cyclictest35966irq/58-eth010:35:213
660899181,10cyclictest35966irq/58-eth012:05:123
660899181,10cyclictest0-21swapper/307:45:023
6608991810,6cyclictest0-21swapper/312:16:523
6608991810,6cyclictest0-21swapper/311:32:483
6608991810,6cyclictest0-21swapper/311:19:373
6608991810,3cyclictest0-21swapper/310:16:423
659999181,9cyclictest6669-21cat07:10:012
658499189,6cyclictest0-21swapper/108:40:251
658499188,7cyclictest1386-21df_inode12:15:151
658499186,9cyclictest1975-21sed07:35:291
658499184,11cyclictest18135-21ps07:55:181
658499182,6cyclictest12719-21df_inode10:15:141
658499181,9cyclictest18237-21kthreadcore09:00:211
658499181,8cyclictest8164-21cut07:10:201
658499181,15cyclictest710-21cron08:25:011
6584991811,5cyclictest0-21swapper/107:25:171
658499181,10cyclictest6170-21cut12:35:201
6584991810,5cyclictest2807-21kthreadcore11:20:221
6584991810,5cyclictest25804-21irqrtprio09:10:181
6584991810,5cyclictest25804-21irqrtprio09:10:171
658499180,9cyclictest18287-21grep11:30:171
657399181,3cyclictest131rcu_preempt07:15:130
6573991811,5cyclictest25619-21ssh12:27:160
657399181,12cyclictest14848-21sed11:10:180
657399180,14cyclictest1792-21find09:15:140
657399180,14cyclictest1792-21find09:15:140
660899179,6cyclictest0-21swapper/310:32:093
660899179,6cyclictest0-21swapper/310:27:403
660899179,6cyclictest0-21swapper/309:21:373
660899172,9cyclictest0-21swapper/308:35:133
660899172,11cyclictest0-21swapper/311:10:133
660899172,10cyclictest0-21swapper/308:40:143
658499179,6cyclictest7740-21sshd07:44:291
658499179,6cyclictest7024-21ntpq11:40:251
658499179,6cyclictest23532-21cron08:35:011
658499174,10cyclictest14327-21wc08:55:281
658499172,12cyclictest8773-21ssh11:25:161
6584991712,3cyclictest32452-21irqcore08:10:191
6584991712,3cyclictest0-21swapper/107:15:231
658499171,14cyclictest7865-21missed_timers08:15:231
658499171,13cyclictest17360-21kthreadcore10:35:181
658499171,12cyclictest24817-21df_inode12:10:161
6584991710,5cyclictest25254-21hwlatdetect10:40:171
6584991710,5cyclictest25254-21hwlatdetect10:40:161
657399179,5cyclictest4790-21grep08:45:280
657399178,7cyclictest32219-21diskstats08:10:150
657399172,13cyclictest2169-21grep07:35:290
6573991712,3cyclictest9579-21idleruntime08:20:170
6573991710,5cyclictest7404-21ssh10:10:270
6573991710,5cyclictest28725-21ssh09:30:070
6573991710,4cyclictest21088-21cat07:25:020
660899162,10cyclictest0-21swapper/309:00:143
660899161,12cyclictest0-21swapper/308:00:153
658499167,7cyclictest1053-21entropy08:45:161
658499167,6cyclictest32395-21grep09:50:131
658499162,5cyclictest7022-21/usr/sbin/munin10:30:151
658499161,12cyclictest20273-21irqcore09:25:191
658499161,12cyclictest14261-21fschecks_count08:25:161
658499160,13cyclictest8378-21tr08:50:221
657399163,11cyclictest25628-21tr08:00:210
657399163,10cyclictest11716-21threads07:45:280
657399163,10cyclictest11716-21threads07:45:270
657399162,11cyclictest6848-21grep07:10:170
657399161,5cyclictest0-21swapper/008:30:020
657399161,12cyclictest24533-21awk07:25:240
660899152,8cyclictest0-21swapper/307:30:153
660899152,8cyclictest0-21swapper/307:30:143
660899152,10cyclictest0-21swapper/307:25:153
660899151,11cyclictest0-21swapper/307:55:183
658499151,11cyclictest23877-21cut08:35:161
658499151,11cyclictest12847-21apt-get07:50:151
660899142,9cyclictest0-21swapper/307:20:123
660899141,7cyclictest0-21swapper/308:20:183
660899141,7cyclictest0-21swapper/307:35:163
660899141,11cyclictest0-21swapper/307:50:233
658499141,10cyclictest6628-21latency_hist07:10:021
657399141,10cyclictest6593-21latency_hist07:10:000
660899131,5cyclictest0-21swapper/308:10:213
660899130,11cyclictest33-21ksoftirqd/308:30:193
660899127,3cyclictest0-21swapper/308:50:133
660899101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional