You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 19:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri Jun 26, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2378099643,49cyclictest17434-21ssh12:30:121
2377999637,9cyclictest131rcu_preempt10:30:110
2377999612,45cyclictest7838-21ssh10:20:110
2377999605,45cyclictest25005-21ssh11:05:130
2378099572,46cyclictest4424-21ssh12:05:131
2379599561,32cyclictest2723-21ssh10:35:132
2377999563,11cyclictest131rcu_preempt08:55:130
2378099552,8cyclictest131rcu_preempt09:00:121
2378099543,40cyclictest22566-21ssh12:15:151
2377999543,10cyclictest131rcu_preempt08:50:120
2379599521,36cyclictest9634-21ssh11:50:122
2378099511,38cyclictest27129-21ssh09:55:131
2378099511,38cyclictest27129-21ssh09:55:121
2379599502,27cyclictest0-21swapper/209:45:132
2378099501,38cyclictest525-21ssh11:45:111
2378099494,36cyclictest1667-21ssh11:10:131
23795994819,22cyclictest27-21ksoftirqd/210:30:112
2378099460,33cyclictest31083-21ssh10:15:121
2379599446,7cyclictest27-21ksoftirqd/209:25:132
2380399433,22cyclictest35966irq/58-eth009:45:133
23803994230,6cyclictest35966irq/58-eth011:45:133
2379599422,31cyclictest22551-21ssh12:15:132
2378099412,32cyclictest18680-21ssh11:55:101
2380399393,22cyclictest35966irq/58-eth009:15:123
2378099392,23cyclictest14811-21ssh11:35:131
2377999397,24cyclictest12-21ksoftirqd/008:25:130
2380399381,22cyclictest35966irq/58-eth012:00:133
2378099371,29cyclictest16940-21ssh10:25:121
23803993510,16cyclictest0-21swapper/310:30:103
2377999353,19cyclictest18056-21ssh09:50:130
2380399332,19cyclictest35966irq/58-eth010:00:133
2378099336,19cyclictest30099-21ssh10:50:131
2377999331,25cyclictest8969-21ssh09:45:140
2378099322,23cyclictest32316-21ssh09:40:131
2377999325,22cyclictest11937-21ssh10:40:120
2377999324,20cyclictest12-21ksoftirqd/008:20:120
2377999323,26cyclictest12-21ksoftirqd/011:15:120
23803993116,10cyclictest0-21swapper/311:30:143
23795993112,12cyclictest27-21ksoftirqd/210:40:132
2379599303,22cyclictest26573-21ssh12:35:132
2379599302,20cyclictest27346irq/40-dwc2_hso07:40:132
2377999304,17cyclictest12-21ksoftirqd/010:55:120
2380399292,18cyclictest0-21swapper/308:25:123
2377999295,18cyclictest151rcuc/011:30:130
23803992817,7cyclictest35966irq/58-eth012:10:143
23803992816,8cyclictest35966irq/58-eth009:50:143
2379599283,16cyclictest27346irq/40-dwc2_hso08:55:132
2379599282,6cyclictest10779-21rm11:15:132
2379599282,19cyclictest0-21swapper/210:00:152
2379599282,19cyclictest0-21swapper/208:20:122
2379599282,17cyclictest27346irq/40-dwc2_hso08:25:122
2379599282,15cyclictest27346irq/40-dwc2_hso09:15:132
2378099281,6cyclictest0-21swapper/108:05:151
23780992813,10cyclictest21-21ksoftirqd/107:30:151
23780992813,10cyclictest21-21ksoftirqd/107:30:141
2378099281,20cyclictest10046-21ssh09:10:141
2378099281,19cyclictest19912-21ssh11:20:141
2377999281,4cyclictest23881-21apt-get11:40:100
2380399275,14cyclictest0-21swapper/310:35:143
2379599273,17cyclictest27346irq/40-dwc2_hso08:15:132
2379599271,14cyclictest31647-21ssh12:20:152
2380399262,15cyclictest0-21swapper/310:10:123
2379599265,14cyclictest21982-21ssh10:10:132
2379599262,16cyclictest27346irq/40-dwc2_hso09:30:112
23795992612,6cyclictest29439-21sshd07:15:232
2380399255,12cyclictest0-21swapper/312:20:153
2380399251,17cyclictest0-21swapper/310:15:133
2379599253,10cyclictest24999-21ssh11:05:142
2379599251,9cyclictest32193-21ssh09:40:052
23795992511,12cyclictest9276-21sshd12:05:322
2377999253,17cyclictest19213-21sh09:15:140
2377999252,17cyclictest12-21ksoftirqd/009:05:120
23779992514,8cyclictest0-21swapper/011:50:120
2380399246,10cyclictest35966irq/58-eth010:45:153
23795992414,7cyclictest0-21swapper/212:10:232
2379599241,20cyclictest5333-21latency_hist11:30:022
2379599241,16cyclictest5790-21ls07:25:012
23779992416,6cyclictest14059-21ssh09:30:110
23779992413,3cyclictest12-21ksoftirqd/011:20:320
2377999241,19cyclictest32613-21find09:00:100
2380399234,11cyclictest0-21swapper/309:30:123
2379599234,7cyclictest131rcu_preempt10:15:112
2379599231,13cyclictest728-21fschecks_count11:45:162
23795992311,3cyclictest1655-21apt11:10:122
2377999235,13cyclictest26557-21ssh12:35:120
2377999232,16cyclictest27765-21ssh12:00:130
2377999231,18cyclictest25861-21ls10:30:020
2377999231,18cyclictest21080-21ssh10:45:150
2380399228,10cyclictest0-21swapper/310:50:313
2380399224,11cyclictest0-21swapper/310:20:113
2380399223,15cyclictest0-21swapper/311:00:133
2380399223,12cyclictest0-21swapper/312:30:123
23803992215,4cyclictest0-21swapper/311:25:283
23803992213,6cyclictest35966irq/58-eth011:10:213
23803992211,7cyclictest0-21swapper/311:39:363
23803992210,6cyclictest0-21swapper/309:40:123
2379599225,10cyclictest131rcu_preempt09:10:312
2379599224,15cyclictest30847-21rm10:50:192
2379599222,7cyclictest25140-21rm12:33:552
2379599222,16cyclictest27346irq/40-dwc2_hso10:45:132
23795992211,9cyclictest16064-21ssh10:24:282
23795992211,5cyclictest25378-21ssh09:53:252
23780992211,8cyclictest9082-21grep09:25:291
2377999221,17cyclictest30105-21ssh10:50:120
2380399219,7cyclictest0-21swapper/311:40:113
2380399217,10cyclictest35966irq/58-eth011:55:113
2380399212,14cyclictest35966irq/58-eth007:30:143
2380399212,14cyclictest35966irq/58-eth007:30:133
2380399212,11cyclictest0-21swapper/311:05:133
2380399211,10cyclictest0-21swapper/312:35:293
2379599215,6cyclictest27-21ksoftirqd/212:00:132
2379599215,13cyclictest15919-21tail08:05:312
2379599214,14cyclictest27346irq/40-dwc2_hso10:55:132
2379599213,12cyclictest23907-21ssh11:40:112
2379599211,4cyclictest0-21swapper/208:50:132
23795992113,6cyclictest24671-21tr07:10:182
2379599211,17cyclictest5135-21cut08:30:212
2379599211,17cyclictest18234-21ssh11:00:212
2379599211,11cyclictest3244866ssh09:56:032
2379599211,11cyclictest3244866ssh09:56:022
2379599211,10cyclictest16904-21timerwakeupswit10:05:272
23795992110,9cyclictest21817-21ssh10:25:312
2378099212,13cyclictest28330-21ssh09:20:131
2378099212,13cyclictest0-21swapper/107:20:141
2378099211,5cyclictest4098-21users12:20:311
2378099211,17cyclictest13047-21uniq10:05:151
2378099211,16cyclictest6172-21ls07:55:291
2377999217,3cyclictest131rcu_preempt11:45:280
2377999211,4cyclictest131rcu_preempt12:05:130
23779992113,6cyclictest4621-21cron09:25:000
23779992112,7cyclictest3536-21ssh09:40:250
23779992112,6cyclictest5760-21cron07:25:020
2377999211,15cyclictest2846-21ssh12:20:250
2380399209,8cyclictest35966irq/58-eth009:55:133
2380399209,8cyclictest35966irq/58-eth009:55:133
2380399207,9cyclictest35966irq/58-eth010:25:133
2380399203,13cyclictest35966irq/58-eth009:25:133
23803992011,6cyclictest0-21swapper/309:20:213
23803992011,5cyclictest35966irq/58-eth011:20:143
2380399201,11cyclictest0-21swapper/307:15:193
2379599209,9cyclictest19762-21ssh11:20:042
2379599206,12cyclictest27-21ksoftirqd/209:20:292
2379599203,14cyclictest27346irq/40-dwc2_hso11:30:132
2379599201,15cyclictest30416-21sed07:50:162
2379599201,15cyclictest18690-21ssh11:55:122
2379599201,11cyclictest12323-21tail08:35:282
2379599201,10cyclictest0-21swapper/209:35:122
2378099206,10cyclictest8395-21diskmemload12:25:141
2378099202,14cyclictest28953-21ssh11:25:121
2378099201,4cyclictest0-21swapper/110:42:071
2378099201,17cyclictest9663-21ls07:25:261
23780992011,7cyclictest29851-21sshd07:49:221
2377999206,11cyclictest31089-21ssh10:15:120
2377999203,9cyclictest0-21swapper/010:00:130
2377999203,12cyclictest0-21swapper/007:40:120
2377999201,11cyclictest29177-21sed07:45:290
2380399193,11cyclictest0-21swapper/312:25:133
2380399191,13cyclictest0-21swapper/307:35:273
2380399191,13cyclictest0-21swapper/307:35:263
2380399191,12cyclictest35966irq/58-eth011:50:123
2380399191,12cyclictest0-21swapper/308:15:223
23803991910,7cyclictest0-21swapper/309:10:143
2379599192,8cyclictest12752-21kthreadcore07:30:212
2379599192,8cyclictest12752-21kthreadcore07:30:212
2379599192,10cyclictest20404-21kthreadcore08:45:212
2379599191,9cyclictest7396-21kthreadcore07:25:192
2379599191,9cyclictest16662-21sh11:35:222
2379599191,9cyclictest10381-21kthreadcore12:25:212
2379599191,8cyclictest25704-21irqcore07:45:192
2379599191,8cyclictest15861-21if_eth007:35:162
2379599191,8cyclictest15861-21if_eth007:35:162
2379599191,3cyclictest131rcu_preempt07:55:142
23795991913,4cyclictest0-21swapper/208:40:122
2379599191,11cyclictest3600-21sshd09:00:232
2379599191,11cyclictest16900-21apt-get08:10:122
2379599191,10cyclictest5771-21kthreadcore09:05:172
2379599191,10cyclictest23826-21latency_hist07:10:022
2379599191,10cyclictest23826-21latency_hist07:10:012
2378099194,12cyclictest15771-21fschecks_time07:35:161
2378099194,12cyclictest15771-21fschecks_time07:35:151
2378099192,15cyclictest1179-21ps07:50:261
2378099191,6cyclictest17649-21munin-run09:50:011
23780991912,5cyclictest7589-21kthreadcore11:30:231
23780991911,5cyclictest9640-21diskmemload10:55:281
2378099191,15cyclictest710-21cron09:55:011
2378099191,15cyclictest28937-21uniq07:15:191
23780991911,5cyclictest20983-21kworker/1:211:40:121
23780991911,5cyclictest0-21swapper/108:40:241
2378099191,10cyclictest7877-21grep10:00:301
23780991910,6cyclictest20777-21kthreadcore09:15:191
2377999199,7cyclictest15198-21ps11:35:190
2377999197,5cyclictest29137-21kthreadcore09:55:250
2377999197,5cyclictest29137-21kthreadcore09:55:240
23779991913,4cyclictest0-21swapper/012:25:110
23779991912,5cyclictest13784-21ssh09:10:270
23779991912,5cyclictest10141-21ssh11:14:450
23779991912,5cyclictest0-21swapper/008:10:230
23779991911,6cyclictest30483-21ssh10:14:470
2377999191,16cyclictest22136-21ls11:55:220
23779991911,6cyclictest13156-21ssh09:29:250
23779991911,6cyclictest10440-21ssh10:38:520
2377999191,14cyclictest12831-21apt-get10:05:110
2377999191,12cyclictest32408-21ls07:15:320
2380399189,4cyclictest0-21swapper/312:18:513
2380399183,12cyclictest35966irq/58-eth009:35:113
2380399182,8cyclictest0-21swapper/310:40:133
2380399182,10cyclictest0-21swapper/308:50:133
2380399182,10cyclictest0-21swapper/308:35:053
2380399182,10cyclictest0-21swapper/308:30:133
2380399182,10cyclictest0-21swapper/307:40:033
2380399181,13cyclictest0-21swapper/308:25:023
2380399181,12cyclictest35966irq/58-eth010:05:243
23803991810,5cyclictest35966irq/58-eth011:15:123
2379599181,13cyclictest7277-21apt-get08:00:122
2378099189,6cyclictest11864-21wc08:35:231
2378099184,8cyclictest0-21swapper/108:45:141
2378099182,13cyclictest15849-21ssh11:00:131
2378099181,9cyclictest23772-21cut10:45:231
23780991811,5cyclictest0-21swapper/112:35:231
23780991811,5cyclictest0-21swapper/109:35:261
2378099181,14cyclictest27887-21grep07:10:291
2378099181,11cyclictest7443-21ls08:30:291
2378099181,10cyclictest11070-21ls11:15:161
2377999187,9cyclictest6136-21kworker/u8:011:00:120
2377999182,13cyclictest12089-21apt-get08:05:140
2377999181,8cyclictest2804-21irqcore07:55:200
23779991811,5cyclictest25913-21ssh12:34:490
23779991810,5cyclictest24551-21kthreadcore12:15:210
2380399179,5cyclictest35966irq/58-eth012:05:133
2380399174,8cyclictest35966irq/58-eth008:45:133
2380399172,9cyclictest0-21swapper/309:05:133
2380399172,11cyclictest35966irq/58-eth010:55:113
2380399172,10cyclictest0-21swapper/308:10:143
2380399171,12cyclictest0-21swapper/308:40:293
2378099179,5cyclictest30385-21head08:20:281
2378099178,7cyclictest31515-21fw_packets08:25:171
2378099177,7cyclictest25301-21basename11:05:161
2378099176,9cyclictest11449-21cron10:40:001
2378099173,12cyclictest15176-21tr12:10:201
2378099173,12cyclictest11263-21ssh11:50:191
2378099173,11cyclictest23994-21ps10:10:201
2378099171,9cyclictest20217-21memory08:10:231
2378099171,6cyclictest9641-21munin-run09:10:011
2378099171,14cyclictest9594-21kthreadcore10:20:191
2378099171,13cyclictest26029-21apt-get10:30:111
2378099171,13cyclictest20160-21apt-get07:40:031
2378099171,12cyclictest23003-21apt-get08:50:111
2378099171,12cyclictest11123-21grep08:00:291
23780991710,4cyclictest14345-21idleruntime09:30:161
2377999179,5cyclictest30253-21cut11:25:180
2377999173,11cyclictest11794-21grep08:35:210
2377999172,11cyclictest13657-21/usr/sbin/munin12:10:160
2377999171,8cyclictest9832-21kthreadcore08:00:200
23779991710,3cyclictest131rcu_preempt08:15:110
2380399162,9cyclictest0-21swapper/308:55:143
2380399161,12cyclictest0-21swapper/307:10:283
2378099168,6cyclictest26255-21sshd08:19:091
2378099163,9cyclictest31424-21switchtime08:55:271
2378099163,10cyclictest29640-21kthreadcore12:00:211
2377999169,5cyclictest29948-21ssh09:37:510
2377999166,7cyclictest16127-21cut08:40:220
2377999164,9cyclictest18369-21kthreadcore07:35:210
2377999164,9cyclictest18369-21kthreadcore07:35:200
2377999163,10cyclictest11104-21irqcore07:30:190
2377999163,10cyclictest11104-21irqcore07:30:180
2377999162,10cyclictest823-21kthreadcore07:50:200
2377999161,12cyclictest20783-21tr08:45:220
2377999160,13cyclictest26281-21kthreadcore07:10:200
2380399152,9cyclictest0-21swapper/307:55:133
2380399152,10cyclictest0-21swapper/309:00:123
2377999156,7cyclictest23778-21cyclictest07:10:010
2377999156,7cyclictest23778-21cyclictest07:10:000
2377999152,10cyclictest0-21swapper/008:30:140
2377999151,10cyclictest6220-21grep07:25:140
2380399149,3cyclictest0-21swapper/307:22:093
2380399141,10cyclictest0-21swapper/307:50:133
2380399141,10cyclictest0-21swapper/307:45:163
2380399141,10cyclictest0-21swapper/307:25:143
2378099141,10cyclictest23945-21latency_hist07:10:021
2378099141,10cyclictest23945-21latency_hist07:10:011
2380399133,7cyclictest0-21swapper/308:00:183
2380399132,9cyclictest0-21swapper/308:05:153
2380399111,7cyclictest0-21swapper/307:10:023
2380399111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional