You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 17:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Jul 13, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1505699543,43cyclictest22805-21ssh09:40:140
1507599539,24cyclictest27-21ksoftirqd/212:05:182
1506599521,3cyclictest0-21swapper/107:35:131
15083995123,17cyclictest35966irq/58-eth012:00:123
1507599501,38cyclictest28290-21ssh09:25:142
1506599501,4cyclictest0-21swapper/109:05:131
1506599501,4cyclictest0-21swapper/109:05:131
1507599494,29cyclictest11665-21ssh11:40:132
15075994931,11cyclictest27-21ksoftirqd/209:20:132
15075994919,15cyclictest27-21ksoftirqd/212:00:122
1506599481,42cyclictest0-21swapper/111:30:131
1506599471,4cyclictest0-21swapper/108:40:111
1506599461,12cyclictest15273-21apt-get07:10:121
1506599441,3cyclictest13595-21ssh11:05:131
1506599441,2cyclictest0-21swapper/110:35:121
1506599434,27cyclictest8210-21ssh11:20:131
1506599431,3cyclictest17797-21apt-get08:20:141
1506599431,3cyclictest17797-21apt-get08:20:141
1506599431,3cyclictest0-21swapper/111:35:111
1506599430,2cyclictest0-21swapper/108:45:141
1506599411,4cyclictest0-21swapper/107:30:131
1506599411,3cyclictest0-21swapper/109:00:141
1506599411,3cyclictest0-21swapper/108:25:131
1506599401,5cyclictest0-21swapper/108:35:121
1506599401,4cyclictest0-21swapper/109:15:141
1505699402,32cyclictest28274-21ssh09:25:130
1508399392,21cyclictest35966irq/58-eth011:15:143
1508399392,21cyclictest35966irq/58-eth011:15:143
1506599392,4cyclictest24087-21apt-get12:05:181
1506599391,2cyclictest17120-21apt-get11:25:131
15056993922,8cyclictest17464-21ssh09:55:150
15075993822,9cyclictest27-21ksoftirqd/210:45:152
1506599382,33cyclictest0-21swapper/107:55:141
1506599382,33cyclictest0-21swapper/107:55:141
1506599381,5cyclictest30991-21apt-get08:00:151
1506599381,5cyclictest12974-21apt-get08:15:141
1506599381,4cyclictest29537-21apt-get11:50:161
1506599381,4cyclictest29537-21apt-get11:50:161
1506599380,4cyclictest27414-21apt-get08:30:121
1506599380,3cyclictest0-21swapper/109:45:141
1506599380,3cyclictest0-21swapper/109:45:141
1506599380,2cyclictest4865-21ssh09:30:121
1506599371,8cyclictest730866ssh11:19:261
1506599371,8cyclictest730866ssh11:19:261
1506599370,7cyclictest22825-21ssh09:40:151
1506599370,5cyclictest11657-21apt-get11:40:121
1508399362,26cyclictest35966irq/58-eth010:45:163
1506599363,30cyclictest2267-21tr10:40:181
1506599362,3cyclictest3353-21apt-get08:05:141
1506599362,2cyclictest8189-21apt-get08:10:121
1506599361,4cyclictest0-21swapper/110:10:131
1506599361,4cyclictest0-21swapper/107:45:151
1506599361,3cyclictest0-21swapper/109:50:121
1506599361,3cyclictest0-21swapper/109:35:111
1505699362,27cyclictest19260-21ssh09:20:130
1506599353,5cyclictest0-21swapper/107:20:121
1506599352,7cyclictest20572-21kthreadcore09:55:231
1506599351,11cyclictest18754-21ntp_states12:00:251
15056993513,17cyclictest12830-21diskmemload12:35:140
1506599341,7cyclictest30250-21cut07:25:191
1506599341,5cyclictest20607-21apt-get11:45:141
1506599341,3cyclictest28263-21apt-get09:25:131
1506599341,24cyclictest28036-21ssh10:55:141
1506599341,10cyclictest30917-21ssh10:20:191
1506599332,2cyclictest19253-21apt-get09:20:131
1506599331,9cyclictest1931-21irqstats09:10:191
1506599331,8cyclictest16483-21ntp_states12:35:271
1506599331,2cyclictest0-21swapper/110:50:131
1506599331,29cyclictest22693-21kthreadcore07:15:241
1506599331,11cyclictest10195-21ntp_192.168.11410:25:261
1506599331,11cyclictest10195-21ntp_192.168.11410:25:261
1506599331,10cyclictest695-21diskstats12:10:151
1506599330,7cyclictest9651-21nfs4_client11:55:251
1506599330,7cyclictest9651-21nfs4_client11:55:251
1505699331,28cyclictest15052-21ssh12:00:110
1508399324,19cyclictest35966irq/58-eth011:55:143
1508399324,19cyclictest35966irq/58-eth011:55:143
1508399323,15cyclictest35966irq/58-eth011:00:133
15083993217,9cyclictest35966irq/58-eth010:50:123
1506599323,6cyclictest20572-21kthreadcore08:55:211
1506599321,9cyclictest4765-21/usr/sbin/munin11:00:161
1506599321,9cyclictest24790-21ntp_states07:50:271
1506599321,8cyclictest14713-21kthreadcore07:40:211
1506599321,7cyclictest17685-21cut08:50:271
1506599321,6cyclictest13733-21ntp_kernel_err10:45:281
1506599320,9cyclictest7575-21ntp_states12:30:281
1506599320,6cyclictest31204-21swap12:25:281
1505699322,26cyclictest4815-21ssh09:10:260
1505699322,25cyclictest26481-21ssh10:00:140
1508399313,15cyclictest35966irq/58-eth009:20:133
1508399312,15cyclictest35966irq/58-eth010:15:153
1507599312,22cyclictest12831-21ssh12:35:142
1506599311,8cyclictest25493-21cat10:15:331
1506599311,7cyclictest19156-21ntp_192.168.11410:30:241
1506599311,7cyclictest18856-21irqcore12:20:181
1506599311,4cyclictest23162-21irqcore11:10:161
1506599311,2cyclictest0-21swapper/110:00:141
1505699313,26cyclictest15519-21ssh10:30:120
15083993015,10cyclictest35966irq/58-eth009:45:143
15083993015,10cyclictest35966irq/58-eth009:45:143
1507599303,10cyclictest0-21swapper/208:05:142
1506599306,22cyclictest13128-21ntp_kernel_err12:15:261
1506599301,3cyclictest0-21swapper/110:05:171
1508399297,13cyclictest35966irq/58-eth010:05:143
1508399297,10cyclictest35966irq/58-eth011:05:133
1508399296,16cyclictest0-21swapper/312:20:143
1507599295,21cyclictest13604-21ssh11:05:132
1507599293,20cyclictest0-21swapper/208:10:132
1507599292,20cyclictest26113-21ssh11:30:122
1507599291,16cyclictest20634-21ssh11:45:152
1505699293,22cyclictest31672-21ssh11:15:140
1505699293,22cyclictest31672-21ssh11:15:140
15083992814,8cyclictest35966irq/58-eth009:25:143
1507599282,15cyclictest27346irq/40-dwc2_hso10:30:132
1505699286,19cyclictest11660-21ssh11:40:120
1505699282,23cyclictest22673-21ssh11:10:110
15056992820,5cyclictest0-21swapper/012:05:180
1505699281,19cyclictest29558-21sh11:50:160
1505699281,19cyclictest29558-21sh11:50:160
15083992710,13cyclictest35966irq/58-eth009:40:153
1507599272,20cyclictest29557-21ssh11:50:162
1507599272,20cyclictest29557-21ssh11:50:162
1505699277,10cyclictest12-21ksoftirqd/008:10:110
1505699271,19cyclictest1078-21ssh10:40:120
1505699271,18cyclictest10234-21ssh09:15:150
1508399266,10cyclictest35966irq/58-eth010:30:123
1507599262,9cyclictest0-21swapper/207:10:132
1507599262,16cyclictest27346irq/40-dwc2_hso08:15:152
1507599261,21cyclictest31746-21ls11:15:152
1507599261,21cyclictest31746-21ls11:15:152
1507599261,17cyclictest18434-21ssh12:20:122
15075992611,12cyclictest0-21swapper/207:50:162
1505699263,16cyclictest12-21ksoftirqd/007:15:160
1508399255,12cyclictest0-21swapper/309:30:133
1508399254,13cyclictest33-21ksoftirqd/310:25:133
1508399254,13cyclictest33-21ksoftirqd/310:25:133
1508399253,15cyclictest0-21swapper/312:35:143
1507599252,18cyclictest8225-21ssh11:20:142
1507599252,15cyclictest27346irq/40-dwc2_hso11:25:132
1507599251,17cyclictest12192-21grep10:10:172
1507599251,15cyclictest9543-21ssh12:15:152
1507599250,17cyclictest31515-21ls10:55:242
1505699256,16cyclictest24506-21ssh10:35:130
1505699256,13cyclictest13786-21sh09:35:110
15056992512,3cyclictest131rcu_preempt10:55:160
1508399244,14cyclictest0-21swapper/310:10:143
1508399243,15cyclictest0-21swapper/310:40:123
1507599244,17cyclictest0-21swapper/207:40:122
1507599243,12cyclictest930-21snmpd11:55:142
1507599243,12cyclictest930-21snmpd11:55:142
15075992417,5cyclictest11587-21ssh10:10:012
15075992416,6cyclictest10214-21ssh09:15:142
1507599241,14cyclictest24996-21apt-get07:20:122
1507599241,11cyclictest9213-21sed09:50:192
1507599240,17cyclictest14166-21find08:50:122
1505699241,16cyclictest27381-21sh12:25:140
1508399234,15cyclictest35966irq/58-eth012:30:143
15083992312,7cyclictest35966irq/58-eth010:20:133
1508399231,10cyclictest35966irq/58-eth008:40:113
1508399230,18cyclictest35966irq/58-eth010:35:133
1507599234,14cyclictest13804-21ssh09:35:132
1507599233,10cyclictest26534-21cut10:35:202
1507599232,14cyclictest27346irq/40-dwc2_hso08:30:122
1507599232,14cyclictest27346irq/40-dwc2_hso08:20:132
1507599232,14cyclictest27346irq/40-dwc2_hso08:20:132
15075992311,10cyclictest30763-21ssh10:00:352
1507599230,6cyclictest6180-21ls10:25:022
1506599230,4cyclictest15055-21cyclictest07:10:011
1505699231,2cyclictest0-21swapper/009:45:120
1505699231,2cyclictest0-21swapper/009:45:120
15056992310,7cyclictest12-21ksoftirqd/011:05:120
1508399225,10cyclictest35966irq/58-eth009:15:163
1508399223,12cyclictest0-21swapper/312:15:153
1508399222,17cyclictest35966irq/58-eth011:10:113
1508399222,14cyclictest0-21swapper/311:45:153
1508399220,13cyclictest35966irq/58-eth009:10:133
1507599227,12cyclictest32692-21sed08:35:182
1507599225,13cyclictest29025-21kworker/2:109:30:122
1507599225,13cyclictest27346irq/40-dwc2_hso08:55:142
1507599223,10cyclictest2802-21ssh12:10:222
1507599221,9cyclictest7377-21kthreadcore11:00:232
15075992211,9cyclictest18167-21ssh09:55:202
1507599221,19cyclictest0-21swapper/210:25:182
1507599221,19cyclictest0-21swapper/210:25:182
1507599221,12cyclictest6407-21kthreadcore12:30:242
1507599221,12cyclictest516-21ssh09:45:202
1507599221,12cyclictest516-21ssh09:45:202
1507599221,11cyclictest28115-21kthreadcore12:25:202
1505699222,15cyclictest20633-21ssh11:45:150
15083992112,6cyclictest35966irq/58-eth011:50:143
15083992112,6cyclictest35966irq/58-eth011:50:143
1507599219,9cyclictest1910-21sshd07:27:312
1507599217,11cyclictest8394-21unixbench_multi08:40:272
1507599215,8cyclictest25111-21kthreadcore11:10:202
1507599215,12cyclictest0-21swapper/211:35:212
1507599214,11cyclictest0-21swapper/207:30:132
1507599213,9cyclictest22532-21kthreadcore10:15:222
1507599211,14cyclictest1210-21ssh09:10:132
1505699212,11cyclictest4651-21ssh11:00:140
15056992113,5cyclictest1-21systemd11:30:260
15056992112,3cyclictest131rcu_preempt11:55:310
15056992112,3cyclictest131rcu_preempt11:55:310
1505699211,14cyclictest18990-21ssh10:50:120
1508399203,12cyclictest0-21swapper/310:55:143
1508399202,11cyclictest0-21swapper/311:20:143
15083992011,6cyclictest0-21swapper/312:10:313
15083992011,5cyclictest35966irq/58-eth012:25:143
1508399201,13cyclictest0-21swapper/312:05:353
15083992010,4cyclictest321rcuc/310:00:143
1507599209,8cyclictest0-21swapper/207:35:292
1507599207,9cyclictest22989-21fschecks_count09:40:172
1507599204,11cyclictest0-21swapper/208:45:142
1507599202,9cyclictest20762-21kthreadcore07:15:212
1507599202,15cyclictest1022-21/usr/sbin/munin10:40:122
1507599202,10cyclictest992-21latency_hist09:10:012
1507599202,10cyclictest992-21latency_hist09:10:012
1507599202,10cyclictest26347-21kthreadcore09:00:222
1507599201,16cyclictest22770-21fschecks_time08:25:172
1507599201,10cyclictest20366-21grep07:45:292
1505699205,12cyclictest3880-21ssh12:30:160
15056992012,5cyclictest12168-21open_inodes09:50:270
1505699201,15cyclictest9656-21ssh12:15:170
1505699201,15cyclictest11962-21ssh10:10:130
1505699200,16cyclictest3630-21cut10:05:200
1505699200,12cyclictest14052-21ls08:50:010
1508399197,8cyclictest35966irq/58-eth008:35:123
1508399192,9cyclictest0-21swapper/309:00:133
1508399192,11cyclictest0-21swapper/308:45:153
1508399192,10cyclictest0-21swapper/308:50:143
15083991911,6cyclictest0-21swapper/309:35:273
15083991911,4cyclictest0-21swapper/311:35:183
1508399191,13cyclictest35966irq/58-eth011:30:143
1508399191,12cyclictest0-21swapper/308:55:293
1508399191,12cyclictest0-21swapper/307:40:273
15083991910,7cyclictest0-21swapper/311:40:093
15083991910,6cyclictest35966irq/58-eth011:25:133
1507599193,8cyclictest15171-21latency_hist07:10:022
1507599191,9cyclictest31780-21kthreadcore08:00:202
1507599191,9cyclictest19228-21fschecks_time10:50:162
1507599191,10cyclictest26258-21fschecks_count07:55:172
1507599191,10cyclictest26258-21fschecks_count07:55:172
1505699199,8cyclictest15360-21df_inode07:10:140
1505699199,7cyclictest11741-21apt-get07:40:120
1505699197,3cyclictest131rcu_preempt08:00:150
1505699193,12cyclictest782-21diskmemload11:35:110
1505699192,11cyclictest0-21swapper/008:30:130
1505699192,10cyclictest20974-21ssh10:15:140
1505699191,9cyclictest0-21swapper/011:26:560
15056991913,3cyclictest30771-21idleruntime-cro08:00:010
15056991913,3cyclictest30771-21idleruntime-cro08:00:010
15056991911,5cyclictest0-21swapper/009:05:020
1508399182,10cyclictest0-21swapper/308:20:143
1508399182,10cyclictest0-21swapper/308:20:143
1508399182,10cyclictest0-21swapper/308:15:153
1508399182,10cyclictest0-21swapper/308:00:153
1508399182,10cyclictest0-21swapper/307:45:153
15083991811,5cyclictest0-21swapper/309:58:403
1508399181,13cyclictest0-21swapper/307:15:183
1508399181,12cyclictest0-21swapper/308:25:293
1508399181,12cyclictest0-21swapper/307:10:153
1505699187,3cyclictest131rcu_preempt09:30:190
1505699186,10cyclictest6709-21cron07:35:010
1505699184,9cyclictest29947-21acpi10:20:120
1505699184,3cyclictest131rcu_preempt12:20:180
1505699183,4cyclictest12-21ksoftirqd/007:49:370
1505699183,10cyclictest0-21swapper/007:20:040
1505699181,8cyclictest9555-21kthreadcore10:25:230
1505699181,8cyclictest9555-21kthreadcore10:25:230
1505699181,15cyclictest27245-21ls08:30:010
1505699181,14cyclictest3962-21kthreadcore12:10:230
15056991811,4cyclictest0-21swapper/008:35:290
1505699180,15cyclictest32269-21ls09:05:280
1505699180,15cyclictest32269-21ls09:05:280
1508399179,6cyclictest0-21swapper/309:52:033
1508399175,6cyclictest0-21swapper/307:50:163
1508399172,9cyclictest0-21swapper/309:05:053
1508399172,9cyclictest0-21swapper/309:05:053
1508399171,11cyclictest0-21swapper/308:05:293
1505699179,6cyclictest23505-21sshd08:57:440
1505699178,7cyclictest9540-21kthreadcore11:20:210
1505699172,10cyclictest0-21swapper/008:15:140
15056991712,3cyclictest0-21swapper/008:40:110
1505699171,11cyclictest11385-21cut10:45:230
1505699171,10cyclictest10265-21missed_timers07:35:220
1508399162,9cyclictest0-21swapper/308:30:143
1505699167,7cyclictest21262-21ntp_states08:20:260
1505699167,7cyclictest21262-21ntp_states08:20:260
1505699162,9cyclictest0-21swapper/008:05:140
1505699161,12cyclictest14181-21apt-get08:50:130
1508399158,5cyclictest0-21swapper/307:25:293
1508399152,10cyclictest0-21swapper/307:35:133
1505699152,9cyclictest12-21ksoftirqd/007:25:150
1505699151,11cyclictest24578-21latency07:50:240
1505699141,10cyclictest15069-21latency_hist07:10:010
1508399131,9cyclictest0-21swapper/307:55:133
1508399131,9cyclictest0-21swapper/307:55:133
1508399131,10cyclictest0-21swapper/307:30:183
1508399122,6cyclictest0-21swapper/307:20:143
1508399121,7cyclictest0-21swapper/307:10:023
1508399121,6cyclictest0-21swapper/308:10:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional