You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 21:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat May 02, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13733996035,17cyclictest0-21swapper/210:40:122
13733996035,17cyclictest0-21swapper/210:40:112
1373399599,16cyclictest131rcu_preempt07:55:122
1373399562,34cyclictest11029-21ssh10:10:112
13724995612,36cyclictest630-21Async09:36:311
1372499522,44cyclictest1905-21apt-get08:05:161
1371799494,6cyclictest131rcu_preempt10:20:120
13743994711,25cyclictest35966irq/58-eth009:55:123
1372499461,3cyclictest0-21swapper/109:05:131
1372499441,5cyclictest25911-21apt-get08:30:131
13733994312,8cyclictest131rcu_preempt12:35:152
1372499433,30cyclictest4923-21ssh12:30:121
1372499431,9cyclictest18016-21ssh09:20:151
13743994217,18cyclictest0-21swapper/311:04:553
1372499423,2cyclictest0-21swapper/112:35:151
1374399412,21cyclictest35966irq/58-eth011:20:133
13733994110,9cyclictest27-21ksoftirqd/211:10:122
1372499412,36cyclictest3666-21ssh09:30:141
1372499412,2cyclictest2006-21apt-get10:05:131
1372499411,2cyclictest0-21swapper/110:50:121
13743994018,11cyclictest35966irq/58-eth011:25:133
1373399401,24cyclictest1295-21ssh12:10:142
1372499401,3cyclictest0-21swapper/108:10:141
1372499401,33cyclictest29077-21ssh10:20:121
13733993913,19cyclictest27-21ksoftirqd/212:30:122
1372499392,6cyclictest0-21swapper/108:25:151
1372499391,3cyclictest0-21swapper/107:35:131
1372499391,2cyclictest22492-21apt-get11:10:121
1372499391,2cyclictest17259-21apt-get11:25:131
1372499383,3cyclictest559-21apt-get10:40:121
1372499383,3cyclictest559-21apt-get10:40:111
1372499381,4cyclictest0-21swapper/108:15:131
1372499381,3cyclictest0-21swapper/109:40:121
1372499381,24cyclictest20052-21sh10:15:141
1371799383,32cyclictest9655-21ssh10:45:140
13733993723,11cyclictest261rcuc/209:45:122
1373399371,27cyclictest930-21snmpd07:15:132
1372499372,4cyclictest0-21swapper/108:55:141
13724993721,9cyclictest30696-21sshd08:35:141
1372499371,4cyclictest0-21swapper/107:55:121
1371799373,31cyclictest23905-21ssh10:35:130
1373399363,24cyclictest27-21ksoftirqd/211:20:142
1373399362,29cyclictest25396-21ssh10:00:132
1373399352,27cyclictest7345-21ssh09:50:132
1372499352,30cyclictest8801-21kthreadcore09:50:181
1372499351,5cyclictest8197-21apt-get11:20:141
1372499351,2cyclictest25379-21apt-get10:00:111
1374399343,27cyclictest35966irq/58-eth010:25:143
13733993414,17cyclictest27-21ksoftirqd/211:25:132
1372499341,26cyclictest5706-21ssh10:25:131
1372499340,4cyclictest7889-21apt-get08:45:131
1372499340,12cyclictest23894-21apt-get10:35:121
13733993320,8cyclictest261rcuc/210:45:352
1372499331,9cyclictest4115-21cut08:40:191
1372499331,9cyclictest26390-21ssh11:30:141
1372499331,9cyclictest26390-21ssh11:30:131
1372499331,9cyclictest10930-21sed07:40:191
1372499331,8cyclictest8408-21cat11:00:281
1372499331,8cyclictest27956-21fschecks_count10:55:151
1372499331,7cyclictest31797-21kthreadcore09:45:201
1372499331,3cyclictest8879-21apt-get09:15:111
1372499331,10cyclictest30320-21ssh12:06:271
1372499331,10cyclictest10936-21kernelversion12:15:181
1374399324,17cyclictest35966irq/58-eth011:55:123
1374399324,17cyclictest35966irq/58-eth011:55:123
13743993224,5cyclictest35966irq/58-eth011:15:133
1373399328,15cyclictest13704-21cyclictest09:10:132
1373399322,25cyclictest29897-21rm12:05:552
1372499327,22cyclictest20985-21sort11:45:121
1372499321,9cyclictest23797-21/usr/sbin/munin07:50:291
1372499321,7cyclictest16419-21grep09:35:271
1372499321,7cyclictest13420-21processes10:45:281
1372499321,6cyclictest594366ssh09:12:001
1372499321,4cyclictest13880-21apt-get07:10:121
1372499321,3cyclictest0-21swapper/107:15:141
1372499321,10cyclictest27068-21ssh09:25:131
1372499321,10cyclictest13579-21ssh11:05:161
1372499320,11cyclictest16571-21fschecks_time09:55:151
1372499320,10cyclictest18208-21ssh11:42:101
1374399313,15cyclictest35966irq/58-eth011:30:123
1374399313,15cyclictest35966irq/58-eth011:30:123
13743993113,9cyclictest35966irq/58-eth011:40:133
1374399311,15cyclictest35966irq/58-eth009:45:123
1373399312,22cyclictest27-21ksoftirqd/210:25:392
1372499312,26cyclictest15403-21irqrtprio10:30:181
1372499311,9cyclictest15796-21ssh12:00:141
1372499311,9cyclictest15249-21cron12:00:011
1372499311,9cyclictest15249-21cron12:00:001
1372499311,9cyclictest15216-21cat10:10:271
1372499311,8cyclictest32143-21sed11:15:181
1372499311,8cyclictest30634-21sed11:50:181
1372499311,7cyclictest417-21cat08:00:231
1372499311,7cyclictest23571-21usage_Anybus12:20:291
1372499311,6cyclictest16497-21fschecks_time08:20:151
1372499311,4cyclictest0-21swapper/112:25:131
1372499311,3cyclictest0-21swapper/108:50:131
1372499310,9cyclictest4027-21ntpq07:30:251
1372499310,7cyclictest23690-21fschecks_time07:20:171
13717993124,3cyclictest12-21ksoftirqd/010:30:140
1374399304,15cyclictest35966irq/58-eth009:50:123
1374399302,22cyclictest35966irq/58-eth012:30:123
1373399305,15cyclictest27346irq/40-dwc2_hso09:15:112
1373399302,8cyclictest16361-21sh09:55:122
1373399302,22cyclictest0-21swapper/211:15:552
1373399302,17cyclictest28266-21apt-get07:25:132
13733993010,13cyclictest0-21swapper/208:10:152
1372499301,6cyclictest31721-21kworker/u8:211:37:341
1372499301,6cyclictest23702-21cut09:00:201
1372499301,3cyclictest1271-21apt-get12:10:121
1371799302,20cyclictest557-21ssh10:40:110
1371799302,20cyclictest557-21ssh10:40:110
1374399294,13cyclictest35966irq/58-eth012:10:133
1373399293,19cyclictest0-21swapper/211:50:542
1373399292,21cyclictest0-21swapper/207:10:142
1372499291,8cyclictest355-21awk07:30:011
1371799293,22cyclictest12-21ksoftirqd/010:15:150
1371799291,23cyclictest3661-21ssh09:30:130
1374399282,14cyclictest35966irq/58-eth010:40:113
1374399282,14cyclictest35966irq/58-eth010:40:113
1373399282,9cyclictest131rcu_preempt11:30:132
1373399282,9cyclictest131rcu_preempt11:30:122
1373399282,11cyclictest0-21swapper/211:42:542
1373399281,14cyclictest30697-21sshd08:35:132
1373399281,14cyclictest2035-21ssh10:05:132
1371799282,20cyclictest22369-21ssh11:25:480
1371799282,17cyclictest12-21ksoftirqd/011:30:120
1371799282,17cyclictest12-21ksoftirqd/011:30:120
1374399274,15cyclictest0-21swapper/311:45:143
1374399271,13cyclictest35966irq/58-eth009:30:143
1373399273,20cyclictest261rcuc/212:20:122
1373399271,16cyclictest27346irq/40-dwc2_hso11:55:132
1373399271,16cyclictest27346irq/40-dwc2_hso11:55:122
1373399271,12cyclictest286166ssh11:35:132
1373399271,11cyclictest9447-21ssh11:00:442
1371799273,18cyclictest30011-21ssh11:50:140
1373399263,13cyclictest14802-21sh10:30:152
1373399263,11cyclictest0-21swapper/209:20:142
13733992612,10cyclictest21295-21cron09:40:012
1373399261,19cyclictest27346irq/40-dwc2_hso12:00:132
1374399255,13cyclictest0-21swapper/312:35:143
1374399252,17cyclictest0-21swapper/312:05:473
1373399252,9cyclictest0-21swapper/208:45:132
1373399252,11cyclictest31373-21ssh10:55:232
1373399251,9cyclictest18436-21ssh11:05:392
1371799252,21cyclictest4361-21ssh11:15:540
1374399242,13cyclictest0-21swapper/311:10:123
13743992411,10cyclictest0-21swapper/311:50:153
1373399248,13cyclictest32638-21cron10:40:012
1373399242,17cyclictest27346irq/40-dwc2_hso10:50:132
1373399242,13cyclictest3685-21ssh09:30:152
1373399242,10cyclictest0-21swapper/207:30:132
1373399241,8cyclictest28697-21ssh11:48:562
1373399241,12cyclictest28438-21diskstats12:25:152
1372499244,3cyclictest13802-21latency_hist07:10:021
1371799242,17cyclictest9610-21ssh11:00:550
13717992416,5cyclictest0-21swapper/011:35:130
13743992314,5cyclictest0-21swapper/308:35:143
1373399233,4cyclictest10203-21ssh12:15:132
1373399231,9cyclictest7656-21sh08:45:012
1371799233,17cyclictest8222-21sh11:20:140
1371799232,17cyclictest0-21swapper/012:09:550
13717992317,3cyclictest0-21swapper/010:50:140
1371799231,19cyclictest12811-21ls09:35:140
1374399228,10cyclictest0-21swapper/310:47:553
1373399224,15cyclictest23341-21kthreadcore08:25:212
1373399222,15cyclictest27346irq/40-dwc2_hso09:25:142
1373399222,15cyclictest27346irq/40-dwc2_hso08:00:142
1373399222,15cyclictest0-21swapper/210:24:552
1373399221,5cyclictest21711-21ssh09:40:122
1373399221,17cyclictest16657-21irqcore08:20:182
1373399221,15cyclictest27346irq/40-dwc2_hso08:15:122
1373399221,15cyclictest20288-21cut10:15:172
1373399221,13cyclictest11738-21kthreadcore07:40:202
1373399221,10cyclictest30483-21latency_hist08:35:012
1373399220,15cyclictest27346irq/40-dwc2_hso09:00:142
1371799224,12cyclictest0-21swapper/008:30:140
1371799223,13cyclictest7326-21ssh09:50:120
13717992211,8cyclictest26095-21awk11:10:220
1371799221,17cyclictest32615-21ssh10:55:300
1371799221,17cyclictest0-21swapper/011:40:140
1371799221,16cyclictest73-21kswapd012:15:140
1371799221,16cyclictest24705-21apt-get07:55:120
1371799221,15cyclictest2025-21ssh10:05:130
13743992112,5cyclictest35966irq/58-eth012:00:143
13743992110,8cyclictest0-21swapper/312:27:013
1373399217,11cyclictest23579-21df_inode07:20:162
1373399213,15cyclictest15211-21uniq07:45:162
1373399211,16cyclictest7374-21kthreadcore07:35:222
1373399211,10cyclictest18076-21irqcore08:55:182
1371799211,16cyclictest21991-21ls07:15:230
13717992110,8cyclictest10604-21vmstat10:25:310
1374399207,9cyclictest0-21swapper/310:05:213
13743992012,5cyclictest35966irq/58-eth009:25:133
1373399202,15cyclictest2043-21fschecks_count08:05:202
1373399201,9cyclictest28036-21kthreadcore09:05:202
1373399201,4cyclictest0-21swapper/207:50:122
1371799208,3cyclictest131rcu_preempt12:35:010
1371799207,4cyclictest131rcu_preempt09:10:230
1371799207,10cyclictest16694-21ssh12:35:240
1371799202,15cyclictest14826-21kthreadcore07:10:200
13717992012,5cyclictest0-21swapper/007:40:290
1371799201,14cyclictest19274-21apt-get12:20:110
1371799201,12cyclictest13030-21sed10:10:190
1374399199,6cyclictest0-21swapper/310:39:323
1374399199,6cyclictest0-21swapper/309:20:223
1374399194,10cyclictest35966irq/58-eth008:10:133
1374399193,11cyclictest0-21swapper/311:05:443
1374399193,11cyclictest0-21swapper/310:50:563
1374399193,11cyclictest0-21swapper/310:30:563
1374399193,11cyclictest0-21swapper/310:24:553
1374399192,10cyclictest35966irq/58-eth009:35:143
13743991911,6cyclictest0-21swapper/311:36:503
1374399191,12cyclictest0-21swapper/310:55:103
1373399191,10cyclictest12887-21cut08:50:162
1371799192,7cyclictest30836-21df_inode09:45:140
1371799192,13cyclictest16360-21ssh09:55:130
1371799192,11cyclictest0-21swapper/008:50:140
1371799191,8cyclictest29615-21munin-run11:50:020
13717991912,5cyclictest0-21swapper/008:10:240
13717991912,4cyclictest12-21ksoftirqd/009:25:020
1371799191,14cyclictest21035-21sshd08:55:250
1371799191,14cyclictest10325-21kthreadcore09:15:200
1371799190,16cyclictest23722-21ls07:20:170
1371799190,16cyclictest22229-21ls11:10:000
1371799190,15cyclictest24313-21ls12:05:010
1374399189,6cyclictest35966irq/58-eth009:15:113
1374399188,8cyclictest0-21swapper/309:43:053
1374399188,6cyclictest35966irq/58-eth012:20:133
1374399188,3cyclictest0-21swapper/310:13:523
1374399182,10cyclictest0-21swapper/308:55:053
1374399182,10cyclictest0-21swapper/307:55:143
1374399182,10cyclictest0-21swapper/307:50:143
1374399182,10cyclictest0-21swapper/307:45:133
1374399182,10cyclictest0-21swapper/307:15:143
1374399181,12cyclictest0-21swapper/309:05:303
1374399181,11cyclictest35966irq/58-eth012:15:243
1374399181,10cyclictest0-21swapper/307:15:013
13743991810,6cyclictest0-21swapper/310:17:263
1371799185,9cyclictest5186-21apt-get07:35:010
1371799184,10cyclictest630-21Async09:36:310
1371799182,12cyclictest19649-21apt-get07:50:030
1371799181,9cyclictest31332-21/usr/sbin/munin08:35:180
1371799181,9cyclictest16902-21kthreadcore07:45:210
1371799181,14cyclictest28341-21ls07:25:150
1371799181,14cyclictest11672-21grep08:45:280
13717991810,6cyclictest14503-21cut08:15:210
13717991810,6cyclictest0-21swapper/011:59:570
13717991810,6cyclictest0-21swapper/011:59:570
1371799180,9cyclictest2527-21cut12:10:200
1374399178,4cyclictest0-21swapper/310:02:203
1374399172,10cyclictest0-21swapper/309:00:153
1374399172,10cyclictest0-21swapper/308:45:143
1374399172,10cyclictest0-21swapper/308:30:143
1374399172,10cyclictest0-21swapper/308:15:143
1374399172,10cyclictest0-21swapper/308:05:183
1374399172,10cyclictest0-21swapper/307:30:143
1374399171,11cyclictest0-21swapper/309:10:143
1374399171,10cyclictest0-21swapper/308:50:163
1374399171,10cyclictest0-21swapper/308:40:173
1373399171,13cyclictest13852-21latency_hist07:10:022
1371799179,6cyclictest782-21users12:25:320
1371799178,7cyclictest29556-21apt-get08:00:130
1371799177,8cyclictest25591-21uniq10:00:160
1371799173,11cyclictest27010-21munin-run09:05:000
1371799172,3cyclictest131rcu_preempt09:25:210
1371799172,11cyclictest22298-21irqrtprio09:40:180
13717991710,5cyclictest16564-21ls08:20:170
13717991710,4cyclictest0-21swapper/007:35:140
1371799170,8cyclictest31902-21munin-run09:10:020
1371799168,6cyclictest21232-21tune2fs08:25:170
1371799161,12cyclictest3159-21fschecks_count08:40:160
1374399141,10cyclictest0-21swapper/308:05:013
1371799141,10cyclictest13792-21latency_hist07:10:020
1374399138,3cyclictest33-21ksoftirqd/307:25:023
1374399132,8cyclictest0-21swapper/307:35:053
1374399131,5cyclictest0-21swapper/308:25:213
1374399121,6cyclictest0-21swapper/308:20:143
1374399112,5cyclictest0-21swapper/307:25:153
1374399111,7cyclictest0-21swapper/307:10:023
1374399111,6cyclictest0-21swapper/307:40:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional