You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-09 - 14:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Nov 09, 2025 12:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23757210793,9sleep10-21swapper/107:08:311
2366429582,9sleep30-21swapper/307:07:203
2173228366,6sleep00-21swapper/007:05:050
2372028164,13sleep20-21swapper/207:08:042
66942630,0sleep30-21swapper/310:20:233
188462510,0sleep00-21swapper/010:04:360
2396299280,28cyclictest0-21swapper/211:14:502
2395799280,17cyclictest0-21swapper/009:13:090
2396299273,22cyclictest25353-21munin-run08:20:012
23959992625,1cyclictest0-21swapper/109:06:071
2395799260,25cyclictest0-21swapper/008:24:300
2396299220,22cyclictest0-21swapper/210:10:572
2395799221,13cyclictest0-21swapper/009:45:150
2396799210,19cyclictest0-21swapper/307:10:143
2396299210,20cyclictest0-21swapper/209:54:342
23957992013,4cyclictest27032-21ssh11:50:370
2395799190,18cyclictest0-21swapper/010:35:010
23967991816,2cyclictest94-21kswapd009:53:453
2396299180,2cyclictest0-21swapper/211:55:112
2396299180,18cyclictest0-21swapper/209:39:552
2395999181,14cyclictest0-21swapper/112:35:161
2395799183,7cyclictest224-21systemd-journal07:52:310
2396799170,1cyclictest0-21swapper/312:02:073
2396799170,14cyclictest0-21swapper/311:43:053
2396299179,1cyclictest141rcu_preempt10:17:382
2396299172,1cyclictest17021-21diskmemload11:36:072
2396299172,14cyclictest11702-21kworker/u8:211:42:262
2395799172,8cyclictest24815-21idleruntime-cro12:30:000
23957991716,1cyclictest0-21swapper/010:32:310
2396799160,2cyclictest141rcu_preempt10:18:313
2396299160,16cyclictest0-21swapper/211:02:072
2395999160,10cyclictest0-21swapper/111:41:071
23957991612,3cyclictest6010-21grep07:39:590
2395799160,3cyclictest0-21swapper/007:25:150
2396799155,6cyclictest224-21systemd-journal09:48:283
2396299150,12cyclictest0-21swapper/210:21:492
2396299150,0cyclictest0-21swapper/212:30:102
2395999155,2cyclictest15193-21perf09:35:001
2395799156,8cyclictest3140-21apt-get11:10:000
23957991511,3cyclictest5463-21wc11:10:260
23957991511,3cyclictest2479-21wc07:30:180
2395799150,13cyclictest31529-21grep09:20:270
23967991414,0cyclictest0-21swapper/310:06:463
23967991410,3cyclictest31504-21sort10:40:193
23967991410,3cyclictest28868-21wc11:30:013
23962991413,1cyclictest0-21swapper/210:41:582
23962991412,1cyclictest30115-21munin-run08:30:002
23962991411,2cyclictest6227-21wc10:45:282
23962991410,1cyclictest31313-21awk07:24:592
2395999147,6cyclictest23219-21apt-get08:15:131
23959991412,1cyclictest0-21swapper/111:57:001
2395999141,1cyclictest141rcu_preempt10:23:161
23959991410,3cyclictest24866-21wc12:30:011
23959991410,3cyclictest19381-21ssh11:45:581
23959991410,2cyclictest19243-21grep09:10:311
2395999140,13cyclictest0-21swapper/110:17:111
2395999140,12cyclictest3566-21grep09:25:171
2395799146,1cyclictest17021-21diskmemload10:05:190
23957991413,1cyclictest0-21swapper/010:42:280
23957991412,1cyclictest27152-21diskstats07:15:150
23957991410,3cyclictest5812-21tr10:45:230
23957991410,3cyclictest16120-21grep08:00:140
2395799140,13cyclictest23805-21tr09:15:160
2396799139,3cyclictest30410-21cut11:05:203
2396799139,3cyclictest29424-21cut07:20:173
2396799133,9cyclictest5676-21ssh10:45:223
23967991310,2cyclictest4342-21grep08:40:303
23967991310,2cyclictest22551-21ssh09:14:133
23967991310,2cyclictest18901-21ssh10:30:193
23967991310,2cyclictest17980-21grep12:05:203
23967991310,2cyclictest11842-21grep07:50:193
2396799130,13cyclictest0-21swapper/309:39:193
2396799130,12cyclictest2017-21fgrep11:55:243
2396299139,3cyclictest24329-21wget07:10:152
2396299137,5cyclictest32440-21awk11:55:002
23962991310,2cyclictest8285-21sleep12:19:592
23962991310,2cyclictest31474-21grep11:05:292
23962991310,2cyclictest2659-21awk07:30:202
23962991310,2cyclictest20985-21grep10:05:242
23962991310,2cyclictest20600-21cat08:10:012
23962991310,2cyclictest16333-21grep08:00:142
2396299130,13cyclictest0-21swapper/210:31:342
2396299130,12cyclictest7456-21cat08:50:012
2396299130,12cyclictest10207-21cat07:45:272
2396299130,12cyclictest0-21swapper/212:10:462
2395999139,3cyclictest1082-21head08:35:171
2395999138,3cyclictest31576-21head09:20:271
2395999138,3cyclictest15808-21awk08:00:001
2395999135,6cyclictest29312-21apt-get11:30:171
23959991310,2cyclictest5514-21grep07:35:261
23959991310,2cyclictest25164-21grep12:10:141
23959991310,2cyclictest18312-21grep10:30:151
2395799139,3cyclictest10317-21wc12:00:270
2395799135,4cyclictest224-21systemd-journal10:30:010
23957991311,2cyclictest23948-21cyclictest08:55:280
23957991310,2cyclictest25012-21head11:00:300
23957991310,2cyclictest24956-21tail07:10:180
23957991310,2cyclictest23479-21sed09:40:320
23957991310,2cyclictest20874-21cut08:10:140
23957991310,2cyclictest15552-21cut09:05:200
23957991310,2cyclictest10767-21cron07:50:000
23957991310,2cyclictest10728-21head12:20:260
23957991310,2cyclictest10341-21grep11:40:180
2396799129,2cyclictest8453-21tail09:55:183
2396799129,2cyclictest3544-21grep09:25:173
2396799129,2cyclictest32445-21perf11:55:003
2396799129,2cyclictest31479-21grep09:20:273
2396799129,2cyclictest30998-21grep08:30:183
2396799129,2cyclictest29162-21grep11:05:003
2396799129,2cyclictest26750-21grep08:20:213
2396799129,2cyclictest25724-21expr10:35:233
2396799129,2cyclictest24844-21sh12:30:013
2396799129,2cyclictest2351-21cat12:15:263
2396799129,2cyclictest23475-21tr08:15:153
2396799129,2cyclictest17688-21sort11:45:183
2396799129,2cyclictest13140-21grep09:00:183
2396799124,5cyclictest11770-21/usr/sbin/munin11:15:283
23967991210,1cyclictest15609-21grep09:05:203
23967991210,1cyclictest14927-21awk07:55:223
2396299129,2cyclictest9813-21basename08:54:592
2396299129,2cyclictest18815-21wc10:55:272
2396299129,2cyclictest13215-21apt-get07:55:012
2396299128,3cyclictest4452-21sed08:40:302
2396299128,3cyclictest16873-21basename11:45:152
2396299127,2cyclictest15129-21ssh12:04:062
23962991211,1cyclictest5555-21kworker/2:110:03:422
23962991210,2cyclictest32-21ksoftirqd/212:23:502
2396299120,11cyclictest0-21swapper/209:30:582
2395999129,2cyclictest3735-21wc07:35:001
2395999129,2cyclictest31143-21sed08:30:181
2395999129,2cyclictest303702sleep110:14:081
2395999129,2cyclictest29671-21tail08:25:281
2395999129,2cyclictest27343-21grep07:15:151
2395999129,2cyclictest22335-21cut09:40:221
2395999129,2cyclictest20474-21head10:05:191
2395999129,2cyclictest10453-21dump-pmu-power10:49:591
2395999128,3cyclictest875-21tr12:15:151
2395999128,3cyclictest18718-21grep08:05:161
2395999125,3cyclictest11711-21apt-get10:25:141
2395999120,7cyclictest0-21swapper/109:45:151
2395999120,5cyclictest0-21swapper/109:55:191
2395799129,2cyclictest874-21grep12:15:160
2395799129,2cyclictest8159-21rm09:55:170
2395799129,2cyclictest7910-21grep07:40:280
2395799129,2cyclictest3068-21tr09:25:140
2395799129,2cyclictest30654-21grep11:30:260
2395799129,2cyclictest26292-21head10:10:180
2395799129,2cyclictest23979-21grep08:15:170
2395799129,2cyclictest11332-21grep11:15:230
2395799128,3cyclictest28826-21cron11:30:000
2395799128,3cyclictest2609-21cut09:50:190
2395799128,3cyclictest2435-21grep11:55:270
2395799128,3cyclictest18157-21grep12:05:220
2395799127,3cyclictest11511-21apache212:11:440
23957991210,2cyclictest13-21ksoftirqd/010:52:370
2396799119,1cyclictest24325-21head09:15:173
2396799119,1cyclictest12309-21wc10:50:223
2396799118,2cyclictest6521-21sh12:38:173
2396799118,2cyclictest4386-21df11:10:163
2396799118,2cyclictest4303-21cat07:35:163
2396799118,2cyclictest3487-21grep11:35:153
2396799118,2cyclictest29328-21chrt10:12:523
2396799118,2cyclictest27725-21ssh12:30:473
2396799118,2cyclictest26818-21expr07:15:123
2396799118,2cyclictest25347-21cat12:10:143
2396799118,2cyclictest21538-21ssh09:40:073
2396799118,2cyclictest21126-21grep08:10:153
2396799118,2cyclictest20511-21rm10:57:303
2396799118,2cyclictest19482-21cat08:05:213
2396799118,2cyclictest1743-21tr07:30:143
2396799118,2cyclictest16190-21tr08:00:153
2396799118,2cyclictest14885-21ssh09:34:373
2396799118,2cyclictest1135-21taskset07:28:503
2396799118,2cyclictest10065-21ntpq12:20:203
2396799111,2cyclictest0-21swapper/310:25:143
23967991110,1cyclictest0-21swapper/308:35:133
2396799110,10cyclictest8630-21chrt07:45:113
2396799110,10cyclictest18175-21cat11:20:273
2396299118,2cyclictest6317-21rm12:38:062
2396299118,2cyclictest4385-21df07:35:162
2396299118,2cyclictest3570-21expr09:25:172
2396299118,2cyclictest29785-21rm09:20:152
2396299118,2cyclictest29302-21apache_volume11:30:162
2396299118,2cyclictest26664-21grep07:15:122
2396299118,2cyclictest26427-21ssh11:27:222
2396299118,2cyclictest23388-21ssh09:15:002
2396299118,2cyclictest21134-21ls08:10:162
2396299118,2cyclictest18143-21grep09:10:222
2396299118,2cyclictest14481-21chrt10:26:262
2396299118,2cyclictest14447-21head07:55:182
2396299118,2cyclictest12864-21ssh11:16:172
2396299118,2cyclictest11680-21cut10:50:182
2396299117,3cyclictest10796-21head08:55:162
2396299115,5cyclictest13774-21ps09:00:232
2396299110,2cyclictest0-21swapper/212:06:492
2396299110,2cyclictest0-21swapper/209:56:452
2396299110,2cyclictest0-21swapper/209:45:162
2396299110,10cyclictest16701-21taskset09:06:112
2395999118,2cyclictest9401-21head07:45:181
2395999118,2cyclictest7882-21wc08:50:141
2395999118,2cyclictest6742-21ssh11:37:201
2395999118,2cyclictest4112-21sleep10:44:591
2395999118,2cyclictest4057-21grep11:10:151
2395999118,2cyclictest30894-21grep07:20:281
2395999118,2cyclictest29499-21chrt11:05:091
2395999118,2cyclictest26794-21cut12:30:241
2395999118,2cyclictest24559-21grep09:15:201
2395999118,2cyclictest24198-21wc11:00:231
2395999118,2cyclictest23821-21head11:25:181
2395999118,2cyclictest17548-21expr10:55:161
2395999118,2cyclictest17201-21cut08:00:201
2395999118,2cyclictest1503-21cut09:50:131
2395999118,2cyclictest14526-21ssh12:03:381
2395999118,2cyclictest14488-21ssh10:52:411
2395999118,2cyclictest13027-21ssh12:22:141
2395999118,2cyclictest10942-21tr11:15:201
2395999118,2cyclictest10664-21sort08:55:161
2395999118,2cyclictest0-21swapper/112:06:121
2395999117,3cyclictest26614-21ssh11:50:281
2395999117,3cyclictest16160-21ssh09:35:161
2395999117,2cyclictest11144-21cut07:50:161
2395999112,5cyclictest0-21swapper/108:40:171
2395999111,2cyclictest0-21swapper/110:35:171
2395999110,10cyclictest15829-21ssh10:00:471
2395799118,2cyclictest7650-21taskset08:50:080
2395799118,2cyclictest4817-21ssh10:19:430
2395799118,2cyclictest4709-21sh11:35:240
2395799118,2cyclictest3733-21ssh12:36:020
2395799118,2cyclictest31776-21smartctl08:30:250
2395799118,2cyclictest31480-21cat07:25:000
2395799118,2cyclictest29442-21ssh12:32:090
2395799118,2cyclictest2448-21apt08:40:010
2395799118,2cyclictest22131-21ssh11:48:140
2395799118,2cyclictest20257-21ssh11:22:350
2395799118,2cyclictest18707-21tr08:05:160
2395799118,2cyclictest17082-21chrt10:55:050
2395799118,2cyclictest15892-21ls08:00:010
2395799118,2cyclictest15465-21ssh09:35:040
2395799118,2cyclictest13833-21taskset09:00:240
2396799108,1cyclictest6714-21expr07:40:163
2396799108,1cyclictest5059-21chrt08:45:063
2396799108,1cyclictest13459-21ssh10:00:023
2396799108,1cyclictest11878-21seq08:55:333
2396799107,2cyclictest29606-21tr11:30:183
2396799100,10cyclictest0-21swapper/308:50:143
2396299108,1cyclictest598-21grep08:35:152
2396299108,1cyclictest30498-21aten2_r7power_v08:30:152
2396299108,1cyclictest24433-21rm10:35:022
2396299108,1cyclictest21475-21chrt09:40:032
2396299108,1cyclictest17135-21ssh12:25:132
2396299107,2cyclictest7024-21tail07:40:192
2396299107,2cyclictest32600-21cat07:25:212
2396299107,2cyclictest26168-21df_abs08:20:172
2396299107,2cyclictest17045-21df_inode11:20:162
2395999108,1cyclictest69682sleep108:47:041
2395999108,1cyclictest261132sleep107:12:441
2395999108,1cyclictest14392-21taskset09:02:051
2395999107,2cyclictest26573-21cut08:20:201
2395999107,2cyclictest21769-21cut08:10:201
2395999107,2cyclictest18609-21ssh11:20:451
2395799107,2cyclictest9369-21cut09:30:160
2395799107,2cyclictest7222-21date10:20:280
2395799107,2cyclictest3665-21grep08:40:230
2395799106,2cyclictest28549-21grep08:25:170
2395799105,3cyclictest5329-21tr08:45:160
239679997,1cyclictest28875-21cat08:25:203
239599997,1cyclictest31525-21sshd07:25:091
239599995,3cyclictest7060-21grep07:40:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional