You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 17:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Dec 09, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8671211083,9sleep30-21swapper/307:09:513
8671211083,9sleep30-21swapper/307:09:503
1264421010,0sleep10-21swapper/108:24:591
835729777,14sleep10-21swapper/107:05:471
835729777,14sleep10-21swapper/107:05:461
836329648,42sleep20-21swapper/207:05:522
836329648,42sleep20-21swapper/207:05:512
837128067,9sleep00-21swapper/007:05:580
837128067,9sleep00-21swapper/007:05:580
243402610,0sleep00-21swapper/012:02:100
876799300,1cyclictest0-21swapper/109:40:131
877199290,29cyclictest0-21swapper/212:23:202
8773992319,3cyclictest39-21ksoftirqd/310:31:433
877399231,1cyclictest0-21swapper/309:55:403
8773992216,1cyclictest39-21ksoftirqd/311:30:183
876799211,19cyclictest0-21swapper/112:10:201
8771992016,3cyclictest12771-21apt-get10:40:012
8771992013,2cyclictest0-21swapper/209:20:002
876699203,6cyclictest21989-21strings11:10:180
8773991917,1cyclictest39-21ksoftirqd/312:07:363
877199190,0cyclictest0-21swapper/210:27:122
8767991918,1cyclictest0-21swapper/109:55:401
876699195,7cyclictest19438-21munin-run09:25:010
877199180,1cyclictest0-21swapper/209:31:492
877199180,18cyclictest0-21swapper/209:22:242
876799187,7cyclictest9449-21/usr/sbin/munin09:15:171
876699186,6cyclictest1428-21awk11:20:010
876699184,13cyclictest0-21swapper/008:40:140
8766991810,7cyclictest0-21swapper/011:30:160
8773991715,2cyclictest39-21ksoftirqd/311:20:173
8773991713,2cyclictest0-21swapper/309:46:203
877399171,1cyclictest0-21swapper/311:45:123
8773991711,5cyclictest0-21swapper/312:15:123
8771991717,0cyclictest0-21swapper/210:04:432
876799175,6cyclictest0-21swapper/111:05:001
8767991715,1cyclictest0-21swapper/110:58:131
876799170,1cyclictest0-21swapper/111:57:051
876699173,7cyclictest0-21swapper/008:50:140
877399162,0cyclictest141rcu_preempt10:12:553
8773991613,2cyclictest94-21kswapd010:00:113
877399160,1cyclictest0-21swapper/308:55:243
8771991613,2cyclictest21065-21cpu12:00:142
8771991612,2cyclictest0-21swapper/210:55:242
877199161,14cyclictest0-21swapper/209:27:282
877199160,15cyclictest0-21swapper/211:07:382
8766991615,1cyclictest0-21swapper/012:05:550
876699160,1cyclictest0-21swapper/010:35:560
877399151,4cyclictest0-21swapper/310:15:213
8773991512,2cyclictest39-21ksoftirqd/310:45:343
877199151,7cyclictest224-21systemd-journal11:04:552
877199151,12cyclictest7744-21snmpd11:37:172
877199150,1cyclictest0-21swapper/209:10:062
877199150,14cyclictest356-21grep09:05:272
876799150,1cyclictest0-21swapper/111:12:201
876699157,7cyclictest20540-21perf12:00:000
876699154,6cyclictest6763-21sendmail11:00:010
876699153,6cyclictest6594-21sendmail12:40:010
876699151,8cyclictest7773-21munin-run08:15:000
876699150,8cyclictest13537-21perf07:20:010
877399146,2cyclictest20895-21apt-get12:00:133
877399141,2cyclictest141rcu_preempt09:21:343
8771991413,1cyclictest0-21swapper/209:58:382
877199141,12cyclictest18667-21head09:50:262
8771991411,2cyclictest13674-21wc10:15:002
877199140,14cyclictest0-21swapper/210:38:132
876799142,12cyclictest0-21swapper/112:27:361
8767991412,1cyclictest25-21ksoftirqd/109:50:571
876799140,9cyclictest0-21swapper/111:45:121
877399139,1cyclictest39-21ksoftirqd/309:40:023
877399137,2cyclictest39-21ksoftirqd/312:25:163
877399135,2cyclictest39-21ksoftirqd/311:03:183
877399132,6cyclictest1884-21diskmemload11:26:313
8773991311,1cyclictest10372-21ssh09:15:273
8773991311,1cyclictest0-21swapper/308:20:203
877399131,10cyclictest14114-21apache_processe11:55:123
8773991310,2cyclictest7030-21tr10:35:163
877399130,2cyclictest0-21swapper/310:21:193
877199139,3cyclictest7655-21ls11:25:002
877199139,3cyclictest2560-21grep10:05:182
877199138,3cyclictest19648-21sh10:19:542
877199138,2cyclictest5884-21grep08:10:152
8771991313,0cyclictest0-21swapper/210:45:252
8771991310,2cyclictest5915-21sh12:39:202
8771991310,2cyclictest32167-21head07:55:282
8771991310,2cyclictest23756-21grep07:40:172
877199130,1cyclictest0-21swapper/210:50:142
877199130,1cyclictest0-21swapper/210:50:132
876799139,3cyclictest30113-21apache212:03:081
876799139,3cyclictest18598-21grep07:30:001
876799138,4cyclictest7305-21sh10:09:591
876799138,3cyclictest0-21swapper/111:08:041
876799134,5cyclictest0-21swapper/111:50:161
8767991311,1cyclictest25-21ksoftirqd/109:45:231
8767991310,2cyclictest7432-21apt-get12:15:121
8767991310,2cyclictest32405-21taskset10:30:111
876699139,3cyclictest5154-21cut09:40:160
876699139,3cyclictest27863-21tr12:30:210
876699139,3cyclictest21679-21tail12:25:200
876699139,3cyclictest20538-21head09:25:180
876699136,6cyclictest0-21swapper/010:05:120
876699136,5cyclictest4350-21idleruntime-cro09:39:590
876699132,6cyclictest1886-21munin-run09:10:010
876699131,1cyclictest1884-21diskmemload09:11:390
8766991310,2cyclictest32028-21cat09:05:000
8766991310,2cyclictest16863-21grep08:30:260
8766991310,2cyclictest16484-21cut07:25:170
8766991310,2cyclictest12677-21sh09:45:290
877399129,2cyclictest39-21ksoftirqd/309:27:003
877399129,2cyclictest2740-21wc12:35:253
877399129,2cyclictest15308-21head10:40:273
877399129,2cyclictest12930-21cat07:15:263
877399127,1cyclictest141rcu_preempt10:25:103
877399124,2cyclictest224-21systemd-journal07:40:293
8773991210,1cyclictest8278-21grep08:15:153
8773991210,1cyclictest16869-21df07:25:173
8773991210,1cyclictest15224-21head12:20:223
8773991210,1cyclictest13054-21tr08:25:143
877399120,8cyclictest1884-21diskmemload11:14:113
877199129,2cyclictest30720-21sh12:32:322
877199129,2cyclictest29117-21cut07:50:162
877199129,2cyclictest23248-21sendmail07:40:002
877199129,2cyclictest17611-21grep07:25:232
877199129,2cyclictest15474-21tail11:30:182
877199129,2cyclictest14959-21cat08:30:002
877199129,1cyclictest11184-21grep08:20:172
877199128,3cyclictest585-21ssh11:19:122
877199128,3cyclictest316552sleep211:42:562
877199127,1cyclictest3303-21basename08:05:142
8771991211,1cyclictest61750irq/125-enp2s0-10:20:332
876799129,3cyclictest25-21ksoftirqd/108:55:001
876799129,2cyclictest3374-21taskset11:20:251
876799129,2cyclictest30964-21tr07:55:131
876799129,2cyclictest30476-21grep09:00:171
876799129,2cyclictest11674-21grep07:15:141
876799124,2cyclictest1884-21diskmemload12:21:191
8767991211,1cyclictest0-21swapper/110:04:511
8767991211,1cyclictest0-21swapper/108:40:131
8767991210,1cyclictest25-21ksoftirqd/109:31:091
876699129,2cyclictest3236-21head11:45:490
876699129,2cyclictest30808-21cat07:55:000
876699129,2cyclictest28253-21sed11:40:190
876699129,2cyclictest21044-21head10:20:180
876699129,2cyclictest19520-21head08:35:250
876699129,2cyclictest17639-21grep09:50:170
876699129,2cyclictest15890-21cron07:25:010
876699129,2cyclictest15259-21wc12:20:220
876699129,2cyclictest14084-21cut08:25:210
876699129,2cyclictest10666-21grep07:10:270
876699128,3cyclictest8555-21cut09:15:130
876699128,3cyclictest26454-21grep10:50:170
876699128,3cyclictest26454-21grep10:50:160
876699124,5cyclictest13626-21turbostat.cron11:05:000
876699123,4cyclictest10444-21apt-get08:20:130
876699122,5cyclictest1723-21ssh12:10:200
877399119,1cyclictest32304-21sed09:35:173
877399119,1cyclictest3125-21head09:10:183
877399119,1cyclictest27010-21head10:50:193
877399119,1cyclictest27010-21head10:50:183
877399119,1cyclictest21982-21grep11:35:173
877399118,2cyclictest9753-21cut07:10:183
877399118,2cyclictest9266-21ssh11:50:313
877399118,2cyclictest5793-21chrt12:13:433
877399118,2cyclictest4621-21ssh10:06:493
877399118,2cyclictest3852-21ssh10:56:533
877399118,2cyclictest28851-21ssh12:30:423
877399118,2cyclictest28727-21grep07:50:133
877399118,2cyclictest28453-21chrt11:15:193
877399118,2cyclictest27367-21ssh11:40:113
877399118,2cyclictest25496-21ssh09:30:093
877399118,2cyclictest25470-21grep08:50:173
877399118,2cyclictest19512-21sh11:09:013
877399118,2cyclictest1869-21sh09:10:003
877399118,2cyclictest18424-21wc09:50:253
877399118,2cyclictest14024-21tr07:20:163
877399116,2cyclictest39-21ksoftirqd/307:35:013
877399110,10cyclictest27945-21chrt07:49:203
877199118,2cyclictest9652-21which11:25:252
877199118,2cyclictest8445-21expr08:15:152
877199118,2cyclictest7832-21expr12:15:142
877199118,2cyclictest7304-21chrt09:41:032
877199118,2cyclictest32181-21grep09:35:172
877199118,2cyclictest2998-21ssh10:31:262
877199118,2cyclictest27499-21cat12:05:132
877199118,2cyclictest23274-21cut08:45:182
877199118,2cyclictest19984-21head07:30:212
877199118,2cyclictest15936-21grep11:55:262
877199118,2cyclictest1491-21grep11:45:152
877199118,2cyclictest12608-21ssh11:53:582
877199118,2cyclictest11708-21grep09:45:202
877199118,2cyclictest10785-21taskset07:10:572
877199117,3cyclictest13501-21idleruntime-cro07:20:002
877199116,3cyclictest1291-21runrttasks12:14:562
877199112,5cyclictest224-21systemd-journal07:23:182
876799119,1cyclictest8807-21grep08:15:171
876799119,1cyclictest18917-21grep08:35:201
876799119,1cyclictest15135-21head09:20:211
876799118,2cyclictest5830-21rm12:39:151
876799118,2cyclictest31012-21ssh11:42:091
876799118,2cyclictest27608-21grep08:55:201
876799118,2cyclictest26640-21apache_volume12:30:131
876799118,2cyclictest2414-21tr09:10:151
876799118,2cyclictest20995-21ssh11:35:051
876799118,2cyclictest18906-21ssh10:44:311
876799118,2cyclictest14898-21uname07:20:211
876799118,2cyclictest14506-21ssh11:30:051
876799118,2cyclictest13862-21ssh10:15:041
876799118,2cyclictest13615-21cut10:15:001
876799118,2cyclictest10221-21ssh11:25:381
876799112,4cyclictest25643-21apt07:45:001
876799110,10cyclictest10874-21ssh10:38:061
876699118,2cyclictest8768-21grep08:15:170
876699118,2cyclictest8429-21grep11:25:170
876699118,2cyclictest8220-21head12:15:160
876699118,2cyclictest32325-21chrt07:55:290
876699118,2cyclictest28961-21head10:00:190
876699118,2cyclictest27088-21df_abs10:25:170
876699118,2cyclictest26530-21grep07:45:180
876699118,2cyclictest26257-21forks09:30:190
876699118,2cyclictest24742-21wc08:50:000
876699118,2cyclictest23852-21ssh11:36:090
876699118,2cyclictest21872-21tail07:35:200
876699118,2cyclictest20576-21head10:45:180
876699118,2cyclictest17551-21ssh10:43:060
876699118,2cyclictest15982-21grep11:05:280
876699118,2cyclictest13063-21ssh10:14:210
876699118,2cyclictest1248-21cat08:00:170
876699110,10cyclictest2499-21ssh10:30:550
876699110,10cyclictest15254-21grep10:15:210
877399108,1cyclictest4439-21ntp_states08:05:223
877399108,1cyclictest21374-21ntpq08:40:203
877399108,1cyclictest19190-21cat08:35:223
877399107,2cyclictest613-21grep08:00:003
877399107,2cyclictest22985-21expr08:45:163
877399107,2cyclictest21265-21hostname07:35:173
877399107,2cyclictest16557-21readlink08:30:213
877399107,2cyclictest1249-21expr08:00:163
877399102,8cyclictest39-21ksoftirqd/309:00:133
877199108,1cyclictest270462sleep208:54:382
877199108,1cyclictest21104-21expr11:10:142
877199108,1cyclictest17123-21taskset08:32:502
877199107,2cyclictest32030-21wc09:05:012
877199107,2cyclictest29548-21date09:00:002
877199107,2cyclictest22061-21sh12:25:252
877199107,2cyclictest20567-21expr08:40:142
877199107,2cyclictest17932-21wget08:35:142
877199107,2cyclictest1355-21expr08:00:172
876799108,2cyclictest23595-21ssh10:47:361
876799108,2cyclictest20846-21taskset07:34:551
876799108,1cyclictest321542sleep109:05:101
876799108,1cyclictest29875-21ssh10:52:221
876799108,1cyclictest29875-21ssh10:52:211
876799108,1cyclictest27726-21ssh11:15:131
876799108,1cyclictest26294-21taskset10:25:021
876799108,1cyclictest22389-21grep07:35:241
876799108,1cyclictest22154-21ssh09:26:091
876799107,2cyclictest26280-21taskset07:45:171
876799107,2cyclictest21969-21grep10:20:291
876799107,2cyclictest15156-21taskset08:30:051
876799107,2cyclictest14920-21cron08:30:001
876799107,2cyclictest1256-21ssh09:35:511
876799104,4cyclictest0-21swapper/112:05:371
876799100,9cyclictest9372sleep108:00:161
876699108,1cyclictest27333-21cat08:55:160
876699107,2cyclictest7326-21aten2_r7power_c11:50:160
876699107,2cyclictest3009-21ssh11:20:220
876699107,2cyclictest22869-21ls09:55:150
876699107,2cyclictest18824-21cut07:30:130
876699102,6cyclictest0-21swapper/008:05:130
87739997,1cyclictest39-21ksoftirqd/308:10:153
87719997,1cyclictest27898-21chrt07:48:422
87679998,1cyclictest9404-21expr07:10:161
87679998,1cyclictest3620-21expr08:05:141
87679998,1cyclictest23280-21uniq08:45:181
87679997,1cyclictest75562sleep108:12:301
87679990,9cyclictest28682-21grep07:50:141
87669997,1cyclictest24717-21taskset07:40:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional