You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 17:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jul 08, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2415029066,8sleep00-21swapper/007:08:430
2415029066,8sleep00-21swapper/007:08:430
2286428975,9sleep20-21swapper/207:05:232
2286428975,9sleep20-21swapper/207:05:232
2410028269,9sleep30-21swapper/307:08:053
2410028269,9sleep30-21swapper/307:08:053
2411528164,13sleep10-21swapper/107:08:161
2411528164,13sleep10-21swapper/107:08:161
2434199300,3cyclictest0-21swapper/110:55:381
2434399290,28cyclictest0-21swapper/209:15:192
2434399280,27cyclictest0-21swapper/210:51:472
2434399280,26cyclictest0-21swapper/210:29:382
2434199281,21cyclictest31114-21awk12:10:211
2434399270,27cyclictest0-21swapper/211:48:012
2434399270,23cyclictest0-21swapper/212:00:032
24346992616,7cyclictest39-21ksoftirqd/310:33:323
2434399260,24cyclictest0-21swapper/210:58:382
24341992618,7cyclictest0-21swapper/108:00:241
2433699261,1cyclictest0-21swapper/010:29:020
24346992525,0cyclictest0-21swapper/309:32:513
24346992514,6cyclictest39-21ksoftirqd/309:29:063
2434199250,0cyclictest0-21swapper/110:46:021
2433699250,19cyclictest0-21swapper/009:20:020
24343992423,1cyclictest0-21swapper/212:18:222
2434399240,24cyclictest0-21swapper/209:54:022
2434399240,0cyclictest0-21swapper/210:03:132
2434399230,23cyclictest0-21swapper/210:49:462
2434399230,22cyclictest0-21swapper/211:38:042
24341992312,8cyclictest537-21snmpd09:18:461
2433699231,21cyclictest0-21swapper/012:26:190
24346992219,2cyclictest39-21ksoftirqd/312:05:113
2434399220,3cyclictest0-21swapper/211:23:202
2434399220,22cyclictest0-21swapper/212:24:492
2434199220,14cyclictest0-21swapper/109:20:061
2433699220,2cyclictest0-21swapper/009:11:510
2433699220,1cyclictest0-21swapper/011:13:390
2434399212,1cyclictest0-21swapper/212:30:542
2434399210,3cyclictest0-21swapper/211:01:592
2434399210,21cyclictest0-21swapper/212:08:502
2434399210,21cyclictest0-21swapper/210:37:132
2434399210,21cyclictest0-21swapper/210:37:122
2433699210,2cyclictest0-21swapper/012:39:300
2433699210,1cyclictest0-21swapper/010:37:140
2433699210,1cyclictest0-21swapper/010:37:140
24343992019,1cyclictest0-21swapper/211:34:472
2434199200,20cyclictest0-21swapper/111:27:041
2434199200,17cyclictest0-21swapper/110:15:131
24336992017,2cyclictest13-21ksoftirqd/008:45:010
2433699200,20cyclictest0-21swapper/011:47:290
2433699200,20cyclictest0-21swapper/009:41:500
2433699200,1cyclictest0-21swapper/011:54:550
2434699193,1cyclictest27827-21apt-get09:35:143
24343991912,7cyclictest0-21swapper/211:43:412
2434399190,6cyclictest0-21swapper/211:12:482
2434399190,2cyclictest0-21swapper/210:17:562
2434399190,0cyclictest0-21swapper/209:12:052
2433699190,1cyclictest0-21swapper/012:17:020
2433699190,17cyclictest0-21swapper/009:19:140
2434399180,1cyclictest0-21swapper/210:34:202
2434399180,17cyclictest0-21swapper/211:50:352
2434399180,17cyclictest0-21swapper/209:38:342
2434399180,16cyclictest0-21swapper/210:09:002
24341991816,1cyclictest0-21swapper/110:05:001
2434199180,3cyclictest141rcu_preempt11:20:501
2434199180,0cyclictest0-21swapper/110:53:161
2433699180,3cyclictest0-21swapper/012:30:020
2433699180,1cyclictest0-21swapper/011:38:390
2433699180,17cyclictest0-21swapper/010:17:530
2434699178,2cyclictest141rcu_preempt12:23:043
2434699172,6cyclictest39-21ksoftirqd/310:02:423
24346991712,5cyclictest39-21ksoftirqd/311:10:123
24343991714,2cyclictest32-21ksoftirqd/210:24:032
2434399170,17cyclictest0-21swapper/209:42:492
2434399170,13cyclictest0-21swapper/210:12:002
2434199173,2cyclictest0-21swapper/108:15:471
2433699170,17cyclictest0-21swapper/011:05:320
2433699170,17cyclictest0-21swapper/009:46:470
2434699162,1cyclictest141rcu_preempt12:01:313
24346991615,1cyclictest39-21ksoftirqd/311:28:563
24346991615,1cyclictest0-21swapper/310:50:143
2434699161,2cyclictest141rcu_preempt12:39:103
24343991615,1cyclictest0-21swapper/211:09:142
2434399160,16cyclictest0-21swapper/210:41:372
2434399160,15cyclictest0-21swapper/209:45:332
2434399160,14cyclictest0-21swapper/209:59:292
2434199160,4cyclictest0-21swapper/110:06:321
2434199160,1cyclictest0-21swapper/110:12:491
2434199160,16cyclictest0-21swapper/112:23:251
2434199160,0cyclictest0-21swapper/109:59:551
2433699160,1cyclictest0-21swapper/009:30:220
2433699160,0cyclictest0-21swapper/011:26:500
2434699159,3cyclictest39-21ksoftirqd/309:20:363
2434699159,2cyclictest39-21ksoftirqd/311:34:263
2434699158,1cyclictest39-21ksoftirqd/312:28:283
2434699151,4cyclictest4775-21mailstats11:35:213
24346991511,1cyclictest39-21ksoftirqd/310:05:313
2434699150,4cyclictest141rcu_preempt11:50:133
2434699150,1cyclictest141rcu_preempt10:55:043
2434399150,2cyclictest0-21swapper/209:25:412
2434399150,1cyclictest0-21swapper/209:30:232
2434399150,13cyclictest7332-21cut12:35:152
2434399150,13cyclictest0-21swapper/211:55:072
24341991511,3cyclictest30526-21cat08:30:001
2434199151,12cyclictest0-21swapper/111:18:291
2434199150,15cyclictest0-21swapper/110:27:431
2434199150,14cyclictest0-21swapper/111:04:191
2434199150,14cyclictest0-21swapper/109:45:131
2433699153,9cyclictest31850-21apache210:20:120
2433699153,7cyclictest26705-21sed07:15:000
2433699150,1cyclictest0-21swapper/010:51:010
2433699150,15cyclictest0-21swapper/009:57:280
2434699145,5cyclictest537-21snmpd11:46:133
24346991411,3cyclictest39-21ksoftirqd/310:48:403
24346991411,3cyclictest39-21ksoftirqd/310:10:573
24346991410,3cyclictest39-21ksoftirqd/309:12:143
2434699140,14cyclictest0-21swapper/310:26:553
24343991411,2cyclictest6467-21wc08:45:202
24343991411,2cyclictest16154-21cron08:00:002
24343991410,3cyclictest5339-21tr07:35:222
24343991410,3cyclictest20049-21head11:25:212
24343991410,3cyclictest10052-21wc07:45:192
2434199149,3cyclictest919-21wc07:25:271
24341991410,3cyclictest27035-21cat08:20:191
2434199140,1cyclictest141rcu_preempt12:15:511
24336991411,2cyclictest12363-21cut09:25:210
2433699140,1cyclictest0-21swapper/011:32:500
2433699140,1cyclictest0-21swapper/010:40:130
2433699140,14cyclictest0-21swapper/011:57:590
2433699140,14cyclictest0-21swapper/011:43:160
2433699140,14cyclictest0-21swapper/010:33:170
2433699140,13cyclictest0-21swapper/010:14:570
2433699140,13cyclictest0-21swapper/010:05:230
2434699139,3cyclictest27570-21grep07:15:173
2434699137,4cyclictest7264-21kworker/3:211:00:183
2434699136,6cyclictest17571-21diskmemload12:33:063
2434699135,6cyclictest12346-21diskstats11:40:163
2434699134,8cyclictest39-21ksoftirqd/310:15:073
2434699132,1cyclictest141rcu_preempt11:22:323
24346991310,2cyclictest8765-21taskset10:20:443
24346991310,2cyclictest6338-21uniq12:15:163
24346991310,2cyclictest6207-21cron07:40:003
24346991310,2cyclictest3279-21chrt12:13:353
24346991310,2cyclictest27603-21grep08:20:263
24346991310,2cyclictest11216-21dpkg07:50:003
2434399139,3cyclictest965-21grep07:25:272
2434399139,3cyclictest16514-21grep08:00:132
2434399132,2cyclictest141rcu_preempt07:40:272
24343991313,0cyclictest0-21swapper/211:15:582
24343991310,2cyclictest31448-21tail08:30:142
24343991310,2cyclictest29765-21grep12:10:132
24343991310,2cyclictest14433-21wc09:00:262
2434399130,12cyclictest4819-21grep08:40:252
2434399130,11cyclictest10745-21wc09:25:012
2434199139,3cyclictest422-21cron08:35:001
2434199139,3cyclictest28309-21sh09:35:151
2434199139,3cyclictest0-21swapper/109:44:081
24341991310,2cyclictest14801-21sed07:55:181
2434199130,12cyclictest8614-21head08:50:171
24336991311,2cyclictest13-21ksoftirqd/009:36:520
24336991310,2cyclictest27721-21grep07:15:180
24336991310,2cyclictest1836-21grep08:35:200
24336991310,2cyclictest13209-21tr12:00:150
24336991310,2cyclictest12841-21seq07:50:210
24336991310,2cyclictest10190-21grep07:45:200
24336991310,1cyclictest0-21swapper/012:08:590
2434699129,2cyclictest8116-21grep07:40:273
2434699129,2cyclictest5322-21head11:55:173
2434699129,2cyclictest32166-21grep10:35:143
2434699129,2cyclictest32166-21grep10:35:143
2434699129,2cyclictest26207-21grep07:10:263
2434699129,2cyclictest18090-21grep08:00:253
2434699129,1cyclictest39-21ksoftirqd/309:53:503
2434699128,3cyclictest39-21ksoftirqd/310:40:133
2434699128,1cyclictest39-21ksoftirqd/309:55:023
2434699126,5cyclictest24060-21df08:15:153
2434699122,2cyclictest141rcu_preempt08:55:233
24346991211,1cyclictest39-21ksoftirqd/309:15:133
24346991210,1cyclictest19572-21grep08:05:163
2434399129,2cyclictest9488-21grep08:50:262
2434399129,2cyclictest2511-21sed07:30:172
2434399129,2cyclictest11071-21sort08:55:172
2434399126,2cyclictest225-21systemd-journal08:40:002
2434399125,6cyclictest311rcuc/208:20:262
2434199129,2cyclictest31671-21tr10:35:121
2434199129,2cyclictest31671-21tr10:35:121
2434199129,2cyclictest21404-21tail09:50:161
2434199129,2cyclictest20870-21head11:45:161
2434199129,2cyclictest15432-21sh10:45:001
2434199129,2cyclictest12616-21grep09:00:001
2434199129,2cyclictest12487-21tail11:40:171
2434199128,3cyclictest3839-21latency_hist07:35:001
2434199128,3cyclictest24059-21cat12:25:251
2434199128,3cyclictest16220-21grep09:05:171
2434199128,3cyclictest14896-21grep12:00:271
2434199128,3cyclictest1346-21sleep11:15:001
2434199126,4cyclictest0-21swapper/112:05:361
2434199126,2cyclictest1400-21rm12:31:121
2434199121,10cyclictest17571-21diskmemload10:31:351
2434199120,1cyclictest0-21swapper/111:09:461
2433699129,2cyclictest8342-21ssh11:18:570
2433699129,2cyclictest30435-21grep07:20:200
2433699129,2cyclictest28384-21grep08:25:120
2433699129,2cyclictest26261-21grep08:20:140
2433699129,2cyclictest2359-21grep10:55:270
2433699129,2cyclictest17215-21grep08:00:160
2433699129,2cyclictest12159-21grep08:55:270
2433699128,3cyclictest30754-21grep08:30:110
2433699128,3cyclictest13-21ksoftirqd/008:10:270
24336991212,0cyclictest0-21swapper/010:02:200
2433699120,8cyclictest0-21swapper/010:45:030
2433699120,7cyclictest7638-21perf08:50:000
2434699118,2cyclictest7206-21sed08:45:283
2434699118,2cyclictest579-21grep07:25:223
2434699118,2cyclictest5671-21rm11:16:513
2434699118,2cyclictest3697-21taskset07:33:173
2434699118,2cyclictest30813-21cut08:30:113
2434699118,2cyclictest21743-21df08:10:153
2434699118,2cyclictest17104-21ssh11:05:073
2434699118,2cyclictest15314-21wc07:55:223
2434699118,2cyclictest1508-21sed08:35:173
2434699118,2cyclictest12983-21grep09:45:173
2434699113,7cyclictest0-21swapper/309:42:423
2434699110,7cyclictest0-21swapper/308:25:133
2434399119,1cyclictest29562-21cut08:25:212
2434399118,2cyclictest26545-21chrt07:13:252
2434399118,2cyclictest22336-21tr08:10:212
2434399118,2cyclictest16129-21sort09:05:162
2434399110,2cyclictest0-21swapper/212:25:092
2434199118,2cyclictest7065-21sh12:35:131
2434199118,2cyclictest6156-21ssh10:20:041
2434199118,2cyclictest3906-21sed08:40:161
2434199118,2cyclictest31641-21wc07:25:011
2434199118,2cyclictest20173-21cut09:30:171
2434199118,2cyclictest19500-21grep09:10:171
2434199118,2cyclictest13591-21head09:00:171
2434199118,2cyclictest11424-21tr09:25:151
2434199118,2cyclictest11113-21ssh11:59:021
2434199117,3cyclictest4822-21grep07:35:171
2434199114,5cyclictest0-21swapper/111:30:141
2434199110,3cyclictest0-21swapper/111:51:251
2434199110,10cyclictest11368-21cat11:40:011
2433699118,2cyclictest8969-21grep08:50:200
2433699118,2cyclictest4830-21head07:35:170
2433699118,2cyclictest3787-21chrt07:34:280
2433699118,2cyclictest29617-21sh12:10:120
2433699118,2cyclictest21624-21ssh09:50:180
2433699118,2cyclictest13970-21apache_processe12:20:120
2433699118,2cyclictest12953-21ssh11:20:590
2433699118,2cyclictest12836-21basename09:00:120
2433699110,10cyclictest9677-21tr11:00:160
2434699108,1cyclictest3523-21expr08:40:143
2434699108,1cyclictest154162sleep309:05:093
2434699108,1cyclictest12864-21apache_volume09:00:143
2434699107,2cyclictest8812-21uname08:50:193
2434699107,2cyclictest13318-21sed07:50:273
2434399108,1cyclictest23536-21taskset08:15:102
2434399108,1cyclictest19176-21expr08:05:142
2434399108,1cyclictest11748-21wget07:50:132
2434399102,4cyclictest28614-21grep07:15:272
2434399101,5cyclictest225-21systemd-journal07:20:012
2434199108,1cyclictest9479-21expr07:45:151
2434199108,1cyclictest726-21apt08:35:131
2434199108,1cyclictest20133-21grep08:05:221
2434199107,2cyclictest6562-21wc08:45:211
2434199107,2cyclictest27455-21sed07:15:161
2434199107,2cyclictest25131-21sed07:10:161
2434199100,9cyclictest11634-21chrt07:50:021
2433699108,1cyclictest6812-21cat07:40:150
2433699108,1cyclictest24304-21head08:15:160
2433699108,1cyclictest15695-21expr09:05:140
2433699107,2cyclictest1656-21ls07:30:010
2433699107,2cyclictest14797-21tr07:55:180
243419996,2cyclictest7087-21entropy07:40:171
243419996,2cyclictest22541-21tr08:10:231
243369998,1cyclictest19229-21expr08:05:140
243469986,1cyclictest39-21ksoftirqd/307:20:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional