You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 21:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Mar 10, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2293721110,1sleep013-21ksoftirqd/010:03:400
451228976,9sleep00-21swapper/007:06:590
452428774,9sleep20-21swapper/207:07:082
456028564,6sleep10-21swapper/107:07:351
440328269,9sleep30-21swapper/307:05:323
308002750,0sleep00-21swapper/012:05:140
482799290,28cyclictest0-21swapper/110:29:481
482799280,1cyclictest0-21swapper/110:46:461
482799280,1cyclictest0-21swapper/110:46:461
4827992423,1cyclictest0-21swapper/111:49:591
483599230,2cyclictest0-21swapper/312:20:523
483599230,1cyclictest0-21swapper/311:55:283
483499230,19cyclictest0-21swapper/211:25:122
483499230,19cyclictest0-21swapper/210:00:122
482799230,2cyclictest0-21swapper/110:10:031
483599220,1cyclictest0-21swapper/312:27:263
483499220,22cyclictest0-21swapper/211:11:382
4822992220,1cyclictest0-21swapper/009:10:120
482299221,2cyclictest141rcu_preempt10:45:030
482299221,2cyclictest141rcu_preempt10:45:020
4835992120,1cyclictest0-21swapper/311:05:253
483499210,3cyclictest0-21swapper/212:34:202
4827992120,1cyclictest0-21swapper/107:19:101
483599200,3cyclictest14890-21kworker/u8:210:38:323
4827992018,2cyclictest25-21ksoftirqd/111:17:401
483599191,18cyclictest0-21swapper/312:00:203
483599190,1cyclictest0-21swapper/309:38:083
4827991917,1cyclictest0-21swapper/109:50:211
482799191,17cyclictest25556-21sed10:25:001
482299190,15cyclictest0-21swapper/010:55:120
483499180,8cyclictest0-21swapper/209:43:512
483499180,17cyclictest0-21swapper/210:17:432
482799183,14cyclictest0-21swapper/109:20:131
482799180,17cyclictest0-21swapper/110:34:491
482799180,16cyclictest0-21swapper/111:23:131
482299183,12cyclictest28643-21sh07:59:590
4822991816,2cyclictest13-21ksoftirqd/010:52:580
4834991714,2cyclictest0-21swapper/208:15:212
482799173,4cyclictest225-21systemd-journal07:50:001
4827991712,2cyclictest0-21swapper/111:10:531
482799171,16cyclictest30054-21diskmemload12:09:331
482799170,16cyclictest0-21swapper/109:16:121
482299170,13cyclictest0-21swapper/009:32:000
482299170,0cyclictest0-21swapper/011:13:240
483599169,6cyclictest5246-21apache210:53:253
483599168,7cyclictest28903-21/usr/sbin/munin08:00:143
483499160,0cyclictest0-21swapper/209:01:332
483499160,0cyclictest0-21swapper/209:01:322
482799169,4cyclictest8664-21/usr/sbin/munin08:25:171
482799167,7cyclictest30054-21diskmemload11:40:201
4827991615,1cyclictest0-21swapper/110:19:021
4827991611,5cyclictest25-21ksoftirqd/112:10:041
482799160,1cyclictest0-21swapper/110:59:441
482799160,15cyclictest0-21swapper/110:08:201
482299169,5cyclictest0-21swapper/010:35:170
4822991615,1cyclictest0-21swapper/009:23:360
4835991515,0cyclictest0-21swapper/312:11:223
4835991514,1cyclictest0-21swapper/310:41:493
4834991511,3cyclictest32-21ksoftirqd/207:50:002
483499150,2cyclictest0-21swapper/210:22:062
482799152,1cyclictest30054-21diskmemload09:46:571
4827991515,0cyclictest0-21swapper/111:52:411
4827991511,3cyclictest8846-21wc12:30:181
482299154,7cyclictest18553-21apt10:40:130
4822991510,3cyclictest21539-21turbostat07:45:000
482299150,1cyclictest0-21swapper/009:25:550
4835991413,1cyclictest0-21swapper/311:10:573
4835991411,2cyclictest9725-21expr10:15:123
4835991411,2cyclictest15059-21grep08:35:263
483599140,12cyclictest22699-21cron09:25:003
483499143,9cyclictest11204-21aten2_r7power_e10:55:152
4834991412,1cyclictest0-21swapper/212:21:432
483499141,1cyclictest0-21swapper/210:31:552
483499141,12cyclictest20672-21chrt11:39:392
483499140,13cyclictest0-21swapper/209:17:282
482799142,3cyclictest225-21systemd-journal12:00:001
4827991411,2cyclictest24423-21ssh12:20:171
482799140,7cyclictest0-21swapper/111:36:231
482299149,3cyclictest29421-21cat09:05:250
482299142,8cyclictest0-21swapper/011:00:140
482299141,7cyclictest8448-21perf08:25:000
4822991410,3cyclictest18961-21sed08:45:190
4822991410,3cyclictest16158-21grep12:35:140
4822991410,2cyclictest24097-21tr08:55:220
483599139,3cyclictest20214-21grep11:00:173
483599131,2cyclictest1142-21runrttasks10:32:083
4835991310,2cyclictest7734-21grep07:15:153
4835991310,2cyclictest31641-21sort11:45:253
4835991310,2cyclictest31339-21df12:05:163
4835991310,2cyclictest26273-21turbostat.cron07:54:593
4835991310,2cyclictest10124-21tr07:20:143
483499131,7cyclictest2967-21runrttasks09:51:102
4834991311,2cyclictest32-21ksoftirqd/210:14:262
4834991310,2cyclictest28719-21head09:05:162
4834991310,2cyclictest28508-21chrt10:25:512
4834991310,2cyclictest2572-21tr08:10:192
4834991310,2cyclictest25386-21grep07:50:202
4834991310,2cyclictest24963-21sh09:25:272
4834991310,2cyclictest20267-21cat10:40:252
4834991310,2cyclictest16849-21ssh12:35:162
4834991310,2cyclictest16841-21wc12:15:252
4834991310,2cyclictest15368-21uname11:55:192
4834991310,2cyclictest15047-21grep08:35:262
4834991310,2cyclictest15040-21cut09:20:132
482799139,3cyclictest5246-21apache210:38:481
482799139,3cyclictest17727-21sh10:00:191
482799139,3cyclictest11005-21ssh11:33:331
482799139,2cyclictest13759-21cat07:25:261
4827991313,0cyclictest0-21swapper/109:40:441
4827991312,1cyclictest0-21swapper/112:28:371
4827991310,2cyclictest25362-21sed07:50:201
4827991310,2cyclictest18131-21wc08:45:001
4827991310,2cyclictest16347-21grep07:30:261
4827991310,2cyclictest14429-21cat08:35:191
482799130,1cyclictest0-21swapper/109:26:351
482799130,12cyclictest16603-21grep12:15:221
482299139,3cyclictest19576-21sed10:20:240
482299139,3cyclictest14381-21ls09:20:010
482299139,2cyclictest12496-21grep11:15:130
482299132,5cyclictest225-21systemd-journal10:30:000
4822991310,2cyclictest7231-21sh07:15:010
4822991310,2cyclictest7231-21sh07:15:000
4822991310,2cyclictest29342-21ssh11:25:140
4822991310,2cyclictest24394-21grep07:50:130
482299130,1cyclictest0-21swapper/012:19:230
482299130,1cyclictest0-21swapper/011:36:090
483599129,2cyclictest7732-21grep11:50:253
483599129,2cyclictest7081-21cut08:20:173
483599129,2cyclictest6898-21grep11:30:263
483599129,2cyclictest31758-21grep08:05:163
483599129,2cyclictest30060-21turbostat.cron09:10:003
483599129,2cyclictest2880-21grep10:10:273
483599129,2cyclictest28544-21wc10:45:283
483599129,2cyclictest28544-21wc10:45:273
483599129,2cyclictest24584-21ssh09:25:223
483599129,2cyclictest1968-21df08:10:153
483599129,2cyclictest17095-21wc08:40:213
483599129,2cyclictest17070-21tr10:00:143
483599129,2cyclictest16844-21grep12:35:163
483599129,2cyclictest10792-21cron08:30:003
483599128,3cyclictest20126-21head07:40:153
483599128,3cyclictest14784-21tr11:35:233
483599126,5cyclictest7381-21sed09:15:153
483599126,3cyclictest24204-21hostname09:45:113
483599122,7cyclictest15803-21apt-get09:40:133
483599121,5cyclictest1142-21runrttasks10:57:423
483599120,11cyclictest6343-21sed07:10:213
483599120,11cyclictest6343-21sed07:10:213
483499129,2cyclictest8419-21cron08:25:012
483499129,2cyclictest7264-21ssh12:30:002
483499129,2cyclictest5835-21head11:30:172
483499129,2cyclictest23693-21grep08:55:182
483499129,2cyclictest20721-21tr08:50:142
483499129,2cyclictest20084-21grep07:40:152
483499129,2cyclictest19136-21idleruntime-cro07:40:002
483499129,2cyclictest18247-21grep08:45:152
483499129,2cyclictest11295-21tr08:30:142
483499129,2cyclictest10375-21sed09:55:252
483499128,3cyclictest26052-21grep09:45:242
483499128,3cyclictest10543-21tr10:35:152
483499121,6cyclictest225-21systemd-journal08:09:592
483499121,5cyclictest0-21swapper/211:50:182
483499121,10cyclictest0-21swapper/210:08:522
483499120,11cyclictest12512-21tr07:25:142
482799129,2cyclictest4106-21grep08:15:141
482799129,2cyclictest32614-21grep08:05:201
482799129,2cyclictest31378-21grep09:10:161
482799129,2cyclictest23263-21grep12:00:161
482799129,2cyclictest21115-21grep07:40:241
482799129,2cyclictest14215-21chrt09:58:241
482799128,3cyclictest3121-21sed08:10:261
482799128,2cyclictest28150-21grep07:55:271
482799128,2cyclictest16373-21tr12:35:151
482799121,9cyclictest19920-21pmu-power10:40:211
482799120,2cyclictest0-21swapper/111:00:041
482799120,11cyclictest0-21swapper/109:38:331
482299129,2cyclictest9917-21grep08:25:230
482299129,2cyclictest7527-21cut12:10:160
482299129,2cyclictest32481-21cut12:25:160
482299129,2cyclictest29168-21tr08:00:140
482299129,2cyclictest27051-21sed10:05:260
482299129,2cyclictest23438-21ssh12:20:100
482299129,2cyclictest22574-21cut12:00:130
482299129,2cyclictest22208-21grep08:50:250
482299129,2cyclictest16545-21head08:40:150
482299129,2cyclictest11342-21grep08:30:130
482299128,3cyclictest23613-21sh09:44:590
482299128,2cyclictest4893-21wc08:15:180
482299128,2cyclictest30259-21tr11:45:160
482299127,3cyclictest0-21swapper/009:56:040
482299124,4cyclictest225-21systemd-journal12:34:550
482299120,12cyclictest0-21swapper/011:42:180
482299120,11cyclictest17181-21cron10:20:010
483599119,1cyclictest21655-21grep11:20:183
483599118,2cyclictest32512-21ssh09:30:183
483599118,2cyclictest31316-21ssh11:25:373
483599118,2cyclictest31077-21rm09:10:153
483599118,2cyclictest28471-21chrt07:57:543
483599118,2cyclictest26136-21cat10:05:183
483599118,2cyclictest2332-21ssh09:50:393
483599118,2cyclictest22850-21date08:55:003
483599118,2cyclictest21714-21basename11:40:133
483599118,2cyclictest21575-21ssh10:21:543
483599118,2cyclictest17834-21hostname07:35:193
483599118,2cyclictest13015-21grep11:15:163
483599118,2cyclictest12377-21ssh12:32:303
483599118,2cyclictest12370-21cut07:25:143
483599118,2cyclictest10888-21ssh09:55:473
483599116,5cyclictest5231-21apache210:25:163
483599115,3cyclictest25841-21df_abs09:00:143
483599115,3cyclictest25841-21df_abs09:00:143
483599110,8cyclictest0-21swapper/312:15:223
483599110,10cyclictest23567-21chrt07:45:513
483499118,2cyclictest8241-21grep07:15:182
483499118,2cyclictest7812-21mii-tool12:10:182
483499118,2cyclictest31638-21basename09:30:132
483499118,2cyclictest31100-21missed_timers11:45:202
483499118,2cyclictest29306-21ssh10:45:482
483499118,2cyclictest29306-21ssh10:45:482
483499118,2cyclictest27948-21tr11:05:142
483499118,2cyclictest2510-21ssh12:07:082
483499118,2cyclictest24409-21ssh11:02:492
483499118,2cyclictest23286-21grep12:00:162
483499118,2cyclictest22374-21ssh11:40:162
483499118,2cyclictest21065-21taskset11:20:162
483499118,2cyclictest1426-21ssh09:11:262
483499118,2cyclictest10705-21sh09:36:262
483499117,3cyclictest31171-21wc08:05:002
483499110,1cyclictest141rcu_preempt10:53:062
483499110,10cyclictest155932sleep211:16:312
482799118,2cyclictest795-21ssh11:27:071
482799118,2cyclictest4640-21awk10:50:251
482799118,2cyclictest31348-21ssh09:30:061
482799118,2cyclictest29260-21wc09:05:211
482799118,2cyclictest27681-21ssh11:05:121
482799118,2cyclictest22229-21tail08:50:261
482799118,2cyclictest18940-21head08:45:191
482799118,2cyclictest13093-21chrt08:34:051
482799118,2cyclictest11428-21cat07:20:241
482299118,2cyclictest9938-21ssh09:35:490
482299118,2cyclictest9243-21ssh10:34:350
482299118,2cyclictest9097-21ssh11:51:240
482299118,2cyclictest8561-21wc07:15:210
482299118,2cyclictest4875-21ssh11:30:090
482299118,2cyclictest32264-21ssh09:50:050
482299118,2cyclictest31559-21grep08:05:140
482299118,2cyclictest27880-21ssh09:46:480
482299118,2cyclictest26067-21grep09:00:150
482299118,2cyclictest26067-21grep09:00:150
482299118,2cyclictest22673-21grep11:20:290
482299118,2cyclictest19002-21chrt07:38:280
482299118,2cyclictest1808-21head10:10:170
482299118,2cyclictest16859-21rm11:55:590
482299118,2cyclictest15240-21taskset08:36:130
482299118,2cyclictest12899-21grep07:25:170
482299117,3cyclictest28467-21tail11:05:160
483599108,1cyclictest18508-21ls08:45:153
483599107,2cyclictest4113-21tr08:15:143
483599107,2cyclictest14884-21expr07:30:163
483599107,2cyclictest11275-21grep08:30:143
483499108,1cyclictest98582sleep207:20:032
483499108,1cyclictest9361-21tune2fs08:25:182
483499108,1cyclictest6983-21chrt07:11:572
483499108,1cyclictest6983-21chrt07:11:562
483499107,2cyclictest28557-21taskset07:58:592
483499100,10cyclictest0-21swapper/208:40:122
482799108,1cyclictest6280-21taskset08:20:061
482799108,1cyclictest5487-21expr07:10:151
482799108,1cyclictest5487-21expr07:10:141
482799108,1cyclictest25492-21grep09:00:141
482799108,1cyclictest25492-21grep09:00:131
482799108,1cyclictest22947-21basename08:55:131
482799107,2cyclictest30043-21cut08:00:201
482299108,1cyclictest21850-21aten2_r7power_e07:45:150
482299107,2cyclictest1618-21tr08:10:140
482299107,2cyclictest15040-21ls07:30:150
482299107,2cyclictest10476-21df07:20:140
48359996,2cyclictest24648-21grep08:55:263
48349997,1cyclictest164332sleep207:30:552
48279981,2cyclictest17294-21tr07:35:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional