You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-03 - 22:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Aug 03, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7536210693,8sleep00-21swapper/007:08:020
747628775,8sleep20-21swapper/207:07:162
243192860,1sleep1241rcuc/112:36:201
737928471,9sleep30-21swapper/307:06:003
737028365,14sleep10-21swapper/107:05:541
7782992825,3cyclictest0-21swapper/210:56:332
778299270,26cyclictest0-21swapper/209:39:262
7784992524,1cyclictest0-21swapper/310:32:053
778299250,24cyclictest0-21swapper/209:48:502
778299240,24cyclictest0-21swapper/211:49:352
778299240,23cyclictest0-21swapper/212:23:302
39652240,1sleep00-21swapper/008:10:000
7784992222,0cyclictest0-21swapper/309:16:313
778299220,21cyclictest0-21swapper/212:15:412
7777992218,4cyclictest13-21ksoftirqd/011:57:370
778499211,0cyclictest0-21swapper/310:36:543
778499210,1cyclictest0-21swapper/310:50:523
778299210,4cyclictest0-21swapper/211:22:092
264612210,0sleep339-21ksoftirqd/310:41:583
7784992020,0cyclictest0-21swapper/311:58:223
778499200,1cyclictest0-21swapper/309:49:563
778499200,0cyclictest0-21swapper/309:40:473
778299200,18cyclictest0-21swapper/211:25:372
7782991913,5cyclictest0-21swapper/211:55:302
778299190,1cyclictest0-21swapper/211:10:392
778299190,19cyclictest0-21swapper/212:34:212
778299190,0cyclictest0-21swapper/212:05:512
7777991913,2cyclictest13-21ksoftirqd/012:02:010
7784991816,1cyclictest0-21swapper/310:21:323
778299182,3cyclictest141rcu_preempt11:01:362
7782991816,2cyclictest0-21swapper/209:41:132
7782991814,3cyclictest0-21swapper/210:36:032
778299180,17cyclictest0-21swapper/210:47:042
778299180,17cyclictest0-21swapper/209:24:272
777999181,8cyclictest0-21swapper/109:30:541
777999180,18cyclictest0-21swapper/110:37:501
7777991816,1cyclictest0-21swapper/011:08:340
142032180,0sleep00-21swapper/010:35:180
7782991717,0cyclictest0-21swapper/211:18:252
7782991716,1cyclictest25690-21ssh10:03:032
7782991716,1cyclictest0-21swapper/211:39:432
7782991715,2cyclictest0-21swapper/209:53:422
778299170,17cyclictest0-21swapper/209:32:492
778299170,0cyclictest0-21swapper/211:52:432
778299170,0cyclictest0-21swapper/211:05:132
777999172,2cyclictest7892-21ssh12:26:411
777999170,15cyclictest0-21swapper/110:48:531
7784991615,1cyclictest0-21swapper/308:20:133
778499160,13cyclictest0-21swapper/311:31:523
7782991615,1cyclictest0-21swapper/210:05:132
777999165,8cyclictest23976-21apache211:00:171
777999162,1cyclictest785-21diskmemload09:38:211
7779991612,3cyclictest25-21ksoftirqd/110:00:151
777999160,1cyclictest0-21swapper/112:09:251
7777991612,2cyclictest20505-21cut07:35:170
778499151,3cyclictest0-21swapper/311:40:143
778499150,15cyclictest0-21swapper/309:33:493
778299154,10cyclictest0-21swapper/212:00:312
7782991514,1cyclictest0-21swapper/212:35:552
7782991514,1cyclictest0-21swapper/211:43:362
7782991514,1cyclictest0-21swapper/210:44:022
7782991511,3cyclictest26166-21wc08:55:012
778299150,1cyclictest0-21swapper/209:58:432
778299150,13cyclictest0-21swapper/212:28:342
777999154,1cyclictest22104-21ssh10:40:001
777999152,13cyclictest785-21diskmemload10:52:501
7779991514,1cyclictest0-21swapper/109:15:361
777999151,1cyclictest785-21diskmemload12:22:381
777999150,1cyclictest0-21swapper/111:07:141
777999150,1cyclictest0-21swapper/110:10:051
777999150,0cyclictest0-21swapper/110:34:481
7777991514,1cyclictest785-21diskmemload10:13:310
275522150,2sleep00-21swapper/009:45:100
778499149,2cyclictest26425-21grep08:55:143
778499149,2cyclictest0-21swapper/312:15:153
778499141,12cyclictest0-21swapper/312:22:213
7784991411,2cyclictest0-21swapper/310:58:343
7784991410,3cyclictest18419-21cat11:15:273
7784991410,3cyclictest16104-21awk11:15:003
7784991410,2cyclictest14230-21grep08:30:143
778499140,3cyclictest0-21swapper/312:28:543
778499140,1cyclictest0-21swapper/310:45:123
7782991414,0cyclictest0-21swapper/210:52:232
7782991413,1cyclictest0-21swapper/210:23:002
7782991412,1cyclictest4653-21chrt10:29:432
7782991410,3cyclictest20966-21ls07:35:212
7782991410,2cyclictest10902-21sort08:20:262
778299140,1cyclictest0-21swapper/210:19:442
778299140,1cyclictest0-21swapper/210:19:442
777999141,12cyclictest785-21diskmemload11:47:151
777999140,1cyclictest0-21swapper/111:15:081
777799146,7cyclictest0-21swapper/009:40:120
7777991411,2cyclictest18623-21cut07:30:210
7777991410,3cyclictest16290-21ssh11:53:140
7777991410,2cyclictest7286-21df08:15:150
777799140,13cyclictest0-21swapper/010:32:520
777799140,12cyclictest0-21swapper/010:15:130
777799140,12cyclictest0-21swapper/010:15:120
27622140,8sleep00-21swapper/010:47:170
778499139,3cyclictest3916-21tr09:50:143
778499132,10cyclictest25916-21hddtemp_smartct11:20:173
7784991311,1cyclictest23976-21apache209:21:093
778499131,11cyclictest0-21swapper/312:01:373
778499131,10cyclictest20638-21latency_hist12:35:003
7784991310,2cyclictest8934-21perf08:20:003
7784991310,2cyclictest6345-21wc10:10:263
7784991310,2cyclictest31874-21grep09:05:193
7784991310,2cyclictest21324-21uname10:00:193
7784991310,2cyclictest10457-21grep07:15:153
778499130,3cyclictest0-21swapper/310:09:123
778499130,12cyclictest17546-21basename07:30:153
778299139,3cyclictest15398-21grep07:25:162
778299138,3cyclictest12919-21tr08:25:212
7782991310,2cyclictest32417-21wc09:05:242
7782991310,2cyclictest3095-21wc08:05:212
7782991310,2cyclictest15796-21grep08:30:272
7782991310,2cyclictest10682-21grep09:15:162
7782991310,2cyclictest10647-21grep07:15:152
778299130,2cyclictest0-21swapper/209:25:052
777999139,4cyclictest0-21swapper/111:24:421
777999139,2cyclictest10119-21sh07:15:011
777999136,6cyclictest0-21swapper/111:29:071
777999132,3cyclictest9662-21grep08:20:141
7779991313,0cyclictest0-21swapper/111:32:471
777999131,2cyclictest785-21diskmemload11:40:281
7779991312,1cyclictest25-21ksoftirqd/110:16:481
7779991312,1cyclictest25-21ksoftirqd/110:16:481
7779991312,1cyclictest0-21swapper/110:24:591
7779991311,1cyclictest0-21swapper/112:02:251
7779991310,2cyclictest9469-21ls11:10:211
7779991310,2cyclictest31519-21grep10:25:261
7779991310,2cyclictest20893-21grep11:55:261
7779991310,2cyclictest20525-21grep09:40:191
7779991310,2cyclictest15485-21ssh10:55:141
7779991310,2cyclictest13874-21tail07:20:211
7779991310,2cyclictest1348-21cut09:10:141
777799131,8cyclictest0-21swapper/012:26:100
7777991310,2cyclictest32664-21sed08:00:200
7777991310,2cyclictest28070-21wc11:40:260
7777991310,2cyclictest27505-21head09:25:170
7777991310,2cyclictest24670-21tail08:50:160
7777991310,2cyclictest22296-21sort08:45:220
7777991310,2cyclictest2019-21grep09:10:170
777799130,13cyclictest0-21swapper/012:33:190
777799130,13cyclictest0-21swapper/012:06:200
777799130,12cyclictest32291-21ls09:05:230
778499129,2cyclictest746-21grep11:05:173
778499129,2cyclictest3539-21head08:05:273
778499129,2cyclictest3145-21tail11:45:183
778499129,2cyclictest17558-21cat08:35:173
778499129,2cyclictest14829-21sed10:15:263
778499129,2cyclictest14829-21sed10:15:263
778499129,1cyclictest0-21swapper/309:11:133
778499128,3cyclictest31985-21cut08:00:173
778499128,3cyclictest20738-21rm12:35:003
778499128,3cyclictest15409-21ssh12:11:493
778499128,2cyclictest22875-21wc08:45:293
778499128,2cyclictest11823-21grep08:25:143
778499127,3cyclictest14093-21grep07:20:243
7784991211,1cyclictest7768-21cyclictest11:27:313
778299129,2cyclictest9627-21grep11:30:132
778299129,2cyclictest8720-21tail07:10:162
778299129,2cyclictest22563-21tr07:40:122
778299129,2cyclictest11633-21ssh12:10:122
778299128,3cyclictest32-21ksoftirqd/210:34:212
778299120,2cyclictest0-21swapper/210:10:052
778299120,12cyclictest0-21swapper/209:11:362
777999129,2cyclictest845-21ssh09:48:161
777999129,2cyclictest7420-21grep08:15:161
777999129,2cyclictest4112-21cat08:10:001
777999129,2cyclictest25020-21tr07:45:151
777999129,2cyclictest20177-21grep07:35:151
777999129,2cyclictest19785-21cat07:35:001
777999129,2cyclictest18072-21wc08:35:231
777999129,2cyclictest10494-21cut07:15:151
777999128,3cyclictest27391-21grep07:50:161
777999128,3cyclictest19798-21tr10:00:001
777999128,3cyclictest16259-21cron08:35:001
777999128,2cyclictest20274-21tr12:15:151
777999127,4cyclictest23964-21apache211:35:071
777999123,3cyclictest225-21systemd-journal09:55:001
777999122,8cyclictest0-21swapper/112:14:041
7779991211,1cyclictest38150irq/131-snd_hda10:06:451
777999120,11cyclictest772-21ls09:10:001
777799129,2cyclictest29505-21tail10:05:170
777799129,2cyclictest24635-21sort11:00:180
777799129,2cyclictest23062-21head07:40:150
777799129,2cyclictest21359-21tr12:35:150
777799128,3cyclictest11979-21grep12:10:150
777799128,3cyclictest10181-21head08:20:160
777799124,4cyclictest16534-21apt-get08:35:110
777799120,9cyclictest0-21swapper/010:51:430
295682124,3sleep329537-21/usr/sbin/munin10:25:113
778499119,1cyclictest24238-21ls11:00:173
778499118,2cyclictest9134-21ssh12:08:243
778499118,2cyclictest4892-21cat08:10:143
778499118,2cyclictest30577-21tr07:55:223
778499118,2cyclictest27471-21ssh09:25:173
778499118,2cyclictest24978-21tr07:45:153
778499118,2cyclictest21985-21chrt07:39:503
778499118,2cyclictest20326-21ssh11:35:553
778499118,2cyclictest15883-21grep07:25:183
778499118,2cyclictest15706-21ssh09:56:513
778499118,2cyclictest12602-21tail11:50:253
778499118,2cyclictest12359-21grep09:35:223
778499115,3cyclictest9477-21/usr/sbin/munin07:10:243
778499110,9cyclictest0-21swapper/307:50:133
778499110,10cyclictest25753-21chrt08:51:143
778299118,2cyclictest32026-21grep08:00:172
778299118,2cyclictest29739-21tr07:55:172
778299118,2cyclictest29323-21cat07:55:002
778299118,2cyclictest28992-21tr09:00:152
778299118,2cyclictest27462-21tr08:55:202
778299118,2cyclictest17574-21awk07:30:152
778299118,2cyclictest16961-21expr08:35:132
778299117,3cyclictest21373-21grep08:45:162
778299117,2cyclictest6901-21grep08:15:142
778299110,2cyclictest0-21swapper/207:20:212
777999118,2cyclictest4846-21expr08:10:141
777999118,2cyclictest32577-21perl08:00:191
777999118,2cyclictest30269-21sed07:55:191
777999118,2cyclictest24595-21fschecks_time08:50:161
777999118,2cyclictest21870-21sh09:21:421
777999118,2cyclictest11394-21cut11:50:161
777799118,2cyclictest9481-21cut11:30:130
777799118,2cyclictest8727-21head07:10:160
777799118,2cyclictest6481-21chrt08:14:330
777799118,2cyclictest5474-21ssh11:46:210
777799118,2cyclictest323-21ssh12:22:200
777799118,2cyclictest31949-21ssh10:25:410
777799118,2cyclictest2850-21expr09:30:150
777799118,2cyclictest27180-21cut07:50:150
777799118,2cyclictest26858-21perf07:50:000
777799118,2cyclictest26609-21grep08:55:150
777799118,2cyclictest24486-21ssh10:02:070
777799118,2cyclictest22997-21ssh10:20:270
777799118,2cyclictest20185-21sh11:16:410
777799118,2cyclictest15225-21ssh11:14:200
777799118,2cyclictest14661-21chrt07:22:120
777799118,2cyclictest12258-21grep09:55:150
777799118,2cyclictest11158-21ssh09:15:200
777799118,2cyclictest11023-21basename09:35:140
777799118,2cyclictest10580-21cpuspeed_turbos07:15:150
777799117,3cyclictest3892-21tr09:50:140
777799113,4cyclictest225-21systemd-journal11:29:590
777799111,8cyclictest21605-21sensors12:15:250
778499107,2cyclictest30576-21chrt09:01:563
778499107,2cyclictest24268-21taskset07:42:363
778499100,5cyclictest0-21swapper/308:40:133
778299107,2cyclictest5658-21grep08:10:212
778299107,2cyclictest26832-21taskset07:50:002
778299100,3cyclictest18982-21sshd08:40:142
777999108,1cyclictest30682-21taskset09:03:191
777999108,1cyclictest26425-21ssh09:25:021
777999108,1cyclictest23348-21missed_timers07:40:171
777999108,1cyclictest22062-21cut08:45:191
777999108,1cyclictest209492sleep108:44:031
777999108,1cyclictest12622-21sh12:30:131
777999107,2cyclictest26445-21grep08:55:141
777999107,2cyclictest17047-21taskset07:27:231
777999107,2cyclictest12872-21needreboot08:25:201
777799108,1cyclictest30726-21taskset09:03:530
777799108,1cyclictest24764-21ssh11:20:000
777799108,1cyclictest20410-21ssh11:35:580
777799108,1cyclictest17008-21chrt07:26:530
777799108,1cyclictest14298-21grep08:30:150
777799108,1cyclictest11785-21apache_accesses08:25:120
777799107,2cyclictest30189-21grep07:55:190
777799107,2cyclictest20280-21cut08:40:230
777799107,2cyclictest18719-21expr09:20:140
777799107,2cyclictest15221-21sh10:55:040
777799100,9cyclictest23347-21cat10:40:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional