You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-11 - 22:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jun 11, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15397210476,9sleep20-21swapper/207:07:122
15288210264,34sleep00-21swapper/007:05:460
1553829266,9sleep30-21swapper/307:09:023
1528428568,13sleep10-21swapper/107:05:421
147352580,0sleep20-21swapper/212:25:402
15707992423,1cyclictest0-21swapper/308:50:133
15700992423,1cyclictest0-21swapper/110:11:531
15697992421,2cyclictest21469-21df_abs10:15:140
1570799231,0cyclictest0-21swapper/310:22:153
1570799220,21cyclictest0-21swapper/312:36:303
1570099220,21cyclictest0-21swapper/109:59:271
1570099220,20cyclictest0-21swapper/111:47:241
1569799220,21cyclictest0-21swapper/010:28:430
15703992119,2cyclictest32-21ksoftirqd/209:48:332
1570399210,20cyclictest0-21swapper/210:30:182
15700992111,1cyclictest0-21swapper/112:24:421
1570099210,2cyclictest0-21swapper/111:51:501
1570099210,2cyclictest0-21swapper/110:57:251
1570099210,0cyclictest0-21swapper/110:48:231
304552200,0sleep30-21swapper/307:40:033
1570099201,1cyclictest0-21swapper/110:22:461
15697992019,0cyclictest0-21swapper/009:19:580
1570099190,2cyclictest0-21swapper/109:51:561
1570099190,19cyclictest0-21swapper/112:12:421
15697991917,2cyclictest13-21ksoftirqd/010:38:000
15707991812,4cyclictest26561-21runrttasks10:18:153
15703991818,0cyclictest0-21swapper/209:12:112
1570399181,15cyclictest0-21swapper/211:55:112
15700991817,1cyclictest0-21swapper/110:01:291
1569799186,6cyclictest24021-21munin-run11:14:590
15697991814,3cyclictest0-21swapper/009:52:550
315912170,1sleep30-21swapper/310:01:203
1570799171,1cyclictest0-21swapper/311:46:513
1570399171,1cyclictest0-21swapper/211:23:402
1570399170,15cyclictest0-21swapper/211:25:222
15700991714,1cyclictest0-21swapper/110:51:541
15700991712,3cyclictest8764-21diskmemload11:00:251
1570099170,12cyclictest0-21swapper/112:25:071
15697991710,4cyclictest26520-21apache210:23:550
15707991611,3cyclictest0-21swapper/309:47:113
15703991610,5cyclictest26520-21apache211:17:342
1570099162,2cyclictest537-21snmpd09:38:101
15700991613,2cyclictest0-21swapper/111:24:321
15700991611,4cyclictest25-21ksoftirqd/109:41:151
1570099160,1cyclictest0-21swapper/108:50:141
1569799169,4cyclictest19487-21ssh12:10:110
1569799160,1cyclictest0-21swapper/010:58:490
1569799160,15cyclictest0-21swapper/010:31:200
134202160,1chrt13421-21/usr/sbin/munin12:25:180
15707991513,1cyclictest0-21swapper/311:35:223
1570399150,15cyclictest0-21swapper/209:30:562
1570099157,5cyclictest4836-21ssh10:05:151
15700991514,1cyclictest8764-21diskmemload09:20:331
1570099150,2cyclictest0-21swapper/111:09:361
1569799152,1cyclictest8764-21diskmemload11:40:540
15697991511,3cyclictest22750-21cron10:55:000
1569799150,13cyclictest15068-21cat08:15:010
1570799149,3cyclictest12026-21cron10:10:013
1570799142,4cyclictest225-21systemd-journal12:20:013
15707991413,1cyclictest0-21swapper/311:44:103
1570799141,1cyclictest0-21swapper/308:55:123
1570799140,13cyclictest3655-21which12:00:183
1570799140,13cyclictest3655-21which12:00:173
1570799140,13cyclictest0-21swapper/309:52:503
1570399143,10cyclictest0-21swapper/210:04:502
1570399142,6cyclictest0-21swapper/210:43:262
15703991412,1cyclictest22117-21kworker/u8:111:40:142
15703991410,3cyclictest4997-21grep10:05:172
1570399140,13cyclictest28281-21tr09:40:162
1570099142,11cyclictest8764-21diskmemload10:19:361
15700991412,1cyclictest38150irq/131-snd_hda11:25:121
15700991411,2cyclictest22206-21head12:30:211
1570099141,11cyclictest8764-21diskmemload12:02:241
1570099141,11cyclictest8764-21diskmemload12:02:241
1570099140,1cyclictest0-21swapper/112:18:481
1570099140,13cyclictest0-21swapper/107:45:141
1570799139,3cyclictest3594-21strings09:25:213
1570799139,3cyclictest2238-21sleep12:00:003
1570799139,2cyclictest19538-21grep11:50:203
15707991310,2cyclictest15964-21sed08:15:153
1570399139,3cyclictest3866-21grep07:50:182
1570399139,2cyclictest18417-21grep07:15:142
1570399133,6cyclictest26440-21cpuspeed_turbos09:20:132
15703991312,1cyclictest0-21swapper/212:34:262
15703991310,2cyclictest7257-21ssh10:45:162
1570399130,13cyclictest0-21swapper/211:36:582
1570099139,3cyclictest3568-21grep09:45:131
1570099139,3cyclictest10497-21cut09:10:191
1570099138,4cyclictest20560-21ssh11:31:321
1570099131,8cyclictest38150irq/131-snd_hda11:40:111
1570099131,11cyclictest4911-21cut09:00:171
15700991310,2cyclictest8603-21grep08:00:171
15700991310,2cyclictest28272-21strings08:40:191
1570099130,1cyclictest0-21swapper/109:31:481
1570099130,1cyclictest0-21swapper/109:17:491
1570099130,12cyclictest18418-21cut07:15:141
1570099130,12cyclictest0-21swapper/107:26:451
1569799139,3cyclictest7414-21sed09:05:180
1569799139,3cyclictest26913-21sort09:20:150
1569799139,3cyclictest20084-21if_err_enp2s009:35:190
1569799139,2cyclictest21477-21sed07:20:150
1569799138,3cyclictest21110-21tr12:30:130
15697991310,2cyclictest992-21tr11:40:000
15697991310,2cyclictest25350-21tail11:15:170
15697991310,2cyclictest21199-21grep08:25:220
15697991310,2cyclictest15456-21grep08:15:130
15697991310,2cyclictest10940-21grep08:05:160
1569799130,3cyclictest0-21swapper/011:26:490
1569799130,11cyclictest0-21swapper/011:08:530
1570799129,2cyclictest8334-21sed09:05:273
1570799129,2cyclictest5531-21wc09:00:233
1570799129,2cyclictest39-21ksoftirqd/309:40:113
1570799129,2cyclictest30529-21grep10:40:153
1570799129,2cyclictest24093-21df10:55:153
1570799129,2cyclictest23848-21head07:25:163
1570799129,2cyclictest23222-21tr10:35:203
1570799129,2cyclictest21463-21sed07:20:153
1570799129,2cyclictest20549-21ssh12:10:183
1570799129,2cyclictest19074-21ssh10:52:073
1570799129,2cyclictest17747-21ssh09:15:123
1570799129,2cyclictest17472-21tr11:30:143
1570799129,2cyclictest14176-21grep10:30:153
1570799129,2cyclictest13056-21tr08:10:133
1570799129,2cyclictest10852-21grep08:05:163
1570799128,3cyclictest26873-21cron08:40:013
1570799124,4cyclictest225-21systemd-journal12:30:003
1570799120,11cyclictest26973-21cron09:39:593
1570399129,2cyclictest8107-21tr11:05:152
1570399129,2cyclictest7104-21sed10:25:252
1570399129,2cyclictest31628-21tr07:40:212
1570399129,2cyclictest21517-21grep09:55:212
1570399129,2cyclictest19106-21ssh12:10:002
1570399129,2cyclictest19004-21head11:50:152
1570399129,2cyclictest18928-21cat09:15:202
1570399129,2cyclictest16617-21taskset07:10:162
1570399129,2cyclictest1480-21cut07:45:172
1570399129,2cyclictest13769-21grep10:10:192
1570399128,3cyclictest803-21rm10:21:562
1570399128,3cyclictest22270-21tr10:35:152
1570399128,3cyclictest2136-21tr08:55:132
1570399128,3cyclictest21213-21sort12:10:252
1570399128,2cyclictest20944-21tr07:20:122
1570399126,2cyclictest225-21systemd-journal08:20:002
1570399123,7cyclictest16157-21/usr/sbin/munin10:50:192
1570399121,5cyclictest0-21swapper/207:55:142
1570399120,8cyclictest0-21swapper/207:35:122
1570399120,10cyclictest22403-21grep10:15:222
1570099129,2cyclictest30102-21cut12:35:161
1570099129,2cyclictest2270-21sendmail-msp12:00:011
1570099129,2cyclictest18371-21cut08:20:171
1570099128,3cyclictest5873-21munin-plugin-st10:45:001
1570099128,3cyclictest3605-21uname09:25:201
1570099128,3cyclictest17606-21tail07:10:251
1570099128,2cyclictest25565-21tr07:30:141
1570099124,4cyclictest225-21systemd-journal07:25:001
1570099124,3cyclictest477-21rs:main1
1570099120,1cyclictest0-21swapper/111:15:131
1569799129,2cyclictest7598-21tr07:59:590
1569799129,2cyclictest3909-21ssh09:25:220
1569799129,2cyclictest3632-21head12:00:170
1569799129,2cyclictest3632-21head12:00:170
1569799129,2cyclictest28662-21grep07:35:160
1569799129,2cyclictest27307-21sed11:55:130
1569799129,2cyclictest26192-21tail07:30:160
1569799129,2cyclictest23123-21sed08:30:170
1569799129,2cyclictest20559-21cat10:15:010
1569799129,2cyclictest17680-21dump-pmu-power11:50:010
1569799129,2cyclictest11668-21ssh12:05:150
1569799128,3cyclictest28290-21grep09:40:150
1569799128,2cyclictest25442-21grep08:35:180
1569799128,2cyclictest10921-21tr09:30:130
1569799124,4cyclictest225-21systemd-journal07:40:000
1569799122,7cyclictest30045-21gltestperf12:35:160
15697991210,1cyclictest20046-21ssh11:31:080
1569799120,12cyclictest0-21swapper/011:23:560
1570799118,2cyclictest9991-21sh11:05:333
1570799118,2cyclictest9707-21expr09:10:143
1570799118,2cyclictest5751-21grep12:20:263
1570799118,2cyclictest5039-21ssh10:25:043
1570799118,2cyclictest29705-21seq07:36:193
1570799118,2cyclictest29409-21ssh11:17:443
1570799118,2cyclictest26572-21missed_timers07:30:183
1570799118,2cyclictest26413-21tr09:20:133
1570799118,2cyclictest22825-21ssh12:30:273
1570799118,2cyclictest2183-21ssh11:00:593
1570799118,2cyclictest21289-21ssh09:55:183
1570799118,2cyclictest19694-21cron08:25:013
1570799118,2cyclictest17978-21taskset07:14:193
1570799118,2cyclictest1738-21ssh11:20:193
1570799118,2cyclictest15145-21ssh12:07:003
1570799118,2cyclictest11758-21sh10:47:563
1570799118,2cyclictest11570-21sort09:30:153
1570799115,5cyclictest12557-21ssh10:10:133
1570799110,11cyclictest0-21swapper/311:10:533
1570399118,2cyclictest5294-21ssh12:00:502
1570399118,2cyclictest5294-21ssh12:00:502
1570399118,2cyclictest4229-21ssh11:02:352
1570399118,2cyclictest3732-21chrt12:20:032
1570399118,2cyclictest30707-21ssh12:16:022
1570399118,2cyclictest30465-21grep08:45:172
1570399118,2cyclictest29257-21ssh12:35:112
1570399118,2cyclictest23755-21cut10:55:142
1570399118,2cyclictest2328-21sh09:25:112
1570399118,2cyclictest19884-21grep09:35:172
1570399118,2cyclictest17392-21cat11:30:142
1570399118,2cyclictest16066-21ssh11:10:082
1570399118,2cyclictest1452-21chrt08:53:102
1570399118,2cyclictest13630-21basename08:10:172
1570399118,2cyclictest12575-21grep09:50:172
1570399118,2cyclictest11143-21open_inodes11:45:202
1570399117,3cyclictest9975-21perf08:04:592
1570399117,3cyclictest6680-21cut09:05:142
1570099118,2cyclictest8430-21rm10:26:211
1570099118,2cyclictest31248-21grep07:40:171
1570099118,2cyclictest30400-21ssh11:37:401
1570099118,2cyclictest29116-21head07:35:211
1570099118,2cyclictest26431-21sed08:35:271
1570099118,2cyclictest20712-21ssh11:12:251
1570099118,2cyclictest20238-21cut08:25:131
1570099118,2cyclictest12250-21cat12:05:191
1570099116,3cyclictest23180-21grep10:35:201
1570099112,6cyclictest30032-21sed08:45:141
1570099111,3cyclictest0-21swapper/109:00:001
1569799119,1cyclictest7421-21ssh12:21:400
1569799118,2cyclictest7941-21sh10:45:210
1569799118,2cyclictest6784-21ssh11:04:350
1569799118,2cyclictest4460-21tr10:05:130
1569799118,2cyclictest30415-21ssh10:00:270
1569799118,2cyclictest30214-21ssh10:40:040
1569799118,2cyclictest29590-21basename08:45:120
1569799118,2cyclictest20077-21ssh09:55:110
1569799118,2cyclictest18217-21grep08:20:170
1569799118,2cyclictest16559-21sort07:10:160
1569799118,2cyclictest11095-21date09:10:250
1569799117,2cyclictest18964-21sort07:15:170
15697991111,0cyclictest0-21swapper/011:50:120
1569799110,3cyclictest0-21swapper/012:15:120
1570799108,1cyclictest5739-21cut07:55:153
1570799108,1cyclictest31410-21taskset08:47:033
1570799108,1cyclictest2579-21taskset07:47:053
1570799107,2cyclictest9158-21cut08:00:223
1570799107,2cyclictest5087-21taskset07:53:463
1570799107,2cyclictest28408-21head08:40:213
1570799107,2cyclictest24037-21grep08:30:263
1570799107,2cyclictest20439-21cat08:25:133
1570799107,2cyclictest18382-21grep07:15:133
1570799107,2cyclictest10415-21seq11:25:243
1570399108,1cyclictest24908-21chrt07:26:272
1570399107,2cyclictest5992-21sed09:00:272
1570399107,2cyclictest27173-21apt-get08:40:122
1570399107,2cyclictest23478-21ntp_kernel_pll_08:30:202
1570399107,2cyclictest17592-21apache_processe08:20:132
1570399107,2cyclictest10876-21grep08:05:152
1570099108,1cyclictest6608-21chrt09:05:071
1570099108,1cyclictest4001-21interrupts07:50:191
1570099108,1cyclictest15662-21cat08:15:131
1570099107,2cyclictest14990-21cat08:15:011
1570099107,2cyclictest10239-21cut08:05:121
1570099100,1cyclictest0-21swapper/108:30:131
1569799108,1cyclictest4476-21grep09:00:140
1569799108,1cyclictest3302-21ssh09:45:080
1569799107,2cyclictest3628-21sed07:50:160
1569799107,2cyclictest31954-21grep08:50:140
1569799107,2cyclictest27844-21grep08:40:160
1569799107,2cyclictest23119-21tr07:25:120
1569799100,9cyclictest1141-21expr07:45:150
157039997,1cyclictest20869-21cat08:25:182
157039996,2cyclictest26243-21sed07:30:162
157039991,4cyclictest24953-21cpuspeed_turbos08:35:152
157009997,1cyclictest5897-21expr07:55:151
156979997,1cyclictest8191-21expr08:00:140
156979997,1cyclictest2930-21cat08:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional