You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-18 - 17:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Aug 18, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
814929278,9sleep00-21swapper/007:08:420
810929178,9sleep20-21swapper/207:08:122
820928265,13sleep10-21swapper/107:09:271
806028269,9sleep30-21swapper/307:07:333
7762500,0sleep00-21swapper/009:29:060
834099313,27cyclictest922-21diskmemload09:30:141
8343992726,1cyclictest0-21swapper/211:17:322
8343992726,1cyclictest0-21swapper/209:53:562
8343992726,1cyclictest0-21swapper/209:53:562
834099261,1cyclictest28905-21egrep10:05:151
834099251,1cyclictest922-21diskmemload10:35:151
834399240,2cyclictest0-21swapper/208:05:142
834099247,10cyclictest0-21swapper/109:55:141
834099242,1cyclictest922-21diskmemload11:51:511
834099240,1cyclictest0-21swapper/112:30:141
834099240,1cyclictest0-21swapper/110:30:141
833899246,1cyclictest922-21diskmemload11:27:350
834099232,1cyclictest8723-21sed11:30:001
834099232,19cyclictest0-21swapper/108:30:001
834099232,17cyclictest0-21swapper/110:40:181
8340992316,7cyclictest0-21swapper/111:55:141
834099230,4cyclictest0-21swapper/111:44:231
834099230,2cyclictest0-21swapper/112:08:261
8348992220,1cyclictest922-21diskmemload11:46:213
834099228,13cyclictest26437-21ssh09:44:231
834099222,1cyclictest922-21diskmemload10:24:451
8340992220,1cyclictest0-21swapper/108:30:221
834099221,3cyclictest922-21diskmemload12:02:391
834099214,1cyclictest922-21diskmemload09:27:001
834099212,5cyclictest922-21diskmemload11:14:431
834099212,4cyclictest2325-21gltestperf09:10:171
834099211,2cyclictest23877-21cpu11:00:151
834099210,3cyclictest0-21swapper/110:04:321
834099210,2cyclictest0-21swapper/108:10:121
834099210,1cyclictest0-21swapper/111:05:131
834099210,1cyclictest0-21swapper/110:50:391
833899210,20cyclictest0-21swapper/011:45:130
834899200,19cyclictest0-21swapper/312:13:303
834399200,17cyclictest0-21swapper/209:21:422
834099206,6cyclictest27421-21ssh12:38:351
834099205,7cyclictest20762-21ssh11:17:161
834099202,17cyclictest922-21diskmemload09:40:001
8340992017,3cyclictest0-21swapper/107:14:221
834099201,3cyclictest19137-21hwlatdetect09:20:181
834099200,4cyclictest922-21diskmemload11:20:031
834099198,8cyclictest3463-21sh10:28:451
834099198,6cyclictest13131-21taskset09:16:271
834099193,15cyclictest922-21diskmemload09:52:251
834099193,15cyclictest922-21diskmemload09:52:251
834099192,2cyclictest29925-21ssh09:45:341
834099192,2cyclictest11694-21sh10:14:241
834099190,2cyclictest0-21swapper/112:25:261
834099190,2cyclictest0-21swapper/110:49:501
834099190,1cyclictest0-21swapper/110:59:241
834099187,6cyclictest16907-21taskset08:35:081
834099182,15cyclictest10631-21grep11:30:211
834099181,1cyclictest7159-21sh11:47:411
834399170,16cyclictest6697-21ssh09:12:582
834099179,7cyclictest21520-21head07:35:171
834099172,1cyclictest22573-21ssh12:15:511
834099171,1cyclictest922-21diskmemload11:38:451
834099170,1cyclictest0-21swapper/108:05:221
834099170,16cyclictest1993-21ssh12:23:231
833899175,6cyclictest15507-21apt-get07:25:000
833899174,12cyclictest13965-21ssh10:35:150
8338991715,2cyclictest13-21ksoftirqd/009:14:340
834899163,8cyclictest5796-21needreboot12:25:193
834899162,14cyclictest0-21swapper/309:40:403
834899160,15cyclictest0-21swapper/310:55:143
8343991615,1cyclictest0-21swapper/211:07:512
834099168,7cyclictest12406-21cat07:15:251
834099162,1cyclictest0-21swapper/108:50:251
834099161,5cyclictest225-21systemd-journal08:00:001
834099161,5cyclictest225-21systemd-journal08:00:001
834099161,1cyclictest12905-21apache_accesses10:15:101
8338991615,1cyclictest0-21swapper/011:40:120
834899150,14cyclictest6922-21kworker/u8:012:30:103
834399155,6cyclictest225-21systemd-journal08:30:002
834399153,5cyclictest55550irq/126-enp2s0-12:15:222
834099158,6cyclictest7618-21tr08:15:141
834099158,6cyclictest22453-21expr08:45:181
834099158,6cyclictest19550-21cpuspeed_turbos08:40:151
834099158,6cyclictest10577-21grep08:20:191
834099157,7cyclictest26750-21expr08:55:161
834099156,8cyclictest30178-21grep09:00:221
834099153,12cyclictest0-21swapper/107:50:271
834099150,2cyclictest922-21diskmemload12:10:101
833899154,3cyclictest225-21systemd-journal11:09:210
8338991510,3cyclictest11807-21rm09:35:170
223192150,0chrt0-21swapper/010:40:020
834899143,2cyclictest141rcu_preempt09:35:563
8348991410,3cyclictest1606-21turbostat.cron11:45:013
8348991410,2cyclictest26188-21cron08:55:003
8348991410,1cyclictest0-21swapper/312:07:163
834899140,1cyclictest0-21swapper/308:25:153
8343991410,3cyclictest17729-21sh12:32:332
834399140,13cyclictest0-21swapper/208:50:152
834099147,6cyclictest32314-21cat09:05:201
834099147,6cyclictest2372-21sleep08:05:001
834099147,6cyclictest15571-21cat07:25:001
833899149,3cyclictest6013-21cat08:10:190
833899149,3cyclictest5060-21wc12:05:270
833899144,4cyclictest225-21systemd-journal07:10:000
833899142,6cyclictest18120-21awk07:30:000
833899141,7cyclictest31524-21munin-run11:05:000
834899139,3cyclictest25620-21cron11:40:003
834899139,3cyclictest1983-21perf09:30:003
834899139,3cyclictest16920-21uname07:25:203
834899139,3cyclictest11427-21tail11:50:173
834899139,2cyclictest9627-21wc07:10:193
834899139,2cyclictest4269-21wc10:10:013
8348991310,2cyclictest31102-21tr10:25:223
8348991310,2cyclictest20927-21grep11:55:273
8348991310,2cyclictest1869-21cat11:25:203
8348991310,2cyclictest16139-21wc08:30:263
834399136,2cyclictest922-21diskmemload10:40:002
8343991312,1cyclictest0-21swapper/211:49:332
8343991310,2cyclictest927-21perf09:10:002
8343991310,2cyclictest7494-21taskset10:50:152
8343991310,2cyclictest4675-21ls12:05:232
8343991310,2cyclictest28457-21grep09:25:282
8343991310,2cyclictest24448-21gzip07:40:222
8343991310,2cyclictest2352-21munin-run08:05:002
834399130,13cyclictest0-21swapper/212:13:392
834399130,12cyclictest22898-21cat07:40:002
834399130,12cyclictest0-21swapper/211:50:062
834099132,2cyclictest25251-21latency_hist07:45:011
834099132,1cyclictest225-21systemd-journal07:32:131
834099131,2cyclictest17606-21grep07:25:271
833899139,3cyclictest7680-21grep10:50:170
833899139,2cyclictest17235-21sed10:55:280
833899138,4cyclictest30561-21grep10:05:270
8338991313,0cyclictest0-21swapper/011:33:540
8338991310,2cyclictest5678-21wc12:25:190
8338991310,2cyclictest31968-21head09:05:180
8338991310,2cyclictest28764-21tr12:00:220
8338991310,2cyclictest17512-21wc11:15:200
833899130,12cyclictest30096-21grep12:20:220
833899130,12cyclictest26791-21cron12:00:000
833899130,10cyclictest11178-21df_abs11:50:150
834899129,2cyclictest5501-21grep09:50:243
834899129,2cyclictest5501-21grep09:50:243
834899129,2cyclictest30574-21tail12:20:263
834899129,2cyclictest26617-21grep07:45:203
834899129,2cyclictest26525-21chrt10:22:543
834899129,2cyclictest17487-21grep08:35:133
834899129,2cyclictest15053-21rm09:56:153
834899129,2cyclictest14073-21cut10:35:153
834899129,2cyclictest12563-21grep07:15:263
834899129,2cyclictest11275-21cut09:15:243
834899128,3cyclictest7422-21cron11:10:003
834899128,3cyclictest2549-21wc09:10:193
834899128,3cyclictest10607-21tr11:30:213
834899128,2cyclictest494-21tr08:00:153
834899125,3cyclictest9499-21dump-pmu-power08:20:003
834899120,9cyclictest6890-21taskset10:50:133
834399129,2cyclictest31073-21grep09:05:002
834399129,2cyclictest30541-21grep10:25:182
834399129,2cyclictest29664-21grep09:45:272
834399129,2cyclictest29575-21grep10:05:192
834399129,2cyclictest22328-21sed10:00:272
834399129,2cyclictest19695-21cut09:40:152
834399129,2cyclictest11703-21sed07:15:172
834399123,8cyclictest7786-21ssh10:11:212
8343991210,1cyclictest22196-21grep12:35:182
834099125,6cyclictest26636-21ntpq07:45:211
833899129,2cyclictest8332-21cyclictest09:45:030
833899129,2cyclictest5483-21sed10:10:180
833899129,2cyclictest3415-21cut09:30:160
833899129,2cyclictest32391-21readlink08:00:000
833899129,2cyclictest32391-21readlink08:00:000
833899129,2cyclictest31074-21wc09:05:000
833899129,2cyclictest29783-21cut10:25:150
833899129,2cyclictest20984-21tr07:35:140
833899129,2cyclictest19138-21grep07:30:200
833899129,2cyclictest16279-21head08:30:260
833899129,2cyclictest12443-21grep12:10:170
833899129,2cyclictest10220-21tr09:15:170
833899128,3cyclictest6460-21wc10:50:000
833899128,3cyclictest18259-21wc08:35:200
833899128,2cyclictest21524-21grep12:35:140
833899124,7cyclictest9493-21munin-run08:20:000
833899121,5cyclictest49-21kcompactd009:52:310
833899121,5cyclictest49-21kcompactd009:52:310
833899121,10cyclictest14344-21/usr/sbin/munin10:15:200
833899120,1cyclictest0-21swapper/011:23:070
834899118,2cyclictest5474-21df08:10:143
834899118,2cyclictest5434-21ssh09:31:013
834899118,2cyclictest5401-21ssh10:30:063
834899118,2cyclictest5360-21grep10:10:183
834899118,2cyclictest4138-21ssh10:48:173
834899118,2cyclictest32385-21ssh09:47:313
834899118,2cyclictest31966-21grep09:05:183
834899118,2cyclictest3087-21cat08:05:153
834899118,2cyclictest30754-21grep07:55:163
834899118,2cyclictest30754-21grep07:55:163
834899118,2cyclictest24972-21tr11:00:203
834899118,2cyclictest24805-21ssh10:40:373
834899118,2cyclictest21861-21wc12:15:263
834899118,2cyclictest21410-21ssh12:35:133
834899118,2cyclictest21189-21chrt08:41:523
834899118,2cyclictest20236-21ssh10:00:063
834899118,2cyclictest17259-21head11:15:183
834899118,2cyclictest13755-21grep10:15:153
834899118,2cyclictest11525-21ssh11:11:403
834899112,6cyclictest27358-21apt-get12:00:143
834899110,1cyclictest0-21swapper/311:20:223
834899110,10cyclictest0-21swapper/309:00:133
834399119,1cyclictest8714-21grep11:30:002
834399119,1cyclictest29993-21idleruntime-cro07:55:002
834399119,1cyclictest24857-21sort11:00:202
834399119,1cyclictest10751-21grep09:15:192
834399118,2cyclictest9489-21grep08:20:002
834399118,2cyclictest8679-21ssh10:31:182
834399118,2cyclictest31670-21taskset11:24:152
834399118,2cyclictest31258-21ssh10:45:162
834399118,2cyclictest27657-21ls08:55:222
834399118,2cyclictest27018-21sed11:40:162
834399118,2cyclictest23249-21date10:20:272
834399118,2cyclictest22384-21ssh10:40:062
834399118,2cyclictest20233-21cat11:55:202
834399118,2cyclictest15805-21head08:30:222
834399118,2cyclictest15191-21rm09:56:232
834399118,2cyclictest14996-21grep07:20:252
834399118,2cyclictest13862-21ssh09:36:062
834399118,2cyclictest11835-21taskset08:24:422
834399118,2cyclictest10700-21basename07:15:002
834399115,5cyclictest14777-21/usr/sbin/munin10:15:242
834399111,3cyclictest0-21swapper/207:30:142
833899118,2cyclictest8312-21sh10:31:010
833899118,2cyclictest27988-21cat12:20:000
833899118,2cyclictest27380-21chrt07:47:040
833899118,2cyclictest24400-21wc07:40:220
833899118,2cyclictest20425-21ssh10:00:130
833899118,2cyclictest20315-21head09:40:180
833899118,2cyclictest19718-21sed11:35:270
833899118,2cyclictest18012-21rm09:20:030
833899118,2cyclictest14543-21ssh09:55:510
833899118,2cyclictest13734-21cut12:30:170
833899118,2cyclictest13577-21sh11:13:160
833899110,10cyclictest29643-21taskset07:50:380
833899110,10cyclictest28035-21ssh10:24:060
834899108,1cyclictest289102sleep307:50:203
834899108,1cyclictest24643-21sed07:40:263
834899108,1cyclictest21223-21df07:35:153
834899108,1cyclictest18704-21expr07:30:153
834899107,2cyclictest18071-21taskset09:20:063
834899107,2cyclictest14019-21sort07:20:153
834899107,2cyclictest10976-21cut08:20:233
834899100,9cyclictest284092sleep308:58:073
834899100,9cyclictest21834-21cut08:45:153
834399108,1cyclictest8468-21expr11:10:152
834399108,1cyclictest6101-21basename12:25:222
834399108,1cyclictest6076-21tr08:10:192
834399108,1cyclictest210452sleep208:40:272
834399108,1cyclictest18739-21taskset08:36:102
834399108,1cyclictest18460-21rm10:56:202
834399108,1cyclictest16574-21ssh11:34:292
834399107,2cyclictest6695-21ssh09:32:012
834399107,2cyclictest29399-21ssh12:00:292
834399107,2cyclictest28427-21sh12:20:042
834399107,2cyclictest18856-21cat11:35:192
833899108,1cyclictest126952sleep007:15:310
833899108,1cyclictest11470-21awk08:20:270
833899107,2cyclictest32670-21basename08:00:140
833899107,2cyclictest24553-21expr08:50:150
833899107,2cyclictest22433-21tail08:45:180
833899107,2cyclictest21405-21chrt08:44:380
833899106,2cyclictest27453-21wc08:55:200
833899103,3cyclictest225-21systemd-journal08:28:530
833899102,5cyclictest4735-21dump-pmu-power08:10:000
83439998,1cyclictest25883-21expr07:45:152
83439997,2cyclictest23764-21taskset08:49:262
83439997,1cyclictest31102-21if_enp0s31f607:55:192
83439997,1cyclictest31102-21if_enp0s31f607:55:192
83439997,1cyclictest16775-21expr07:25:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional