You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 15:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Nov 19, 2025 12:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3128921050,0sleep00-21swapper/010:56:100
10478210187,10sleep30-21swapper/307:08:543
10425210188,9sleep10-21swapper/107:08:161
1039429365,9sleep00-21swapper/007:07:500
104852750,0sleep00-21swapper/012:01:220
1022227565,6sleep20-21swapper/207:05:362
10658992826,2cyclictest39-21ksoftirqd/311:40:003
10658992821,4cyclictest39-21ksoftirqd/312:08:103
10658992520,2cyclictest39-21ksoftirqd/309:23:223
10655992524,1cyclictest0-21swapper/212:26:132
1065599240,23cyclictest0-21swapper/212:10:032
10658992319,3cyclictest94-21kswapd010:00:123
10658992319,2cyclictest39-21ksoftirqd/312:11:553
1065899221,1cyclictest0-21swapper/311:53:093
1064899220,2cyclictest0-21swapper/011:10:200
10658992018,1cyclictest39-21ksoftirqd/312:23:543
1065899201,2cyclictest141rcu_preempt09:55:483
1065899200,2cyclictest141rcu_preempt10:14:093
1065899191,15cyclictest39-21ksoftirqd/309:46:473
1065599190,19cyclictest0-21swapper/209:42:122
1065899188,3cyclictest39-21ksoftirqd/308:41:273
1065899182,2cyclictest39-21ksoftirqd/311:41:563
1065899181,3cyclictest0-21swapper/311:47:093
10658991813,4cyclictest39-21ksoftirqd/312:20:003
10658991813,2cyclictest39-21ksoftirqd/310:47:113
1065899180,2cyclictest0-21swapper/310:36:213
1065599180,1cyclictest0-21swapper/207:55:142
1065399180,2cyclictest0-21swapper/111:17:401
1065399180,2cyclictest0-21swapper/111:17:391
12502170,1chrt0-21swapper/311:26:153
1065899175,11cyclictest22508-21idleruntime-cro09:55:013
1065899172,6cyclictest39-21ksoftirqd/312:00:183
1065899171,2cyclictest141rcu_preempt11:56:083
1065899171,13cyclictest0-21swapper/310:23:423
1065599171,15cyclictest0-21swapper/212:30:442
10648991715,1cyclictest0-21swapper/012:31:060
1065899167,1cyclictest2745-21taskset08:00:103
10658991614,2cyclictest39-21ksoftirqd/311:10:143
10655991614,2cyclictest37850irq/131-snd_hda11:51:262
1065599160,16cyclictest0-21swapper/210:35:132
1065399164,1cyclictest141rcu_preempt10:25:071
1064899160,15cyclictest0-21swapper/012:08:100
1064899160,15cyclictest0-21swapper/009:32:140
1065899154,6cyclictest39-21ksoftirqd/309:12:473
1065899154,2cyclictest141rcu_preempt12:33:213
10658991512,1cyclictest381rcuc/310:44:513
10658991511,3cyclictest39-21ksoftirqd/311:24:143
1065899150,1cyclictest0-21swapper/310:50:333
1065599153,12cyclictest13284-21kworker/2:210:52:582
10655991514,1cyclictest0-21swapper/208:36:092
1065599150,15cyclictest0-21swapper/212:20:182
10653991511,3cyclictest8597-21wc08:10:231
10653991510,3cyclictest6725-21wc12:25:251
1065399150,14cyclictest0-21swapper/109:36:221
1064899150,2cyclictest0-21swapper/010:35:020
1064899150,10cyclictest0-21swapper/012:20:200
1065899149,4cyclictest3717-21diskmemload09:36:143
1065899148,2cyclictest11040-21cron12:30:003
1065899145,7cyclictest39-21ksoftirqd/312:35:153
10658991411,2cyclictest6219-21head09:40:213
10658991410,2cyclictest39-21ksoftirqd/310:26:073
10658991410,2cyclictest29279-21cut08:55:163
1065899140,2cyclictest0-21swapper/310:57:293
1065599149,3cyclictest24920-21grep08:45:162
1065599143,11cyclictest3717-21diskmemload11:10:312
1065599140,13cyclictest0-21swapper/210:30:142
1065399146,4cyclictest224-21systemd-journal10:24:591
1065399144,6cyclictest15210-21ssh11:10:201
1065399143,8cyclictest20139-21/usr/sbin/munin09:25:131
10653991410,3cyclictest13149-21cat07:15:011
10653991410,2cyclictest7475-21basename12:00:001
1065399140,13cyclictest8085-21sh11:05:001
1065399140,12cyclictest6187-21wc08:05:201
10648991410,3cyclictest7259-21cron08:10:000
1064899140,8cyclictest1274-21awk10:05:000
1064899140,3cyclictest0-21swapper/009:46:460
1064899140,2cyclictest0-21swapper/009:26:300
102512140,1chrt0-21swapper/111:34:581
1065899139,3cyclictest486-21grep07:55:163
1065899139,3cyclictest3262-21grep11:00:213
1065899139,3cyclictest23499-21grep07:35:193
1065899139,3cyclictest2287-21grep09:05:173
1065899133,2cyclictest5526-21ssh10:08:143
1065899132,1cyclictest141rcu_preempt10:19:003
1065899131,5cyclictest39-21ksoftirqd/308:20:253
10658991311,2cyclictest39-21ksoftirqd/311:16:263
10658991311,2cyclictest39-21ksoftirqd/311:16:253
10658991310,2cyclictest19943-21tr08:35:153
10658991310,2cyclictest19667-21grep07:25:363
1065599139,1cyclictest1964-21sendmail11:00:002
1065599138,5cyclictest24969-21apache211:30:122
1065599138,4cyclictest37850irq/131-snd_hda07:30:002
1065599136,1cyclictest8155-21grep12:00:152
10655991310,2cyclictest3545-21head10:05:302
10655991310,2cyclictest30156-21cron07:50:012
10655991310,2cyclictest20979-21sed11:15:222
10655991310,2cyclictest20979-21sed11:15:222
10655991310,2cyclictest18963-21sed12:35:312
10655991310,2cyclictest18439-21wc09:50:232
10655991310,2cyclictest16515-21grep08:25:252
1065399139,2cyclictest0-21swapper/110:11:221
10653991310,2cyclictest9385-21ssh12:00:251
10653991310,2cyclictest5125-21grep10:35:251
10653991310,2cyclictest3467-21awk10:34:591
10653991310,2cyclictest25576-21grep08:45:201
10653991310,2cyclictest22138-21grep11:45:171
10653991310,2cyclictest13228-21grep12:30:291
1065399130,12cyclictest28680-21sort07:45:171
1065399130,12cyclictest0-21swapper/107:50:141
1064899139,3cyclictest7438-21cron12:00:000
1064899136,5cyclictest30986-21taskset12:19:410
1064899132,3cyclictest21907-21ssh10:21:500
1064899131,4cyclictest28572-21df_inode11:50:190
10648991310,2cyclictest5863-21wc09:10:280
10648991310,2cyclictest27598-21basename07:45:010
10648991310,2cyclictest25250-21sed08:45:180
10648991310,2cyclictest20443-21sed12:10:190
10648991310,2cyclictest11564-21grep08:15:290
10648991310,2cyclictest11484-21head11:35:180
1064899130,3cyclictest32726-21sensors11:25:280
1064899130,12cyclictest14163-21cut10:15:220
1065899129,2cyclictest32469-21grep10:30:293
1065899129,2cyclictest10291-21grep09:15:143
1065899127,3cyclictest39-21ksoftirqd/309:25:153
1065899121,5cyclictest224-21systemd-journal07:17:003
1065599129,2cyclictest31351-21turbostat09:00:002
1065599129,2cyclictest3071-21grep08:00:182
1065599129,2cyclictest26142-21sed11:20:182
1065599129,2cyclictest22318-21tr11:45:172
1065599129,2cyclictest18604-21fgrep08:30:262
1065599129,2cyclictest17445-21cut10:45:222
1065599129,2cyclictest14544-21sh09:20:012
1065599128,3cyclictest27466-21cat09:30:272
1065599128,1cyclictest0-21swapper/210:14:282
1065599124,7cyclictest24969-21apache210:40:392
1065599122,9cyclictest61850irq/126-enp2s0-11:43:582
1065599120,4cyclictest224-21systemd-journal10:18:472
1065399129,2cyclictest30026-21sed08:55:191
1065399129,2cyclictest26542-21uname11:20:211
1065399129,2cyclictest1874-21ssh10:05:151
1065399129,2cyclictest13255-21head08:20:181
1065399129,2cyclictest12022-21head10:40:301
1065399129,2cyclictest1025-21uniq07:55:181
1065399128,2cyclictest25723-21cut07:40:151
1065399128,2cyclictest15196-21tr08:25:141
1065399121,2cyclictest3717-21diskmemload12:35:221
1064899129,2cyclictest7930-21sort10:10:200
1064899129,2cyclictest31362-21run-parts08:59:590
1064899129,2cyclictest28824-21sed07:45:170
1064899129,2cyclictest20363-21grep08:35:180
1064899129,2cyclictest17468-21tr10:45:220
1064899129,2cyclictest16758-21grep07:20:240
1064899129,2cyclictest11616-21head07:10:180
1064899129,2cyclictest11088-21wc09:45:010
1064899128,3cyclictest0-21swapper/010:30:130
1064899128,2cyclictest17532-21grep08:30:180
1064899128,2cyclictest17340-21tr09:50:160
1064899126,6cyclictest3717-21diskmemload10:43:130
1064899124,4cyclictest224-21systemd-journal07:55:010
264822110,1sleep20-21swapper/209:57:592
1065899118,2cyclictest9893-21grep11:05:273
1065899118,2cyclictest7767-21tr08:10:183
1065899118,2cyclictest4556-21sort11:30:003
1065899118,2cyclictest28782-21head07:45:183
1065899118,2cyclictest27848-21ssh09:30:463
1065899118,2cyclictest18793-21cat08:30:293
1065899118,2cyclictest10139-21expr08:15:183
1065899117,3cyclictest12577-21grep07:10:273
1065899110,7cyclictest32763-21ntpq09:00:253
1065899110,7cyclictest21336-21grep07:30:213
1065599118,2cyclictest6364-21cut08:05:222
1065599118,2cyclictest4382-21sort09:10:172
1065599118,2cyclictest31983-21expr09:00:192
1065599118,2cyclictest31545-21ssh09:35:092
1065599118,2cyclictest27142-21tr08:50:142
1065599118,2cyclictest26875-21ssh10:25:492
1065599118,2cyclictest26067-21fschecks_time12:15:192
1065599118,2cyclictest24154-21cron08:44:592
1065599118,2cyclictest20859-21sort09:25:182
1065599118,2cyclictest20820-21ssh10:20:392
1065599118,2cyclictest17993-21ssh12:08:332
1065599118,2cyclictest15464-21cat09:20:182
1065599118,2cyclictest14057-21ls11:10:012
1065599118,2cyclictest12741-21sh11:35:372
1065599118,2cyclictest11350-21taskset09:45:132
1065599118,2cyclictest11296-21expr07:10:152
1065599118,2cyclictest10714-21hddtemp_smartct08:15:202
1065599117,3cyclictest8628-21tail08:10:232
1065599117,3cyclictest14160-21sed08:20:262
1065399119,1cyclictest25438-21cut12:15:161
1065399118,2cyclictest7439-21ssh09:40:421
1065399118,2cyclictest4160-21ssh12:23:521
1065399118,2cyclictest411-21grep09:00:261
1065399118,2cyclictest32111-21ssh10:02:521
1065399118,2cyclictest29722-21grep11:50:301
1065399118,2cyclictest28628-21ssh10:55:041
1065399118,2cyclictest28556-21head08:50:261
1065399118,2cyclictest27850-21ssh10:54:211
1065399118,2cyclictest25536-21ssh09:30:041
1065399118,2cyclictest19175-21ssh09:50:481
1065399118,2cyclictest18911-21ssh09:23:341
1065399118,2cyclictest18628-21taskset10:45:591
1065399118,2cyclictest1846-21ssh11:26:541
1065399118,2cyclictest17608-21wc11:40:271
1065399118,2cyclictest16750-21ssh12:07:001
1065399118,2cyclictest15767-21ssh10:16:391
1065399118,2cyclictest12952-21ssh11:08:441
1065399118,2cyclictest10705-21sed08:15:201
1065399118,2cyclictest10548-21ssh11:35:021
1065399118,2cyclictest10116-21chrt09:15:131
1065399117,3cyclictest23148-21ssh09:55:211
1065399117,3cyclictest22756-21awk07:35:001
1065399114,4cyclictest20472-21/usr/sbin/munin08:35:191
1065399110,10cyclictest4585-21chrt08:01:171
1065399110,10cyclictest11207-21ssh09:45:041
1064899118,2cyclictest610-21cut07:55:160
1064899118,2cyclictest5890-21cut12:25:200
1064899118,2cyclictest5214-21tr11:30:160
1064899118,2cyclictest32469-21cut09:35:180
1064899118,2cyclictest2865-21basename08:00:170
1064899118,2cyclictest27428-21ssh11:20:300
1064899118,2cyclictest27046-21ssh10:53:220
1064899118,2cyclictest24313-21sendmail_mailqu09:55:290
1064899118,2cyclictest2275-21ssh11:00:040
1064899118,2cyclictest21957-21sed11:15:310
1064899118,2cyclictest21957-21sed11:15:300
1064899118,2cyclictest20894-21tr07:30:180
1064899118,2cyclictest20095-21taskset07:29:280
1064899118,2cyclictest18087-21basename12:35:210
1064899118,2cyclictest15620-21grep08:25:160
1064899118,2cyclictest10096-21ssh11:05:300
1064899114,5cyclictest13930-21python08:20:260
1064899110,7cyclictest0-21swapper/011:45:160
1064899110,6cyclictest0-21swapper/008:10:130
1064899110,3cyclictest29066-21ssh10:28:390
1064899110,10cyclictest18566-21ssh11:41:180
1064899110,10cyclictest15420-21expr09:20:180
1064899110,10cyclictest0-21swapper/010:07:020
1064899110,10cyclictest0-21swapper/007:15:150
1065899108,1cyclictest27320-21chrt07:41:543
1065899108,1cyclictest15399-21sed08:25:153
1065899107,2cyclictest31210-21grep07:50:183
1065899107,2cyclictest27953-21grep08:50:193
1065899106,3cyclictest39-21ksoftirqd/308:05:133
1065899104,5cyclictest7744-21snmpd08:45:033
1065899100,9cyclictest176192sleep307:23:123
1065599108,1cyclictest28496-21cut10:00:142
1065599108,1cyclictest15887-21wc07:20:192
1065599108,1cyclictest15449-21date07:20:002
1065599107,2cyclictest3585-21chrt09:08:402
1065599107,2cyclictest32607-21cat07:55:012
1065599107,2cyclictest32395-21cat11:25:222
1065599107,2cyclictest2762-21rm11:55:222
1065599107,2cyclictest25469-21aten2_r7power_c07:40:142
1065599107,2cyclictest2521-21tr11:00:162
1065599107,2cyclictest24869-21taskset07:37:012
1065599100,9cyclictest20465-21apt-config07:30:002
1065399108,1cyclictest23252-21expr07:35:181
1065399108,1cyclictest18246-21ls07:25:261
1065399108,1cyclictest17494-21chrt07:21:341
1065399108,1cyclictest15295-21chrt07:19:041
1065399107,2cyclictest6474-21ssh09:11:031
1065399107,2cyclictest2235-21tune2fs09:05:171
1065399107,2cyclictest17655-21expr08:30:181
1065399102,4cyclictest22062-21apt-get08:40:161
1065399102,3cyclictest224-21systemd-journal12:10:011
1064899108,1cyclictest23119-21awk07:35:170
1064899108,1cyclictest1550-21chrt09:05:010
1064899108,1cyclictest10478-21expr09:15:160
1064899107,2cyclictest32477-21basename09:00:220
1064899107,2cyclictest26823-21chrt08:50:040
1064899107,2cyclictest22840-21sed08:40:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional