You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 11:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Nov 25, 2025 00:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8912210680,9sleep10-21swapper/119:07:121
8792210275,9sleep30-21swapper/319:05:393
908828770,13sleep20-21swapper/219:09:272
901428266,11sleep00-21swapper/019:08:280
9217992422,2cyclictest25-21ksoftirqd/122:50:011
9213992420,1cyclictest141rcu_preempt22:30:060
9213992419,3cyclictest13-21ksoftirqd/021:44:460
9213992419,2cyclictest13-21ksoftirqd/021:31:190
9219992320,2cyclictest37850irq/131-snd_hda22:10:182
9223992222,0cyclictest0-21swapper/322:56:113
9219992220,2cyclictest37850irq/131-snd_hda22:03:342
9219992220,2cyclictest37850irq/131-snd_hda00:14:392
9219992220,1cyclictest37850irq/131-snd_hda23:50:422
9219992220,1cyclictest37850irq/131-snd_hda23:21:482
9219992220,1cyclictest37850irq/131-snd_hda23:05:272
9219992220,1cyclictest37850irq/131-snd_hda22:57:552
9219992220,1cyclictest37850irq/131-snd_hda22:52:432
9219992220,1cyclictest37850irq/131-snd_hda22:44:322
9219992220,1cyclictest37850irq/131-snd_hda22:37:242
9219992220,1cyclictest37850irq/131-snd_hda22:28:372
9219992220,1cyclictest37850irq/131-snd_hda22:20:022
9219992220,1cyclictest37850irq/131-snd_hda22:05:112
9219992220,1cyclictest37850irq/131-snd_hda21:43:552
9219992220,1cyclictest37850irq/131-snd_hda21:16:162
9219992220,1cyclictest37850irq/131-snd_hda00:17:372
9219992219,2cyclictest37850irq/131-snd_hda21:20:242
9219992219,2cyclictest37850irq/131-snd_hda21:20:232
9219992120,1cyclictest37850irq/131-snd_hda23:56:422
9219992120,1cyclictest37850irq/131-snd_hda23:41:392
9219992120,1cyclictest37850irq/131-snd_hda23:37:162
9219992120,1cyclictest37850irq/131-snd_hda23:27:532
9219992120,1cyclictest37850irq/131-snd_hda23:10:362
9219992120,1cyclictest37850irq/131-snd_hda23:02:362
9219992120,1cyclictest37850irq/131-snd_hda22:46:272
9219992120,1cyclictest37850irq/131-snd_hda22:33:472
9219992120,1cyclictest37850irq/131-snd_hda21:56:572
9219992120,1cyclictest37850irq/131-snd_hda21:53:522
9219992120,1cyclictest37850irq/131-snd_hda21:48:242
9219992120,1cyclictest37850irq/131-snd_hda21:30:362
9219992120,1cyclictest37850irq/131-snd_hda21:25:412
9219992120,1cyclictest37850irq/131-snd_hda21:11:232
9219992120,1cyclictest37850irq/131-snd_hda00:37:342
9219992120,1cyclictest37850irq/131-snd_hda00:26:132
9219992120,1cyclictest37850irq/131-snd_hda00:21:372
9219992120,1cyclictest37850irq/131-snd_hda00:07:392
9219992120,1cyclictest37850irq/131-snd_hda00:01:582
9219992119,2cyclictest37850irq/131-snd_hda23:46:062
9219992119,1cyclictest37850irq/131-snd_hda23:19:282
9219992119,1cyclictest37850irq/131-snd_hda22:18:192
9219992119,1cyclictest37850irq/131-snd_hda21:35:222
9219992119,1cyclictest37850irq/131-snd_hda00:30:162
9217992120,1cyclictest0-21swapper/122:06:131
9217992119,1cyclictest0-21swapper/121:20:141
9217992119,1cyclictest0-21swapper/121:20:131
921799211,7cyclictest141rcu_preempt00:15:241
9213992116,1cyclictest13-21ksoftirqd/022:35:080
921399210,5cyclictest0-21swapper/023:20:420
922399200,16cyclictest0-21swapper/319:20:143
9219992018,1cyclictest37850irq/131-snd_hda23:30:152
921799202,2cyclictest141rcu_preempt00:00:221
9217992017,2cyclictest25-21ksoftirqd/119:15:011
921399202,6cyclictest13-21ksoftirqd/000:35:550
921399202,15cyclictest0-21swapper/021:56:180
9213992013,4cyclictest13-21ksoftirqd/023:27:180
9213991916,2cyclictest13-21ksoftirqd/000:09:460
922399180,16cyclictest0-21swapper/322:23:003
922399180,11cyclictest0-21swapper/321:50:113
9219991817,1cyclictest0-21swapper/219:25:132
921799184,3cyclictest141rcu_preempt22:25:151
9217991816,1cyclictest25-21ksoftirqd/123:45:031
9217991813,2cyclictest25-21ksoftirqd/121:41:271
921799181,1cyclictest141rcu_preempt22:35:181
921399189,1cyclictest13-21ksoftirqd/020:06:320
921399185,1cyclictest141rcu_preempt22:44:360
921399181,14cyclictest32650-21ssh23:02:530
921799176,4cyclictest25-21ksoftirqd/100:35:181
9217991716,1cyclictest25-21ksoftirqd/123:40:011
9217991716,1cyclictest0-21swapper/100:27:101
9217991713,1cyclictest25-21ksoftirqd/123:33:011
921399177,5cyclictest21768-21ssh00:30:000
921399174,7cyclictest26092-21munin-run19:45:000
921399174,1cyclictest141rcu_preempt23:07:320
921399173,9cyclictest0-21swapper/023:50:140
921399172,2cyclictest141rcu_preempt22:52:410
9213991710,4cyclictest13-21ksoftirqd/021:27:240
921399170,2cyclictest0-21swapper/022:02:180
922399163,11cyclictest0-21swapper/300:25:133
9223991614,2cyclictest0-21swapper/322:03:543
921799164,5cyclictest25-21ksoftirqd/122:04:451
921799164,1cyclictest141rcu_preempt23:23:371
921799161,1cyclictest141rcu_preempt22:34:011
9213991613,2cyclictest24904-21awk20:50:010
9213991610,6cyclictest13-21ksoftirqd/020:20:150
921399160,4cyclictest0-21swapper/022:29:260
921399160,1cyclictest0-21swapper/021:21:290
921399160,1cyclictest0-21swapper/021:21:280
9223991514,1cyclictest0-21swapper/300:20:073
921999159,5cyclictest23052-21grep20:45:172
921799155,3cyclictest25-21ksoftirqd/121:51:431
9217991514,0cyclictest0-21swapper/121:19:041
9217991513,2cyclictest25-21ksoftirqd/123:02:521
9217991512,2cyclictest25-21ksoftirqd/122:40:241
9217991512,2cyclictest25-21ksoftirqd/122:16:311
9217991512,2cyclictest25-21ksoftirqd/121:56:181
921799150,0cyclictest141rcu_preempt23:08:541
921399152,2cyclictest141rcu_preempt23:44:030
921399151,2cyclictest0-21swapper/000:31:070
921399151,1cyclictest141rcu_preempt22:47:200
9213991511,3cyclictest2419-21wc21:10:000
9213991510,1cyclictest13-21ksoftirqd/000:20:470
921399150,15cyclictest0-21swapper/022:19:190
922399149,3cyclictest29556-21tr23:00:233
922399147,3cyclictest9363-21cat22:45:263
922399144,6cyclictest29861-21apt21:00:013
9223991413,0cyclictest0-21swapper/322:11:443
9223991411,2cyclictest6885-21ssh21:35:223
921999140,1cyclictest0-21swapper/220:25:142
921799142,8cyclictest224-21systemd-journal23:25:011
9217991411,2cyclictest25-21ksoftirqd/122:14:591
9217991410,3cyclictest25-21ksoftirqd/123:16:351
921799140,14cyclictest0-21swapper/122:50:371
921399149,3cyclictest13-21ksoftirqd/000:18:180
921399145,5cyclictest161rcuc/021:35:290
921399144,1cyclictest141rcu_preempt23:32:030
921399140,13cyclictest17350-21tr22:05:190
922399139,3cyclictest25857-21wc22:35:003
922399139,3cyclictest2023-21tr21:55:153
922399139,2cyclictest4467-21cat00:15:213
922399138,4cyclictest13767-21ntpq22:25:223
9223991311,1cyclictest16469-21ssh22:05:143
9223991310,2cyclictest6151-21grep23:30:163
9223991310,2cyclictest31292-21tr21:00:193
9223991310,2cyclictest20113-21cron20:39:593
9223991310,2cyclictest10783-21wc21:15:213
922399130,12cyclictest27136-21sed22:35:173
922399130,12cyclictest16365-21grep20:30:183
922399130,12cyclictest0-21swapper/323:08:123
9219991310,2cyclictest29952-21head19:50:242
921999130,13cyclictest0-21swapper/221:00:122
921799135,5cyclictest141rcu_preempt21:36:341
921799135,4cyclictest224-21systemd-journal20:05:011
921799131,9cyclictest26256-21rm22:58:471
9217991310,2cyclictest29876-21systemctl21:00:011
9217991310,2cyclictest18300-21grep20:35:151
921799130,1cyclictest141rcu_preempt23:57:251
921799130,0cyclictest0-21swapper/123:40:551
921399136,6cyclictest2920-21munin-run00:14:590
921399135,7cyclictest18941-21munin-run19:30:010
921399133,3cyclictest141rcu_preempt23:47:280
921399131,3cyclictest141rcu_preempt22:24:090
9213991310,2cyclictest4622-21grep21:10:280
9213991310,2cyclictest12891-21wc19:15:190
921399130,12cyclictest13077-21ssh23:35:140
922399129,2cyclictest4261-21sed20:05:163
922399129,2cyclictest27956-21grep19:45:263
922399129,2cyclictest23515-21awk23:20:003
922399129,2cyclictest22844-21tr00:05:173
922399129,2cyclictest2008-21grep23:50:163
922399128,3cyclictest9153-21dump-pmu-power21:15:013
922399128,3cyclictest828-21cron20:00:003
922399128,3cyclictest39-21ksoftirqd/319:25:173
922399128,3cyclictest26474-21grep20:50:193
922399128,3cyclictest17172-21ssh21:20:163
922399128,3cyclictest17172-21ssh21:20:163
922399128,3cyclictest10107-21tr23:10:153
922399123,6cyclictest0-21swapper/323:45:153
922399122,9cyclictest0-21swapper/323:29:213
921999129,2cyclictest22123-21grep20:40:272
921999129,2cyclictest1796-21tail21:05:262
921999128,3cyclictest23696-21cat19:40:012
921999120,9cyclictest0-21swapper/220:20:152
921799129,2cyclictest6760-21grep22:20:201
921799129,2cyclictest26117-21grep20:50:171
921799129,2cyclictest21638-21grep20:40:221
921799129,2cyclictest19459-21tr19:30:161
921799129,2cyclictest13407-21ls19:15:251
921799129,2cyclictest10161-21sed20:15:291
921799128,3cyclictest28828-21grep19:50:161
921799128,3cyclictest21732-21grep21:45:281
921799126,2cyclictest25-21ksoftirqd/100:20:181
921799123,4cyclictest224-21systemd-journal19:25:011
921799123,1cyclictest141rcu_preempt21:25:041
921799122,1cyclictest1486-21sh21:31:301
921799121,7cyclictest3511-21apt-get20:05:141
921799120,4cyclictest19126-21latency_hist19:30:001
921799120,1cyclictest141rcu_preempt21:14:161
921399129,2cyclictest27261-21grep21:50:140
921399129,2cyclictest26672-21cat21:50:000
921399129,2cyclictest1189-21wget20:00:150
921399126,3cyclictest13-21ksoftirqd/022:58:220
921399121,10cyclictest31161-21apt-get19:55:120
9213991210,1cyclictest9838-21apt-get23:10:140
921399120,1cyclictest0-21swapper/022:10:330
922399118,2cyclictest9607-21cat23:55:183
922399118,2cyclictest9200-21grep20:15:203
922399118,2cyclictest31697-21grep21:30:193
922399118,2cyclictest28948-21tr19:50:163
922399118,2cyclictest2827-21grep20:00:283
922399118,2cyclictest25061-21cut19:40:213
922399118,2cyclictest24530-21ssh00:31:013
922399118,2cyclictest24278-21wc23:20:153
922399118,2cyclictest23884-21taskset23:42:183
922399118,2cyclictest23677-21cron19:40:013
922399118,2cyclictest21831-21grep21:45:283
922399118,2cyclictest21120-21grep20:40:183
922399118,2cyclictest17249-21ssh23:37:393
922399118,2cyclictest15882-21ssh21:41:333
922399118,2cyclictest15095-21tail22:50:163
922399118,2cyclictest12866-21taskset20:23:283
922399118,2cyclictest11790-21taskset19:15:013
922399117,3cyclictest23017-21tr20:45:163
922399117,3cyclictest1684-21cut22:40:183
922399113,4cyclictest224-21systemd-journal20:11:283
922399113,3cyclictest2270-21dump-pmu-power21:10:003
922399110,1cyclictest0-21swapper/300:37:413
922399110,10cyclictest23423-21chrt21:25:023
921999118,2cyclictest9406-21chrt19:10:022
921999118,2cyclictest834-21idleruntime-cro20:00:002
921999118,2cyclictest5969-21sed20:10:012
921999118,2cyclictest29060-21gzip20:55:232
921999118,2cyclictest24900-21apt-get20:50:022
921999118,2cyclictest17262-21grep20:30:272
921999117,3cyclictest12062-21grep19:15:142
921999112,4cyclictest224-21systemd-journal19:35:002
921799118,2cyclictest6222-21aten2_r7power_v20:10:141
921799118,2cyclictest3212-21basename23:50:241
921799118,2cyclictest31289-21grep21:00:201
921799118,2cyclictest25182-21cut19:40:221
921799118,2cyclictest2491-21ssh00:14:321
921799118,2cyclictest23230-21expr20:45:171
921799118,2cyclictest21912-21expr19:35:181
921799118,2cyclictest12613-21ssh23:11:181
921799117,3cyclictest24378-21grep00:05:291
921799116,1cyclictest25-21ksoftirqd/120:30:271
921799114,2cyclictest15382-21munin-run20:29:591
921399119,1cyclictest31340-21ntpq21:00:210
921399119,1cyclictest161rcuc/023:16:130
921399118,2cyclictest9919-21expr19:10:160
921399118,2cyclictest28026-21tr20:55:150
921399118,2cyclictest20962-21chrt19:30:430
921399118,2cyclictest15746-21cut00:00:150
921399118,2cyclictest13987-21head20:25:170
921399118,2cyclictest10312-21grep21:15:190
921399118,2cyclictest10105-21grep20:15:280
921399117,4cyclictest13-21ksoftirqd/019:40:010
921399110,10cyclictest20631-21cut20:40:150
43802110,1sleep30-21swapper/322:19:293
922399108,1cyclictest21161-21taskset19:33:173
922399107,2cyclictest15769-21ssh00:00:153
922399107,2cyclictest13997-21grep20:25:173
922399105,3cyclictest9493-21/usr/sbin/munin19:10:143
922399102,3cyclictest0-21swapper/300:10:143
921999108,1cyclictest5096-21cat20:05:252
921999108,1cyclictest200592sleep220:39:212
921999108,1cyclictest14354-21sort19:20:012
921999107,2cyclictest26953-21cut19:45:172
921999107,2cyclictest24596-21sort19:40:182
921999107,2cyclictest1075-21date20:00:012
921999106,3cyclictest32-21ksoftirqd/220:15:222
921799108,1cyclictest32239-21needreboot19:55:191
921799107,2cyclictest773-21sort21:05:161
921799107,2cyclictest22240-21ssh00:30:031
921799107,2cyclictest12415-21cat20:20:281
921799101,7cyclictest28252-21sshd19:47:311
921399108,1cyclictest5928-21chmod20:10:010
921399108,1cyclictest26392-21cut19:45:130
921399108,1cyclictest15862-21wc19:20:260
921399108,1cyclictest15612-21cat20:30:120
921399107,3cyclictest6472-21kworker/0:219:50:040
921399106,2cyclictest13-21ksoftirqd/020:50:080
921399105,4cyclictest161rcuc/020:35:170
921399102,6cyclictest9460-21/usr/sbin/munin23:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional