You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 12:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Nov 30, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10855210087,9sleep30-21swapper/319:06:333
1081129783,10sleep10-21swapper/119:05:581
1080228767,16sleep20-21swapper/219:05:522
1103028572,9sleep00-21swapper/019:08:460
215422570,0sleep30-21swapper/322:35:373
1121399250,23cyclictest0-21swapper/223:44:332
1120899250,1cyclictest0-21swapper/121:45:361
11208992422,2cyclictest25-21ksoftirqd/122:20:491
1120899231,2cyclictest0-21swapper/123:20:191
1121399210,2cyclictest0-21swapper/222:25:552
1121399210,21cyclictest0-21swapper/222:33:122
1121399210,21cyclictest0-21swapper/200:03:532
1120499217,7cyclictest16169-21munin-run19:20:000
1121399200,19cyclictest37850irq/131-snd_hda00:30:142
55772190,0sleep00-21swapper/023:12:410
1121399185,11cyclictest4060-21diskmemload22:24:262
1120499186,6cyclictest20257-21awk23:00:000
1121599174,8cyclictest0-21swapper/322:45:153
1121399174,3cyclictest224-21systemd-journal21:10:012
1121599164,5cyclictest0-21swapper/323:55:143
11215991613,2cyclictest0-21swapper/323:20:183
1121399160,15cyclictest0-21swapper/222:01:212
1121399160,14cyclictest0-21swapper/223:28:392
1120899160,2cyclictest0-21swapper/123:13:491
1121599152,2cyclictest141rcu_preempt21:57:183
1121399159,5cyclictest10778-21basename21:15:002
1121399159,3cyclictest31476-21chrt00:20:142
1121399152,5cyclictest224-21systemd-journal21:47:312
1121399152,12cyclictest0-21swapper/222:54:482
1121399150,15cyclictest0-21swapper/223:15:192
1121399150,15cyclictest0-21swapper/222:38:302
1120899151,10cyclictest0-21swapper/100:05:141
1120499158,4cyclictest16653-21apt-get23:45:140
11215991412,1cyclictest39-21ksoftirqd/300:14:363
11215991410,3cyclictest27722-21grep22:40:263
11215991410,3cyclictest20124-21wc21:20:293
11215991410,2cyclictest0-21swapper/319:45:163
1121399149,3cyclictest1195-21latency_hist23:10:002
1121399148,4cyclictest0-21swapper/222:16:412
1121399144,5cyclictest516-21gdbus20:40:002
11213991411,2cyclictest14493-21head22:05:302
11213991411,1cyclictest0-21swapper/223:56:462
11213991410,3cyclictest5722-21tr00:25:182
11213991410,3cyclictest31049-21dump-pmu-power21:30:012
1120899143,3cyclictest224-21systemd-journal19:55:001
1120899141,12cyclictest26994-21cat20:50:001
11208991411,2cyclictest11195-21cyclictest23:32:121
1120499141,7cyclictest22137-21idleruntime-cro20:39:590
11204991410,3cyclictest3074-21basename20:00:000
1120499140,1cyclictest0-21swapper/021:51:090
1120499140,1cyclictest0-21swapper/021:30:130
1121599139,3cyclictest881-21head00:20:273
1121599139,3cyclictest30764-21munin-node21:32:303
1121599139,3cyclictest22127-21basename20:40:003
1121599139,2cyclictest6461-21expr22:00:183
1121599139,2cyclictest39-21ksoftirqd/300:07:463
1121599133,7cyclictest495-21cron00:29:563
11215991310,2cyclictest31995-21grep21:00:153
11215991310,2cyclictest25655-21sort21:25:173
11215991310,2cyclictest15744-21grep19:15:293
11215991310,2cyclictest1180-21cron23:10:003
1121399138,4cyclictest3052-21taskset21:32:402
1121399138,3cyclictest1862-21wc21:05:002
11213991310,2cyclictest604-21dump-pmu-power19:55:002
11213991310,2cyclictest28860-21grep20:50:272
11213991310,2cyclictest21342-21ssh22:10:452
11213991310,2cyclictest13577-21awk19:15:002
1121399130,1cyclictest0-21swapper/200:11:342
1121399130,11cyclictest0-21swapper/221:44:362
1120899139,3cyclictest26239-21tr22:40:161
1120899139,3cyclictest24626-21cat23:50:211
1120899139,3cyclictest11890-21tail23:40:311
1120899139,2cyclictest10577-21ssh00:30:151
1120899138,1cyclictest9352-21gpgv23:40:001
11208991310,2cyclictest28941-21grep19:45:181
11208991310,2cyclictest28805-21fgrep23:05:231
11208991310,2cyclictest23499-21sed20:40:221
11208991310,2cyclictest19238-21grep20:30:281
11208991310,2cyclictest16359-21cat19:20:001
11208991310,2cyclictest14488-21grep20:20:281
11208991310,2cyclictest14151-21tr22:55:161
1120899130,12cyclictest7699-21tail21:35:271
1120899130,12cyclictest7699-21tail21:35:261
1120899130,12cyclictest0-21swapper/123:55:011
1120499136,6cyclictest24910-21munin-run21:50:000
11204991310,2cyclictest5607-21ssh22:00:000
11204991310,2cyclictest29951-21grep19:45:270
11204991310,2cyclictest25190-21cat23:50:290
11204991310,2cyclictest1913-21grep22:45:280
11204991310,2cyclictest12719-21grep21:40:170
1120499130,0cyclictest0-21swapper/023:44:160
1121599129,2cyclictest4259-21grep23:35:203
1121599129,2cyclictest27575-21sed22:15:293
1121599129,2cyclictest26768-21grep21:50:233
1121599129,2cyclictest21953-21grep19:30:193
1121599129,2cyclictest21758-21ssh23:25:073
1121599129,2cyclictest20572-21grep22:10:223
1121599129,2cyclictest18501-21grep23:45:283
1121599129,2cyclictest14248-21grep21:40:293
1121599129,2cyclictest13149-21ssh22:05:193
1121599129,2cyclictest10176-21grep23:15:273
1121599128,3cyclictest1178-21grep19:55:163
1121599124,4cyclictest224-21systemd-journal21:17:013
1121599122,2cyclictest224-21systemd-journal19:55:003
1121599120,12cyclictest0-21swapper/323:54:493
1121599120,11cyclictest3566-21cut20:00:183
1121399129,2cyclictest3002-21wc23:10:232
1121399129,2cyclictest26201-21grep19:40:162
1121399129,2cyclictest25256-21sed19:35:292
1121399129,2cyclictest18862-21head20:30:232
1121399129,2cyclictest18240-21grep00:10:012
1121399129,2cyclictest16832-21grep00:35:192
1121399129,2cyclictest16832-21grep00:35:182
1121399129,2cyclictest16016-21uname23:20:202
1121399129,2cyclictest14100-21grep20:20:232
1121399129,2cyclictest12359-21wc21:15:212
1121399128,3cyclictest26369-21ssh22:40:162
1121399128,3cyclictest11573-21grep20:15:212
1121399124,4cyclictest224-21systemd-journal22:00:002
1121399121,2cyclictest19283-21kworker/2:023:50:482
1120899129,2cyclictest991-21sed21:30:261
1120899129,2cyclictest7082-21head22:25:161
1120899129,2cyclictest31605-21cut00:20:161
1120899129,2cyclictest3054-21cut21:05:231
1120899129,2cyclictest25517-21cat22:39:591
1120899129,2cyclictest24517-21basename21:25:001
1120899129,2cyclictest18784-21wc22:10:001
1120899129,2cyclictest11354-21basename21:15:161
1120899129,2cyclictest10753-21cron21:15:001
1120899129,2cyclictest10220-21perf20:15:011
1120899128,3cyclictest14504-21cron23:20:001
1120899120,11cyclictest26614-21grep19:40:181
1120499129,2cyclictest9681-21wc23:15:230
1120499129,2cyclictest5369-21head21:10:210
1120499129,2cyclictest4946-21grep20:00:300
1120499129,2cyclictest28142-21grep23:05:170
1120499129,2cyclictest25426-21idleruntime-cro22:39:590
1120499129,2cyclictest24325-21grep19:35:190
1120499129,2cyclictest19596-21grep19:25:170
1120499129,2cyclictest18634-21grep21:20:170
1120499129,2cyclictest15820-21grep23:20:180
1120499129,2cyclictest13743-21grep22:30:170
1120499127,4cyclictest16945-21grep20:25:260
1120499125,3cyclictest224-21systemd-journal21:10:000
1120499124,7cyclictest15006-21idleruntime-cro20:25:000
1120499120,12cyclictest0-21swapper/023:34:510
1121599119,1cyclictest15340-21apt-get00:35:003
1121599118,2cyclictest8191-21grep22:50:223
1121599118,2cyclictest5909-21tr21:35:143
1121599118,2cyclictest5909-21tr21:35:143
1121599118,2cyclictest3554-21grep21:05:293
1121599118,2cyclictest31775-21ssh23:31:443
1121599118,2cyclictest26697-21cat00:15:273
1121599118,2cyclictest2186-21df_abs23:10:163
1121599118,2cyclictest20385-21ssh23:00:003
1121599118,2cyclictest18949-21ssh21:45:143
1121599118,2cyclictest18884-21ssh00:36:513
1121599118,2cyclictest18884-21ssh00:36:503
1121599118,2cyclictest15898-21ssh22:55:343
1121599118,2cyclictest15050-21ssh22:30:363
1121599117,3cyclictest5821-21cron22:25:003
1121599117,3cyclictest39-21ksoftirqd/322:25:203
1121599114,5cyclictest31646-21systemctl21:00:003
1121599114,2cyclictest224-21systemd-journal20:31:053
1121599110,10cyclictest6169-21cat00:00:223
1121399119,1cyclictest3627-21expr20:00:182
1121399118,2cyclictest9317-21head20:10:242
1121399118,2cyclictest7108-21tr21:35:212
1121399118,2cyclictest7108-21tr21:35:202
1121399118,2cyclictest6482-21cron22:49:592
1121399118,2cyclictest3128-21basename23:35:132
1121399118,2cyclictest303862sleep221:53:532
1121399118,2cyclictest28214-21ssh00:16:532
1121399118,2cyclictest2425-21expr19:55:272
1121399118,2cyclictest23638-21tr20:40:242
1121399118,2cyclictest23049-21ssh23:00:502
1121399118,2cyclictest18645-21ssh23:45:342
1121399118,2cyclictest18254-21ssh21:20:092
1121399118,2cyclictest18083-21tail19:20:262
1121399118,2cyclictest14640-21grep22:55:192
1121399113,4cyclictest224-21systemd-journal19:30:012
1121399111,7cyclictest62050irq/128-enp2s0-20:45:152
1121399110,10cyclictest28863-21sort23:30:142
1120899118,2cyclictest9571-21ssh22:51:111
1120899118,2cyclictest6141-21cut22:00:171
1120899118,2cyclictest5885-21ssh00:00:201
1120899118,2cyclictest3484-21ssh21:57:441
1120899118,2cyclictest30310-21ssh21:29:091
1120899118,2cyclictest28342-21ssh21:51:381
1120899118,2cyclictest25722-21sort00:15:171
1120899118,2cyclictest20790-21head20:35:181
1120899118,2cyclictest20622-21ssh23:00:091
1120899118,2cyclictest20531-21sed19:25:261
1120899118,2cyclictest19125-21seq00:37:111
1120899118,2cyclictest19125-21seq00:37:111
1120899118,2cyclictest18540-21cron19:25:001
1120899118,2cyclictest17372-21basename20:29:591
1120899118,2cyclictest13877-21run-parts21:40:251
1120899118,2cyclictest13369-21expr22:30:161
1120899118,2cyclictest1216-21sed21:00:271
1120899117,3cyclictest23294-21cron19:35:011
1120899114,3cyclictest4060-21diskmemload22:15:231
1120899112,5cyclictest1291-21runrttasks00:12:111
1120899111,6cyclictest7364-21unixbench_multi20:05:281
1120899110,9cyclictest0-21swapper/122:45:141
1120899110,10cyclictest22299-21ssh22:11:521
1120899110,10cyclictest16507-21ssh23:45:081
1120499118,2cyclictest8800-21grep22:50:300
1120499118,2cyclictest6904-21ssh23:37:300
1120499118,2cyclictest6224-21ssh00:00:230
1120499118,2cyclictest5655-21ssh21:35:020
1120499118,2cyclictest5655-21ssh21:35:020
1120499118,2cyclictest32677-21head21:00:190
1120499118,2cyclictest31886-21ssh23:55:230
1120499118,2cyclictest30053-21ssh00:18:580
1120499118,2cyclictest27642-21df20:50:170
1120499118,2cyclictest27235-21ssh21:25:480
1120499118,2cyclictest24005-21ssh23:01:500
1120499118,2cyclictest23779-21cut20:40:260
1120499118,2cyclictest22672-21head23:25:200
1120499118,2cyclictest21040-21grep22:10:270
1120499118,2cyclictest16788-21rm22:07:580
1120499118,2cyclictest12271-21taskset20:16:080
1120499118,2cyclictest11985-21grep21:15:180
1120499118,2cyclictest11864-21chrt00:05:140
1120499117,3cyclictest16854-21head00:35:180
1120499117,3cyclictest16854-21head00:35:180
1120499116,5cyclictest690-21apache222:40:440
1120499114,5cyclictest0-21swapper/022:20:290
1120499113,7cyclictest7744-21snmpd00:26:300
11204991110,1cyclictest0-21swapper/022:26:410
1120499110,10cyclictest9992-21ssh22:02:410
1121599108,1cyclictest9801-21ssh23:40:043
1121599108,1cyclictest8302-21cat20:10:173
1121599108,1cyclictest26172-21cat19:40:163
1121599108,1cyclictest25422-21fschecks_time20:45:193
1121599108,1cyclictest22399-21apache_processe20:40:153
1121599108,1cyclictest19115-21cat19:25:163
1121599108,1cyclictest16438-21apache_processe19:20:123
1121599108,1cyclictest15227-21basename20:25:133
1121599108,1cyclictest13255-21chrt19:11:213
1121599108,1cyclictest11682-21cut20:15:233
1121599107,2cyclictest7493-21ssh21:11:323
1121599107,2cyclictest5822-21tr20:05:173
1121599107,2cyclictest23585-21basename19:35:173
1121599107,2cyclictest13652-21grep20:20:183
1121599101,8cyclictest30764-21munin-node20:50:043
1121399108,1cyclictest30924-21smart_sda20:55:272
1121399108,1cyclictest302912sleep219:48:072
1121399108,1cyclictest17361-21latency_hist20:30:002
1121399107,2cyclictest6422-21grep20:05:202
1121399101,7cyclictest0-21swapper/219:30:582
1121399100,9cyclictest160172sleep219:18:262
1120899108,1cyclictest5569-21cat20:05:011
1120899108,1cyclictest27002sleep119:55:391
1120899108,1cyclictest25408-21taskset19:36:421
1120899108,1cyclictest11587-21egrep19:10:141
1120899107,2cyclictest5341-21ssh00:25:101
1120899107,2cyclictest29808-21expr20:55:161
1120899107,2cyclictest21676-21ssh23:25:021
1120899107,2cyclictest11230-21sed20:15:181
1120499108,2cyclictest487-21grep00:20:240
1120499108,1cyclictest9543-21cat20:10:290
1120499108,1cyclictest5991-21expr20:05:160
1120499108,1cyclictest26891-21head22:15:230
1120499108,1cyclictest26839-21taskset20:49:310
1120499108,1cyclictest230232sleep019:31:390
1120499108,1cyclictest16901-21expr19:20:160
1120499107,2cyclictest31179-21cut19:50:160
1120499107,2cyclictest30753-21grep20:55:230
1120499107,2cyclictest19724-21cron20:35:000
1120499107,2cyclictest19011-21expr00:10:170
1120499107,2cyclictest13128-21ssh00:31:460
1120499107,2cyclictest12335-21cat19:10:190
112089997,1cyclictest27846-21tune2fs20:50:171
112049997,1cyclictest25968-21apache_volume19:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional