You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 07:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Oct 20, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2140199616,6cyclictest131rcu_preempt20:45:130
2141199502,37cyclictest7645-21ssh21:30:151
2141199502,37cyclictest7645-21ssh21:30:141
2140199501,39cyclictest20486-21ssh23:35:140
2142199495,26cyclictest17807-21ssh22:15:142
2141199471,35cyclictest2148-21ssh22:25:141
21401994717,11cyclictest20442-21ssh21:15:410
21401994715,10cyclictest7484-21ssh21:30:020
21401994715,10cyclictest7484-21ssh21:30:020
2140199462,34cyclictest16861-21ssh21:55:130
21421994521,18cyclictest28265-21kworker/u8:223:35:122
2141199456,28cyclictest6825-21diskmemload23:10:151
2141199456,28cyclictest6825-21diskmemload23:10:151
21411994516,21cyclictest21-21ksoftirqd/119:40:161
2141199451,34cyclictest23637-21ssh21:20:131
21401994516,9cyclictest6518-21ssh21:47:410
21401994515,10cyclictest29138-21ssh21:21:330
21421994414,15cyclictest26199-21ssh00:38:512
2141199448,4cyclictest131rcu_preempt22:20:161
21401994415,11cyclictest26895-21ssh23:19:010
21401994415,10cyclictest18905-21ssh00:33:150
21401994318,8cyclictest912-21ssh23:21:190
21401994317,10cyclictest17254-21ssh22:33:240
21401994316,9cyclictest1582-21ssh23:41:050
21401994316,12cyclictest10307-21ssh00:05:470
21401994315,12cyclictest9137-21ssh22:48:310
21401994315,11cyclictest31670-21ssh22:02:510
2143199424,21cyclictest25366irq/58-eth000:05:153
21421994228,9cyclictest32317-21apt-get19:55:132
21411994210,5cyclictest131rcu_preempt21:10:151
21401994230,8cyclictest8512-21apt-get21:50:140
21401994215,10cyclictest1158-21ssh23:02:510
21421994118,20cyclictest0-21swapper/220:00:122
21401994135,3cyclictest12927-21apt-get23:50:160
21401994117,6cyclictest19606-21ssh00:12:080
21401994115,11cyclictest6740-21ssh21:29:130
21401994115,11cyclictest16610-21ssh23:11:120
21401994115,11cyclictest16610-21ssh23:11:110
21401994115,10cyclictest14525-21sh22:10:540
21401994114,8cyclictest32064-21ssh22:22:010
21401994114,11cyclictest6630-21usage_EthcTask21:05:300
21401994114,10cyclictest17590-21sort20:10:240
21401994113,11cyclictest24433-21usage_EthcTask20:50:300
21421994026,10cyclictest18839-21apt-get22:55:142
21421994015,20cyclictest0-21swapper/223:05:132
21411994013,18cyclictest21-21ksoftirqd/123:00:151
21401994017,10cyclictest3731-21ssh00:23:440
21401994017,10cyclictest3731-21ssh00:23:440
21401994016,7cyclictest10656-21ssh00:27:150
21401994015,10cyclictest25265-21ssh00:36:490
21401994015,10cyclictest10125-21ssh23:27:310
21401994014,15cyclictest8953-21cat20:00:300
2141199395,26cyclictest2372-21ssh22:45:131
21401993932,2cyclictest29432-21apt-get00:00:130
21401993930,6cyclictest21571-21apt-get19:10:150
21401993919,13cyclictest0-21swapper/022:35:150
21401993916,8cyclictest27110-21ssh00:16:210
21401993915,8cyclictest854-21ssh22:43:130
21401993915,8cyclictest5551-21users20:30:300
21401993915,8cyclictest30920-21awk19:50:240
21401993915,8cyclictest26238-21ssh22:59:160
21401993914,8cyclictest6825-21diskmemload21:43:110
21401993913,9cyclictest7745-21/usr/sbin/munin19:25:310
2142199386,16cyclictest4620-21ssh23:45:122
21401993827,8cyclictest2145-21apt-get22:25:140
21401993815,9cyclictest16487-21ssh22:52:160
21401993815,9cyclictest16487-21ssh22:52:160
21401993813,8cyclictest6825-21diskmemload23:31:320
21401993812,9cyclictest9142-21/usr/sbin/munin20:35:230
21401993726,8cyclictest31172-21apt-get19:20:140
21401993715,8cyclictest14679-21ssh21:14:210
21401993715,7cyclictest9810-21ssh23:46:010
21401993714,7cyclictest6825-21diskmemload22:05:490
21401993711,9cyclictest22631-21ssh22:15:330
21401993711,10cyclictest28154-21ssh23:58:470
21421993628,5cyclictest5307-21apt-get00:05:152
21411993616,16cyclictest21-21ksoftirqd/100:25:141
2141199361,27cyclictest5330-21ssh00:05:161
21401993615,8cyclictest27382-21latency_hist19:50:010
21401993614,8cyclictest13449-21irqcore19:35:180
21401993614,7cyclictest6825-21diskmemload23:07:360
21401993613,10cyclictest32319-21dpkg19:55:120
21401993612,8cyclictest22477-21awk20:15:270
21401993611,9cyclictest23102-21ssh21:38:530
2142199359,13cyclictest2059-21sh23:22:562
2142199357,15cyclictest4491-21ssh00:04:232
2142199357,14cyclictest21594-21cat19:40:292
2142199357,12cyclictest0-21swapper/223:02:402
21421993528,4cyclictest15975-21apt-get21:35:132
21421993527,5cyclictest1037-21apt-get20:30:142
21421993519,12cyclictest0-21swapper/222:05:142
21421993510,12cyclictest17228-21ssh22:53:192
21421993510,12cyclictest17228-21ssh22:53:182
21401993514,8cyclictest8580-21if_err_eth019:30:170
21401993514,8cyclictest8580-21if_err_eth019:30:160
21401993512,7cyclictest27008-21irqstats19:15:190
2143199342,18cyclictest25366irq/58-eth022:35:143
2142199347,16cyclictest8866-21ssh22:48:062
21421993422,9cyclictest26181-21apt-get22:20:162
21421993418,12cyclictest0-21swapper/223:30:142
21421993411,8cyclictest10166-21ssh00:26:272
21401993419,12cyclictest0-21swapper/020:40:140
21401993413,8cyclictest12873-21ntp_192.168.11420:05:220
21401993412,8cyclictest26876-21kthreadcore20:55:230
2143199332,23cyclictest25366irq/58-eth023:00:143
2142199339,7cyclictest4140-21ssh00:24:252
2142199339,7cyclictest4140-21ssh00:24:252
2142199336,12cyclictest16755-21ssh23:11:242
2142199336,12cyclictest16755-21ssh23:11:232
21421993318,12cyclictest21846irq/40-dwc2_hso00:15:132
2140199338,11cyclictest1692-21cat21:00:310
2142199329,8cyclictest30429-21ssh21:23:232
2142199329,8cyclictest24788-21ssh21:59:562
2142199329,8cyclictest19616-21ssh00:12:092
2142199329,7cyclictest2101-21ssh22:25:112
2142199327,12cyclictest0-21swapper/222:00:082
21421993216,10cyclictest21139-21apt-get23:55:152
21421993215,8cyclictest26757-21ssh22:40:142
21421993214,13cyclictest0-21swapper/223:50:172
21421993213,13cyclictest0-21swapper/221:50:142
21421993212,15cyclictest0-21swapper/220:35:122
2141199324,22cyclictest8531-21ssh21:50:131
2143199313,16cyclictest25366irq/58-eth021:10:153
2143199312,12cyclictest25366irq/58-eth000:20:153
2143199312,12cyclictest25366irq/58-eth000:20:143
2142199316,7cyclictest6825-21diskmemload21:11:332
2142199315,10cyclictest0-21swapper/223:26:092
21421993119,9cyclictest12684-21apt-get00:30:142
21421993116,11cyclictest0-21swapper/221:00:132
21421993115,13cyclictest0-21swapper/221:25:132
21411993115,10cyclictest21392-21cyclictest22:40:131
2142199309,7cyclictest6027-21ssh21:47:002
2142199305,16cyclictest21846irq/40-dwc2_hso19:15:122
21421993014,13cyclictest0-21swapper/222:35:122
21421993014,12cyclictest32161-21/usr/sbin/munin20:25:282
21421993014,12cyclictest15557-21ssh21:34:582
21421993014,12cyclictest15557-21ssh21:34:572
21421993012,14cyclictest25214-21switchtime19:10:312
21421993012,14cyclictest12812-21grep20:05:212
2141199303,21cyclictest15535-21ssh21:15:161
21401993019,8cyclictest32514-21sed20:25:300
21401993019,8cyclictest18413-21/usr/sbin/munin19:40:200
2143199292,17cyclictest25366irq/58-eth020:10:153
21431992911,13cyclictest0-21swapper/322:25:143
21421992923,4cyclictest15524-21apt-get21:15:152
21421992923,3cyclictest28778-21apt-get23:40:132
21421992914,6cyclictest0-21swapper/220:10:142
21421992914,12cyclictest27285-21ntpq20:20:252
21421992914,11cyclictest14501-21ssh22:30:302
21421992913,12cyclictest19666-21/usr/sbin/munin23:15:302
21421992912,14cyclictest26634-21kthreadcore21:40:242
2143199283,18cyclictest0-21swapper/300:38:503
21431992813,10cyclictest25366irq/58-eth000:29:053
21421992819,6cyclictest20228-21apt-get20:50:122
21421992813,8cyclictest14999-21usage_EthcTask20:40:322
21421992812,8cyclictest19043-21/usr/sbin/munin20:15:152
21421992811,8cyclictest9273-21ssh22:10:032
21401992816,9cyclictest28076-21cat20:20:290
2143199272,16cyclictest25366irq/58-eth021:40:153
21421992718,6cyclictest21846irq/40-dwc2_hso19:20:152
21421992712,12cyclictest12326-21usage_Anybus19:30:302
21421992712,12cyclictest12326-21usage_Anybus19:30:302
2141199272,20cyclictest1212-21ssh23:21:431
2141199272,19cyclictest9420-21ssh22:10:151
2143199264,18cyclictest25366irq/58-eth022:40:133
2143199263,17cyclictest0-21swapper/323:57:433
21431992620,4cyclictest321rcuc/323:50:163
2143199261,15cyclictest25366irq/58-eth023:20:123
2142199268,13cyclictest0-21swapper/219:45:162
21421992614,9cyclictest0-21swapper/219:35:142
21421992612,7cyclictest8082-21sshd19:27:152
21421992612,10cyclictest2249-21apt-get21:05:122
21411992611,6cyclictest131rcu_preempt19:45:141
21411992611,4cyclictest21-21ksoftirqd/121:55:301
2143199252,17cyclictest0-21swapper/300:00:423
21421992513,9cyclictest0-21swapper/219:50:122
2143199244,16cyclictest25366irq/58-eth023:30:153
21431992418,4cyclictest321rcuc/300:10:393
2141199249,7cyclictest21-21ksoftirqd/122:05:341
2141199249,3cyclictest21-21ksoftirqd/123:40:191
2141199244,3cyclictest131rcu_preempt20:55:161
2141199243,16cyclictest21-21ksoftirqd/122:30:141
21411992411,4cyclictest21-21ksoftirqd/123:25:261
21401992415,7cyclictest21404-21latency_hist19:10:010
21401992415,7cyclictest21404-21latency_hist19:10:010
21431992314,6cyclictest25366irq/58-eth021:45:113
2143199231,14cyclictest25366irq/58-eth023:10:143
2143199231,14cyclictest25366irq/58-eth023:10:133
2142199232,14cyclictest21846irq/40-dwc2_hso20:55:152
21421992313,7cyclictest21846irq/40-dwc2_hso20:45:132
2141199232,4cyclictest131rcu_preempt19:55:131
2141199232,16cyclictest5804-21find20:35:121
21411992316,4cyclictest201rcuc/121:45:111
21411992311,8cyclictest29457-21ssh00:20:141
21411992311,8cyclictest29457-21ssh00:20:131
21431992217,3cyclictest321rcuc/322:31:423
2141199226,9cyclictest21-21ksoftirqd/100:35:101
2143199217,9cyclictest25366irq/58-eth021:25:133
21431992115,4cyclictest321rcuc/319:30:153
21431992115,4cyclictest321rcuc/319:30:143
2143199211,10cyclictest0-21swapper/321:55:143
2141199218,7cyclictest15629-21latency23:30:221
2141199218,6cyclictest21-21ksoftirqd/122:50:131
2141199218,6cyclictest21-21ksoftirqd/122:50:131
2141199217,8cyclictest21-21ksoftirqd/122:35:251
2141199212,4cyclictest21-21ksoftirqd/119:25:251
21411992115,4cyclictest0-21swapper/122:55:031
21411992112,6cyclictest31938-21ssh21:25:141
2141199211,17cyclictest27514-21apt-get19:50:131
2141199211,17cyclictest21679-21cut19:10:191
2143199209,8cyclictest0-21swapper/322:48:303
2143199204,11cyclictest0-21swapper/323:43:433
21431992015,3cyclictest321rcuc/323:48:273
21431992015,3cyclictest321rcuc/322:13:433
21431992015,3cyclictest321rcuc/320:45:263
21431992012,5cyclictest25366irq/58-eth022:15:153
2143199201,13cyclictest0-21swapper/319:45:023
2141199207,7cyclictest21-21ksoftirqd/119:15:181
2141199206,10cyclictest21-21ksoftirqd/123:50:171
2141199202,2cyclictest131rcu_preempt21:41:301
2141199201,9cyclictest29440-21ssh00:00:141
21411992015,3cyclictest131rcu_preempt23:15:171
21411992013,4cyclictest21-21ksoftirqd/100:10:331
21411992012,5cyclictest0-21swapper/119:35:321
2141199201,16cyclictest21167-21ssh23:55:171
2141199201,12cyclictest8717-21unixbench_multi23:45:281
2141199201,10cyclictest16706-21sed00:30:281
21411992010,7cyclictest4010-21ssh23:05:191
2143199199,8cyclictest0-21swapper/323:19:183
2143199199,6cyclictest0-21swapper/322:07:233
2143199193,11cyclictest0-21swapper/322:50:443
2143199193,11cyclictest0-21swapper/322:50:433
2143199192,11cyclictest0-21swapper/322:55:043
21431991914,3cyclictest321rcuc/321:39:573
21431991914,3cyclictest321rcuc/319:20:143
2143199191,12cyclictest0-21swapper/319:50:283
21431991910,6cyclictest25366irq/58-eth000:15:153
2143199190,12cyclictest25366irq/58-eth021:30:163
2143199190,12cyclictest25366irq/58-eth021:30:163
2141199198,8cyclictest8327-21apt-get19:30:131
2141199198,8cyclictest8327-21apt-get19:30:131
2141199198,3cyclictest21-21ksoftirqd/120:25:151
2141199198,2cyclictest21-21ksoftirqd/121:10:021
21411991913,5cyclictest21-21ksoftirqd/122:00:141
2143199189,6cyclictest0-21swapper/322:00:183
2143199182,10cyclictest0-21swapper/323:35:133
2143199182,10cyclictest0-21swapper/300:30:153
21431991813,3cyclictest321rcuc/320:40:313
21431991813,3cyclictest321rcuc/320:20:303
21431991812,4cyclictest321rcuc/321:20:133
21431991812,4cyclictest321rcuc/319:45:143
21431991812,4cyclictest0-21swapper/323:25:483
21431991812,4cyclictest0-21swapper/323:05:063
21431991812,4cyclictest0-21swapper/322:20:433
21431991811,5cyclictest321rcuc/320:30:123
2143199181,10cyclictest0-21swapper/320:00:013
21431991810,6cyclictest0-21swapper/321:52:243
21431991810,6cyclictest0-21swapper/321:15:163
2141199188,7cyclictest24097-21ntpq23:35:241
2141199186,3cyclictest21-21ksoftirqd/121:35:281
2141199185,10cyclictest14512-21grep20:40:291
2141199182,8cyclictest15578-21fschecks_time20:45:161
2141199181,2cyclictest131rcu_preempt22:18:581
21411991810,5cyclictest27453-21mailstats20:20:271
2141199180,12cyclictest21800-21find00:15:131
2143199172,10cyclictest0-21swapper/320:35:143
21431991713,2cyclictest321rcuc/321:05:163
21431991711,4cyclictest321rcuc/319:10:153
2143199171,13cyclictest0-21swapper/320:05:143
2143199171,11cyclictest0-21swapper/320:15:263
2141199179,6cyclictest2011-21cron21:05:001
2141199177,8cyclictest131rcu_preempt20:30:141
2141199172,12cyclictest19656-21irqrtprio20:15:211
2141199171,3cyclictest131rcu_preempt19:25:011
21411991712,2cyclictest21-21ksoftirqd/120:05:241
21411991710,2cyclictest21-21ksoftirqd/120:10:291
2143199162,10cyclictest0-21swapper/321:00:153
21431991611,3cyclictest321rcuc/320:57:293
21431991611,3cyclictest321rcuc/320:03:433
21431991611,3cyclictest0-21swapper/320:25:133
21431991610,4cyclictest321rcuc/319:35:133
2143199160,9cyclictest25366irq/58-eth020:50:123
2141199161,11cyclictest23503-21cat20:50:211
21411991610,4cyclictest9046-21usage_EthcTask20:00:311
21431991510,3cyclictest321rcuc/319:15:133
2143199149,3cyclictest0-21swapper/319:25:273
2143199141,11cyclictest0-21swapper/319:10:023
2143199141,11cyclictest0-21swapper/319:10:023
2142199141,9cyclictest21507-21latency_hist19:10:022
2142199141,9cyclictest21507-21latency_hist19:10:012
2141199131,10cyclictest21392-21cyclictest19:10:011
2141199131,10cyclictest21392-21cyclictest19:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional