You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 05:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 10, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7397996517,8cyclictest131rcu_preempt23:35:130
739799589,7cyclictest12-21ksoftirqd/022:55:110
741499513,6cyclictest20131-21ssh00:25:132
741499513,17cyclictest22346irq/40-dwc2_hso20:05:102
741499493,33cyclictest13634-21ssh22:15:152
741499492,17cyclictest22346irq/40-dwc2_hso20:55:142
740599491,41cyclictest24424-21ssh23:15:151
740599491,41cyclictest24424-21ssh23:15:151
739799496,6cyclictest131rcu_preempt21:30:130
739799496,6cyclictest131rcu_preempt21:30:130
741499481,36cyclictest20604-21ssh21:25:142
741499481,34cyclictest2935-21ssh22:45:142
740599472,32cyclictest851-21sshd21:40:141
741499431,28cyclictest29047-21ssh00:30:142
741499421,23cyclictest11171-21ssh00:20:162
740599403,26cyclictest16674-21ssh00:05:121
740599401,32cyclictest15387-21ssh23:10:141
740599401,32cyclictest15387-21ssh23:10:131
742299393,26cyclictest28366irq/58-eth020:10:123
740599383,24cyclictest25628-21diskmemload21:30:151
740599383,24cyclictest25628-21diskmemload21:30:151
742299372,22cyclictest28366irq/58-eth020:35:133
741499362,18cyclictest4767-21ssh22:10:142
740599363,23cyclictest31798-21ssh22:25:141
740599352,25cyclictest6221-21ssh21:35:121
7397993510,8cyclictest131rcu_preempt22:45:150
741499345,19cyclictest0-21swapper/220:40:112
741499341,11cyclictest10258-21ssh21:55:182
740599341,24cyclictest5535-21ssh00:35:121
741499332,19cyclictest19031-21rm23:30:132
740599333,23cyclictest2135-21sh00:15:131
7397993211,4cyclictest131rcu_preempt00:35:100
741499314,20cyclictest22346irq/40-dwc2_hso21:00:132
741499312,24cyclictest27015-21ssh22:04:132
741499312,19cyclictest22346irq/40-dwc2_hso20:20:132
741499312,19cyclictest22346irq/40-dwc2_hso20:20:122
740599312,21cyclictest22678-21sshd22:20:141
742299303,16cyclictest28366irq/58-eth021:35:123
741499304,17cyclictest22346irq/40-dwc2_hso19:50:122
740599304,17cyclictest7735-21sh00:00:121
739799303,18cyclictest24485-21ssh21:45:150
742299290,16cyclictest28366irq/58-eth022:00:133
741499292,21cyclictest2929-21ssh23:38:342
741499291,21cyclictest26354-21ssh22:40:132
742299283,16cyclictest0-21swapper/321:33:153
742299283,16cyclictest0-21swapper/321:33:143
741499282,21cyclictest4294-21ssh21:33:122
741499282,21cyclictest4294-21ssh21:33:122
741499282,21cyclictest0-21swapper/223:13:292
741499282,21cyclictest0-21swapper/223:13:292
741499282,19cyclictest22346irq/40-dwc2_hso20:10:102
741499282,19cyclictest0-21swapper/222:55:122
741499282,10cyclictest30651-21ssh22:24:142
740599286,17cyclictest21-21ksoftirqd/122:45:161
739799289,7cyclictest12-21ksoftirqd/019:25:130
739799284,16cyclictest24401-21sshd23:15:140
739799284,16cyclictest24401-21sshd23:15:130
7397992812,8cyclictest0-21swapper/022:30:270
742299271,14cyclictest28366irq/58-eth023:05:123
741499272,19cyclictest0-21swapper/222:50:142
741499272,19cyclictest0-21swapper/220:25:142
741499272,18cyclictest22346irq/40-dwc2_hso20:00:132
741499272,15cyclictest22346irq/40-dwc2_hso23:50:142
741499272,15cyclictest22346irq/40-dwc2_hso23:50:132
741499272,14cyclictest10105-21ssh23:25:142
741499272,10cyclictest0-21swapper/223:24:142
741499272,10cyclictest0-21swapper/223:24:142
741499271,16cyclictest11601-21sh21:20:142
741499271,11cyclictest24390-21apt-get23:15:142
741499271,11cyclictest24390-21apt-get23:15:142
740599278,16cyclictest21-21ksoftirqd/122:30:221
739799277,11cyclictest7385-21cyclictest21:20:140
739799271,19cyclictest992-21apt-get21:50:110
741499263,18cyclictest22346irq/40-dwc2_hso22:35:132
741499262,17cyclictest22346irq/40-dwc2_hso19:10:122
7414992618,5cyclictest10914-21ssh22:30:222
7414992612,9cyclictest0-21swapper/221:50:122
741499261,21cyclictest28991-21ls20:40:012
740599262,18cyclictest29960-21df_inode23:00:141
740599262,18cyclictest26176-21ssh22:03:141
7397992621,3cyclictest7736-21ssh00:00:120
742299252,17cyclictest0-21swapper/323:24:143
742299252,17cyclictest0-21swapper/323:24:143
742299252,17cyclictest0-21swapper/321:24:143
741499257,6cyclictest711-21rs:main2
741499254,10cyclictest131rcu_preempt23:05:192
741499253,17cyclictest27-21ksoftirqd/221:35:142
741499252,17cyclictest22346irq/40-dwc2_hso19:15:132
740599256,10cyclictest21-21ksoftirqd/120:20:131
740599256,10cyclictest21-21ksoftirqd/120:20:121
740599253,13cyclictest21-21ksoftirqd/120:50:121
740599251,2cyclictest131rcu_preempt21:17:371
740599251,17cyclictest24484-21sh21:45:141
740599250,3cyclictest131rcu_preempt21:00:121
739799251,18cyclictest2138-21ssh00:15:140
741499243,17cyclictest26521-21irqcore21:10:202
741499243,16cyclictest19579-21sshd20:30:142
741499243,11cyclictest22346irq/40-dwc2_hso20:45:132
741499242,16cyclictest22346irq/40-dwc2_hso23:00:132
741499242,16cyclictest22346irq/40-dwc2_hso22:25:132
741499240,15cyclictest7322-21latency_hist00:00:012
740599249,3cyclictest131rcu_preempt23:55:141
740599244,3cyclictest131rcu_preempt21:50:191
740599242,6cyclictest131rcu_preempt00:20:161
740599241,2cyclictest131rcu_preempt00:12:281
740599240,2cyclictest131rcu_preempt22:51:501
7414992311,9cyclictest8597-21kthreadcore00:00:202
741499231,15cyclictest22346irq/40-dwc2_hso19:35:132
740599235,4cyclictest21-21ksoftirqd/123:29:061
740599231,4cyclictest131rcu_preempt22:55:151
740599231,3cyclictest131rcu_preempt22:41:461
740599231,2cyclictest131rcu_preempt23:51:101
740599231,2cyclictest131rcu_preempt23:51:091
740599231,2cyclictest131rcu_preempt22:06:351
740599231,2cyclictest131rcu_preempt21:57:421
740599231,2cyclictest131rcu_preempt21:23:271
7405992310,11cyclictest0-21swapper/100:30:301
739799232,15cyclictest13496-21kworker/u8:021:42:130
7397992310,2cyclictest12-21ksoftirqd/021:25:250
742299227,10cyclictest0-21swapper/323:14:143
742299227,10cyclictest0-21swapper/323:14:143
742299223,12cyclictest0-21swapper/323:00:143
7422992210,7cyclictest28366irq/58-eth022:40:143
741499228,5cyclictest6522-21kthreadcore00:35:182
741499222,16cyclictest10933-21grep00:20:022
741499221,8cyclictest17166systemd-network22:05:102
741499221,10cyclictest25380-21ls00:10:022
740599226,3cyclictest131rcu_preempt20:35:221
740599223,3cyclictest131rcu_preempt23:40:331
740599222,3cyclictest131rcu_preempt23:10:011
740599222,3cyclictest131rcu_preempt20:30:281
740599221,8cyclictest131rcu_preempt19:50:131
740599221,7cyclictest17108-21kworker/1:000:26:061
740599221,6cyclictest18315-21/usr/sbin/munin19:55:261
740599221,4cyclictest131rcu_preempt22:18:091
740599221,3cyclictest131rcu_preempt21:26:421
740599221,3cyclictest0-21swapper/123:37:271
740599221,2cyclictest131rcu_preempt23:31:041
740599221,17cyclictest28082-21cut20:05:121
740599220,8cyclictest0-21swapper/122:36:411
739799229,3cyclictest12-21ksoftirqd/022:17:190
739799222,15cyclictest14249-21ssh21:39:140
7397992213,6cyclictest151rcuc/023:30:110
739799221,2cyclictest131rcu_preempt22:05:200
742299219,5cyclictest0-21swapper/322:25:263
742299213,12cyclictest0-21swapper/322:10:153
741499214,9cyclictest5128-21kthreadcore19:40:192
741499213,16cyclictest22346irq/40-dwc2_hso00:10:122
741499213,15cyclictest27796-21kthreadcore21:45:252
741499213,15cyclictest25414-21proc_pri19:25:262
741499213,13cyclictest22346irq/40-dwc2_hso19:30:142
741499212,16cyclictest15293-21/usr/sbin/munin21:40:252
741499212,13cyclictest0-21swapper/221:05:112
740599217,3cyclictest26009-21kthreadcore20:00:211
740599214,2cyclictest131rcu_preempt23:45:321
740599211,9cyclictest8996-21tr19:10:191
740599211,7cyclictest131rcu_preempt22:10:121
740599211,4cyclictest131rcu_preempt21:10:381
740599211,4cyclictest131rcu_preempt20:15:201
740599211,3cyclictest131rcu_preempt20:55:221
739799212,2cyclictest131rcu_preempt23:44:260
739799212,16cyclictest11145-21apt-get00:20:140
739799212,14cyclictest29956-21ssh23:00:130
7397992115,4cyclictest0-21swapper/022:40:010
7397992113,6cyclictest9737-21cron20:20:010
7397992113,5cyclictest18668-21kthreadcore00:05:220
7397992111,7cyclictest1-21systemd20:53:010
739799211,15cyclictest26306-21ssh22:20:260
7397992110,9cyclictest5164-21cron00:35:010
742299209,8cyclictest0-21swapper/322:30:163
742299209,6cyclictest0-21swapper/323:46:253
742299209,6cyclictest0-21swapper/321:15:473
742299203,12cyclictest0-21swapper/322:24:143
7422992011,7cyclictest0-21swapper/300:37:503
741499207,11cyclictest27-21ksoftirqd/219:55:222
741499202,9cyclictest17232-21fschecks_count19:20:152
741499202,9cyclictest16264-21kthreadcore23:45:222
741499202,15cyclictest10218-21grep20:50:292
741499201,6cyclictest2570-21ssh21:15:132
741499200,8cyclictest12247-21wc19:45:282
740599207,10cyclictest1398-21cron20:45:021
740599201,3cyclictest131rcu_preempt20:45:281
740599201,2cyclictest131rcu_preempt23:20:291
740599201,2cyclictest131rcu_preempt23:20:281
740599201,2cyclictest131rcu_preempt20:10:191
740599201,2cyclictest131rcu_preempt19:35:121
740599201,17cyclictest21-21ksoftirqd/119:25:291
740599201,15cyclictest14797-21apt-get20:25:131
740599201,12cyclictest20927-21/usr/sbin/munin19:20:281
740599201,11cyclictest15611-21memory19:15:221
739799209,8cyclictest25699-21cat21:10:010
739799204,12cyclictest26360-21ssh22:40:130
739799203,13cyclictest31814-21ssh22:25:140
739799202,4cyclictest12-21ksoftirqd/021:57:590
739799201,3cyclictest131rcu_preempt22:01:440
739799201,2cyclictest131rcu_preempt23:11:480
739799201,2cyclictest131rcu_preempt23:11:470
739799201,2cyclictest131rcu_preempt19:40:130
7397992011,6cyclictest0-21swapper/019:30:180
739799201,15cyclictest19343-21latency_hist20:30:010
739799201,11cyclictest13381-21latency_hist19:50:020
7397992010,8cyclictest8571-21sed22:10:280
739799200,2cyclictest131rcu_preempt23:48:000
739799200,17cyclictest2476-21ls19:35:230
742299199,8cyclictest0-21swapper/300:15:043
742299199,8cyclictest0-21swapper/300:08:183
742299198,8cyclictest0-21swapper/322:06:003
742299198,6cyclictest0-21swapper/322:35:093
742299193,11cyclictest0-21swapper/323:38:343
742299192,12cyclictest0-21swapper/321:40:143
742299192,11cyclictest0-21swapper/323:25:043
742299192,10cyclictest28366irq/58-eth019:45:143
7422991913,4cyclictest0-21swapper/300:30:283
7422991911,5cyclictest28366irq/58-eth022:45:163
7422991910,6cyclictest0-21swapper/322:57:453
741499196,11cyclictest5159-21sshd20:15:132
740599191,9cyclictest8721-21/usr/sbin/munin19:45:171
740599191,2cyclictest131rcu_preempt19:30:251
739799198,8cyclictest7528-21apt-get19:10:110
739799198,7cyclictest12-21ksoftirqd/019:55:140
739799192,2cyclictest131rcu_preempt20:40:280
739799191,9cyclictest9816-21latency23:05:220
7397991914,3cyclictest0-21swapper/020:30:130
7397991913,4cyclictest25628-21diskmemload21:15:150
739799191,2cyclictest131rcu_preempt23:20:490
739799191,2cyclictest131rcu_preempt23:20:480
739799191,2cyclictest131rcu_preempt19:50:290
739799191,2cyclictest131rcu_preempt00:25:290
7397991911,5cyclictest31969-21kthreadcore23:55:200
739799191,15cyclictest22301-21apt-get23:50:130
739799191,15cyclictest22301-21apt-get23:50:130
742299189,6cyclictest0-21swapper/321:49:073
742299189,6cyclictest0-21swapper/300:20:233
742299185,9cyclictest28366irq/58-eth000:25:153
742299183,11cyclictest0-21swapper/321:25:143
742299183,10cyclictest0-21swapper/323:18:143
742299183,10cyclictest0-21swapper/323:18:143
742299182,11cyclictest0-21swapper/321:10:163
742299182,10cyclictest0-21swapper/321:55:153
742299182,10cyclictest0-21swapper/320:45:143
742299182,10cyclictest0-21swapper/320:30:143
742299182,10cyclictest0-21swapper/319:55:143
742299182,10cyclictest0-21swapper/319:30:143
742299182,10cyclictest0-21swapper/319:20:133
742299182,10cyclictest0-21swapper/319:15:143
7422991813,3cyclictest33-21ksoftirqd/323:55:143
7422991811,5cyclictest0-21swapper/323:30:173
7422991811,4cyclictest0-21swapper/300:10:133
742299181,11cyclictest0-21swapper/300:00:153
7422991810,6cyclictest0-21swapper/322:50:223
7422991810,5cyclictest28366irq/58-eth022:15:143
7422991810,5cyclictest28366irq/58-eth021:50:223
740599181,2cyclictest131rcu_preempt21:05:121
740599180,2cyclictest131rcu_preempt19:40:271
739799183,12cyclictest12004-21kthreadcore23:25:210
739799181,6cyclictest21498-21usage_EthcTask19:20:300
739799181,5cyclictest28608-21irqstats20:05:160
739799181,2cyclictest131rcu_preempt21:10:190
739799181,2cyclictest131rcu_preempt20:45:130
739799181,2cyclictest131rcu_preempt20:35:150
739799181,2cyclictest131rcu_preempt20:10:160
739799181,2cyclictest131rcu_preempt20:00:190
7397991812,4cyclictest0-21swapper/019:15:280
7397991812,4cyclictest0-21swapper/000:10:120
739799181,10cyclictest20731-21munin-run21:05:020
739799180,2cyclictest131rcu_preempt22:50:450
742299179,5cyclictest28366irq/58-eth023:41:223
742299179,5cyclictest0-21swapper/319:35:303
742299172,10cyclictest0-21swapper/323:50:153
742299172,10cyclictest0-21swapper/323:50:153
742299172,10cyclictest0-21swapper/320:25:153
742299172,10cyclictest0-21swapper/319:50:133
742299172,10cyclictest0-21swapper/319:40:133
742299171,12cyclictest0-21swapper/320:05:203
740599170,2cyclictest131rcu_preempt19:10:021
739799177,7cyclictest11525-21fschecks_time20:55:160
739799171,2cyclictest131rcu_preempt20:20:230
739799171,2cyclictest131rcu_preempt20:20:230
742299152,9cyclictest0-21swapper/320:20:133
742299152,9cyclictest0-21swapper/320:20:133
742299152,9cyclictest0-21swapper/319:10:133
742299152,10cyclictest0-21swapper/320:40:123
742299151,10cyclictest0-21swapper/320:55:143
742299142,8cyclictest0-21swapper/320:15:143
742299141,9cyclictest0-21swapper/320:00:143
741499141,10cyclictest7430-21latency_hist19:10:022
742299138,3cyclictest0-21swapper/320:50:293
739799131,9cyclictest7400-21latency_hist19:10:020
742299127,3cyclictest33-21ksoftirqd/319:25:153
742299127,3cyclictest28366irq/58-eth021:10:013
742299111,7cyclictest0-21swapper/321:00:213
74229991,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional