You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-24 - 07:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Jun 24, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29567997021,6cyclictest131rcu_preempt20:20:141
29584996033,19cyclictest35966irq/58-eth000:30:153
2956799592,44cyclictest19009-21ssh23:00:121
29567995915,33cyclictest131rcu_preempt22:45:131
2956799583,40cyclictest1079-21ssh21:20:131
2955899551,42cyclictest30029-21ssh22:30:140
2956799534,12cyclictest21-21ksoftirqd/123:15:121
2956799533,39cyclictest20561-21ssh23:55:141
2956799482,33cyclictest12120-21ssh22:20:141
2956799473,14cyclictest21-21ksoftirqd/119:45:111
2955899472,37cyclictest19220-21ssh21:30:150
2955899472,37cyclictest19220-21ssh21:30:150
2957699466,14cyclictest0-21swapper/223:15:122
2956799461,38cyclictest22825-21ssh21:50:121
2958499452,28cyclictest35966irq/58-eth021:10:153
29576994514,14cyclictest27-21ksoftirqd/221:55:172
2957699442,32cyclictest10046-21ssh22:55:142
2955899442,37cyclictest23929-21ssh00:15:130
29584994337,3cyclictest35966irq/58-eth023:00:123
2955899431,35cyclictest27911-21ssh23:05:140
2956799424,28cyclictest24460-21ssh21:15:141
2957699414,11cyclictest30550-21irqcore22:30:202
2955899401,29cyclictest1186-21ssh22:50:140
2958499394,22cyclictest35966irq/58-eth023:15:133
2958499392,25cyclictest35966irq/58-eth021:55:153
2955899394,31cyclictest22816-21ssh21:50:120
29558993931,4cyclictest2580-21apt-get23:45:120
29558993926,9cyclictest11712-21apt-get19:25:130
29558993924,11cyclictest24452-21apt-get21:15:150
2958499381,21cyclictest35966irq/58-eth020:50:133
2956799381,24cyclictest131rcu_preempt19:15:121
2955899384,25cyclictest12-21ksoftirqd/000:30:140
2955899382,32cyclictest27533-21ssh00:35:130
2955899381,5cyclictest131rcu_preempt22:05:110
29567993711,18cyclictest21-21ksoftirqd/100:05:181
2957699362,19cyclictest26391-21ssh22:10:112
2957699361,28cyclictest0-21swapper/200:30:142
2956799368,19cyclictest21-21ksoftirqd/119:20:121
29576993525,7cyclictest6039-21ssh00:05:182
2955899356,26cyclictest9551-21ssh00:25:120
2955899356,26cyclictest9551-21ssh00:25:120
29558993522,9cyclictest0-21swapper/023:15:130
2956799342,17cyclictest4578-21ssh23:10:141
2955899346,9cyclictest25011-21sh22:45:180
29558993425,5cyclictest14060-21apt-get20:35:140
29558993423,5cyclictest17679-21apt-get20:05:120
29558993411,19cyclictest12-21ksoftirqd/023:25:140
2956799332,25cyclictest17434-21ssh22:05:131
2956799332,23cyclictest18439-21rm00:10:211
2956799331,21cyclictest8419-21ssh22:00:121
29558993325,4cyclictest14915-21ssh00:10:120
29558993323,5cyclictest22487-21apt-get20:10:140
2958499321,17cyclictest35966irq/58-eth021:30:153
2958499321,17cyclictest35966irq/58-eth021:30:143
2957699324,14cyclictest7956-21ssh23:30:112
2957699322,17cyclictest13630-21cut23:50:212
2957699321,26cyclictest15556-21ssh22:40:132
2957699321,18cyclictest6570-21sh22:35:132
29567993214,3cyclictest131rcu_preempt23:40:141
2955899326,10cyclictest21501-21ssh22:41:520
2955899324,9cyclictest30067-21ntp_192.168.11422:10:230
2955899322,11cyclictest3364-21sed21:55:310
29558993219,8cyclictest15474-21ssh21:10:140
29584993119,9cyclictest35966irq/58-eth023:55:153
2957699314,18cyclictest27346irq/40-dwc2_hso23:10:132
29576993115,12cyclictest27346irq/40-dwc2_hso23:35:112
2956799319,17cyclictest21-21ksoftirqd/123:25:141
2955899315,11cyclictest21490-21irqcore22:25:210
2955899315,11cyclictest11096-21sed21:05:190
2955899314,9cyclictest29607-21df_inode00:00:160
2955899314,8cyclictest9015-21usage_Anybus21:40:310
29558993123,3cyclictest3265-21apt-get19:50:110
2958499303,17cyclictest35966irq/58-eth021:40:123
2957699305,14cyclictest27346irq/40-dwc2_hso19:10:132
2957699304,14cyclictest27346irq/40-dwc2_hso20:45:122
2957699304,14cyclictest27346irq/40-dwc2_hso20:45:122
2957699302,26cyclictest17575-21ssh00:29:222
2957699302,26cyclictest17575-21ssh00:29:212
2957699302,20cyclictest0-21swapper/219:40:132
2957699302,19cyclictest27346irq/40-dwc2_hso20:20:142
2955899306,10cyclictest28336-21ssh21:35:150
2955899304,8cyclictest18950-21ssh22:23:000
2955899304,10cyclictest26628-21ssh23:03:570
2955899303,10cyclictest8543-21ssh00:24:140
29558993022,4cyclictest6025-21apt-get00:05:170
29558993018,7cyclictest10148-21kworker/u8:223:35:120
2955899301,12cyclictest15129-21kthreadcore21:45:200
2957699294,19cyclictest19018-21ssh23:00:132
2957699293,19cyclictest27346irq/40-dwc2_hso22:05:122
2957699292,18cyclictest0-21swapper/200:35:132
29576992913,11cyclictest15490-21sh21:10:162
2957699291,12cyclictest4825-21ssh21:40:122
2957699291,10cyclictest10608-21cut21:05:162
2956799292,21cyclictest21067-21ssh22:25:161
2956799292,19cyclictest28245-21ssh21:35:151
2955899297,10cyclictest13451-21ssh23:31:140
2955899296,9cyclictest11892-21rm22:36:040
2955899295,8cyclictest1927-21ntp_states19:45:270
2955899294,11cyclictest22842-21usage_Anybus20:40:300
29558992924,2cyclictest27268-21apt-get20:15:100
2955899292,10cyclictest5914-21cut23:10:210
29558992917,8cyclictest0-21swapper/021:20:130
29558992911,13cyclictest0-21swapper/021:25:130
2957699285,14cyclictest27346irq/40-dwc2_hso19:55:132
2957699282,20cyclictest0-21swapper/219:25:152
2955899286,8cyclictest20396-21usage_Anybus19:30:280
2955899285,10cyclictest10478-21rm22:18:310
2955899283,8cyclictest31217-21ssh23:40:550
2955899283,7cyclictest2085-21apt-get19:15:120
2955899281,9cyclictest24550-21kworker/u8:322:59:250
29558992817,8cyclictest20552-21apt-get23:55:150
2958499272,10cyclictest35966irq/58-eth021:35:153
2957699273,14cyclictest27346irq/40-dwc2_hso00:20:142
2957699272,19cyclictest0-21swapper/222:20:142
2955899274,9cyclictest24822-21tr19:35:270
2955899274,9cyclictest24822-21tr19:35:270
2955899274,8cyclictest9159-21date20:30:020
2955899274,10cyclictest7325-21cut19:20:170
2955899273,13cyclictest0-21swapper/020:50:180
2955899272,10cyclictest24201-21kthreadcore23:20:200
2957699263,12cyclictest27346irq/40-dwc2_hso21:45:122
2957699262,20cyclictest0-21swapper/220:05:132
2957699261,18cyclictest27346irq/40-dwc2_hso19:50:112
2957699261,14cyclictest2595-21ssh23:45:142
2957699261,10cyclictest31159-21ssh00:18:202
2956799262,18cyclictest29550-21ssh00:16:251
2955899265,9cyclictest29902-21fw_packets19:10:160
2955899264,8cyclictest16514-21ssh23:50:410
2955899263,9cyclictest25361-21tr20:45:210
2955899263,9cyclictest25361-21tr20:45:200
2955899263,7cyclictest1042-21fschecks_time20:55:180
29558992619,4cyclictest8412-21sh22:00:120
2958499253,11cyclictest0-21swapper/323:05:133
2958499251,17cyclictest35966irq/58-eth022:55:143
2957699253,15cyclictest27346irq/40-dwc2_hso23:20:122
2957699252,19cyclictest26069-21ssh23:40:122
2957699252,19cyclictest22995-21ssh00:14:212
2957699252,17cyclictest0-21swapper/223:55:152
2957699251,18cyclictest28094-21ssh21:35:052
2957699251,15cyclictest27418-21usage_Anybus21:50:452
2956799254,15cyclictest528-21ssh00:20:141
2955899258,12cyclictest0-21swapper/021:00:140
2955899254,8cyclictest12035-21usage_Anybus19:55:280
2955899252,14cyclictest13506-21irqstats20:00:270
2955899252,14cyclictest13506-21irqstats20:00:270
29584992411,7cyclictest0-21swapper/321:20:143
2958499241,14cyclictest35966irq/58-eth022:20:133
29584992410,9cyclictest35966irq/58-eth023:35:113
2957699243,16cyclictest10150-21ssh21:25:142
2957699241,21cyclictest9662-21latency_hist21:25:022
2957699241,12cyclictest28635-21cut23:05:192
29576992411,11cyclictest27737-21ssh21:35:022
29576992411,11cyclictest27737-21ssh21:35:012
2956799247,3cyclictest131rcu_preempt22:50:141
29567992412,4cyclictest21-21ksoftirqd/121:30:271
29567992412,4cyclictest21-21ksoftirqd/121:30:271
29567992411,10cyclictest21-21ksoftirqd/100:30:011
29567992411,10cyclictest21-21ksoftirqd/100:30:011
2958499234,14cyclictest0-21swapper/322:35:133
2958499232,16cyclictest35966irq/58-eth021:15:143
2957699239,11cyclictest31419-21apt-get23:25:132
2957699239,11cyclictest30481-21kthreadcore20:50:222
2957699231,12cyclictest9552-21grep21:00:282
2957699231,11cyclictest11877-21latency22:00:242
29576992310,6cyclictest26931-21ssh22:26:492
2956799232,16cyclictest0-21swapper/122:10:121
29567992311,3cyclictest21-21ksoftirqd/123:30:131
29558992311,9cyclictest26083-21apt-get19:40:110
2958499225,13cyclictest35966irq/58-eth022:40:123
2958499222,11cyclictest0-21swapper/323:20:123
2958499221,4cyclictest0-21swapper/322:10:123
2958499221,14cyclictest0-21swapper/321:45:143
2957699228,11cyclictest22574-21df_inode20:10:152
2957699224,9cyclictest18645-21kthreadcore19:30:202
2957699223,10cyclictest31177-21kthreadcore00:00:222
2957699222,9cyclictest0-21swapper/219:15:132
2957699222,11cyclictest4546-21awk20:55:302
2957699222,11cyclictest19204-21irqcore20:40:182
2957699221,4cyclictest24450-21ssh21:15:132
2957699221,12cyclictest1723-21cut22:50:192
2956799221,4cyclictest131rcu_preempt21:10:141
2956799221,18cyclictest27893-21find23:05:121
29567992211,2cyclictest21-21ksoftirqd/119:35:251
29567992211,2cyclictest21-21ksoftirqd/119:35:251
29567992210,7cyclictest0-21swapper/122:40:131
29567992210,3cyclictest21-21ksoftirqd/123:55:021
2955899220,19cyclictest12878-21ls20:30:290
2958499213,13cyclictest35966irq/58-eth023:10:143
29584992113,6cyclictest35966irq/58-eth023:40:123
29584992111,7cyclictest35966irq/58-eth000:35:133
2958499211,11cyclictest35966irq/58-eth020:15:103
2957699216,12cyclictest4474-21acpi20:25:112
2957699213,13cyclictest0-21swapper/220:15:102
2957699213,11cyclictest27346irq/40-dwc2_hso19:35:132
2957699213,11cyclictest27346irq/40-dwc2_hso19:35:132
2957699211,16cyclictest3311-21grep22:15:162
2957699211,16cyclictest3213-21latency_hist19:50:022
2957699211,16cyclictest12874-21cpu20:00:162
2957699211,16cyclictest12874-21cpu20:00:152
2957699211,10cyclictest7097-21fschecks_time19:20:162
2957699211,10cyclictest25630-21cut22:45:202
2956799219,9cyclictest21-21ksoftirqd/122:18:411
2956799219,2cyclictest21-21ksoftirqd/123:20:221
2956799219,2cyclictest21-21ksoftirqd/121:00:281
2956799214,3cyclictest131rcu_preempt21:45:141
2956799214,10cyclictest21-21ksoftirqd/120:15:101
2956799212,5cyclictest131rcu_preempt20:55:151
2956799211,9cyclictest29652-21mailstats19:40:261
2955899219,8cyclictest32159-21/usr/sbin/munin20:20:150
29584992012,6cyclictest0-21swapper/323:50:203
29584992011,6cyclictest0-21swapper/322:05:193
2958499201,11cyclictest35966irq/58-eth022:30:143
2957699206,7cyclictest14141-21df20:35:152
2957699201,9cyclictest9386-21fschecks_count20:30:172
2956799209,3cyclictest21-21ksoftirqd/123:46:381
2956799209,3cyclictest21-21ksoftirqd/123:00:011
2956799209,3cyclictest21-21ksoftirqd/100:35:541
2956799203,3cyclictest131rcu_preempt22:35:271
2956799203,2cyclictest131rcu_preempt21:40:271
29567992013,5cyclictest0-21swapper/121:25:331
29567992012,5cyclictest0-21swapper/120:40:151
29567992011,6cyclictest0-21swapper/119:50:231
2956799201,15cyclictest13003-21cut20:00:241
2956799201,15cyclictest13003-21cut20:00:241
2958499194,9cyclictest35966irq/58-eth022:25:163
2958499193,10cyclictest0-21swapper/300:15:133
2958499192,11cyclictest0-21swapper/320:55:043
29584991912,4cyclictest0-21swapper/300:10:123
29584991911,6cyclictest0-21swapper/322:48:343
29584991911,3cyclictest0-21swapper/323:45:273
2958499191,13cyclictest0-21swapper/300:25:153
2958499191,13cyclictest0-21swapper/300:25:143
2958499191,12cyclictest0-21swapper/321:25:183
2958499191,11cyclictest0-21swapper/319:35:293
2958499191,11cyclictest0-21swapper/319:35:283
2956799198,3cyclictest21-21ksoftirqd/120:35:291
2956799198,3cyclictest131rcu_preempt22:30:251
2956799196,3cyclictest131rcu_preempt21:55:331
2956799196,3cyclictest131rcu_preempt19:55:111
29567991913,3cyclictest21-21ksoftirqd/120:30:271
29567991912,4cyclictest21-21ksoftirqd/100:30:201
2956799191,14cyclictest16890-21ssh23:35:041
2956799190,16cyclictest28405-21find20:50:131
2955899196,10cyclictest29564-21cat19:10:010
2958499189,6cyclictest0-21swapper/300:01:303
2958499188,6cyclictest0-21swapper/322:19:543
2958499182,11cyclictest35966irq/58-eth000:05:173
2958499182,11cyclictest0-21swapper/319:25:153
2958499182,10cyclictest0-21swapper/320:30:153
2958499182,10cyclictest0-21swapper/320:20:153
2958499182,10cyclictest0-21swapper/319:40:133
2958499181,13cyclictest0-21swapper/320:45:173
2958499181,13cyclictest0-21swapper/320:45:173
29584991810,6cyclictest0-21swapper/323:28:483
29584991810,6cyclictest0-21swapper/321:54:253
2956799187,3cyclictest21-21ksoftirqd/120:25:221
2956799182,8cyclictest131rcu_preempt00:00:441
2956799182,11cyclictest0-21swapper/121:05:151
29567991811,4cyclictest21-21ksoftirqd/120:05:201
2958499178,7cyclictest0-21swapper/300:21:243
2958499173,7cyclictest0-21swapper/319:30:123
2958499172,9cyclictest0-21swapper/319:55:133
2958499172,10cyclictest35966irq/58-eth023:30:123
29584991711,4cyclictest0-21swapper/322:51:413
2958499171,11cyclictest0-21swapper/321:05:013
2958499171,10cyclictest0-21swapper/319:20:283
2956799171,6cyclictest11845-21fschecks_count19:25:171
29567991711,3cyclictest21-21ksoftirqd/119:30:211
2958499168,6cyclictest0-21swapper/322:00:023
2956799167,2cyclictest21-21ksoftirqd/119:10:011
2956799164,9cyclictest24048-21irqcore20:45:191
2956799164,9cyclictest24048-21irqcore20:45:181
2956799162,10cyclictest0-21swapper/120:10:141
2956799161,12cyclictest29698-21apt-get19:10:131
2958499152,10cyclictest0-21swapper/319:15:133
2958499149,3cyclictest0-21swapper/320:44:273
2958499142,8cyclictest0-21swapper/320:15:013
2958499142,8cyclictest0-21swapper/319:50:133
2957699145,6cyclictest29652-21latency_hist19:10:022
2958499138,3cyclictest0-21swapper/320:27:493
2958499138,3cyclictest0-21swapper/319:48:253
2958499135,5cyclictest0-21swapper/319:10:133
2958499122,7cyclictest0-21swapper/320:35:163
2958499122,6cyclictest0-21swapper/320:05:143
2958499121,9cyclictest0-21swapper/321:05:203
2958499121,7cyclictest0-21swapper/320:05:003
2958499121,7cyclictest0-21swapper/320:05:003
2958499111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional