You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 13:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Nov 22, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1003599477,23cyclictest29566irq/58-eth000:35:153
10027994739,6cyclictest27-21ksoftirqd/219:55:132
1002799454,22cyclictest932-21snmpd22:35:152
1002799441,33cyclictest24410-21ssh00:00:132
1002799433,34cyclictest21913-21ssh23:20:112
1001999434,34cyclictest26507-21ssh22:25:131
10035994227,11cyclictest29566irq/58-eth021:30:143
10035994122,9cyclictest0-21swapper/322:45:173
1003599411,21cyclictest29566irq/58-eth023:30:133
1001999411,31cyclictest13419-21ssh23:15:121
1000799404,32cyclictest15106-21ssh21:40:120
1003599394,21cyclictest29566irq/58-eth000:05:133
1000799394,32cyclictest30558-21ssh21:30:140
1003599382,23cyclictest29566irq/58-eth000:10:123
1002799381,32cyclictest15097-21ssh21:40:122
1001999381,30cyclictest7747-21ssh23:50:121
10027993718,5cyclictest0-21swapper/200:15:142
10019993712,15cyclictest21-21ksoftirqd/121:30:311
1000799372,26cyclictest32103-21ssh21:50:130
1003599361,18cyclictest29566irq/58-eth021:10:133
1001999351,25cyclictest21-21ksoftirqd/121:05:121
1003599342,19cyclictest29566irq/58-eth000:15:143
1000799343,20cyclictest23918-21ssh00:20:130
1003599331,17cyclictest29566irq/58-eth000:20:133
1002799335,20cyclictest26346irq/40-dwc2_hso23:05:132
10027993324,4cyclictest932-21snmpd20:30:132
10027993324,4cyclictest932-21snmpd20:30:132
1001999332,6cyclictest23244-21apt-get23:40:161
10019993322,6cyclictest530-21usb-storage20:10:111
1002799321,11cyclictest1865466ssh23:35:272
1000799325,19cyclictest16272-21ssh00:15:160
1000799322,9cyclictest12-21ksoftirqd/023:20:130
1003599311,17cyclictest29566irq/58-eth023:50:123
1001999318,5cyclictest131rcu_preempt22:15:121
1001999312,7cyclictest131rcu_preempt22:35:131
1001999312,18cyclictest4958-21ssh23:10:171
10019993117,8cyclictest0-21swapper/120:45:141
1000799317,3cyclictest131rcu_preempt20:00:140
1000799313,18cyclictest1365-21kworker/u8:222:55:130
1000799313,18cyclictest1365-21kworker/u8:222:55:130
1003599302,12cyclictest29566irq/58-eth022:55:133
1003599302,12cyclictest29566irq/58-eth022:55:133
1003599301,16cyclictest29566irq/58-eth000:00:133
1002799303,5cyclictest271ksoftirqd/222:25:022
1002799303,18cyclictest26346irq/40-dwc2_hso21:25:152
1002799302,11cyclictest1818-21grep23:25:302
1000799309,14cyclictest12-21ksoftirqd/022:35:150
1000799306,18cyclictest8182-21ssh00:10:130
1000799304,7cyclictest12-21ksoftirqd/021:20:170
1000799303,5cyclictest12-21ksoftirqd/022:40:170
1000799302,18cyclictest24400-21ssh00:00:120
1002799295,14cyclictest26346irq/40-dwc2_hso20:25:122
1002799293,9cyclictest27-21ksoftirqd/223:40:172
1002799293,15cyclictest26346irq/40-dwc2_hso20:55:142
1002799292,19cyclictest0-21swapper/220:05:152
10027992911,15cyclictest3206-21kworker/u8:121:20:142
10019992923,3cyclictest131rcu_preempt19:15:141
10019992912,6cyclictest131rcu_preempt20:00:141
1001999291,20cyclictest13616-21sh21:20:141
1001999291,18cyclictest21-21ksoftirqd/120:30:141
1001999291,18cyclictest21-21ksoftirqd/120:30:141
1000799292,5cyclictest12-21ksoftirqd/023:00:310
10007992919,8cyclictest0-21swapper/022:30:130
1003599284,16cyclictest29566irq/58-eth022:30:123
1003599284,16cyclictest0-21swapper/323:20:123
1002799285,9cyclictest27-21ksoftirqd/221:35:212
1002799285,14cyclictest26346irq/40-dwc2_hso21:30:152
1002799284,5cyclictest27-21ksoftirqd/219:25:182
1002799284,14cyclictest0-21swapper/222:49:432
1002799283,9cyclictest27-21ksoftirqd/219:45:182
1002799283,14cyclictest26346irq/40-dwc2_hso21:50:142
1002799283,10cyclictest27-21ksoftirqd/200:35:242
1002799282,8cyclictest27-21ksoftirqd/223:00:232
1002799280,12cyclictest11429-21ntp_states23:50:262
1001999288,12cyclictest25804-21diskmemload21:25:261
1000799282,19cyclictest22202-21sh21:25:140
1003599272,13cyclictest29566irq/58-eth000:25:113
10035992710,11cyclictest0-21swapper/323:15:133
1002799276,5cyclictest27-21ksoftirqd/223:15:152
1002799276,12cyclictest0-21swapper/222:30:142
1002799274,4cyclictest27-21ksoftirqd/222:10:272
1002799272,10cyclictest0-21swapper/220:15:132
1001999274,5cyclictest131rcu_preempt00:35:151
1000799277,3cyclictest131rcu_preempt23:35:150
1000799275,10cyclictest131rcu_preempt22:22:110
1000799273,6cyclictest12-21ksoftirqd/022:55:010
1000799273,6cyclictest12-21ksoftirqd/019:10:260
1000799273,5cyclictest12-21ksoftirqd/020:40:010
1000799273,12cyclictest12-21ksoftirqd/019:40:240
1000799273,12cyclictest12-21ksoftirqd/019:40:240
1000799272,18cyclictest6465-21ssh23:30:140
1000799272,11cyclictest12-21ksoftirqd/020:05:150
1000799271,7cyclictest7920-21cut21:15:230
1000799271,21cyclictest23256-21ssh23:40:160
1000799271,16cyclictest842-21munin-node20:25:130
1002799266,4cyclictest27-21ksoftirqd/221:55:222
1002799265,4cyclictest27-21ksoftirqd/222:03:062
1002799264,10cyclictest27-21ksoftirqd/219:15:312
1002799262,19cyclictest0-21swapper/222:55:142
1002799262,19cyclictest0-21swapper/222:55:142
10027992615,7cyclictest3771-21apt-get22:50:162
1002799261,11cyclictest20786-21kthreadcore21:00:202
1001999262,18cyclictest23640-21ssh21:45:131
1001999262,18cyclictest23640-21ssh21:45:131
1001999262,18cyclictest19849-21ssh00:15:261
1001999262,18cyclictest16083-21ssh23:55:121
1000799267,10cyclictest131rcu_preempt22:13:240
1000799263,4cyclictest12-21ksoftirqd/020:55:010
1000799262,4cyclictest12-21ksoftirqd/022:05:120
10007992614,4cyclictest12-21ksoftirqd/023:50:200
1003599251,15cyclictest29566irq/58-eth022:25:133
1002799253,14cyclictest27-21ksoftirqd/222:05:152
1002799252,5cyclictest27-21ksoftirqd/221:15:262
1002799251,4cyclictest27-21ksoftirqd/200:05:132
1002799251,12cyclictest29504-21missed_timers20:00:252
1002799251,11cyclictest16617-21ssh23:55:172
1002799250,11cyclictest9703-21/usr/sbin/munin20:50:112
1002799250,11cyclictest27-21ksoftirqd/221:10:302
1001999253,17cyclictest31439-21ssh00:25:111
1001999250,9cyclictest131rcu_preempt23:31:141
1000799259,3cyclictest131rcu_preempt00:05:210
1000799257,10cyclictest12-21ksoftirqd/000:25:210
1000799256,9cyclictest12-21ksoftirqd/023:56:510
1000799256,6cyclictest12-21ksoftirqd/022:19:400
1000799256,3cyclictest131rcu_preempt23:47:290
1000799256,3cyclictest131rcu_preempt20:45:170
1000799255,9cyclictest131rcu_preempt23:25:310
1000799254,6cyclictest12-21ksoftirqd/021:45:220
1000799254,6cyclictest12-21ksoftirqd/021:45:220
1000799254,4cyclictest12-21ksoftirqd/021:35:180
1000799253,7cyclictest12-21ksoftirqd/000:35:160
1000799251,5cyclictest12-21ksoftirqd/023:05:270
1000799251,5cyclictest12-21ksoftirqd/022:00:260
1000799251,4cyclictest12-21ksoftirqd/022:30:010
1002799246,5cyclictest27-21ksoftirqd/223:30:242
1002799244,5cyclictest27-21ksoftirqd/219:35:192
1002799243,7cyclictest27-21ksoftirqd/222:25:232
1002799243,13cyclictest27-21ksoftirqd/223:47:142
1002799243,13cyclictest27-21ksoftirqd/223:11:052
1002799243,13cyclictest27-21ksoftirqd/220:10:172
1002799243,13cyclictest27-21ksoftirqd/219:30:302
1002799242,4cyclictest27-21ksoftirqd/221:05:252
1002799241,8cyclictest27-21ksoftirqd/219:50:222
1002799241,6cyclictest13741-21rm00:11:362
1002799241,4cyclictest27-21ksoftirqd/221:45:032
1002799241,4cyclictest27-21ksoftirqd/221:45:022
10027992414,4cyclictest23923-21ssh00:20:132
10027992412,3cyclictest10222-21apt-get19:10:122
1002799240,16cyclictest6620-21ls00:30:152
1000799248,3cyclictest131rcu_preempt20:30:220
1000799248,3cyclictest131rcu_preempt20:30:220
1000799247,2cyclictest131rcu_preempt19:15:260
1000799245,3cyclictest131rcu_preempt20:55:220
1000799244,8cyclictest131rcu_preempt20:20:220
1000799244,6cyclictest12-21ksoftirqd/021:55:180
1000799244,5cyclictest12-21ksoftirqd/023:10:260
1000799244,2cyclictest131rcu_preempt00:30:050
1000799244,10cyclictest12-21ksoftirqd/022:45:270
1000799243,3cyclictest131rcu_preempt21:10:240
1003599233,13cyclictest0-21swapper/321:20:153
1003599232,11cyclictest0-21swapper/321:25:143
1002799236,4cyclictest27-21ksoftirqd/220:25:012
1002799235,6cyclictest27-21ksoftirqd/222:15:202
1002799231,4cyclictest27-21ksoftirqd/222:40:182
1002799231,15cyclictest27-21ksoftirqd/200:27:352
1002799230,4cyclictest91-21pr/ttyS020:45:022
1001999238,3cyclictest0-21swapper/122:45:461
1001999232,13cyclictest28908-21ssh23:05:111
10019992313,5cyclictest131rcu_preempt19:30:141
10019992311,3cyclictest0-21swapper/122:11:361
1000799237,3cyclictest131rcu_preempt19:45:190
1000799236,3cyclictest131rcu_preempt20:10:250
1000799236,3cyclictest131rcu_preempt19:55:130
1000799235,2cyclictest131rcu_preempt19:30:200
1000799234,3cyclictest131rcu_preempt20:40:100
1000799233,7cyclictest12-21ksoftirqd/021:00:250
1000799233,5cyclictest12-21ksoftirqd/019:50:220
1000799232,6cyclictest12-21ksoftirqd/023:15:150
1000799231,8cyclictest12-21ksoftirqd/021:05:150
1000799231,4cyclictest131rcu_preempt19:20:150
1003599225,6cyclictest0-21swapper/320:50:113
1003599224,10cyclictest0-21swapper/322:05:133
1002799221,16cyclictest29505-21grep20:35:232
1001999229,4cyclictest21-21ksoftirqd/122:32:591
1001999229,10cyclictest21-21ksoftirqd/122:55:221
1001999229,10cyclictest21-21ksoftirqd/122:55:221
1001999222,14cyclictest18622-21ssh23:35:261
1001999222,10cyclictest30366-21ssh23:25:141
1001999221,6cyclictest131rcu_preempt20:05:131
1001999221,19cyclictest19593-21ls21:00:171
1001999221,18cyclictest4792-21ls20:45:021
10019992211,8cyclictest25590-21awk23:20:261
1001999221,14cyclictest6663-21ssh21:35:161
10019992211,3cyclictest0-21swapper/119:10:211
10019992210,8cyclictest3375-21unixbench_multi23:45:291
1000799225,5cyclictest12-21ksoftirqd/019:35:140
1000799225,3cyclictest131rcu_preempt20:15:190
1000799225,3cyclictest131rcu_preempt19:25:310
1003599212,13cyclictest0-21swapper/323:10:323
1003599212,13cyclictest0-21swapper/322:40:263
1002799217,11cyclictest10681-21unixbench_multi19:40:322
1002799217,11cyclictest10681-21unixbench_multi19:40:312
1002799213,4cyclictest27-21ksoftirqd/219:20:202
1001999219,4cyclictest0-21swapper/100:32:581
1001999215,6cyclictest0-21swapper/121:43:541
1001999212,16cyclictest24242-21ssh23:00:281
1001999211,9cyclictest131rcu_preempt22:52:521
10019992113,4cyclictest0-21swapper/122:00:151
1001999211,13cyclictest131rcu_preempt22:43:401
1001999211,12cyclictest131rcu_preempt21:53:121
1001999211,12cyclictest131rcu_preempt21:10:231
1001999210,2cyclictest131rcu_preempt21:57:211
1003599203,11cyclictest0-21swapper/321:40:123
1002799200,5cyclictest27-21ksoftirqd/220:45:172
1001999208,10cyclictest0-21swapper/122:07:071
1001999202,2cyclictest131rcu_preempt00:20:241
1001999201,8cyclictest131rcu_preempt22:24:291
10019992011,7cyclictest7829-21latency_hist00:10:021
1001999200,2cyclictest131rcu_preempt21:15:191
1001999200,11cyclictest131rcu_preempt00:10:041
1000799202,4cyclictest12-21ksoftirqd/019:10:020
1003599199,3cyclictest0-21swapper/323:43:383
1003599199,3cyclictest0-21swapper/322:22:293
1003599194,10cyclictest0-21swapper/321:45:133
1003599194,10cyclictest0-21swapper/321:45:123
1003599192,11cyclictest0-21swapper/320:25:133
10035991912,4cyclictest0-21swapper/323:45:143
10035991911,6cyclictest0-21swapper/322:15:423
1003599191,13cyclictest0-21swapper/321:35:253
1003599191,12cyclictest0-21swapper/320:15:233
1003599191,12cyclictest0-21swapper/319:25:253
1003599191,11cyclictest0-21swapper/319:15:023
10035991910,6cyclictest0-21swapper/322:53:553
1002799191,4cyclictest27-21ksoftirqd/219:10:022
1001999192,8cyclictest131rcu_preempt00:00:251
1001999191,9cyclictest20095-21switchtime19:50:281
1001999191,9cyclictest131rcu_preempt20:20:241
1001999191,3cyclictest131rcu_preempt20:15:171
1001999191,2cyclictest131rcu_preempt20:50:241
10019991911,5cyclictest0-21swapper/120:35:291
1003599189,6cyclictest0-21swapper/322:10:073
1003599189,6cyclictest0-21swapper/321:57:133
1003599188,6cyclictest0-21swapper/321:52:003
1003599187,7cyclictest29566irq/58-eth021:15:143
1003599186,8cyclictest29566irq/58-eth023:35:133
1003599184,11cyclictest0-21swapper/322:35:193
1003599182,4cyclictest0-21swapper/319:40:163
1003599182,4cyclictest0-21swapper/319:40:153
1003599182,11cyclictest0-21swapper/320:05:153
1003599182,10cyclictest0-21swapper/320:40:123
1003599182,10cyclictest0-21swapper/320:30:053
1003599182,10cyclictest0-21swapper/320:30:043
1003599182,10cyclictest0-21swapper/319:50:143
1003599182,10cyclictest0-21swapper/319:35:123
1003599182,10cyclictest0-21swapper/319:15:043
1003599181,13cyclictest0-21swapper/323:30:013
1003599181,10cyclictest0-21swapper/319:45:143
1003599181,10cyclictest0-21swapper/319:30:143
10035991810,6cyclictest0-21swapper/323:05:243
1001999187,3cyclictest131rcu_preempt20:25:261
1001999184,5cyclictest9478-21cut19:40:251
1001999184,5cyclictest9478-21cut19:40:251
1001999181,2cyclictest131rcu_preempt20:55:261
10019991812,4cyclictest0-21swapper/119:45:181
1003599179,6cyclictest0-21swapper/322:02:333
1003599179,3cyclictest0-21swapper/323:55:483
1003599177,6cyclictest29566irq/58-eth023:00:153
1003599172,9cyclictest0-21swapper/319:20:153
1003599172,10cyclictest0-21swapper/320:35:153
1003599172,10cyclictest0-21swapper/320:20:163
1003599172,10cyclictest0-21swapper/300:30:143
1001999171,7cyclictest5762-21timerandwakeup19:35:271
1001999171,2cyclictest131rcu_preempt19:55:121
1003599162,9cyclictest0-21swapper/321:05:143
1003599161,12cyclictest0-21swapper/320:00:203
1001999163,9cyclictest10059-21latency_hist19:10:021
1001999163,10cyclictest28375-21grep19:25:281
1001999160,5cyclictest19981-21cpu19:20:151
1003599152,9cyclictest0-21swapper/320:10:123
1003599152,10cyclictest0-21swapper/320:55:143
1003599151,10cyclictest0-21swapper/320:45:193
1003599141,6cyclictest0-21swapper/321:00:223
1003599122,6cyclictest0-21swapper/319:55:133
1003599101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional