You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 04:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue May 19, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2838899733,56cyclictest9261-21ssh21:25:130
2838899685,45cyclictest3612-21ssh23:10:130
28417995933,23cyclictest35966irq/58-eth000:35:143
28388995710,8cyclictest131rcu_preempt22:45:130
2839899541,43cyclictest5675-21sh22:35:131
2839899541,43cyclictest5675-21sh22:35:121
28388995431,10cyclictest12-21ksoftirqd/023:20:140
2839899533,39cyclictest23618-21ssh21:15:151
28398995241,8cyclictest0-21swapper/123:15:151
2840699494,9cyclictest131rcu_preempt22:20:122
2838899494,35cyclictest23066-21ssh00:15:120
2838899481,35cyclictest12963-21sh21:45:160
2838899475,39cyclictest18145-21ssh23:00:140
2841799462,29cyclictest35966irq/58-eth000:15:123
2839899463,37cyclictest6994-21ssh23:30:131
2839899463,37cyclictest6994-21ssh23:30:121
2839899451,39cyclictest11208-21ssh22:20:121
28417994321,13cyclictest35966irq/58-eth023:25:123
2841799424,24cyclictest35966irq/58-eth022:10:123
2840699422,15cyclictest16568-21sh22:05:122
2839899423,30cyclictest20143-21ssh22:25:121
2839899422,30cyclictest24916-21ssh23:40:141
2839899422,30cyclictest24916-21ssh23:40:131
2838899422,5cyclictest131rcu_preempt19:20:120
2841799413,21cyclictest35966irq/58-eth023:15:153
2841799402,19cyclictest35966irq/58-eth021:45:163
28406993919,17cyclictest0-21swapper/200:25:092
28398993911,19cyclictest21-21ksoftirqd/119:55:131
28417993819,8cyclictest35966irq/58-eth022:00:143
2841799381,30cyclictest0-21swapper/323:35:123
28398993817,17cyclictest21-21ksoftirqd/122:50:131
2838899372,26cyclictest32559-21ssh21:20:120
2841799361,21cyclictest35966irq/58-eth022:05:133
2841799355,23cyclictest35966irq/58-eth019:10:113
2841799355,23cyclictest35966irq/58-eth019:10:113
2840699354,19cyclictest6999-21ssh23:30:132
2840699354,19cyclictest6999-21ssh23:30:122
2838899352,23cyclictest26406-21ssh00:35:150
28388993513,9cyclictest12-21ksoftirqd/019:40:150
2841799341,29cyclictest0-21swapper/321:20:113
28388993422,7cyclictest12-21ksoftirqd/021:15:150
2841799332,21cyclictest35966irq/58-eth019:45:153
2841799331,29cyclictest0-21swapper/300:05:133
2841799322,24cyclictest0-21swapper/322:55:143
2841799322,21cyclictest35966irq/58-eth020:00:113
2841799321,20cyclictest0-21swapper/300:30:113
2841799321,20cyclictest0-21swapper/300:30:113
2840699322,22cyclictest28303-21ssh00:00:042
2840699322,22cyclictest28303-21ssh00:00:042
2839899322,6cyclictest17507-21apt-get00:30:111
2839899322,6cyclictest17507-21apt-get00:30:111
2838899323,4cyclictest131rcu_preempt22:40:110
2841799312,19cyclictest35966irq/58-eth023:45:163
2841799311,23cyclictest0-21swapper/320:40:123
2838899316,14cyclictest21879-21ssh21:50:110
2841799301,26cyclictest0-21swapper/322:45:153
2839899304,20cyclictest29136-21ssh22:30:151
2838899302,21cyclictest19388-21ssh23:55:110
28388993010,9cyclictest14092-21ssh00:10:130
2841799292,16cyclictest35966irq/58-eth023:10:133
2841799291,24cyclictest0-21swapper/322:40:103
2841799291,22cyclictest0-21swapper/323:30:143
2841799291,22cyclictest0-21swapper/323:30:133
2841799284,21cyclictest35966irq/58-eth023:55:093
2841799281,24cyclictest0-21swapper/321:50:103
2840699282,19cyclictest32582-21ssh22:50:112
2841799271,21cyclictest0-21swapper/321:55:123
2841799271,21cyclictest0-21swapper/321:55:113
2841799271,21cyclictest0-21swapper/320:35:143
28417992710,9cyclictest0-21swapper/323:05:113
2841799270,24cyclictest0-21swapper/322:35:133
2841799270,24cyclictest0-21swapper/322:35:123
2840699272,22cyclictest30219-21latency_hist23:25:012
2840699272,17cyclictest27346irq/40-dwc2_hso22:35:152
2840699272,17cyclictest27346irq/40-dwc2_hso22:35:142
2841799263,19cyclictest0-21swapper/300:25:093
2841799262,16cyclictest35966irq/58-eth020:55:143
2841799261,22cyclictest0-21swapper/321:30:123
2841799261,21cyclictest0-21swapper/323:20:263
2841799261,21cyclictest0-21swapper/321:00:143
2841799261,19cyclictest0-21swapper/319:20:113
2840699264,13cyclictest30494-21ssh23:25:132
2840699262,8cyclictest1508-21ssh23:45:162
2840699262,16cyclictest19598-21kthreadcore00:30:212
2840699262,16cyclictest19598-21kthreadcore00:30:212
2840699260,16cyclictest2138-21ssh21:55:252
2840699260,16cyclictest2138-21ssh21:55:242
2838899268,8cyclictest12-21ksoftirqd/021:55:120
2838899268,8cyclictest12-21ksoftirqd/021:55:110
28388992613,7cyclictest536-21usb-storage00:25:100
2841799253,17cyclictest0-21swapper/323:40:123
2841799253,17cyclictest0-21swapper/323:40:113
2841799252,17cyclictest35966irq/58-eth021:35:133
28417992516,6cyclictest0-21swapper/323:50:143
2841799251,21cyclictest0-21swapper/319:30:123
2841799251,21cyclictest0-21swapper/300:10:153
2841799251,20cyclictest0-21swapper/321:15:173
2841799251,18cyclictest0-21swapper/319:35:153
2841799251,17cyclictest0-21swapper/320:20:113
2841799251,17cyclictest0-21swapper/320:20:103
2841799251,16cyclictest0-21swapper/319:50:133
2841799250,20cyclictest0-21swapper/321:40:103
2840699252,20cyclictest5113-21kthreadcore21:40:182
2840699251,21cyclictest913-21kthreadcore20:55:212
2840699251,20cyclictest15353-21apt-get19:30:132
2840699251,17cyclictest25601-21kthreadcore20:45:222
2840699251,16cyclictest17049-21grep21:45:332
2840699251,15cyclictest29044-21grep23:40:292
2840699251,15cyclictest29044-21grep23:40:282
2839899254,12cyclictest0-21swapper/119:50:121
2841799243,16cyclictest0-21swapper/322:50:113
2841799242,19cyclictest0-21swapper/300:20:153
2841799241,19cyclictest0-21swapper/321:11:583
2841799241,19cyclictest0-21swapper/319:40:143
2840699242,19cyclictest15147-21irqcore22:40:182
2840699241,20cyclictest25643-21tr00:15:222
2840699241,20cyclictest11071-21kthreadcore20:30:222
2840699241,16cyclictest27346irq/40-dwc2_hso19:25:122
2840699241,10cyclictest346966ssh21:40:022
2838899245,13cyclictest10510-21ssh23:50:140
2838899242,16cyclictest9125-21ssh22:55:150
2838899242,15cyclictest14130-21diskmemload21:40:050
28388992412,3cyclictest14130-21diskmemload23:28:100
2841799232,18cyclictest0-21swapper/321:25:113
2841799232,17cyclictest0-21swapper/323:00:143
2841799231,19cyclictest0-21swapper/322:15:233
2841799231,19cyclictest0-21swapper/320:50:123
2841799231,18cyclictest0-21swapper/322:30:123
2841799231,18cyclictest0-21swapper/322:20:113
2841799231,17cyclictest0-21swapper/320:15:113
2841799230,19cyclictest0-21swapper/320:30:113
2841799230,19cyclictest0-21swapper/300:00:143
2841799230,19cyclictest0-21swapper/300:00:133
2841799230,18cyclictest0-21swapper/320:25:123
2841799230,17cyclictest0-21swapper/319:25:123
2840699239,11cyclictest14130-21diskmemload21:30:242
2840699232,18cyclictest5579-21ssh00:05:202
2840699231,20cyclictest25298-21sh22:45:212
2840699231,19cyclictest28807-21tail19:40:312
2840699231,14cyclictest0-21swapper/220:25:112
2840699231,13cyclictest28515-21kthreadcore22:10:222
2840699231,13cyclictest1211566ssh23:50:212
2839899233,17cyclictest4971-21idleruntime21:00:191
2839899232,9cyclictest131rcu_preempt00:29:211
28398992315,4cyclictest21-21ksoftirqd/123:10:141
28398992314,6cyclictest21-21ksoftirqd/121:25:111
28398992310,5cyclictest0-21swapper/122:55:121
2838899233,16cyclictest15923-21ssh23:35:130
2838899231,15cyclictest14534-21ssh21:10:130
2841799221,18cyclictest0-21swapper/322:25:123
2840699229,4cyclictest27-21ksoftirqd/219:55:022
2840699228,5cyclictest30922-21latency_hist20:20:022
2840699227,6cyclictest27-21ksoftirqd/219:15:252
2840699225,8cyclictest131rcu_preempt21:27:402
2840699223,7cyclictest131rcu_preempt23:15:252
28406992212,8cyclictest27-21ksoftirqd/221:52:572
2840699221,18cyclictest17291-21cut21:10:212
2840699220,20cyclictest22681-21latency_hist00:15:012
2839899223,16cyclictest32208-21cron22:50:011
2839899223,16cyclictest27561-21kthreadcore19:40:231
2838899227,5cyclictest13324-21wc20:35:180
2838899225,4cyclictest131rcu_preempt22:05:190
2838899222,14cyclictest28466-21ssh00:00:140
2838899222,14cyclictest28466-21ssh00:00:140
2838899221,6cyclictest5647-21/usr/sbin/munin22:35:260
2838899221,6cyclictest5647-21/usr/sbin/munin22:35:250
2841799210,18cyclictest0-21swapper/319:15:133
2840699219,8cyclictest20108-21/usr/sbin/munin22:25:112
2840699216,12cyclictest9380-21hwlatdetect22:55:182
2840699213,15cyclictest20015-21kthreadcore23:00:222
2840699213,10cyclictest3273-21grep20:25:012
2840699213,10cyclictest3273-21grep20:25:002
2840699212,15cyclictest8293-21latency_hist00:25:012
2840699212,12cyclictest8585-21tail21:00:302
2840699211,7cyclictest32394-21kthreadcore22:30:222
2840699211,7cyclictest131rcu_preempt23:14:062
2840699211,6cyclictest131rcu_preempt00:36:552
2840699211,16cyclictest13301-21fschecks_time20:35:172
28406992111,6cyclictest0-21swapper/222:00:212
2840699211,13cyclictest18573-21tr23:35:202
2840699211,12cyclictest20123-21tail23:55:172
2840699211,11cyclictest27366-21processes21:15:302
2840699211,11cyclictest21488-21latency_hist20:10:032
2840699211,10cyclictest29962-21/usr/sbin/munin19:45:162
2840699211,10cyclictest2302-21sh22:15:152
28406992110,3cyclictest27-21ksoftirqd/221:20:502
2839899213,16cyclictest13976-21ntp_kernel_pll_19:25:241
2839899213,15cyclictest24659-21latency_hist23:40:021
2839899213,15cyclictest21876-21ssh23:55:181
2839899213,15cyclictest19432-21ssh21:30:191
2839899213,15cyclictest18637-21cat19:30:221
2838899219,3cyclictest131rcu_preempt23:45:250
2838899213,3cyclictest131rcu_preempt22:00:140
2838899213,13cyclictest27078-21ssh23:05:130
2838899211,17cyclictest1108-21latency_hist23:45:030
2838899211,17cyclictest1108-21latency_hist23:45:020
2838899211,15cyclictest2217-21ssh22:15:160
2838899210,18cyclictest1903-21ls22:15:010
2841799201,16cyclictest0-21swapper/321:05:133
2841799201,16cyclictest0-21swapper/320:05:183
2841799201,16cyclictest0-21swapper/319:55:143
2841799200,17cyclictest0-21swapper/320:45:133
2841799200,17cyclictest0-21swapper/320:10:153
2840699202,15cyclictest867-21latency_hist19:15:022
2840699202,15cyclictest867-21latency_hist19:15:012
2840699201,9cyclictest24866-21latency_hist19:40:012
2840699201,9cyclictest21696-21fschecks_time20:10:162
2840699201,9cyclictest10046-21sed21:05:192
2840699201,7cyclictest131rcu_preempt23:06:182
2840699201,10cyclictest18739-21kthreadcore20:40:192
2839899208,3cyclictest131rcu_preempt22:15:181
2839899208,2cyclictest131rcu_preempt22:02:241
2839899203,14cyclictest31957-21sh23:25:191
2839899203,14cyclictest13130-21kthreadcore20:00:181
2839899202,4cyclictest131rcu_preempt00:10:061
2839899201,16cyclictest0-21swapper/122:05:131
28398992011,6cyclictest0-21swapper/120:10:251
2839899201,14cyclictest18391-21ssh22:40:271
2839899201,12cyclictest5376-21kthreadcore21:40:191
2838899203,14cyclictest151rcuc/000:20:150
2838899202,15cyclictest7203-21grep19:55:170
2838899201,7cyclictest11335-21diskstats22:20:140
2838899201,16cyclictest26091-21latency_hist20:15:010
2838899201,15cyclictest2254-21apt-get19:50:120
2838899200,14cyclictest20015-21ls20:05:220
2840699193,9cyclictest13330-21kthreadcore20:00:182
2840699192,14cyclictest27346irq/40-dwc2_hso19:55:132
2840699191,9cyclictest5888-21fschecks_time19:20:162
2840699191,15cyclictest27539-21/usr/sbin/munin20:50:172
2839899197,2cyclictest131rcu_preempt21:36:251
2839899194,3cyclictest131rcu_preempt19:35:161
2839899193,14cyclictest22626-21kthreadcore21:50:201
2839899193,13cyclictest9782-21idleruntime21:05:171
2839899193,13cyclictest28716-21fw_packets00:00:181
2839899193,13cyclictest28716-21fw_packets00:00:171
2839899193,13cyclictest13956-21kthreadcore20:35:191
2839899191,9cyclictest131rcu_preempt22:10:101
2839899191,3cyclictest131rcu_preempt23:50:171
28398991913,4cyclictest0-21swapper/123:05:111
28398991911,6cyclictest6201-21sshd00:22:421
2839899191,15cyclictest3776-21ssh21:20:251
2838899199,7cyclictest22637-21cat21:30:290
2838899197,3cyclictest131rcu_preempt00:05:190
2838899195,4cyclictest131rcu_preempt21:40:030
2838899192,9cyclictest0-21swapper/023:15:390
28388991912,4cyclictest0-21swapper/022:30:280
2838899191,15cyclictest1978-21kthreadcore20:55:220
2840699181,8cyclictest28415-21latency_hist19:10:022
2839899184,11cyclictest23284-21fschecks_time00:15:171
2839899183,13cyclictest531-21sort20:20:181
2839899183,13cyclictest531-21sort20:20:171
2839899183,13cyclictest20669-21kthreadcore23:00:231
2839899182,13cyclictest17719-21ssh21:10:231
2839899182,13cyclictest1015-21apt-get23:45:041
2839899182,12cyclictest21680-21diskstats23:20:161
28398991812,4cyclictest0-21swapper/119:15:201
2839899180,16cyclictest1152-21timerandwakeup19:45:301
2838899186,3cyclictest9868-21ps21:05:190
2838899182,14cyclictest14221-21kthreadcore20:00:200
2838899182,11cyclictest12-21ksoftirqd/020:45:150
2838899181,9cyclictest0-21swapper/000:30:040
2838899181,9cyclictest0-21swapper/000:30:040
2838899181,14cyclictest30062-21tr20:50:210
2838899181,14cyclictest21166-21kthreadcore20:40:220
28388991811,4cyclictest14130-21diskmemload23:30:140
28388991811,4cyclictest14130-21diskmemload23:30:130
2838899181,13cyclictest29772-21ntp_states20:15:250
2838899181,13cyclictest20255-21diskstats22:25:150
2838899181,11cyclictest29858-21find19:45:120
2838899180,7cyclictest3742-21nfs_client22:50:230
2839899179,5cyclictest0-21swapper/120:25:111
2839899176,9cyclictest21-21ksoftirqd/119:10:121
2839899176,9cyclictest21-21ksoftirqd/119:10:111
2839899173,11cyclictest9027-21kthreadcore20:30:191
2839899173,11cyclictest27129-21kthreadcore00:35:201
2839899173,10cyclictest29784-21kthreadcore20:50:211
2839899172,11cyclictest5044-21ssh00:05:131
2839899172,11cyclictest0-21swapper/120:05:041
2839899171,13cyclictest1217-21tr21:55:221
2839899171,13cyclictest1217-21tr21:55:211
2839899171,12cyclictest14504-21tr21:45:221
2839899170,8cyclictest26433-21fschecks_count20:15:141
2838899179,5cyclictest685-21in:imuxsock19:40:000
2838899178,6cyclictest1055-21grep19:15:170
2838899176,5cyclictest6500-21cut20:25:210
2838899172,10cyclictest0-21swapper/020:20:100
2838899172,10cyclictest0-21swapper/020:20:100
2838899171,7cyclictest8417-21threads21:00:290
2838899171,5cyclictest19938-21munin-run19:35:000
2839899163,11cyclictest32421-21sort20:55:141
2839899162,12cyclictest9139-21/usr/sbin/munin19:20:221
2839899162,11cyclictest20056-21kthreadcore20:40:221
2839899161,12cyclictest23872-21tr20:45:191
2838899167,6cyclictest8444-21grep20:30:150
2838899167,2cyclictest131rcu_preempt19:10:180
2838899167,2cyclictest131rcu_preempt19:10:170
2838899152,10cyclictest10763-21grep19:25:150
2841799141,11cyclictest0-21swapper/319:10:023
2839899141,10cyclictest28416-21latency_hist19:10:011
2838899141,10cyclictest28519-21latency_hist19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional