You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 05:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 01, 2026 00:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1737099753,7cyclictest0-21swapper/220:35:132
1737099694,12cyclictest0-21swapper/221:00:152
1736099613,45cyclictest21789-21ssh21:20:131
17370995216,30cyclictest0-21swapper/223:50:112
1737099502,26cyclictest28597-21ssh22:35:132
1734699472,37cyclictest15325-21ssh00:35:120
1737099452,31cyclictest29600-21ssh22:00:132
1737099452,31cyclictest29600-21ssh22:00:132
17360994510,26cyclictest21-21ksoftirqd/119:35:141
1734699444,36cyclictest25716-21ssh21:40:120
1734699441,32cyclictest2842-21ssh23:50:120
1736099432,34cyclictest19456-21ssh22:30:121
1736099432,34cyclictest19456-21ssh22:30:121
1737099407,7cyclictest2547-21sed21:25:302
1737099402,32cyclictest12631-21ssh00:32:172
1736099402,28cyclictest29604-21ssh22:00:131
1736099402,28cyclictest29604-21ssh22:00:131
17360994018,13cyclictest17337-21cyclictest20:00:121
1734699402,32cyclictest11990-21ssh23:55:130
17346994015,21cyclictest12-21ksoftirqd/019:10:140
17346994015,21cyclictest12-21ksoftirqd/019:10:140
1734699401,28cyclictest5313-21ssh22:40:110
1737099391,8cyclictest21214-21grep19:10:292
1737099391,8cyclictest21214-21grep19:10:282
1737099383,21cyclictest924-21snmpd23:05:132
1737099381,9cyclictest14931-21missed_timers20:45:232
1737099381,6cyclictest3155-21diskmemload23:00:022
1737099381,10cyclictest14689-21cut21:15:202
1736099382,30cyclictest14473-21ssh22:45:131
1738199375,20cyclictest28366irq/58-eth022:45:143
17381993713,13cyclictest28366irq/58-eth020:25:143
1737099371,13cyclictest21918-21sh21:20:162
1737099371,13cyclictest13888-21irqrtprio23:20:192
1737099371,11cyclictest11710-21grep21:30:292
1738199362,18cyclictest28366irq/58-eth021:20:143
1738199361,22cyclictest28366irq/58-eth022:50:123
1737099362,8cyclictest11644-21uniq21:50:162
1737099362,28cyclictest23878-21tr00:00:222
1737099362,11cyclictest20539-21ssh21:55:142
1737099362,11cyclictest20539-21ssh21:55:142
1737099361,9cyclictest4724-21grep20:00:282
1737099361,8cyclictest7438-21/usr/sbin/munin21:10:292
1737099361,7cyclictest24348-21/usr/sbin/munin22:15:252
1737099361,13cyclictest21540-21kthreadcore19:45:202
1736099363,27cyclictest3586-21ssh21:10:141
17346993610,7cyclictest131rcu_preempt20:45:120
1738199351,24cyclictest28366irq/58-eth021:15:133
1737099358,5cyclictest20947-21proc_pri23:40:292
1737099351,13cyclictest14962-21ssh22:10:012
1737099350,8cyclictest16802-21cut19:40:212
1737099350,8cyclictest16802-21cut19:40:212
1737099350,8cyclictest15056-21sort20:15:012
1734699352,23cyclictest6281-21ssh00:30:130
1734699351,26cyclictest9369-21ssh23:00:140
1737099341,9cyclictest10092-21kthreadcore23:35:202
1737099341,7cyclictest6323-21switchtime21:45:292
1737099341,15cyclictest32364-21kthreadcore00:25:222
1737099341,12cyclictest20651-21fschecks_count00:20:152
1737099341,10cyclictest6762-21kthreadcore20:05:202
1737099341,10cyclictest6762-21kthreadcore20:05:202
1737099340,13cyclictest26994-21latency_hist19:20:002
1736099342,24cyclictest17248-21ssh23:40:161
1734699344,22cyclictest28582-21ssh22:35:120
1737099336,7cyclictest20071-21apt-get20:20:132
1737099331,9cyclictest15320-21kthreadcore22:45:192
1737099331,5cyclictest8206-21tail19:30:292
1737099331,5cyclictest1939-21ssh22:20:192
1737099331,12cyclictest6457-21kthreadcore22:40:182
1736099332,22cyclictest22377-21ssh23:25:141
1734699334,20cyclictest12-21ksoftirqd/019:25:130
1737099326,9cyclictest16290-21cat23:55:292
1737099323,19cyclictest19466-21ssh22:30:132
1737099323,19cyclictest19466-21ssh22:30:132
17360993210,16cyclictest23585-21ssh22:50:131
17346993218,8cyclictest0-21swapper/021:45:140
17346993217,6cyclictest131rcu_preempt21:30:130
1737099311,14cyclictest18337-21kthreadcore21:35:202
1737099311,12cyclictest4253-21sh23:15:142
1737099310,8cyclictest12589-21ntpq19:35:252
1736099311,21cyclictest32695-21ssh22:55:151
1734699313,22cyclictest26260-21ssh23:45:130
1737099305,6cyclictest23555-21acpi22:50:122
1734699306,17cyclictest12-21ksoftirqd/020:35:130
1734699302,5cyclictest131rcu_preempt22:50:130
1738199291,15cyclictest28366irq/58-eth022:55:153
1737099291,8cyclictest3621-21usage_Anybus19:25:292
1737099291,10cyclictest12774-21kthreadcore23:00:242
1734699294,14cyclictest12-21ksoftirqd/000:25:150
1734699292,22cyclictest23875-21kthreadcore00:00:220
1734699292,21cyclictest21414-21ssh23:24:220
1734699292,19cyclictest12-21ksoftirqd/021:00:150
17346992910,4cyclictest131rcu_preempt22:05:140
17346992910,12cyclictest9635-1kworker/0:2H23:15:120
1738199281,23cyclictest28366irq/58-eth021:40:113
1737099285,13cyclictest22346irq/40-dwc2_hso20:55:132
1737099282,11cyclictest0-21swapper/200:36:282
1736099282,24cyclictest21110-21ssh00:00:131
17360992814,8cyclictest666-21ntpd23:20:141
17381992710,11cyclictest0-21swapper/300:32:173
1737099271,6cyclictest1969166ssh22:10:312
1736099273,16cyclictest8141-21ssh23:35:121
1736099271,19cyclictest30866-21ssh21:25:141
1734699271,19cyclictest2192-21ssh00:10:100
1738199263,18cyclictest0-21swapper/323:25:223
1738199262,15cyclictest28366irq/58-eth022:30:113
1738199262,15cyclictest28366irq/58-eth022:30:113
1736099262,16cyclictest10378-21ssh22:25:131
17360992612,9cyclictest0-21swapper/123:15:121
1734699263,3cyclictest131rcu_preempt20:30:120
1734699262,18cyclictest32690-21ssh22:55:150
1734699262,17cyclictest27533-21ssh23:10:130
1734699261,18cyclictest31498-21ssh23:30:130
17381992511,10cyclictest28366irq/58-eth000:39:283
1737099252,18cyclictest1905-21ssh23:49:222
1736099252,19cyclictest21-21ksoftirqd/100:20:121
1736099252,17cyclictest20555-21ssh21:55:151
1736099252,17cyclictest20555-21ssh21:55:151
17360992511,8cyclictest21-21ksoftirqd/120:50:121
17346992513,9cyclictest16788-21cron23:40:010
1738199244,17cyclictest0-21swapper/300:20:123
17381992413,5cyclictest0-21swapper/322:10:133
1737099245,13cyclictest0-21swapper/200:15:142
17370992413,8cyclictest0-21swapper/220:15:182
1736099247,4cyclictest131rcu_preempt00:10:171
1736099243,6cyclictest0-21swapper/100:38:281
1734699249,8cyclictest633-21Async12:11:530
1734699242,17cyclictest16662-21ssh21:35:130
1734699242,17cyclictest15374-21ssh22:10:140
1734699242,14cyclictest11304-21ssh00:15:140
17346992414,7cyclictest3155-21diskmemload22:15:120
17381992313,7cyclictest0-21swapper/300:15:193
1737099231,5cyclictest29566-21ls20:30:012
17370992311,9cyclictest28446-21ssh23:27:012
17360992313,7cyclictest18422-21apt-get23:05:131
17360992310,10cyclictest7692-21df_inode21:30:151
1734699231,18cyclictest15442-21cut20:15:160
1734699231,15cyclictest20062-21find20:20:120
1738199225,7cyclictest0-21swapper/320:55:133
1738199225,11cyclictest0-21swapper/323:57:113
1738199225,10cyclictest0-21swapper/323:20:133
1738199223,15cyclictest28366irq/58-eth022:25:133
1738199222,12cyclictest28366irq/58-eth021:50:133
1737099221,8cyclictest3207-21unixbench_singl23:30:292
1737099220,13cyclictest2139-21/usr/sbin/munin00:10:202
1736099229,9cyclictest12687-21sh21:15:141
1736099221,16cyclictest3155-21diskmemload22:20:141
1736099220,15cyclictest15391-21cut20:15:151
1734699222,15cyclictest30865-21ssh21:25:140
1734699221,16cyclictest18115-21ssh22:45:250
1738199219,6cyclictest28366irq/58-eth023:50:123
17381992112,6cyclictest28366irq/58-eth000:25:143
1737099212,9cyclictest10582-21fschecks_time22:25:172
1737099211,16cyclictest6932-21grep20:40:142
1737099211,16cyclictest3754-21latency_hist23:15:012
1737099211,12cyclictest27694-21kthreadcore21:40:202
1736099218,8cyclictest21-21ksoftirqd/100:32:171
1736099218,3cyclictest131rcu_preempt21:50:201
17360992113,5cyclictest28547-21/usr/sbin/munin22:35:191
1736099211,17cyclictest2349-21tr20:35:171
1736099211,16cyclictest32009-21apt-get19:25:131
1736099211,15cyclictest839-21munin-node22:05:141
1736099211,10cyclictest9458-21unixbench_multi20:05:301
1736099211,10cyclictest9458-21unixbench_multi20:05:301
1736099210,18cyclictest9536-21cut23:00:171
1734699213,4cyclictest131rcu_preempt22:25:150
1734699211,4cyclictest131rcu_preempt21:55:250
1734699211,4cyclictest131rcu_preempt21:55:250
1734699211,15cyclictest1178-21cut20:00:170
1734699211,14cyclictest5656-21find20:05:130
1734699211,14cyclictest5656-21find20:05:130
1734699210,3cyclictest131rcu_preempt00:05:400
1738199209,8cyclictest0-21swapper/322:24:593
1738199209,5cyclictest0-21swapper/323:10:203
1738199203,11cyclictest0-21swapper/321:55:143
1738199203,11cyclictest0-21swapper/321:55:143
1738199202,12cyclictest0-21swapper/300:01:283
17381992014,4cyclictest0-21swapper/321:25:133
17381992013,5cyclictest28466irq/59-eth023:35:133
17381992011,7cyclictest0-21swapper/321:10:173
1738199201,12cyclictest0-21swapper/320:15:013
1737099202,9cyclictest30689-21kthreadcore19:55:232
1737099202,9cyclictest20018-21wc20:50:262
1737099201,16cyclictest29763-21fschecks_count20:30:152
1737099201,15cyclictest25285-21kthreadcore19:50:182
1737099201,10cyclictest31031-21unixbench_multi19:20:302
1736099209,8cyclictest28214-21basename22:15:251
1736099209,6cyclictest0-21swapper/100:07:121
1736099205,11cyclictest616-21dbus-daemon21:46:431
1736099201,9cyclictest31649-21tune2fs23:30:151
1736099201,9cyclictest29472-21grep21:40:261
17360992011,5cyclictest0-21swapper/120:50:021
17360992010,7cyclictest21643-21grep20:55:181
1736099200,17cyclictest1073-21ls00:25:301
1736099200,15cyclictest2454-21latency_hist23:50:021
1734699205,5cyclictest131rcu_preempt21:20:190
1734699201,5cyclictest0-21swapper/000:23:170
1734699201,4cyclictest0-21swapper/022:23:480
17346992011,6cyclictest0-21swapper/020:50:170
1734699201,14cyclictest11407-21latency_hist20:45:020
1734699200,17cyclictest22919-21latency22:30:210
1734699200,17cyclictest22919-21latency22:30:210
1738199198,8cyclictest0-21swapper/322:02:373
1738199198,8cyclictest0-21swapper/322:02:373
1738199195,10cyclictest28366irq/58-eth023:45:113
1738199193,10cyclictest0-21swapper/320:45:143
1738199191,9cyclictest28366irq/58-eth019:55:153
17381991911,6cyclictest0-21swapper/321:46:053
1738199191,13cyclictest0-21swapper/322:05:303
1737099191,9cyclictest17401-21latency_hist19:10:012
1737099191,9cyclictest17401-21latency_hist19:10:012
1737099191,8cyclictest31057-21sort21:05:122
1737099191,8cyclictest31057-21sort21:05:122
1737099191,8cyclictest1915-21latency_hist00:10:012
1736099196,10cyclictest15633-21ssh23:55:241
1736099195,10cyclictest11728-21irqcore00:15:181
1736099191,15cyclictest17777-21wc19:40:281
1736099191,15cyclictest17777-21wc19:40:281
1736099191,14cyclictest16674-21sh21:35:141
1736099191,10cyclictest6448-21sshd23:50:221
1734699198,8cyclictest29575-21apt-get22:00:120
1734699198,8cyclictest29575-21apt-get22:00:120
1734699195,8cyclictest0-21swapper/020:10:130
1734699195,7cyclictest131rcu_preempt23:40:180
1734699193,9cyclictest0-21swapper/023:27:040
1734699191,9cyclictest28460-21/usr/sbin/munin19:55:170
1734699191,3cyclictest131rcu_preempt19:40:120
1734699191,3cyclictest131rcu_preempt19:40:120
1734699191,14cyclictest7200-21sh21:10:260
1734699191,14cyclictest31004-21apt-get21:05:100
1734699191,14cyclictest31004-21apt-get21:05:100
1734699190,13cyclictest20121-21latency_hist21:55:010
1738199183,11cyclictest0-21swapper/300:07:123
1738199182,11cyclictest0-21swapper/300:13:293
1738199182,10cyclictest0-21swapper/319:35:143
17381991813,3cyclictest28466irq/59-eth023:15:373
17381991812,4cyclictest28466irq/59-eth023:44:023
17381991812,4cyclictest28466irq/59-eth022:15:193
1738199181,12cyclictest0-21swapper/320:50:273
17381991810,6cyclictest0-21swapper/323:05:083
17381991810,5cyclictest28366irq/58-eth023:00:223
1736099189,6cyclictest21738-21sshd19:45:211
1736099188,6cyclictest1109-21unixbench_singl20:30:281
1736099187,9cyclictest10346-21wc20:40:251
1736099185,10cyclictest29461-21latency_hist20:30:011
1736099183,8cyclictest15373-21diskmemload22:10:141
1736099183,5cyclictest31326-21grep21:05:151
1736099183,5cyclictest31326-21grep21:05:151
1736099181,4cyclictest53146usb-storage21:00:111
1736099181,13cyclictest31147-21ntp_192.168.11423:10:241
1736099181,12cyclictest5428-21df_inode22:40:131
17360991810,6cyclictest10651-21idleruntime20:10:161
1734699182,13cyclictest29446-21kthreadcore19:20:230
1734699181,6cyclictest13874-21latency_hist19:40:020
1734699181,14cyclictest0-21swapper/019:15:140
1734699181,13cyclictest16286-21sh21:15:250
1738199179,6cyclictest0-21swapper/323:30:183
1738199172,10cyclictest0-21swapper/321:35:143
1738199172,10cyclictest0-21swapper/321:05:123
1738199172,10cyclictest0-21swapper/321:05:123
1738199171,7cyclictest28366irq/58-eth021:30:133
17381991712,3cyclictest0-21swapper/322:45:023
1738199171,10cyclictest28366irq/58-eth022:35:133
1736099178,6cyclictest24181-21kthreadcore19:15:201
1736099173,11cyclictest6659-21cut19:30:211
1736099171,12cyclictest22617-21kthreadcore20:20:211
1736099171,12cyclictest20750-21grep19:10:231
1736099171,12cyclictest20750-21grep19:10:231
1736099171,12cyclictest0-21swapper/119:20:161
1734699173,9cyclictest12-21ksoftirqd/023:05:120
1734699171,7cyclictest27111-21ntp_kernel_pll_19:50:240
1734699171,3cyclictest0-21swapper/019:30:280
1734699171,13cyclictest25897-21http21:00:020
1734699171,11cyclictest26892-21kthreadcore20:25:220
1738199162,10cyclictest0-21swapper/320:15:143
17381991610,4cyclictest28366irq/58-eth020:30:133
1736099162,11cyclictest26963-21latency19:50:221
1736099161,12cyclictest28460-21/usr/sbin/munin19:55:181
1736099153,10cyclictest17417-21latency_hist19:10:011
1736099153,10cyclictest17417-21latency_hist19:10:011
1738199149,3cyclictest0-21swapper/319:21:343
1738199149,3cyclictest0-21swapper/319:15:143
1738199141,8cyclictest0-21swapper/320:20:193
1738199141,7cyclictest0-21swapper/320:40:213
1738199141,10cyclictest0-21swapper/320:00:133
1734699142,9cyclictest17358-21latency_hist19:10:010
1734699142,9cyclictest17358-21latency_hist19:10:010
1738199138,3cyclictest28366irq/58-eth021:00:273
1738199138,3cyclictest0-21swapper/319:10:303
1738199138,3cyclictest0-21swapper/319:10:303
1738199136,5cyclictest28366irq/58-eth019:30:133
1738199131,8cyclictest0-21swapper/319:45:113
1738199131,10cyclictest0-21swapper/319:50:273
1738199121,9cyclictest0-21swapper/319:30:013
1738199121,6cyclictest0-21swapper/320:35:133
1738199121,5cyclictest0-21swapper/319:40:203
1738199121,5cyclictest0-21swapper/319:40:203
1738199111,7cyclictest0-21swapper/320:05:193
1738199111,7cyclictest0-21swapper/320:05:193
173819991,6cyclictest0-21swapper/319:10:013
173819991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional