You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 08:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Jul 14, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2409299623,50cyclictest24623-21ssh22:30:122
24092995936,9cyclictest9935-21diskmemload23:20:132
24092994925,17cyclictest24770-21apt-get00:00:142
2407299492,36cyclictest19273-21ssh22:45:120
24092994632,9cyclictest28372-21apt-get21:20:162
2409299451,37cyclictest1481-21sshd19:20:132
24092994434,3cyclictest5388-21apt-get21:05:142
2407299443,8cyclictest131rcu_preempt21:10:120
2410199432,25cyclictest35966irq/58-eth021:45:133
24092994330,9cyclictest10308-21apt-get21:10:132
24092994233,3cyclictest2772-21apt-get19:55:152
2410199414,24cyclictest35966irq/58-eth022:25:133
24092994112,15cyclictest19553-21ssh23:01:322
2408499413,27cyclictest6715-21ssh23:50:131
2407299411,33cyclictest2956-21ssh23:30:110
2410199402,22cyclictest35966irq/58-eth021:40:133
24072994020,9cyclictest12-21ksoftirqd/022:05:130
2407299401,8cyclictest1311-21ssh00:05:150
24092993931,4cyclictest24283-21apt-get19:10:112
2409299392,29cyclictest8465-21df23:15:152
24101993811,15cyclictest35966irq/58-eth020:10:113
2409299384,14cyclictest0-21swapper/222:40:142
24092993831,3cyclictest15714-21apt-get23:55:152
24092993812,6cyclictest9935-21diskmemload00:17:272
2408499381,30cyclictest11959-21ssh23:35:131
24092993717,12cyclictest0-21swapper/200:10:142
2407299372,26cyclictest13895-21ssh23:00:140
2410199365,18cyclictest35966irq/58-eth000:05:143
2410199363,21cyclictest35966irq/58-eth022:45:103
2410199363,20cyclictest35966irq/58-eth022:50:133
2409299367,8cyclictest30105-21ntp_states23:25:272
2409299367,15cyclictest31264-21usage_EthcTask20:20:302
2409299366,14cyclictest4414-21irqcore20:30:182
24092993616,15cyclictest0-21swapper/200:20:102
2408499361,29cyclictest10330-21ssh21:10:121
24101993518,8cyclictest0-21swapper/323:55:133
2409299359,20cyclictest22820-21sh23:05:132
24092993517,9cyclictest1299-21apt-get00:05:172
2408499353,28cyclictest1212-21ssh22:35:131
2410199344,24cyclictest35966irq/58-eth022:00:123
2409299348,16cyclictest0-21swapper/219:50:142
2409299347,13cyclictest21159-21ssh21:33:272
2408499344,27cyclictest20925-21ssh22:10:131
2408499343,27cyclictest19372-21ssh21:15:151
2410199332,21cyclictest35966irq/58-eth000:20:113
2410199332,18cyclictest35966irq/58-eth023:20:123
2409299339,18cyclictest0-21swapper/222:05:132
2409299337,12cyclictest135-21systemd-journal19:40:012
2409299337,12cyclictest135-21systemd-journal19:40:012
2409299335,23cyclictest0-21swapper/222:17:142
24092993327,3cyclictest11948-21apt-get23:35:152
24092993317,13cyclictest0-21swapper/222:20:132
24092993312,18cyclictest9935-21diskmemload00:25:332
2408499333,26cyclictest22853-21ssh21:35:131
2408499332,27cyclictest22815-21ssh23:05:141
2409299329,6cyclictest23748-21kthreadcore23:40:212
2409299329,6cyclictest23748-21kthreadcore23:40:212
2409299327,10cyclictest14135-21ssh23:53:342
2409299326,10cyclictest5235-21rm23:48:442
2408499321,28cyclictest4789-21ssh00:25:151
2407299324,19cyclictest13708-21apt-get00:30:120
2407299322,17cyclictest4893-21ssh21:25:130
2407299322,17cyclictest4893-21ssh21:25:130
24072993218,12cyclictest12-21ksoftirqd/000:15:110
2410199313,23cyclictest35966irq/58-eth022:30:143
2410199312,20cyclictest35966irq/58-eth022:40:133
24101993115,10cyclictest0-21swapper/320:20:123
2409299315,9cyclictest0-21swapper/221:47:482
2409299315,7cyclictest0-21swapper/221:53:192
2409299313,15cyclictest167-21systemd-network19:15:122
2409299312,11cyclictest930-21snmpd22:12:132
24084993124,3cyclictest201rcuc/119:50:141
24084993119,9cyclictest21-21ksoftirqd/119:20:141
2408499311,20cyclictest0-21swapper/121:05:131
2407299318,17cyclictest13871-21diskmemload21:30:120
2407299312,22cyclictest21127-21ssh23:40:140
2407299312,22cyclictest21127-21ssh23:40:140
2407299311,19cyclictest22670-21ssh00:35:130
2407299311,19cyclictest22670-21ssh00:35:130
2410199304,22cyclictest35966irq/58-eth020:30:143
2410199301,19cyclictest35966irq/58-eth019:20:133
2409299308,8cyclictest17000-21cron20:10:012
2409299306,10cyclictest14589-21ntp_192.168.11419:30:252
2409299305,12cyclictest30960-21ssh00:39:452
2409299305,12cyclictest30960-21ssh00:39:442
24092993017,10cyclictest27346irq/40-dwc2_hso21:35:142
24092993014,10cyclictest0-21swapper/220:40:132
24092993014,10cyclictest0-21swapper/220:40:132
2408499303,24cyclictest10276-21ssh22:40:141
2408499302,25cyclictest13857-21ssh21:30:111
2407299306,11cyclictest31811-21ssh23:10:120
2407299302,22cyclictest2935-21ssh22:00:120
2407299302,22cyclictest12076-21ssh22:21:150
2407299301,5cyclictest131rcu_preempt19:50:140
2410199295,13cyclictest35966irq/58-eth019:30:143
2409299299,10cyclictest23435-21/usr/sbin/munin21:15:322
2409299295,10cyclictest0-21swapper/221:42:172
2409299293,21cyclictest9935-21diskmemload22:50:142
24092992915,11cyclictest0-21swapper/221:00:132
2409299291,20cyclictest4919-21ssh21:25:152
2409299291,20cyclictest4919-21ssh21:25:142
2409299291,16cyclictest19276-21rm22:45:132
2407299292,20cyclictest9935-21diskmemload00:20:110
2410199284,18cyclictest0-21swapper/323:07:233
24101992811,9cyclictest0-21swapper/300:35:143
24101992811,9cyclictest0-21swapper/300:35:143
2409299284,11cyclictest0-21swapper/222:35:252
2409299282,19cyclictest0-21swapper/221:55:152
2409299282,18cyclictest27346irq/40-dwc2_hso20:50:142
24092992821,4cyclictest4891-21apt-get22:55:152
24092992813,8cyclictest2895-21ssh23:30:102
24092992813,7cyclictest10480-21usage_EthcTask19:26:022
24092992813,12cyclictest24414-21/usr/sbin/munin19:40:292
24092992813,12cyclictest18684-21/usr/sbin/munin20:45:162
24092992811,14cyclictest7650-21apt20:00:132
24092992810,15cyclictest6753-21mailstats22:00:282
2408499289,15cyclictest21-21ksoftirqd/122:25:131
2408499285,19cyclictest17377-21ssh21:50:131
2407299282,20cyclictest29241-21ssh23:07:230
2407299282,20cyclictest16612-21ssh22:42:180
2410199274,16cyclictest35966irq/58-eth021:35:143
2410199273,17cyclictest0-21swapper/322:37:163
2410199272,18cyclictest0-21swapper/300:17:323
24101992715,8cyclictest35966irq/58-eth021:30:123
24101992712,9cyclictest35966irq/58-eth023:10:113
2410199271,13cyclictest35966irq/58-eth023:00:143
2409299272,18cyclictest27346irq/40-dwc2_hso20:55:122
24092992710,14cyclictest17239-21ssh00:30:222
2408499274,19cyclictest201rcuc/123:25:141
2408499271,4cyclictest28174-21apt-get00:20:121
2407299272,20cyclictest4921-21ssh22:55:140
2407299272,19cyclictest31850-21ssh23:26:140
2407299272,19cyclictest21082-21ssh22:26:130
2410199269,11cyclictest0-21swapper/300:10:143
2409299269,13cyclictest135-21systemd-journal20:16:492
2409299265,3cyclictest9935-21diskmemload23:13:262
2409299263,14cyclictest27346irq/40-dwc2_hso22:25:132
24092992610,13cyclictest21803-21sshd20:14:142
2408499264,14cyclictest21-21ksoftirqd/122:20:131
2408499262,18cyclictest26394-21ssh21:55:151
24084992616,8cyclictest21-21ksoftirqd/122:50:141
2408499261,21cyclictest31802-21ssh23:10:121
2407299269,8cyclictest12-21ksoftirqd/019:10:120
2407299268,4cyclictest131rcu_preempt23:35:140
2410199252,11cyclictest35966irq/58-eth021:00:133
24101992515,3cyclictest0-21swapper/319:25:123
2409299259,13cyclictest30137-21sshd19:46:452
24092992511,11cyclictest8849-21/usr/sbin/munin20:35:112
2408499252,16cyclictest0-21swapper/123:00:131
24084992515,7cyclictest9935-21diskmemload22:15:151
2407299252,18cyclictest4055-21sshd20:30:140
2410199243,18cyclictest0-21swapper/322:21:153
24101992418,4cyclictest321rcuc/300:30:133
24101992414,7cyclictest0-21swapper/300:00:223
24101992412,9cyclictest0-21swapper/320:25:253
2410199241,16cyclictest0-21swapper/321:55:153
24084992417,4cyclictest21-21ksoftirqd/122:00:121
2408499241,18cyclictest9935-21diskmemload21:45:131
2407299243,14cyclictest0-21swapper/019:40:150
2407299242,16cyclictest17352-21ssh23:20:120
24072992413,6cyclictest24065-21cyclictest00:00:130
2407299241,16cyclictest8103-21ls21:45:020
2410199233,13cyclictest0-21swapper/319:45:143
24101992311,6cyclictest0-21swapper/321:50:233
2409299230,8cyclictest0-21swapper/220:25:132
2408499233,17cyclictest24629-21ssh22:30:131
2408499232,19cyclictest4166-21ssh23:47:281
2408499232,18cyclictest3072-21df_inode23:30:151
2408499232,16cyclictest9193-21kworker/u8:022:09:141
2408499232,15cyclictest5020-21ssh00:05:301
24084992311,3cyclictest0-21swapper/119:55:201
2407299232,3cyclictest131rcu_preempt21:50:130
2407299231,8cyclictest28846-21ssh22:14:140
2410199223,13cyclictest0-21swapper/323:40:143
2410199223,13cyclictest0-21swapper/323:40:143
24101992214,5cyclictest35966irq/58-eth021:15:163
24101992213,6cyclictest0-21swapper/321:21:303
2410199221,18cyclictest0-21swapper/321:25:303
2410199221,18cyclictest0-21swapper/321:25:303
24101992211,7cyclictest0-21swapper/323:15:143
24084992212,8cyclictest21-21ksoftirqd/119:45:131
2408499221,16cyclictest4918-21sh21:25:151
2408499221,16cyclictest4918-21sh21:25:151
24072992213,7cyclictest22045-21sshd20:15:140
2407299221,14cyclictest9935-21diskmemload21:20:150
2410199219,7cyclictest35966irq/58-eth022:05:123
24101992116,3cyclictest0-21swapper/300:25:263
24101992115,4cyclictest0-21swapper/322:10:213
24101992114,5cyclictest0-21swapper/322:55:143
24101992112,7cyclictest0-21swapper/323:30:133
2410199210,15cyclictest321rcuc/320:45:143
2408499213,4cyclictest131rcu_preempt23:20:141
2408499212,3cyclictest131rcu_preempt00:15:161
2408499212,14cyclictest13928-21idleruntime20:40:171
2408499212,14cyclictest13928-21idleruntime20:40:161
2408499211,16cyclictest3836-21ls21:00:231
2408499211,14cyclictest28666-21idleruntime21:20:191
2407299212,14cyclictest21114-21ssh23:56:140
2407299211,4cyclictest4493-21awk22:55:010
2407299211,18cyclictest31526-21ls20:25:020
2407299211,17cyclictest19219-21latency19:35:230
2407299211,17cyclictest19219-21latency19:35:220
2407299211,16cyclictest8371-21ssh23:15:140
24072992110,3cyclictest131rcu_preempt21:55:270
2407299210,18cyclictest5339-21ls21:05:020
2410199204,11cyclictest0-21swapper/323:26:143
24101992014,4cyclictest0-21swapper/323:50:243
24101992014,4cyclictest0-21swapper/322:15:193
24101992014,4cyclictest0-21swapper/320:40:193
24101992014,4cyclictest0-21swapper/320:40:193
24101992014,4cyclictest0-21swapper/319:20:023
24101992013,5cyclictest0-21swapper/323:35:133
24101992011,6cyclictest0-21swapper/319:50:313
2408499202,2cyclictest131rcu_preempt22:55:581
2408499202,10cyclictest21736-21kworker/1:100:37:541
2408499202,10cyclictest21736-21kworker/1:100:37:541
2408499201,17cyclictest21038-21ssh23:40:141
2408499201,17cyclictest21038-21ssh23:40:131
2408499201,16cyclictest6250-21find19:25:121
2407299207,3cyclictest131rcu_preempt20:55:010
2407299201,9cyclictest26567-21/usr/sbin/munin21:35:270
2407299201,15cyclictest19371-21sh21:15:160
2410199192,11cyclictest0-21swapper/319:35:143
2410199192,11cyclictest0-21swapper/319:35:143
2410199192,10cyclictest0-21swapper/320:00:043
24101991913,4cyclictest0-21swapper/323:45:153
24101991913,4cyclictest0-21swapper/321:05:123
24101991913,4cyclictest0-21swapper/320:50:283
24101991913,4cyclictest0-21swapper/320:05:203
24101991912,5cyclictest0-21swapper/321:10:123
2409299191,10cyclictest24103-21latency_hist19:10:022
2408499194,10cyclictest15724-21ssh23:55:131
2408499192,11cyclictest10269-21ssh00:10:141
2407299196,10cyclictest4968-21wc19:20:270
2407299194,3cyclictest131rcu_preempt21:45:150
2407299191,7cyclictest23138-21munin-run20:50:010
2407299191,6cyclictest13815-21ls00:10:240
2407299191,2cyclictest131rcu_preempt23:45:200
2407299190,15cyclictest31812-21ls20:55:280
2410199189,6cyclictest35966irq/58-eth020:15:143
24101991813,3cyclictest0-21swapper/320:00:013
24101991813,3cyclictest0-21swapper/319:10:223
24101991812,4cyclictest0-21swapper/320:55:203
24101991812,4cyclictest0-21swapper/319:40:263
2410199181,11cyclictest0-21swapper/320:35:113
2408499189,6cyclictest28393-21ntpq00:00:261
2408499185,9cyclictest5924-21sed20:30:211
2408499183,9cyclictest0-21swapper/123:15:151
2408499182,4cyclictest131rcu_preempt20:55:141
2408499182,3cyclictest131rcu_preempt21:40:141
2408499182,14cyclictest13856-21kthreadcore20:05:201
2408499182,11cyclictest0-21swapper/119:15:141
2408499182,11cyclictest0-21swapper/119:10:131
2408499181,9cyclictest29973-21kthreadcore20:20:221
2408499181,9cyclictest22874-21cut20:15:191
2408499181,8cyclictest21536-21kthreadcore22:45:201
24084991812,4cyclictest0-21swapper/120:00:131
2408499181,14cyclictest13686-21/usr/sbin/munin00:30:151
2407299182,2cyclictest0-21swapper/022:17:140
2407299182,12cyclictest31758-21ssh22:33:140
2407299181,5cyclictest7328-21munin-run20:00:010
2407299181,14cyclictest1162-21/usr/sbin/munin22:35:220
24072991810,6cyclictest8269-21memory00:25:240
2408499179,3cyclictest21-21ksoftirqd/120:45:041
2408499178,6cyclictest25325-21kthreadcore20:50:211
2408499175,4cyclictest167-21systemd-network19:35:171
2408499175,4cyclictest167-21systemd-network19:35:171
2408499172,12cyclictest18320-21kthreadcore20:10:181
2408499171,3cyclictest131rcu_preempt19:35:011
2408499171,13cyclictest20990-21fschecks_time19:40:171
2408499170,6cyclictest32024-21tail20:25:171
2407299176,8cyclictest12-21ksoftirqd/019:30:140
2407299171,7cyclictest26826-21tr19:45:200
2407299171,3cyclictest131rcu_preempt20:00:140
2407299171,13cyclictest10460-21ntpq23:50:260
2407299170,10cyclictest9905-21switchtime19:25:290
2408499162,10cyclictest0-21swapper/120:35:131
2407299162,2cyclictest131rcu_preempt19:15:120
2407299162,10cyclictest0-21swapper/020:25:140
2407299162,10cyclictest0-21swapper/020:05:150
2407299161,11cyclictest17467-21wc20:40:290
2407299161,11cyclictest17467-21wc20:40:280
2407299161,11cyclictest10512-21kthreadcore20:35:190
2407299153,4cyclictest12-21ksoftirqd/021:05:160
2407299151,11cyclictest24114-21latency_hist19:10:020
2407299151,11cyclictest17439-21fschecks_time20:10:150
2408499141,10cyclictest24193-21wc19:10:021
2410199111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional