You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-05 - 16:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue May 05, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
221122221101,99sleep20-21swapper/207:06:192
221122221101,99sleep20-21swapper/207:06:192
223612182105,29sleep10-21swapper/107:09:311
223612182105,29sleep10-21swapper/107:09:311
222522165114,28sleep30-21swapper/307:08:083
222522165114,28sleep30-21swapper/307:08:083
222492164109,29sleep00-21swapper/007:08:050
222492164109,29sleep00-21swapper/007:08:050
283552690,8sleep228-21ksoftirqd/212:14:452
190592650,3sleep10-21swapper/108:15:141
148082570,3chrt0-21swapper/109:13:131
71472540,3sleep27148-21vmstat07:45:262
71352520,2chrt0-21swapper/212:22:112
61022520,3chrt22-21ksoftirqd/112:21:101
140202520,2sleep30-21swapper/309:12:263
33582500,1sleep20-21swapper/207:40:122
314412500,4sleep10-21swapper/110:17:251
266102470,1sleep20-21swapper/209:46:172
230262460,3chrt0-21swapper/209:19:362
165602460,1sleep00-21swapper/009:14:560
103412430,1sleep30-21swapper/312:25:093
20802420,1sleep30-21swapper/312:19:073
288402410,2sleep20-21swapper/211:26:412
219142410,1sleep10-21swapper/109:18:301
99652400,2sleep122-21ksoftirqd/111:36:401
302592400,9sleep228-21ksoftirqd/211:04:462
261862400,1chrt0-21swapper/111:00:441
29182390,2chrt0-21swapper/309:54:443
97802370,3chrt0-21swapper/312:24:493
184332350,9sleep3141rcu_preempt08:15:003
297472340,1sleep30-21swapper/312:39:143
22671993223,6cyclictest1781-21/usr/sbin/munin07:35:180
279362310,5sleep334-21ksoftirqd/311:50:143
240832310,2sleep012-21ksoftirqd/009:20:140
2267499301,26cyclictest3158-21latency_hist07:40:003
2267199302,12cyclictest3808-21apt-get10:45:110
2267199301,26cyclictest27937-21ntp_states07:20:200
323922290,1sleep30-21swapper/308:45:193
2267299282,17cyclictest8903-21ssh10:25:131
2267299281,25cyclictest0-21swapper/107:23:051
2267199280,26cyclictest52750irq/139-enp3s0-07:29:280
157132280,3sleep228-21ksoftirqd/209:14:072
149322280,1sleep00-21swapper/010:53:000
6452270,5sleep334-21ksoftirqd/310:42:173
41012270,2chrt0-21swapper/211:55:502
2267299271,24cyclictest0-21swapper/108:10:161
2267299271,24cyclictest0-21swapper/107:35:181
2267299271,24cyclictest0-21swapper/107:30:301
2267299271,24cyclictest0-21swapper/107:25:361
22671992723,2cyclictest31871-21sed07:30:160
2267199271,25cyclictest0-21swapper/008:13:300
40332260,2sleep20-21swapper/209:29:272
40332260,2sleep20-21swapper/209:29:272
2267499261,24cyclictest0-21swapper/307:30:393
2267499261,23cyclictest0-21swapper/308:16:353
2267499261,23cyclictest0-21swapper/307:26:223
2267399261,23cyclictest0-21swapper/208:11:412
2267399261,23cyclictest0-21swapper/207:25:502
2267299262,16cyclictest5085-21ssh09:30:031
2267199264,18cyclictest18458-21grep09:40:000
22671992623,2cyclictest0-21swapper/007:50:160
22671992623,1cyclictest0-21swapper/007:40:190
2267199261,23cyclictest0-21swapper/008:15:590
253462250,4sleep2271rcuc/207:15:162
2267499251,18cyclictest0-21swapper/307:23:393
22673992522,2cyclictest0-21swapper/207:35:212
2267399251,23cyclictest0-21swapper/207:20:222
2267399251,1cyclictest0-21swapper/208:18:402
2267399250,23cyclictest0-21swapper/207:30:172
22672992515,7cyclictest10968-21df10:50:141
2267299250,1cyclictest0-21swapper/107:40:191
2267199255,18cyclictest8317-21run-parts12:00:000
22671992514,8cyclictest13310-21/usr/sbin/munin08:00:240
22674992413,7cyclictest25463-21df_inode08:30:143
2267199248,4cyclictest66050lldpd10:29:280
22671992416,6cyclictest29879-21sh12:39:210
22671992414,8cyclictest5175-21apt-get07:45:040
22674992313,7cyclictest25233-21python309:20:243
22674992311,8cyclictest560-21dbus-daemon11:25:003
22673992315,6cyclictest28-21ksoftirqd/210:44:592
22673992314,7cyclictest6339-21ls09:00:192
2267399231,10cyclictest316472ssh09:51:042
2267299239,5cyclictest18201-21expr10:55:171
22672992316,5cyclictest31765-21sh11:29:361
22672992313,3cyclictest23393-21sh09:44:481
2267299231,15cyclictest5314-21ssh11:10:031
2267299231,10cyclictest297482sh12:39:151
2267199236,13cyclictest31071-21tr11:05:130
2267199232,15cyclictest3658-21ssh09:55:130
2267199231,10cyclictest66050lldpd09:41:570
22671992310,11cyclictest32250-21ssh12:16:520
22674992220,1cyclictest0-21swapper/307:50:173
22674992220,1cyclictest0-21swapper/307:40:193
22674992213,5cyclictest52950irq/141-enp3s0-11:10:123
22674992212,8cyclictest3655-21ssh11:32:073
22673992220,1cyclictest0-21swapper/207:59:142
22673992220,1cyclictest0-21swapper/207:50:182
22673992214,6cyclictest28127-21perl10:15:122
2267399220,10cyclictest201482sh11:44:472
2267299229,11cyclictest9164-21ssh12:00:151
2267299227,4cyclictest22-21ksoftirqd/111:40:131
2267299222,6cyclictest640-21snmpd10:34:401
22672992215,5cyclictest22-21ksoftirqd/110:15:001
22672992213,7cyclictest30365-21cron09:50:001
2267299221,19cyclictest3384-21cron10:44:591
22671992213,7cyclictest9975-21cron12:25:010
178602220,2sleep317879-21ssh11:42:323
314452210,2sleep012-21ksoftirqd/011:29:160
279202210,1sleep00-21swapper/012:14:190
2267499216,3cyclictest245782grep09:45:153
2267499213,16cyclictest3210-21pmu-power10:20:203
22674992115,3cyclictest9833-21diskmemload10:53:003
22674992115,3cyclictest9833-21diskmemload09:29:213
22674992115,3cyclictest9833-21diskmemload09:29:203
22674992113,5cyclictest9833-21diskmemload10:05:263
22674992112,5cyclictest52950irq/141-enp3s0-11:25:123
22674992111,8cyclictest12662-21grep11:15:133
22674992111,7cyclictest28768-21sed08:35:243
22674992111,7cyclictest2612-21timerwakeupswit08:50:243
2267399211,4cyclictest0-21swapper/212:33:592
22673992113,6cyclictest15110-21pluginstate12:05:002
22673992113,6cyclictest11666-21cat09:35:002
2267399211,10cyclictest9833-21diskmemload11:30:132
2267299218,3cyclictest141rcu_preempt08:35:001
22672992113,6cyclictest28614-21ntpq11:50:201
22672992113,6cyclictest26183-21seq10:37:111
22672992113,6cyclictest22648-21seq08:24:001
2267299211,13cyclictest141rcu_preempt12:27:281
2267299210,7cyclictest24221-21users07:10:271
2267299210,13cyclictest141rcu_preempt09:35:051
2267199210,9cyclictest12-21ksoftirqd/009:46:510
60682200,2sleep30-21swapper/311:57:483
2267499209,5cyclictest20913-21cat08:20:133
2267499207,11cyclictest10437-21ssh10:00:143
2267499204,3cyclictest9833-21diskmemload11:47:513
2267499203,4cyclictest34-21ksoftirqd/312:10:143
2267499203,3cyclictest9833-21diskmemload10:27:373
2267499201,5cyclictest4875-21ntp_states10:45:203
22674992014,4cyclictest24933-21cpuspeed_turbos07:15:133
22674992012,6cyclictest18754-21uptime09:15:253
22674992012,6cyclictest16846-21python312:05:233
22674992011,7cyclictest24273-21seq07:10:573
22674992011,6cyclictest4321-21ntp_kernel_pll_08:55:203
2267499201,11cyclictest0-21swapper/310:15:423
2267399206,4cyclictest37392chrt10:45:072
2267399202,2cyclictest141rcu_preempt10:55:112
2267399202,13cyclictest19062-21ssh11:20:112
22673992013,4cyclictest164672chrt10:05:202
22673992012,6cyclictest23015-21seq11:45:532
22673992012,6cyclictest14648-21ls08:05:142
22673992012,5cyclictest23430-21tune2fs08:25:162
22673992012,5cyclictest22102-21timerandwakeup08:20:242
22673992011,7cyclictest28310-21cat11:50:172
22673992011,6cyclictest10611-21cat12:25:132
2267299202,9cyclictest141rcu_preempt08:25:141
2267299202,13cyclictest30948-21ssh11:05:121
22672992013,4cyclictest27759-21sh12:14:091
22672992012,6cyclictest8460-21pluginstate10:25:001
22672992012,6cyclictest28653-21grep08:35:221
22672992011,7cyclictest11545-21seq07:57:001
22672992011,6cyclictest3333-21seq09:28:451
22672992011,6cyclictest3333-21seq09:28:451
22672992011,6cyclictest21327-21tune2fs11:45:161
2267299201,14cyclictest29052-21ssh12:15:111
2267199209,2cyclictest12-21ksoftirqd/009:17:010
2267199208,4cyclictest52750irq/139-enp3s0-10:01:350
2267199204,14cyclictest8908-21cstates12:00:130
2267199202,13cyclictest27639-21seq10:15:080
2267199201,5cyclictest18004-21smartctl10:55:160
22671992014,4cyclictest1033-21sh10:42:380
22671992012,6cyclictest32251-21sed09:25:260
22671992012,6cyclictest32251-21sed09:25:260
22671992012,6cyclictest27002-21seq11:49:510
22671992011,6cyclictest26016-21python307:15:230
104472200,2sleep30-21swapper/312:00:253
90442190,1sleep20-21swapper/210:25:142
2267499198,9cyclictest9863-21latency_hist09:10:003
2267499191,5cyclictest0-21swapper/310:30:353
22674991913,4cyclictest52950irq/141-enp3s0-10:35:143
22674991913,4cyclictest11550-21ssh11:38:133
22674991912,4cyclictest9833-21diskmemload09:43:483
22674991911,6cyclictest5905-21idleruntime09:30:163
22674991911,6cyclictest5571-21cat09:00:113
22674991911,6cyclictest31193-21cat11:05:143
22674991911,6cyclictest278-21systemd-journal09:59:593
22674991911,6cyclictest13365-21cat09:35:243
22674991911,6cyclictest12346-21expr08:00:133
22674991910,7cyclictest6458-21needreboot07:45:193
22674991910,7cyclictest25514-21cat11:00:243
2267399194,2cyclictest141rcu_preempt10:35:112
2267399192,12cyclictest15860-21ssh10:30:112
2267399192,12cyclictest11983-21ssh09:35:142
2267399191,4cyclictest0-21swapper/211:12:122
22673991912,5cyclictest15856-21expr12:05:132
22673991912,4cyclictest9833-21diskmemload10:21:562
22673991911,6cyclictest11467-21memory10:50:182
2267299198,7cyclictest53050irq/142-enp3s0-11:30:121
2267299196,11cyclictest22-21ksoftirqd/111:15:141
2267299192,5cyclictest0-21swapper/110:06:021
2267299192,12cyclictest12066-21apt-get08:00:121
22672991917,1cyclictest0-21swapper/107:50:171
22672991913,3cyclictest31338-21sh09:50:461
22672991911,6cyclictest4113-21cut09:55:161
22672991911,6cyclictest32176-21if_enp3s008:45:161
22672991911,6cyclictest20343-21run-parts11:20:221
22672991910,6cyclictest24922-21cat07:15:131
22672991910,5cyclictest0-21swapper/110:45:111
2267199199,8cyclictest1019-21ssh09:52:520
2267199197,4cyclictest52750irq/139-enp3s0-11:01:350
2267199197,4cyclictest52750irq/139-enp3s0-10:07:470
2267199193,3cyclictest9833-21diskmemload11:42:190
2267199191,9cyclictest103432rm12:25:090
2267199191,5cyclictest0-21swapper/009:30:130
22671991913,4cyclictest30377-21sh11:51:290
2267199191,2cyclictest1795-21ls11:30:260
22671991912,5cyclictest8321-21cut09:05:160
22671991912,5cyclictest22922-21expr10:35:120
22671991911,4cyclictest19064-21cpu11:20:120
22671991910,7cyclictest24529-21seq08:25:380
22671991910,7cyclictest17392-21cstates12:30:140
2267499182,11cyclictest17355-21apt-get10:55:123
22674991813,3cyclictest9833-21diskmemload12:30:203
2267399189,7cyclictest3331-21cut08:55:012
2267399182,11cyclictest7941-21apt-get09:05:122
2267399182,11cyclictest24036-21taskset12:35:032
2267399181,5cyclictest0-21swapper/210:11:572
2267399181,4cyclictest0-21swapper/210:02:072
22673991813,3cyclictest6929-21/usr/sbin/munin11:35:192
22673991813,3cyclictest0-21swapper/211:09:012
22673991811,5cyclictest30834-21grep08:40:262
22673991811,5cyclictest26453-21expr08:30:242
22673991811,5cyclictest12657-21expr11:15:132
22673991810,6cyclictest23069-21grep07:10:162
2267299189,6cyclictest30129-21latency08:40:191
2267299188,6cyclictest53050irq/142-enp3s0-12:05:131
2267299187,8cyclictest17472-21cstates12:30:131
22672991812,4cyclictest1826-21sh11:55:061
22672991812,4cyclictest11465-21sh10:01:141
22672991811,5cyclictest7456-21seq07:49:251
22672991811,5cyclictest4729-21cat08:55:241
2267299181,14cyclictest7892-21packagekitd09:05:011
22672991810,6cyclictest24698-21cat09:20:191
22672991810,5cyclictest5908-21cat09:00:131
22672991810,5cyclictest2445-21sensors_temp08:50:221
2267199186,9cyclictest0-21swapper/011:10:030
2267199182,11cyclictest14415-21apt-get08:05:140
2267199181,3cyclictest9833-21diskmemload11:35:150
22671991813,3cyclictest20377-21sh12:08:330
22671991812,4cyclictest23414-21sh10:11:120
22671991811,5cyclictest2746-21if_enp3s010:20:150
22671991811,5cyclictest27251-21find08:35:090
22671991810,6cyclictest29759-21df08:40:150
22671991810,6cyclictest16901-21ntp_states10:30:200
22671991810,6cyclictest11799-21ls08:00:010
22671991810,5cyclictest6348-21memory09:00:180
2267199180,16cyclictest6509-21kworker/u8:111:15:110
2267499179,5cyclictest9833-21diskmemload10:12:313
22674991711,4cyclictest9885-21/usr/sbin/munin07:55:173
22674991710,5cyclictest23581-21basename08:25:183
2267399179,6cyclictest560-21dbus-daemon09:00:002
2267399177,7cyclictest27273-21apt-get08:35:092
22673991712,3cyclictest12041-21/usr/sbin/munin08:00:222
22673991711,4cyclictest52850irq/140-enp3s0-09:21:312
22673991711,3cyclictest9833-21diskmemload09:43:142
22673991710,5cyclictest52850irq/140-enp3s0-09:55:262
22673991710,5cyclictest30594-21ls12:15:242
22673991710,4cyclictest32223-21kernelversion08:45:172
2267199179,6cyclictest26423-21cat08:30:240
2267199179,5cyclictest1898-21if_enp3s008:50:170
2267199172,9cyclictest636-21runrttasks08:20:120
2267199171,14cyclictest4740-21grep08:55:240
22671991710,5cyclictest24811-21sed07:15:010
22671991710,5cyclictest12-21ksoftirqd/008:45:100
2267499161,10cyclictest29437-21apt-get08:40:143
2267299161,11cyclictest13958-21apt-get08:05:021
2267499152,9cyclictest14379-21apt-get08:05:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional