You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-26 - 19:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Jul 26, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
274002163104,27sleep20-21swapper/207:06:362
274982161106,28sleep30-21swapper/307:07:523
276632160104,30sleep00-21swapper/007:09:590
275952158105,30sleep10-21swapper/107:09:061
286542760,1sleep30-21swapper/310:07:163
2766627557,10sleep30-21swapper/307:10:023
327372610,10sleep012-21ksoftirqd/010:10:190
68942600,9sleep334-21ksoftirqd/311:27:193
267902560,1chrt0-21swapper/209:41:392
248272560,4sleep00-21swapper/009:40:210
206082560,1sleep220610-21rm10:00:582
206082560,1sleep220610-21rm10:00:572
270862480,1sleep20-21swapper/211:42:192
238052480,2sleep30-21swapper/310:28:003
264252470,1sleep00-21swapper/010:54:200
39942450,1sleep30-21swapper/312:37:173
188732450,1sleep30-21swapper/312:24:583
177252420,1sleep20-21swapper/210:47:212
288052410,1sleep10-21swapper/109:43:381
78612380,2chrt0-21swapper/011:28:160
49102370,3chrt0-21swapper/009:50:120
128332340,4chrt12828-21ssh12:20:120
320322310,3sleep227936-21cyclictest10:10:132
135062310,2chrt0-21swapper/311:55:593
133982290,2sleep20-21swapper/210:20:162
161202280,2sleep30-21swapper/309:58:143
53302270,2chrt0-21swapper/112:15:011
27941992717,8cyclictest32013-21cron07:19:593
27940992714,8cyclictest13030-21diskstats12:20:142
2793899271,24cyclictest0-21swapper/007:20:530
2793899271,24cyclictest0-21swapper/007:18:140
127722270,2sleep10-21swapper/111:07:551
2794199261,23cyclictest0-21swapper/307:20:543
27940992622,2cyclictest0-21swapper/207:20:472
2794099261,23cyclictest0-21swapper/207:18:112
2793999261,24cyclictest0-21swapper/107:18:331
27938992618,6cyclictest15589-21ssh11:58:060
64212250,1sleep10-21swapper/111:03:161
292842250,6sleep215092-21diskmemload12:31:552
2794199251,23cyclictest0-21swapper/307:27:163
2793899257,8cyclictest141rcu_preempt09:12:030
317212240,2sleep00-21swapper/010:57:540
27940992413,7cyclictest53050irq/142-enp3s0-10:35:132
27939992422,1cyclictest0-21swapper/107:20:421
2793999241,16cyclictest22245-21ssh11:15:131
27939992410,5cyclictest20403-21cut10:25:231
206332240,1chrt0-21swapper/112:25:231
47882230,1sleep20-21swapper/209:50:072
2794199231,13cyclictest15102-21apt-get09:10:003
27940992313,7cyclictest2754-21ls07:25:182
27940992313,6cyclictest6059-21expr12:15:152
27940992313,6cyclictest6059-21expr12:15:142
2793999231,7cyclictest636-21runrttasks12:20:061
2793999231,7cyclictest25697-21hddtemp_smartct10:05:171
2793999231,6cyclictest23858-21df11:40:151
2793899238,7cyclictest141rcu_preempt10:30:380
122382230,2sleep10-21swapper/110:43:351
27941992213,7cyclictest3197-21sensors_temp11:00:243
27940992214,5cyclictest53050irq/142-enp3s0-09:10:032
27940992211,7cyclictest7228-21cut10:15:212
2793999223,11cyclictest4534-21ssh11:50:131
2793999221,15cyclictest10056-21sh11:30:131
2793999221,15cyclictest10056-21sh11:30:121
2793899227,8cyclictest12-21ksoftirqd/010:17:440
2793899227,4cyclictest0-21swapper/011:14:580
2793899226,4cyclictest15092-21diskmemload09:35:110
2793899224,5cyclictest12-21ksoftirqd/010:43:030
2793899223,7cyclictest12-21ksoftirqd/009:47:280
2793899222,10cyclictest12-21ksoftirqd/012:00:460
27938992217,3cyclictest26323-21/usr/sbin/munin12:05:270
233512220,2sleep30-21swapper/310:51:153
20042220,3sleep30-21swapper/312:11:463
51922210,1sleep00-21swapper/008:45:150
2794199216,3cyclictest141rcu_preempt10:50:003
27941992114,5cyclictest21346-21sh08:09:593
27941992112,7cyclictest7511-21date08:50:173
2794099219,9cyclictest3749-21sh09:49:172
27940992113,6cyclictest27029-21ntp_states08:20:202
27940992113,6cyclictest10406-21seq07:44:522
27940992113,5cyclictest13686-21idleruntime-cro10:45:002
27940992112,7cyclictest12673-21ntp_states09:55:212
27940992111,6cyclictest6166-21tail11:50:272
2793999211,5cyclictest25146-21perl12:05:151
27939992113,6cyclictest3907-21seq08:41:501
27939992112,6cyclictest7072-21cat08:50:131
27939992111,8cyclictest278-21systemd-journal10:50:001
27939992111,8cyclictest13353-21df07:50:141
27939992111,6cyclictest52850irq/140-enp3s0-12:30:111
27939992111,6cyclictest31483-21sed09:45:201
2793899217,7cyclictest12-21ksoftirqd/009:28:200
2793899215,2cyclictest141rcu_preempt11:30:560
2793899215,2cyclictest141rcu_preempt11:30:550
2793899214,9cyclictest10425-21ssh09:30:150
2793899214,7cyclictest141rcu_preempt12:31:310
2793899214,2cyclictest12-21ksoftirqd/011:45:550
2793899213,16cyclictest3896-21ls07:29:590
27938992113,6cyclictest30621-21python309:20:220
27938992112,7cyclictest4479-21diskstats07:30:130
27938992112,5cyclictest52950irq/141-enp3s0-10:05:230
27938992110,7cyclictest23952-21apt-get08:15:120
149532210,3sleep3341ksoftirqd/310:20:493
148522210,1sleep20-21swapper/211:10:012
27941992013,5cyclictest25328-21ssh11:16:493
27941992013,5cyclictest13114-21grep07:50:133
27941992013,4cyclictest0-21swapper/311:10:133
27941992012,6cyclictest6786-21load12:15:213
27941992012,6cyclictest6786-21load12:15:203
27941992012,6cyclictest25108-21python308:15:233
27941992012,6cyclictest12144-21sendmail_mailqu09:00:233
27941992011,7cyclictest4603-21perl07:30:133
27941992011,7cyclictest4168-21tune2fs09:25:173
27941992011,7cyclictest29652-21smartctl11:20:163
27941992011,7cyclictest28059-21cut08:25:013
27941992011,7cyclictest10804-21seq10:18:263
27941992011,6cyclictest19466-21ls12:00:253
27940992012,6cyclictest22233-21sed10:50:242
27940992010,7cyclictest10954-21grep09:30:182
2794099200,17cyclictest28994-21find11:20:122
2793999202,13cyclictest27574-21ssh10:55:141
2793999202,12cyclictest20827-21chrt10:50:131
2793999201,7cyclictest30197-21apt-get11:45:131
2793999201,6cyclictest29390-21df_inode11:20:141
27939992015,3cyclictest6026-21sh07:35:001
27939992014,4cyclictest13615-21sh11:56:051
27939992013,5cyclictest23638-21seq10:04:001
27939992013,5cyclictest23638-21seq10:04:001
27939992012,6cyclictest7843-21seq12:15:391
27939992012,6cyclictest7843-21seq12:15:391
27939992012,6cyclictest3239-21date07:25:251
27939992012,6cyclictest11344-21cat09:30:211
27939992011,6cyclictest7435-21sensors_temp07:35:231
27939992011,6cyclictest7435-21sensors_temp07:35:221
27939992011,6cyclictest31966-21seq08:31:031
2793999201,14cyclictest26269-21ssh10:30:121
2793899208,3cyclictest12-21ksoftirqd/012:25:120
2793899207,5cyclictest12-21ksoftirqd/011:39:020
2793899207,3cyclictest12-21ksoftirqd/012:13:100
2793899207,2cyclictest12-21ksoftirqd/010:35:420
2793899204,3cyclictest141rcu_preempt11:20:120
27938992014,5cyclictest12-21ksoftirqd/008:10:000
27938992014,4cyclictest12-21ksoftirqd/011:05:010
27938992012,6cyclictest14693-21smartctl10:45:150
27938992012,5cyclictest278-21systemd-journal08:05:000
27938992010,7cyclictest4523-21apt-get11:50:140
27938992010,2cyclictest12-21ksoftirqd/012:15:530
27938992010,2cyclictest12-21ksoftirqd/012:15:520
2794199192,12cyclictest30401-21apt-get08:30:153
2794199192,12cyclictest16753-21apt-get11:35:133
2794199191,4cyclictest0-21swapper/309:48:213
27941991914,3cyclictest30956-21kernelversion11:45:193
27941991914,3cyclictest22935-21sendmail_mailtr08:10:243
27941991913,4cyclictest636-21runrttasks12:26:513
27941991913,4cyclictest26007-21sh12:05:233
27941991912,5cyclictest4691-21cat08:45:113
27941991912,4cyclictest15092-21diskmemload10:55:043
27941991911,6cyclictest5068-21expr09:50:133
27941991911,6cyclictest3785-21users08:40:253
27941991911,6cyclictest26395-21expr10:30:133
27941991911,6cyclictest23694-21cat11:40:143
27941991911,6cyclictest19525-21seq09:36:073
27941991911,6cyclictest19207-21date10:00:193
27941991911,6cyclictest19207-21date10:00:183
27941991911,6cyclictest17207-21cat08:00:013
2794199191,15cyclictest25228-21switchtime09:40:253
27941991910,7cyclictest31953-21expr10:10:123
27941991910,7cyclictest10411-21diskstats11:30:153
27941991910,7cyclictest10411-21diskstats11:30:143
27941991910,6cyclictest7117-21grep10:40:013
2794099192,2cyclictest141rcu_preempt11:35:132
2794099192,12cyclictest22072-21ssh11:15:032
27940991913,4cyclictest28690-21latency07:10:172
27940991912,4cyclictest636-21runrttasks09:35:142
27940991912,4cyclictest15092-21diskmemload12:26:072
27940991911,6cyclictest9706-21cat08:55:222
27940991911,6cyclictest14626-21seq07:51:272
2794099191,15cyclictest28-21ksoftirqd/207:45:192
27940991910,6cyclictest19897-21cut08:05:182
2793999195,10cyclictest52850irq/140-enp3s0-11:35:291
2793999191,7cyclictest15092-21diskmemload11:13:581
27939991913,4cyclictest22468-21/usr/sbin/munin09:15:241
27939991913,4cyclictest19079-21sh10:24:581
27939991913,4cyclictest1287-21sh09:23:051
27939991912,5cyclictest4983-21awk11:25:271
27939991912,5cyclictest15456-21expr07:55:151
27939991912,4cyclictest2322-21unixbench_singl10:35:261
27939991912,4cyclictest15092-21diskmemload10:13:201
27939991911,6cyclictest19397-21seq09:35:591
27939991911,6cyclictest1937-21seq08:39:171
27939991911,6cyclictest11964-21cat07:45:241
2793999191,15cyclictest52850irq/140-enp3s0-09:10:171
27939991910,7cyclictest29469-21awk07:10:241
2793899199,8cyclictest6257-21/usr/sbin/munin07:35:200
2793899199,8cyclictest6257-21/usr/sbin/munin07:35:200
2793899198,9cyclictest25221-21ssh09:16:260
2793899197,2cyclictest12-21ksoftirqd/012:35:130
2793899193,13cyclictest11979-21sh09:55:140
2793899191,2cyclictest141rcu_preempt10:25:200
27938991911,6cyclictest8120-21cat08:50:240
27938991911,6cyclictest28313-21awk08:25:140
27938991911,6cyclictest28313-21awk08:25:130
27938991911,6cyclictest23090-21date11:15:190
27938991911,6cyclictest22155-21entropy08:10:150
27938991911,6cyclictest10696-21seq08:58:590
27938991910,7cyclictest18444-21sort10:00:140
27938991910,7cyclictest18444-21sort10:00:130
27938991910,6cyclictest26199-21grep08:20:130
2793899190,8cyclictest11225-21head07:45:140
264772190,16sleep20-21swapper/209:17:412
71262180,2sleep20-21swapper/207:35:192
71262180,2sleep20-21swapper/207:35:192
2794199189,6cyclictest7621-21cat10:40:143
2794199189,5cyclictest52750irq/139-enp3s0-12:33:433
2794199182,11cyclictest8546-21apt-get07:40:133
2794199182,10cyclictest29295-21sh09:20:113
27941991813,3cyclictest26337-21sh09:17:303
27941991813,3cyclictest10674-21sh11:05:473
27941991812,4cyclictest18794-21sh09:11:453
27941991812,4cyclictest12823-21sh09:31:093
27941991812,4cyclictest10385-21sh11:54:363
27941991811,5cyclictest7434-21mailstats07:35:233
27941991811,5cyclictest7434-21mailstats07:35:223
27941991810,6cyclictest1273-21python308:35:233
2794099189,7cyclictest7230-21fw_packets08:50:162
2794099189,7cyclictest13197-21apt09:05:092
2794099188,7cyclictest278-21systemd-journal07:35:012
2794099187,7cyclictest53050irq/142-enp3s0-11:25:122
2794099181,8cyclictest11837-21/usr/sbin/munin11:55:162
27940991813,3cyclictest31339-21/usr/sbin/munin12:10:182
27940991813,3cyclictest2844-21hwlatdetect08:40:162
27940991812,4cyclictest5743-21sh09:25:482
27940991812,4cyclictest15680-21sh11:10:172
27940991812,4cyclictest15092-21diskmemload10:28:012
27940991812,3cyclictest10649-21sh11:30:162
27940991812,3cyclictest10649-21sh11:30:162
27940991811,5cyclictest15382-21expr07:55:152
27940991811,4cyclictest4951-21sh08:45:122
2794099181,14cyclictest25990-21ssh12:05:242
27940991811,4cyclictest15092-21diskmemload11:49:212
27940991811,4cyclictest15092-21diskmemload10:55:572
2794099181,12cyclictest32662-21latency_hist10:35:012
2793999188,8cyclictest12420-21idleruntime-cro10:20:011
27939991813,3cyclictest9116-21sh09:52:551
27939991813,3cyclictest21984-21sh12:02:461
27939991813,3cyclictest14103-21ntp_offset09:05:181
27939991812,4cyclictest15610-21sh09:57:421
27939991811,5cyclictest12753-21seq09:02:401
27939991811,5cyclictest10149-21threads08:55:281
2793999181,12cyclictest381-21apt-get12:35:151
27939991810,6cyclictest5369-21cat08:45:171
27939991810,6cyclictest4203-21fw_packets09:25:171
27939991810,6cyclictest29213-21pluginstate08:25:231
27939991810,6cyclictest29213-21pluginstate08:25:221
27939991810,6cyclictest10485-21date07:45:001
27939991810,5cyclictest17317-21cat08:00:121
2793899189,7cyclictest29047-21cut07:10:210
2793899189,7cyclictest10574-21grep07:45:010
2793899184,4cyclictest1450rcu_preempt11:41:140
2793899182,12cyclictest12865-21ssh10:20:120
27938991813,3cyclictest30376-21/usr/sbin/munin08:30:250
27938991813,3cyclictest14124-21sendmail_mailst07:50:220
27938991813,3cyclictest12-21ksoftirqd/008:45:010
27938991812,4cyclictest14096-21ntp_kernel_pll_09:05:190
27938991811,5cyclictest13101-21date09:05:000
27938991810,6cyclictest16265-21which07:55:240
2794199173,11cyclictest10681-21apt-get07:45:103
2794199170,4cyclictest9108-21expr08:55:173
2794099179,6cyclictest3613-21awk11:00:282
2794099179,6cyclictest24735-21sed08:15:202
2794099179,6cyclictest23762-21wc08:15:002
2794099177,8cyclictest1674-21cut12:35:242
2794099171,11cyclictest17947-21apt-get12:00:132
27940991710,4cyclictest15092-21diskmemload10:05:042
2793999178,6cyclictest0-21swapper/108:09:591
27939991712,3cyclictest26089-21/usr/sbin/munin08:20:231
2793899173,8cyclictest8561-21apt-get11:05:120
27938991712,3cyclictest32514-21/usr/sbin/munin08:35:140
27941991611,3cyclictest17304-21/usr/sbin/munin08:00:223
2794099169,5cyclictest858-21sed08:35:192
2794099169,5cyclictest11942-21awk09:00:202
27940991612,2cyclictest17304-21/usr/sbin/munin08:00:132
2794099161,13cyclictest0-21swapper/209:20:032
27940991610,4cyclictest28972-21latency08:25:192
27940991610,4cyclictest28972-21latency08:25:182
2793999169,5cyclictest22387-21cat08:10:171
2793999162,4cyclictest22-21ksoftirqd/108:19:591
27941991510,3cyclictest28611-21forks08:25:153
27941991510,3cyclictest28611-21forks08:25:153
27940991510,3cyclictest31794-21unixbench-2d08:30:272
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional