You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 01:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Fri May 22, 2026 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30475217097,26sleep30-21swapper/307:05:523
286602168100,28sleep00-21swapper/007:05:000
307172162104,27sleep20-21swapper/207:09:002
307162162105,28sleep10-21swapper/107:08:591
502421500,1sleep30-21swapper/311:24:103
25682700,1chrt0-21swapper/209:45:212
138532700,2chrt0-21swapper/210:18:032
6692680,2sleep10-21swapper/112:08:341
324272630,2chrt0-21swapper/011:20:140
83002560,9sleep334-21ksoftirqd/311:25:423
213892520,1sleep20-21swapper/209:11:022
148562520,2sleep30-21swapper/309:30:213
112562500,4sleep111250-21cat12:40:011
128212490,2chrt0-21swapper/009:53:210
66752480,2sleep30-21swapper/309:48:553
253592430,2chrt0-21swapper/109:38:381
73952390,3chrt0-21swapper/111:50:081
3107199381,29cyclictest141rcu_preempt09:16:562
74132360,24sleep228-21ksoftirqd/211:50:092
31069993530,3cyclictest18431-21diskmemload10:30:120
300742340,2sleep00-21swapper/010:54:410
3107099321,28cyclictest18431-21diskmemload10:53:561
3107299316,23cyclictest0-21swapper/310:53:103
3107199311,27cyclictest31813-21/usr/sbin/munin10:55:192
3107199311,27cyclictest31813-21/usr/sbin/munin10:55:192
31069993120,9cyclictest1934-21ssh10:09:330
3106999310,29cyclictest29448-21ssh09:17:220
133982310,3chrt0-21swapper/207:40:252
3107299301,26cyclictest5908-21hddtemp_smartct07:25:143
31071993020,3cyclictest27138-21ssh12:27:332
3107099301,27cyclictest27151-21ssh09:15:231
3107099301,26cyclictest32456-21apt-get09:20:101
125232300,2sleep10-21swapper/107:40:161
3107299291,25cyclictest3863-21irqstats07:20:163
31070992923,4cyclictest52950irq/141-enp3s0-10:56:021
31070992923,4cyclictest52950irq/141-enp3s0-10:56:021
31069992925,2cyclictest30660-21ssh10:55:040
31069992925,2cyclictest30660-21ssh10:55:040
3106999291,4cyclictest28505-21ssh10:29:170
3107299281,3cyclictest30688-21/usr/sbin/munin10:30:203
3107299280,26cyclictest31224-21ssh09:19:123
31071992823,3cyclictest31270-21sed10:30:262
3107099281,2cyclictest0-21swapper/110:27:431
3107099281,25cyclictest23129-21ssh09:12:471
3107099281,25cyclictest0-21swapper/110:35:241
3107099281,25cyclictest0-21swapper/108:16:151
3107099281,25cyclictest0-21swapper/107:27:041
3106999281,25cyclictest0-21swapper/009:20:090
3106999281,25cyclictest0-21swapper/007:22:390
52472270,2chrt0-21swapper/111:24:241
313372270,2sleep10-21swapper/111:43:341
313372270,2sleep10-21swapper/111:43:341
3107299271,24cyclictest18431-21diskmemload10:55:033
3107299271,24cyclictest18431-21diskmemload10:55:033
3107299270,24cyclictest23747-21ssh09:13:233
3107299270,24cyclictest1763-21cut08:30:163
3107199271,25cyclictest29176-21ssh10:53:482
3107199271,24cyclictest0-21swapper/208:07:482
31070992724,2cyclictest0-21swapper/108:08:181
31070992723,2cyclictest0-21swapper/107:48:001
31070992723,2cyclictest0-21swapper/107:20:201
3107099271,24cyclictest0-21swapper/110:30:131
31069992724,2cyclictest0-21swapper/009:10:180
31069992723,2cyclictest0-21swapper/008:07:210
3106999271,25cyclictest0-21swapper/008:15:540
3106999271,24cyclictest0-21swapper/007:43:360
109832270,4sleep20-21swapper/212:39:582
3107299261,23cyclictest0-21swapper/308:28:383
3107299261,23cyclictest0-21swapper/308:10:093
3107299261,23cyclictest0-21swapper/308:07:273
3107299261,23cyclictest0-21swapper/308:03:593
3107299261,23cyclictest0-21swapper/307:43:283
3107299260,3cyclictest16861-21ssh10:44:533
31071992623,2cyclictest0-21swapper/210:46:082
3107199261,2cyclictest0-21swapper/207:25:082
3107199261,23cyclictest0-21swapper/209:20:242
3107199261,23cyclictest0-21swapper/208:30:212
3107199261,23cyclictest0-21swapper/208:15:092
3107199261,23cyclictest0-21swapper/208:10:302
3107199261,15cyclictest31525-21sendmail_mailqu12:30:222
3107199261,15cyclictest18431-21diskmemload11:42:232
3107199261,15cyclictest18431-21diskmemload11:42:232
31070992618,6cyclictest18640-21ssh09:57:291
3107099261,1cyclictest0-21swapper/110:45:221
3107099261,1cyclictest0-21swapper/108:22:171
3107099261,1cyclictest0-21swapper/108:10:281
3107099260,24cyclictest0-21swapper/108:01:451
31069992624,1cyclictest0-21swapper/008:21:540
31069992623,2cyclictest0-21swapper/010:44:440
31069992623,2cyclictest0-21swapper/008:30:190
31069992623,2cyclictest0-21swapper/007:25:400
31069992619,5cyclictest7267-21ssh12:13:210
31069992614,6cyclictest12-21ksoftirqd/011:09:580
3106999261,24cyclictest0-21swapper/010:45:220
3106999261,1cyclictest0-21swapper/010:35:220
3106999261,1cyclictest0-21swapper/007:48:590
3107299251,1cyclictest0-21swapper/308:15:593
3107299250,2cyclictest27069-21ssh10:27:503
3107299250,23cyclictest0-21swapper/310:45:053
31071992523,1cyclictest0-21swapper/207:48:402
3107199251,23cyclictest0-21swapper/210:27:422
3107199251,1cyclictest0-21swapper/210:35:112
3107199251,14cyclictest18431-21diskmemload10:24:012
3107199251,10cyclictest0-21swapper/212:15:232
3107199250,23cyclictest0-21swapper/208:20:232
3107199250,23cyclictest0-21swapper/207:20:202
31070992523,1cyclictest0-21swapper/111:00:591
3106999259,5cyclictest12-21ksoftirqd/010:25:000
31069992523,1cyclictest0-21swapper/008:10:270
31069992523,1cyclictest0-21swapper/008:01:460
185932250,1chrt0-21swapper/009:33:350
31072992422,1cyclictest0-21swapper/310:35:173
31072992422,1cyclictest0-21swapper/308:22:013
3107299240,1cyclictest0-21swapper/309:20:173
31070992422,1cyclictest0-21swapper/110:44:351
31070992422,1cyclictest0-21swapper/108:30:191
31070992421,2cyclictest0-21swapper/108:27:471
3106999248,3cyclictest66050lldpd10:18:180
31069992413,8cyclictest16860-21df09:05:140
93082230,1sleep10-21swapper/112:15:091
31072992321,1cyclictest0-21swapper/307:48:203
31072992314,7cyclictest14458-21cron09:55:013
31070992316,5cyclictest22-21ksoftirqd/107:55:001
31070992316,5cyclictest22-21ksoftirqd/107:55:001
31069992314,6cyclictest560-21dbus-daemon11:40:000
31069992311,10cyclictest13660-21sh11:30:170
271002230,8sleep334-21ksoftirqd/309:40:083
169242230,3sleep334-21ksoftirqd/309:55:453
12612230,2sleep30-21swapper/310:08:543
31072992212,7cyclictest16798-21cat12:20:143
3107199229,10cyclictest4575-21ntp_offset08:35:202
3107199224,10cyclictest0-21swapper/210:41:592
3107199221,7cyclictest21658-21switchtime08:00:232
31071992217,2cyclictest18431-21diskmemload09:30:252
31071992213,5cyclictest636-21runrttasks09:50:212
3107199221,13cyclictest716-21open_inodes11:20:212
3107199221,12cyclictest18431-21diskmemload11:27:492
3107099224,4cyclictest18431-21diskmemload09:52:481
3107099224,3cyclictest18431-21diskmemload10:14:221
3107099223,3cyclictest18431-21diskmemload11:35:151
3107099223,17cyclictest20025-21awk09:34:591
3107099222,4cyclictest18431-21diskmemload12:12:511
194402220,2sleep20-21swapper/207:55:222
31072992113,6cyclictest32040-21open_files07:10:213
31072992113,6cyclictest28746-21seq12:29:083
31072992112,7cyclictest18949-21diskstats11:10:143
31072992112,6cyclictest53050irq/142-enp3s0-10:00:113
3107199212,13cyclictest31528-21ssh12:07:032
3107199212,13cyclictest2571-21taskset12:10:122
3107199212,10cyclictest0-21swapper/211:10:252
3107199211,9cyclictest0-21swapper/210:05:382
3107199211,9cyclictest0-21swapper/209:40:132
31071992112,6cyclictest278-21systemd-journal12:20:032
3107199211,11cyclictest0-21swapper/211:57:552
3107199211,10cyclictest0-21swapper/211:31:382
3107199210,12cyclictest13120-21grep11:05:222
3107199210,11cyclictest24652-21taskset11:38:382
3107099213,3cyclictest18431-21diskmemload11:59:481
3107099213,3cyclictest18431-21diskmemload10:22:111
3107099212,3cyclictest18431-21diskmemload11:27:191
3107099212,3cyclictest18431-21diskmemload09:43:171
3107099211,6cyclictest0-21swapper/111:31:321
3107099211,6cyclictest0-21swapper/110:03:391
31070992113,6cyclictest11025-21ntp_states08:50:211
3106999217,10cyclictest52750irq/139-enp3s0-09:25:160
3106999217,10cyclictest52750irq/139-enp3s0-09:25:160
31069992119,1cyclictest0-21swapper/011:00:590
3106999211,3cyclictest141rcu_preempt09:58:440
3106999211,3cyclictest141rcu_preempt09:38:170
31069992113,6cyclictest7065-21ls08:40:240
31069992113,6cyclictest7065-21ls08:40:240
31069992113,6cyclictest1392-21cat07:15:150
3106999211,2cyclictest141rcu_preempt12:23:590
272342210,10sleep122-21ksoftirqd/112:04:451
157692210,2sleep00-21swapper/011:55:250
31072992014,4cyclictest5342-21sh11:48:193
31072992013,5cyclictest1610-21smartctl07:15:173
31072992013,5cyclictest12297-21sh09:29:133
31072992013,5cyclictest12297-21sh09:29:133
31072992013,4cyclictest15078-21sh10:19:183
31072992012,6cyclictest560-21dbus-daemon09:40:003
31072992012,6cyclictest3810-21ntpq10:10:193
31072992012,5cyclictest5804-21smartctl11:00:173
31072992011,7cyclictest8883-21ntp_states08:45:213
31072992011,7cyclictest16342-21expr07:50:133
31072992011,7cyclictest16342-21expr07:50:133
31072992011,6cyclictest19218-21ntp_states07:55:203
31072992010,7cyclictest53050irq/142-enp3s0-11:40:133
31072992010,7cyclictest53050irq/142-enp3s0-11:40:133
31072992010,6cyclictest53050irq/142-enp3s0-12:07:033
3107199204,10cyclictest0-21swapper/211:18:342
3107199201,9cyclictest0-21swapper/210:10:222
3107199201,8cyclictest0-21swapper/212:00:202
3107199201,7cyclictest5259-21ssh11:00:132
31071992015,3cyclictest31982-21idleruntime08:25:172
31071992011,7cyclictest3147-21cat07:20:012
3107199201,11cyclictest17193-21ssh09:56:012
3107199200,11cyclictest21487-21ssh09:35:202
3107099201,5cyclictest0-21swapper/112:21:341
3107099201,5cyclictest0-21swapper/111:46:391
3107099201,5cyclictest0-21swapper/110:15:101
3107099201,5cyclictest0-21swapper/109:28:271
3107099201,5cyclictest0-21swapper/109:28:271
31070992015,3cyclictest14132-21sh09:00:001
31070992012,6cyclictest31380-21ls07:10:121
31070992012,6cyclictest278-21systemd-journal08:40:061
31070992012,6cyclictest278-21systemd-journal08:40:061
3107099200,6cyclictest0-21swapper/111:09:281
3107099200,5cyclictest22-21ksoftirqd/112:27:331
3106999207,9cyclictest52750irq/139-enp3s0-12:38:240
3106999207,9cyclictest52750irq/139-enp3s0-09:41:430
3106999202,3cyclictest141rcu_preempt11:50:550
3106999202,14cyclictest18431-21diskmemload11:10:150
31069992018,1cyclictest0-21swapper/008:27:470
31069992013,5cyclictest29630-21apt-get12:30:020
31069992013,5cyclictest10891-21ls12:15:240
3106999201,2cyclictest141rcu_preempt12:25:110
3106999201,2cyclictest141rcu_preempt12:05:220
3106999201,2cyclictest141rcu_preempt08:50:170
31069992012,6cyclictest26621-21sensors_temp11:15:220
31069992012,6cyclictest2301-21cat11:45:230
31069992012,6cyclictest1853-21df09:45:140
31069992012,6cyclictest18303-21apt-get07:55:100
31072991913,4cyclictest16107-21sh11:55:323
31072991913,4cyclictest11346-21sh12:15:403
31072991913,4cyclictest10578-21sh12:39:343
31072991912,5cyclictest7070-21find08:40:243
31072991912,5cyclictest7070-21find08:40:233
31072991912,5cyclictest3677-21awk12:10:223
31072991912,5cyclictest13195-21pmu-power08:55:213
31072991911,6cyclictest4789-21grep08:35:243
31072991911,6cyclictest16539-21apt-get09:05:113
3107199198,6cyclictest21608-21chrt10:00:022
3107199191,9cyclictest0-21swapper/208:40:222
3107199191,9cyclictest0-21swapper/208:40:222
3107199191,8cyclictest0-21swapper/207:39:552
31071991911,6cyclictest8040-21cat08:45:132
31071991911,6cyclictest1499-21sed11:45:162
3107099193,7cyclictest26312-21ntpq11:15:191
3107099192,4cyclictest18431-21diskmemload10:07:371
3107099192,15cyclictest1852-21ssh09:45:141
3106999198,8cyclictest9633-21apt-get07:35:140
3106999192,2cyclictest141rcu_preempt11:42:060
3106999192,2cyclictest141rcu_preempt11:42:060
3106999192,12cyclictest14401-21apt-get09:00:120
3106999191,2cyclictest141rcu_preempt11:25:110
31069991912,5cyclictest9366-21cron07:35:000
31069991912,5cyclictest8871-21ntpq08:45:200
3106999191,16cyclictest21614-21cut12:00:160
3106999190,2cyclictest141rcu_preempt10:00:330
31072991813,3cyclictest24059-21sh12:01:383
31072991813,3cyclictest18094-21sh10:20:343
31072991813,3cyclictest17356-21sh11:09:123
31072991813,3cyclictest11739-21sh11:52:563
31072991812,4cyclictest25118-21sh11:15:023
31072991812,4cyclictest22563-21sh11:36:313
31072991812,4cyclictest10459-21needreboot07:35:213
31072991812,3cyclictest1814-21sh12:32:373
31072991811,5cyclictest8347-21netstat07:30:183
31072991811,5cyclictest34-21ksoftirqd/311:30:163
31072991811,5cyclictest16331-21date09:05:013
31072991810,6cyclictest10114-21grep08:50:123
31071991811,5cyclictest16128-21seq07:50:022
31071991811,5cyclictest16128-21seq07:50:022
31071991810,6cyclictest28-21ksoftirqd/208:55:012
31071991810,5cyclictest278-21systemd-journal07:33:032
3107199180,8cyclictest0-21swapper/207:10:242
3107099189,6cyclictest52950irq/141-enp3s0-12:30:101
3107099189,6cyclictest4128-21tune2fs08:35:171
3107099182,14cyclictest3027-21wc07:20:011
3107099181,7cyclictest22214-21ssh11:12:211
31070991811,5cyclictest8679-21sed08:45:181
31070991811,5cyclictest18127-21seq09:06:221
31070991811,5cyclictest10416-21sed07:35:211
3106999181,8cyclictest6749-21ssh10:12:390
31069991812,3cyclictest12-21ksoftirqd/007:10:010
31069991811,5cyclictest13380-21tr08:55:220
3106999181,12cyclictest16167-21apt-get07:50:120
3106999181,12cyclictest16167-21apt-get07:50:110
3107199176,9cyclictest0-21swapper/209:25:372
3107199176,9cyclictest0-21swapper/209:25:372
3107199170,8cyclictest16369-21cat09:05:012
3107099179,6cyclictest15316-21seq09:00:201
3107099177,2cyclictest22-21ksoftirqd/107:57:371
3106999172,10cyclictest3213-21apt-get08:35:030
3107199169,5cyclictest12354-21expr08:55:122
31071991610,4cyclictest17136-21basename09:05:172
31070991510,3cyclictest7466-21/usr/sbin/munin07:30:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional