You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-18 - 11:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Tue Aug 18, 2026 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75482175153,15sleep00-21swapper/019:06:020
7614217096,25sleep20-21swapper/219:06:532
5953215864,15sleep10-21swapper/119:05:081
76352141105,25sleep30-21swapper/319:07:103
47292610,2sleep122-21ksoftirqd/120:15:201
53502600,1sleep00-21swapper/000:29:180
53502600,1sleep00-21swapper/000:29:170
123862580,2sleep00-21swapper/000:10:160
243152570,3sleep1211rcuc/122:43:011
175612550,2sleep00-21swapper/000:38:010
135422540,6sleep3331rcuc/300:10:263
271592520,1sleep10-21swapper/100:20:361
151602500,2sleep00-21swapper/022:35:370
55802490,2sleep10-21swapper/122:53:161
235322480,2sleep122-21ksoftirqd/121:55:011
311282460,1sleep10-21swapper/121:35:351
92232450,2sleep20-21swapper/222:31:262
812699342,22cyclictest12481-21ssh22:10:140
812699314,9cyclictest11074-21/usr/sbin/munin23:45:170
812699314,9cyclictest11074-21/usr/sbin/munin23:45:170
812699311,20cyclictest12-21ksoftirqd/023:57:480
74572310,2chrt0-21swapper/321:19:103
812699305,7cyclictest14713-21ssh23:24:210
201892300,2sleep10-21swapper/123:04:241
87032290,2sleep30-21swapper/322:30:543
8126992926,2cyclictest131rcuc/021:56:310
812699290,9cyclictest9044-21grep22:55:230
812699283,9cyclictest24574-21ssh00:19:470
8126992824,3cyclictest131rcuc/021:51:370
812699280,10cyclictest12577-21sh21:22:310
812999271,24cyclictest0-21swapper/319:16:013
8128992719,6cyclictest10783-21ssh21:20:432
812799271,24cyclictest0-21swapper/119:15:051
812799271,12cyclictest243392tail23:55:171
812699277,7cyclictest12-21ksoftirqd/022:31:230
812699274,6cyclictest27183-21diskmemload23:04:550
812699273,7cyclictest21781-21grep23:05:140
812699272,6cyclictest5497-21ssh21:17:120
812699271,9cyclictest27183-21diskmemload21:13:550
297992270,2chrt0-21swapper/320:00:013
18932270,2chrt0-21swapper/221:15:092
812699264,7cyclictest25091-21perf19:50:000
812699263,8cyclictest12-21ksoftirqd/022:01:070
812699263,6cyclictest12-21ksoftirqd/000:02:420
812699263,16cyclictest4897-21ssh00:05:150
812699262,8cyclictest9062-21ntpq20:25:220
812699262,8cyclictest20328-21sed19:35:250
812699262,6cyclictest10605-21cstates21:45:150
812699262,6cyclictest0-21swapper/021:42:240
812699262,5cyclictest0-21swapper/000:32:440
8126992622,3cyclictest131rcuc/021:34:280
812699261,18cyclictest0-21swapper/019:16:150
812699260,9cyclictest27183-21diskmemload23:38:300
8129992515,7cyclictest14938-21df19:25:163
812999251,12cyclictest201902ssh23:04:243
8128992523,1cyclictest0-21swapper/219:18:482
8128992515,6cyclictest30494-21apt-get00:00:112
812799258,5cyclictest22-21ksoftirqd/123:54:381
812799256,9cyclictest141rcu_preempt22:15:531
812799256,9cyclictest141rcu_preempt22:15:521
812699253,8cyclictest25409-21ssh23:31:370
812699252,9cyclictest28795-21ssh22:45:460
812699252,9cyclictest12-21ksoftirqd/000:21:130
812699251,7cyclictest1511-21ssh23:14:410
812699251,6cyclictest20081-21ssh21:28:140
812699251,6cyclictest19034-21cat20:50:140
812699250,9cyclictest22231-21ssh23:53:560
812699250,9cyclictest20108-21ssh22:15:240
812699250,9cyclictest20108-21ssh22:15:240
812699250,7cyclictest12-21ksoftirqd/023:17:390
812699250,6cyclictest0-21swapper/022:20:250
8129992415,6cyclictest6423-21sed00:05:283
812899246,3cyclictest66050lldpd00:29:242
812899246,3cyclictest66050lldpd00:29:242
8128992414,8cyclictest25418-21ssh23:31:372
812799243,10cyclictest22-21ksoftirqd/122:38:341
8127992415,5cyclictest0-21swapper/121:15:131
812699244,6cyclictest27183-21diskmemload23:29:310
812699243,8cyclictest3833-21awk20:15:120
812699243,7cyclictest12-21ksoftirqd/022:28:200
812699242,7cyclictest12-21ksoftirqd/022:41:070
8126992419,3cyclictest131rcuc/022:07:100
812699240,8cyclictest3469-21sh23:40:070
20152240,2sleep10-21swapper/120:10:151
143062240,1sleep30-21swapper/322:11:003
8129992310,6cyclictest23975-21sed23:30:243
812799239,2cyclictest22-21ksoftirqd/121:58:481
812799238,2cyclictest141rcu_preempt00:05:151
812799237,10cyclictest22-21ksoftirqd/122:05:221
812799236,9cyclictest141rcu_preempt21:12:301
812799235,8cyclictest141rcu_preempt23:17:161
8127992315,6cyclictest22-21ksoftirqd/121:25:001
812799231,10cyclictest31662sh00:27:071
812799231,10cyclictest31662sh00:27:061
812699233,9cyclictest12-21ksoftirqd/021:36:280
812699233,17cyclictest6519-21sed20:20:170
812699232,6cyclictest14968-21fschecks_count19:25:170
812699232,5cyclictest0-21swapper/022:50:160
812699231,7cyclictest12-21ksoftirqd/019:55:170
20172230,9sleep122-21ksoftirqd/123:38:551
163382230,4sleep2271rcuc/221:25:202
133552230,2sleep30-21swapper/323:23:003
8129992214,6cyclictest18608-21ls23:50:263
812899229,4cyclictest181072sed20:45:262
812899223,13cyclictest27183-21diskmemload00:35:132
812799226,4cyclictest22-21ksoftirqd/121:45:181
812799224,6cyclictest22-21ksoftirqd/122:33:171
812799223,3cyclictest17384-21runrttasks00:14:191
8127992213,6cyclictest8497-21df_inode19:10:151
8126992218,3cyclictest131rcuc/021:06:470
8126992218,3cyclictest131rcuc/020:38:450
327172220,9sleep334-21ksoftirqd/321:37:093
812999212,13cyclictest21875-21apt-get21:30:043
8129992112,6cyclictest6273-21cat00:30:013
8129992112,6cyclictest28828-21sendmail_mailqu21:10:203
8129992111,8cyclictest3196-21tr23:15:183
812899213,12cyclictest22343-21apt-get21:30:042
812899212,3cyclictest25009-21sendmail22:20:002
812899212,3cyclictest25009-21sendmail22:19:592
812899212,17cyclictest13848-21grep20:35:252
812899212,13cyclictest25479-21ssh22:20:132
812899211,5cyclictest0-21swapper/223:03:052
8128992113,6cyclictest11207-21ntpq20:30:212
8128992111,8cyclictest20380-21df22:40:152
812799219,3cyclictest22-21ksoftirqd/123:30:271
812799219,2cyclictest141rcu_preempt00:00:181
812799218,10cyclictest22-21ksoftirqd/123:20:121
812799217,2cyclictest141rcu_preempt22:02:411
812799213,5cyclictest22-21ksoftirqd/123:29:361
8127992115,5cyclictest22-21ksoftirqd/122:24:591
8127992113,6cyclictest25675-21forks19:50:141
8127992113,6cyclictest25675-21forks19:50:131
8127992112,6cyclictest15541-21acpi21:25:131
812699212,6cyclictest12-21ksoftirqd/019:21:430
812699212,10cyclictest12-21ksoftirqd/019:30:120
8126992118,2cyclictest131rcuc/019:53:390
8126992118,2cyclictest131rcuc/019:53:390
812699211,7cyclictest12-21ksoftirqd/021:01:450
812699211,7cyclictest12-21ksoftirqd/019:43:490
8126992117,3cyclictest131rcuc/020:45:280
8126992117,3cyclictest131rcuc/020:10:040
8126992117,3cyclictest131rcuc/020:02:260
812699210,7cyclictest12-21ksoftirqd/020:42:400
812699210,7cyclictest12-21ksoftirqd/020:42:390
812699210,7cyclictest12-21ksoftirqd/020:08:580
6712210,1sleep30-21swapper/322:25:213
324722210,1sleep30-21swapper/323:36:563
812999203,15cyclictest173932sleep300:37:523
812999202,13cyclictest5058-21ssh22:05:123
812999202,13cyclictest28014-21ssh23:10:143
812999201,6cyclictest27547-21munin-run23:09:593
8129992013,5cyclictest4348-21grep20:15:173
8129992012,6cyclictest28043-21seq21:57:493
8129992012,6cyclictest15124-21seq23:48:333
8129992012,6cyclictest15124-21seq23:48:323
8129992012,6cyclictest12421-21apt20:35:103
8129992012,5cyclictest15165-21idleruntime20:40:183
8129992012,5cyclictest15165-21idleruntime20:40:173
8129992011,7cyclictest22934-21ls21:00:013
8129992011,7cyclictest22195-21wc23:30:013
8129992011,7cyclictest13275-21cat19:20:183
8129992011,6cyclictest638-21cut22:50:003
8129992011,6cyclictest18024-21missed_timers21:50:183
812899203,14cyclictest8325-21gltestperf22:55:162
812899201,6cyclictest9052-21ssh22:07:422
812899201,5cyclictest0-21swapper/223:50:202
8128992015,4cyclictest28-21ksoftirqd/200:25:002
8128992015,3cyclictest27185-21sh21:10:012
8128992013,5cyclictest16778-21date21:50:002
8128992012,6cyclictest8968-21ls20:25:212
8128992012,6cyclictest31108-21diskstats22:00:162
8128992012,6cyclictest30290-21awk21:35:212
8128992011,7cyclictest23016-21cut21:00:012
8128992011,6cyclictest28301-21cat22:45:262
8128992011,6cyclictest24346-21sensors_temp19:45:222
8128992010,6cyclictest52750irq/139-enp3s0-23:10:132
812899200,7cyclictest8908-21ssh00:07:442
812799209,3cyclictest22-21ksoftirqd/120:20:201
812799205,3cyclictest141rcu_preempt21:32:581
812799204,2cyclictest141rcu_preempt22:55:121
8127992014,4cyclictest22-21ksoftirqd/120:49:591
812799201,3cyclictest141rcu_preempt00:38:101
8127992012,6cyclictest9697-21seq20:25:541
8127992012,6cyclictest14860-21seq22:11:331
8127992010,8cyclictest560-21dbus-daemon21:05:011
8127992010,7cyclictest21783-21cat19:40:191
8126992011,6cyclictest8636-21grep19:10:170
812699200,7cyclictest12-21ksoftirqd/020:57:050
812999191,9cyclictest20790-21munin-run19:40:003
8129991913,4cyclictest8685-21sh21:20:093
8129991913,4cyclictest4171-21grep23:40:153
8129991913,4cyclictest20927-21sh21:29:053
8129991913,4cyclictest14160-21sh21:47:283
8129991912,5cyclictest28051-21idleruntime19:55:183
8129991912,5cyclictest24989-21sed00:20:003
8129991911,6cyclictest32003-21cat00:00:263
8129991911,6cyclictest24401-21cat23:55:183
8129991910,7cyclictest17675-21pluginstate20:45:233
812899193,12cyclictest14585-21apt-get20:40:142
812899193,12cyclictest14585-21apt-get20:40:132
812899193,11cyclictest21056-21apt-get19:40:122
812899192,13cyclictest2240-21apt-get23:15:092
812899191,8cyclictest13110-21python300:10:232
8128991914,3cyclictest8143-21unixbench_singl00:30:252
8128991914,3cyclictest5648-21sh21:40:452
8128991914,3cyclictest29410-21sh23:59:232
8128991914,3cyclictest15812-21sh23:25:112
8128991913,5cyclictest28-21ksoftirqd/220:00:002
8128991911,6cyclictest27962-21forks21:10:112
8128991911,5cyclictest14278-21ntpq22:35:202
8128991910,6cyclictest560-21dbus-daemon23:25:012
8128991910,6cyclictest560-21dbus-daemon22:50:232
8128991910,6cyclictest1001-21grep20:05:242
812799198,2cyclictest22-21ksoftirqd/119:25:251
812799196,4cyclictest22-21ksoftirqd/123:10:521
812799194,3cyclictest141rcu_preempt22:25:241
812799193,2cyclictest141rcu_preempt23:05:111
812799191,7cyclictest141rcu_preempt23:45:131
812799191,7cyclictest141rcu_preempt23:45:131
812799191,4cyclictest0-21swapper/120:55:161
812799191,4cyclictest0-21swapper/119:45:101
8127991914,3cyclictest16801-21cpu19:30:131
8127991913,4cyclictest3636-21sh21:40:131
8127991913,4cyclictest30166-21sh20:00:151
8127991912,5cyclictest7594-21seq23:42:421
8127991912,5cyclictest20472-21seq19:36:111
8127991912,5cyclictest19164-21basename00:15:161
8127991911,6cyclictest14515-21ls20:40:001
8127991911,6cyclictest13700-21cat19:20:221
8127991911,6cyclictest11453-21awk20:30:251
8127991910,6cyclictest466-21pluginstate20:05:211
8126991915,3cyclictest131rcuc/020:32:120
812999181,8cyclictest13264-21ssh22:59:123
8129991813,3cyclictest31952-21/usr/sbin/munin20:05:223
8129991813,3cyclictest30689-21/usr/sbin/munin22:00:173
8129991813,3cyclictest27540-21sh22:20:463
8129991813,3cyclictest25244-21sh00:20:123
8129991813,3cyclictest25012-21dump-pmu-power22:20:003
8129991813,3cyclictest25012-21dump-pmu-power22:20:003
8129991813,3cyclictest22029-21pmu-power19:40:223
8129991811,5cyclictest11174-21ntpq20:30:213
812999181,14cyclictest335-21grep00:25:193
8129991810,6cyclictest25638-21entropy21:05:153
8129991810,6cyclictest23656-21fw_packets19:45:143
8129991810,6cyclictest18987-21expr20:50:153
8129991810,6cyclictest1878-21cat22:50:193
8129991810,6cyclictest13355-21ls22:35:133
8129991810,5cyclictest7401-21cat00:30:183
8129991810,5cyclictest10214-21date19:15:013
812899189,5cyclictest52750irq/139-enp3s0-22:25:132
812899187,3cyclictest141rcu_preempt00:15:172
812899182,11cyclictest1699-21apt-get20:10:152
812899181,8cyclictest30997-21python320:00:252
812899181,4cyclictest0-21swapper/220:20:162
8128991813,3cyclictest29226-21sh23:35:102
8128991813,3cyclictest19423-21sh21:50:582
8128991813,3cyclictest12423-21sh23:45:472
8128991813,3cyclictest12423-21sh23:45:462
8128991811,5cyclictest9088-21ntpq19:10:222
8128991811,5cyclictest22025-21cat23:05:172
8128991811,5cyclictest17851-21which19:30:222
812899181,14cyclictest4700-21grep23:40:212
8128991810,5cyclictest278-21systemd-journal19:29:552
812799184,4cyclictest22-21ksoftirqd/100:34:181
812799182,11cyclictest27471-21apt-get19:55:141
812799181,7cyclictest0-21swapper/122:45:251
8127991811,5cyclictest20865-21sed20:55:011
8127991810,6cyclictest25728-21tune2fs21:05:161
8127991810,6cyclictest14741-21cat20:40:141
8127991810,6cyclictest14741-21cat20:40:141
812999178,7cyclictest25752-21tune2fs19:50:143
812999178,7cyclictest25752-21tune2fs19:50:133
8129991712,3cyclictest23142-21acpi21:00:133
8129991711,4cyclictest23248-21sh22:41:573
8129991711,4cyclictest17877-21mailstats19:30:213
8129991710,5cyclictest6214-21expr20:20:153
8129991710,5cyclictest3558-21kworker/u8:021:40:333
8129991710,5cyclictest2348-21uname20:10:183
812899178,7cyclictest19216-21df19:35:142
812899173,12cyclictest12130-21cut22:10:112
8128991711,4cyclictest26569-21sh21:56:212
812899171,13cyclictest278-21systemd-journal20:20:012
812899171,11cyclictest18862-21apt-get20:50:152
8128991710,5cyclictest27257-21date19:55:012
8128991710,5cyclictest27257-21date19:55:002
8128991710,5cyclictest13169-21ls19:20:182
812999169,5cyclictest8456-21expr20:25:143
8128991611,3cyclictest23133-21/usr/sbin/munin21:00:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional