You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 11:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Fri May 08, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1150623726,0sleep30-21swapper/318:51:303
1722623526,0sleep20-21swapper/218:53:212
1108523223,0sleep10-21swapper/118:51:241
1742923026,0sleep00-21swapper/018:55:430
175489988,0cyclictest0-21swapper/020:56:350
175489976,0cyclictest0-21swapper/021:11:300
339960,0migration/37697-21sh21:10:563
339960,0migration/35915-21sh22:47:533
339960,0migration/35844-21sh22:26:593
339960,0migration/331474-21sh22:25:403
339960,0migration/330217-21sh00:25:393
339960,0migration/323000-21sh21:38:163
339960,0migration/322881-21sh23:00:253
339960,0migration/315308-21sh22:15:193
29715060,0irq/26-eth0-rx-4364-21pmu-power23:46:383
29715060,0irq/26-eth0-rx-29131-21needreboot22:41:343
29715060,0irq/26-eth0-rx-22802-21munin-run23:41:163
175509960,0cyclictest0-21swapper/223:47:512
149960,0migration/030065-21users22:21:400
149960,0migration/026332-21kthreadcore22:01:350
149960,0migration/022874-21sh22:39:430
149960,0migration/022730-21aten_r1power_en00:21:280
149960,0migration/022435-21latency_hist00:21:160
149960,0migration/013492-21sh23:12:170
693250,0sleep01153-21cat21:06:330
339950,0migration/3771-21kthreadcore00:06:353
339950,0migration/3640-21munin-node22:06:313
339950,0migration/330948-21idleruntime-cro23:26:163
339950,0migration/32867-21kthreadcore20:21:323
339950,0migration/328230-21kthreadcore22:01:363
339950,0migration/327247-21switchtime20:36:393
339950,0migration/323007-21runrttasks21:58:423
339950,0migration/32245-21latency_hist20:46:163
339950,0migration/322062-21taskset22:37:193
339950,0migration/32163-21kthreadcore21:46:323
339950,0migration/321030-21taskset20:33:173
339950,0migration/319790-21kthreadcore21:16:343
339950,0migration/318914-21cpuspeed_turbos19:21:283
339950,0migration/318406-21taskset19:21:163
339950,0migration/317219-21kthreadcore23:56:323
339950,0migration/314820-21sendmail_mailtr20:51:383
339950,0migration/314057-21timerwakeupswit20:26:383
339950,0migration/312603-21sh23:31:363
339950,0migration/31182-21taskset20:20:283
339950,0migration/311778-21kthreadcore22:51:343
339950,0migration/311275-21kthreadcore23:11:343
29725050,0irq/27-eth0-tx-6788-21munin-node23:51:252
29725050,0irq/27-eth0-tx-32470-21df_abs21:06:282
29725050,0irq/27-eth0-tx-30968-21munin-node00:06:332
29725050,0irq/27-eth0-tx-24603-21diskstats22:01:312
29725050,0irq/27-eth0-tx-24328-21munin-node21:41:392
29725050,0irq/27-eth0-tx-23911-21sshd18:58:290
29725050,0irq/27-eth0-tx-23461-21munin-node23:01:322
29725050,0irq/27-eth0-tx-20984-21python23:16:402
29725050,0irq/27-eth0-tx-15694-21grep23:36:272
29725050,0irq/27-eth0-tx-15673-21munin-node22:36:322
29725050,0irq/27-eth0-tx-0-21swapper/223:08:312
29725050,0irq/27-eth0-tx-0-21swapper/222:31:392
29725050,0irq/27-eth0-tx-0-21swapper/222:22:002
29725050,0irq/27-eth0-tx-0-21swapper/200:21:352
29725050,0irq/27-eth0-tx-0-21swapper/020:37:250
29725050,0irq/27-eth0-tx-0-21swapper/020:06:220
29725050,0irq/27-eth0-tx-0-21swapper/019:26:350
29725050,0irq/27-eth0-tx-0-21swapper/019:06:070
29715050,0irq/26-eth0-rx-8872-21kthreadcore19:36:333
29715050,0irq/26-eth0-rx-8246-21iostat_ios20:01:333
29715050,0irq/26-eth0-rx-25148-21idleruntime21:41:303
29715050,0irq/26-eth0-rx-23162-21ssh20:59:573
29715050,0irq/26-eth0-rx-19483-21kthreadcore22:16:353
29715050,0irq/26-eth0-rx-14891-21ssh21:53:403
29715050,0irq/26-eth0-rx-14040-21ssh22:32:243
29715050,0irq/26-eth0-rx-0-21swapper/323:44:433
29715050,0irq/26-eth0-rx-0-21swapper/323:06:273
29715050,0irq/26-eth0-rx-0-21swapper/323:02:173
29715050,0irq/26-eth0-rx-0-21swapper/321:11:533
29715050,0irq/26-eth0-rx-0-21swapper/321:04:433
29715050,0irq/26-eth0-rx-0-21swapper/300:19:003
29715050,0irq/26-eth0-rx-0-21swapper/300:11:193
29715050,0irq/26-eth0-rx-0-21swapper/300:11:193
27150,0ktimersoftd/20-21swapper/219:50:442
259950,0migration/28425-21munin-node21:11:282
259950,0migration/229613-21runrttasks23:22:272
259950,0migration/214236-21sh22:12:112
249950,0watchdog/223801-21if_err_eth100:01:302
249950,0watchdog/213136-21sendmail_mailst22:51:382
19150,0ktimersoftd/10-21swapper/121:50:281
175509955,0cyclictest0-21swapper/222:27:552
175509955,0cyclictest0-21swapper/221:34:102
175509955,0cyclictest0-21swapper/221:22:302
175509955,0cyclictest0-21swapper/221:05:282
175509955,0cyclictest0-21swapper/200:15:272
175509955,0cyclictest0-21swapper/200:15:272
175509954,0cyclictest3514-21snmpd22:58:472
175509950,0cyclictest0-21swapper/223:56:412
175509950,0cyclictest0-21swapper/223:15:422
175509950,0cyclictest0-21swapper/221:39:032
175499955,0cyclictest15602-21diskmemload23:21:551
175499955,0cyclictest0-21swapper/123:13:241
175499955,0cyclictest0-21swapper/123:07:241
175499955,0cyclictest0-21swapper/120:46:281
175499950,0cyclictest0-21swapper/122:39:021
175499950,0cyclictest0-21swapper/122:24:501
175499950,0cyclictest0-21swapper/121:09:291
175489955,0cyclictest0-21swapper/023:33:140
175489955,0cyclictest0-21swapper/021:48:040
175489954,0cyclictest11757-21sadc19:16:160
175489950,0cyclictest0-21swapper/023:39:320
175489950,0cyclictest0-21swapper/022:15:590
179950,0migration/16683-21latency_hist23:31:161
179950,0migration/132622-21munin-node22:06:281
179950,0migration/131811-21aten_r1power_vo22:46:271
179950,0migration/131608-21fschecks_time00:06:311
179950,0migration/130820-21sh22:03:081
179950,0migration/127440-21kthreadcore21:41:331
179950,0migration/126839-21taskset20:14:471
179950,0migration/121617-21taskset23:58:191
179950,0migration/116625-21munin-node20:56:281
169950,0watchdog/112311-21kthreadcore22:51:341
149950,0migration/09668-21kthreadcore20:26:320
149950,0migration/08538-21taskset20:46:520
149950,0migration/08332-21fschecks_time22:31:290
149950,0migration/06883-21taskset23:51:260
149950,0migration/06033-21taskset00:10:230
149950,0migration/04227-21kthreadcore22:26:330
149950,0migration/032196-21gltestperf23:06:290
149950,0migration/031117-21latency_hist23:26:160
149950,0migration/030837-21kthreadcore20:41:330
149950,0migration/030125-21taskset23:44:240
149950,0migration/028611-21missed_timers23:21:340
149950,0migration/027610-21kthreadcore22:41:330
149950,0migration/025590-21kthreadcore20:11:330
149950,0migration/02174-21kthreadcore21:26:340
149950,0migration/021174-21sh23:57:000
149950,0migration/017675-21munin-node19:16:390
149950,0migration/01731-21kthreadcore19:06:340
149950,0migration/016937-21sh22:16:330
149950,0migration/014533-21sh21:31:530
149950,0migration/012563-21kthreadcore21:51:330
149950,0migration/011820-21kthreadcore22:51:340
339940,0migration/38613-21taskset20:47:353
339940,0migration/36613-21sh21:27:123
339940,0migration/35039-21taskset19:10:433
339940,0migration/33919-21kthreadcore19:31:353
339940,0migration/330043-21kthreadcore19:51:333
339940,0migration/328276-21kthreadcore19:26:333
339940,0migration/325543-21kthreadcore00:01:323
339940,0migration/321210-21kthreadcore19:46:323
339940,0migration/320682-21kthreadcore18:56:343
29725040,0irq/27-eth0-tx-32145-21ssh22:06:172
29725040,0irq/27-eth0-tx-14895-21munin-node20:31:340
29725040,0irq/27-eth0-tx-0-21swapper/222:47:272
29725040,0irq/27-eth0-tx-0-21swapper/221:56:222
29725040,0irq/27-eth0-tx-0-21swapper/221:52:072
29725040,0irq/27-eth0-tx-0-21swapper/019:46:270
29725040,0irq/27-eth0-tx-0-21swapper/019:36:310
29725040,0irq/27-eth0-tx-0-21swapper/019:21:270
29715040,0irq/26-eth0-rx-12963-21aten_r1power_cu19:41:283
29715040,0irq/26-eth0-rx-0-21swapper/323:53:003
29715040,0irq/26-eth0-rx-0-21swapper/323:26:363
29715040,0irq/26-eth0-rx-0-21swapper/323:16:343
29715040,0irq/26-eth0-rx-0-21swapper/321:31:293
29715040,0irq/26-eth0-rx-0-21swapper/321:21:343
259940,0migration/28326-21kthreadcore19:36:332
259940,0migration/2373-21fschecks_count21:46:292
259940,0migration/232722-21kthreadcore23:26:322
259940,0migration/23207-21kthreadcore19:31:352
259940,0migration/231330-21runrttasks19:27:082
259940,0migration/231132-21kthreadcore20:41:332
259940,0migration/230153-21kthreadcore20:16:332
259940,0migration/229092-21kthreadcore19:51:322
259940,0migration/22541-21kthreadcore19:56:322
259940,0migration/223498-21kthreadcore19:21:332
259940,0migration/219037-21kthreadcore20:31:342
259940,0migration/218909-21kthreadcore20:06:332
259940,0migration/217318-21kthreadcore20:56:312
259940,0migration/21445-21latency_hist20:21:162
259940,0migration/211302-21kthreadcore20:26:322
249940,0watchdog/23634-21kthreadcore19:06:352
249940,0watchdog/223483-21diskstats23:41:302
249940,0watchdog/217610-21kthreadcore22:16:342
249940,0watchdog/210366-21grep20:01:352
249940,0watchdog/20-21swapper/221:26:342
175509944,0cyclictest0-21swapper/200:16:332
175509943,0cyclictest3556-21lldpd19:12:232
175509943,0cyclictest3514-21snmpd20:51:172
175509943,0cyclictest3514-21snmpd20:46:172
175509943,0cyclictest3514-21snmpd20:39:022
175509943,0cyclictest3514-21snmpd19:42:322
175509943,0cyclictest3514-21snmpd19:02:312
175509943,0cyclictest3241-21irqbalance23:31:242
175509943,0cyclictest24565-21awk22:41:302
175509943,0cyclictest18742-21awk18:56:322
175509943,0cyclictest12827-21awk19:16:312
175509940,0cyclictest15602-21diskmemload21:16:172
175499944,0cyclictest15602-21diskmemload23:41:091
175499943,0cyclictest3556-21lldpd22:11:261
175499943,0cyclictest3514-21snmpd23:55:331
175499943,0cyclictest3514-21snmpd23:42:331
175499943,0cyclictest3514-21snmpd23:32:031
175499943,0cyclictest3514-21snmpd19:57:171
175499943,0cyclictest3514-21snmpd19:29:461
175499943,0cyclictest29044-21awk20:41:301
175499943,0cyclictest25260-21awk19:01:341
175499943,0cyclictest19497-21awk19:21:301
175499943,0cyclictest17184-21awk21:36:311
175499943,0cyclictest12896-21awk19:16:321
175489944,0cyclictest0-21swapper/021:41:180
175489943,0cyclictest3514-21snmpd23:16:170
175489943,0cyclictest3514-21snmpd22:56:470
175489943,0cyclictest3514-21snmpd21:22:470
175489943,0cyclictest3514-21snmpd00:11:330
175489943,0cyclictest3514-21snmpd00:11:330
175489943,0cyclictest2345-21awk20:21:300
179940,0migration/19678-21kthreadcore21:51:321
179940,0migration/19114-21kthreadcore19:11:351
179940,0migration/130702-21kthreadcore19:51:331
179940,0migration/129418-21sendmail_mailtr22:41:371
179940,0migration/126298-21kthreadcore00:21:351
179940,0migration/124541-21aten_r1power_cu21:21:271
179940,0migration/122693-21sh22:18:291
179940,0migration/120036-21runrttasks20:08:031
179940,0migration/119275-21kthreadcore00:16:321
179940,0migration/114165-21kthreadcore20:51:341
179940,0migration/112647-21kthreadcore21:31:351
169940,0watchdog/18971-21if_err_eth1.221:11:301
169940,0watchdog/13636-21kthreadcore19:06:351
169940,0watchdog/131982-21ssh22:26:261
169940,0watchdog/130727-21ssh21:03:181
169940,0watchdog/128415-21ssh00:01:341
169940,0watchdog/12570-21kthreadcore23:46:341
169940,0watchdog/12474-21interrupts20:21:311
169940,0watchdog/119851-21cut21:16:341
169940,0watchdog/116592-21cut23:16:341
169940,0watchdog/114237-21users20:26:391
169940,0watchdog/114215-21ssh22:32:541
169940,0watchdog/10-21swapper/123:01:341
169940,0watchdog/10-21swapper/120:31:311
169940,0watchdog/10-21swapper/100:11:181
169940,0watchdog/10-21swapper/100:11:181
149940,0migration/05984-21ntp_kernel_err19:56:350
339930,0migration/329186-21kthreadcore19:01:363
29715030,0irq/26-eth0-rx-0-21swapper/320:12:013
29715030,0irq/26-eth0-rx-0-21swapper/319:56:273
29715030,0irq/26-eth0-rx-0-21swapper/319:11:293
175509932,0cyclictest3514-21snmpd20:13:322
175499932,0cyclictest3514-21snmpd19:47:171
175499932,0cyclictest3514-21snmpd19:41:171
175499932,0cyclictest16396-21awk22:56:321
175489933,0cyclictest32246-21kworker/0:000:01:330
175489933,0cyclictest19663-21kworker/0:223:01:250
175489933,0cyclictest19663-21kworker/0:222:46:270
175489933,0cyclictest19663-21kworker/0:222:06:230
175489933,0cyclictest19663-21kworker/0:221:16:360
175489933,0cyclictest19663-21kworker/0:220:51:200
175489933,0cyclictest19663-21kworker/0:220:16:180
175489933,0cyclictest19663-21kworker/0:220:01:220
175489933,0cyclictest19663-21kworker/0:219:51:260
175489933,0cyclictest19663-21kworker/0:219:41:180
175489933,0cyclictest19663-21kworker/0:219:31:240
175489933,0cyclictest16969-21awk21:56:320
175489932,0cyclictest31965-21awk23:46:310
175489932,0cyclictest17229-21awk21:36:310
175489930,0cyclictest24537-21perl21:01:290
169930,0watchdog/123018-21munin-node18:56:351
169930,0watchdog/10-21swapper/121:56:261
169930,0watchdog/10-21swapper/121:26:341
169930,0watchdog/10-21swapper/120:01:351
169930,0watchdog/10-21swapper/119:36:271
169930,0watchdog/10-21swapper/119:31:391
29715020,0irq/26-eth0-rx-0-21swapper/320:07:103
169920,0watchdog/10-21swapper/120:36:191
169920,0watchdog/10-21swapper/120:16:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional