You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 15:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Thu Nov 27, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350970,0irq/25-eth10-21swapper/306:35:083
2020927470,0sleep20-21swapper/206:37:222
2002523430,0sleep00-21swapper/006:35:160
2030323127,0sleep10-21swapper/106:38:291
8225060,0irq/24-0000:00:0-21swapper/207:59:252
339960,0migration/332353-21sh11:52:163
339960,0migration/324921-21sh09:02:203
339960,0migration/324472-21sh11:26:273
339960,0migration/313815-21sh09:57:263
28415060,0irq/27-eth0-tx-30253-21munin-node07:59:370
28415060,0irq/27-eth0-tx-0-21swapper/212:05:332
28415060,0irq/27-eth0-tx-0-21swapper/211:57:282
28415060,0irq/27-eth0-tx-0-21swapper/211:39:312
28415060,0irq/27-eth0-tx-0-21swapper/211:31:202
28415060,0irq/27-eth0-tx-0-21swapper/211:24:082
28415060,0irq/27-eth0-tx-0-21swapper/211:18:572
28415060,0irq/27-eth0-tx-0-21swapper/211:10:262
28415060,0irq/27-eth0-tx-0-21swapper/210:59:482
28415060,0irq/27-eth0-tx-0-21swapper/210:50:432
28415060,0irq/27-eth0-tx-0-21swapper/210:38:442
28415060,0irq/27-eth0-tx-0-21swapper/210:31:072
28415060,0irq/27-eth0-tx-0-21swapper/210:10:352
28415060,0irq/27-eth0-tx-0-21swapper/210:07:592
28415060,0irq/27-eth0-tx-0-21swapper/209:59:372
28415060,0irq/27-eth0-tx-0-21swapper/209:56:532
28415060,0irq/27-eth0-tx-0-21swapper/209:52:122
28415060,0irq/27-eth0-tx-0-21swapper/209:43:012
28415060,0irq/27-eth0-tx-0-21swapper/209:31:132
28415060,0irq/27-eth0-tx-0-21swapper/209:26:092
28415060,0irq/27-eth0-tx-0-21swapper/209:07:482
28415060,0irq/27-eth0-tx-0-21swapper/209:01:472
28415060,0irq/27-eth0-tx-0-21swapper/208:51:112
28415060,0irq/27-eth0-tx-0-21swapper/208:47:102
28415060,0irq/27-eth0-tx-0-21swapper/007:44:500
28405060,0irq/26-eth0-rx-4829-21cut10:34:383
28405060,0irq/26-eth0-rx-4304-21ntpq08:24:453
28405060,0irq/26-eth0-rx-18188-21if_eth109:19:413
204529965,0cyclictest1550-21ps09:29:420
169960,0watchdog/1326482sleep109:06:481
149960,0migration/02799-21taskset10:54:070
149960,0migration/016966-21irqcore07:24:440
8225050,0irq/24-0000:00:0-21swapper/209:44:452
339950,0migration/38910-21kthreadcore08:04:413
339950,0migration/36883-21munin-node11:34:443
339950,0migration/35906-21munin-node08:29:373
339950,0migration/34817-21munin-node07:39:403
339950,0migration/331286-21latency_hist09:49:253
339950,0migration/330364-21kthreadcore11:09:423
339950,0migration/328393-21fschecks_time10:49:403
339950,0migration/328012-21taskset10:26:473
339950,0migration/327960-21fschecks_time06:44:403
339950,0migration/326164-21irqrtprio09:24:443
339950,0migration/325960-21missed_timers06:39:453
339950,0migration/325564-21kthreadcore09:44:433
339950,0migration/324285-21ksmtuned08:19:223
339950,0migration/323262-21munin-node07:29:363
339950,0migration/322734-21irqcore07:04:413
339950,0migration/320787-21taskset10:23:193
339950,0migration/320490-21kthreadcore12:04:413
339950,0migration/319248-21kthreadcore06:59:423
339950,0migration/316858-21latency_hist07:49:253
339950,0migration/316275-21kthreadcore09:59:433
339950,0migration/315639-21munin-node11:19:453
339950,0migration/315342-21timerwakeupswit09:14:483
339950,0migration/312936-21kthreadcore07:19:453
339950,0migration/312106-21kthreadcore07:44:423
339950,0migration/311710-21kthreadcore11:39:423
339950,0migration/311488-21kthreadcore08:54:433
339950,0migration/31085-21munin-node10:09:443
339950,0migration/310576-21kthreadcore09:34:433
339950,0migration/310216-21kthreadcore10:14:443
28415050,0irq/27-eth0-tx-6372-21iceccd08:19:360
28415050,0irq/27-eth0-tx-0-21swapper/212:00:092
28415050,0irq/27-eth0-tx-0-21swapper/211:49:132
28415050,0irq/27-eth0-tx-0-21swapper/211:05:232
28415050,0irq/27-eth0-tx-0-21swapper/210:49:212
28415050,0irq/27-eth0-tx-0-21swapper/210:27:092
28415050,0irq/27-eth0-tx-0-21swapper/210:18:062
28415050,0irq/27-eth0-tx-0-21swapper/209:38:352
28415050,0irq/27-eth0-tx-0-21swapper/209:22:182
28415050,0irq/27-eth0-tx-0-21swapper/209:15:552
28415050,0irq/27-eth0-tx-0-21swapper/209:09:272
28415050,0irq/27-eth0-tx-0-21swapper/208:55:062
28415050,0irq/27-eth0-tx-0-21swapper/208:43:002
28415050,0irq/27-eth0-tx-0-21swapper/007:37:150
28405050,0irq/26-eth0-rx-5395-21tr08:49:433
28405050,0irq/26-eth0-rx-32011-21ps11:30:233
28405050,0irq/26-eth0-rx-2093-21ntpq10:29:453
28405050,0irq/26-eth0-rx-19785-21kthreadcore11:44:423
28405050,0irq/26-eth0-rx-18361-21sh10:40:073
28405050,0irq/26-eth0-rx-1694-21kthreadcore09:29:433
28405050,0irq/26-eth0-rx-1413-21diskstats11:54:363
28405050,0irq/26-eth0-rx-12507-21cpuspeed_turbos08:34:353
28405050,0irq/26-eth0-rx-0-21swapper/312:00:273
28405050,0irq/26-eth0-rx-0-21swapper/311:08:353
28405050,0irq/26-eth0-rx-0-21swapper/310:57:533
28405050,0irq/26-eth0-rx-0-21swapper/310:07:173
28405050,0irq/26-eth0-rx-0-21swapper/309:04:403
28405050,0irq/26-eth0-rx-0-21swapper/308:44:253
28405050,0irq/26-eth0-rx-0-21swapper/308:39:493
259950,0migration/27114-21kthreadcore08:29:402
259950,0migration/227888-21mdstat07:29:412
259950,0migration/227092-21kthreadcore11:49:412
259950,0migration/226122-21kthreadcore07:54:412
259950,0migration/221480-21latency_hist07:04:252
259950,0migration/213154-21if_eth108:34:382
249950,0watchdog/27346-21unixbench_multi11:34:472
204549955,0cyclictest0-21swapper/206:54:572
204549950,0cyclictest0-21swapper/210:19:362
204539955,0cyclictest0-21swapper/111:33:521
204539955,0cyclictest0-21swapper/110:59:301
204539955,0cyclictest0-21swapper/110:26:051
204539950,0cyclictest0-21swapper/110:05:161
204539950,0cyclictest0-21swapper/109:26:011
204529952,0cyclictest18731-21awk11:24:400
204529950,0cyclictest0-21swapper/009:44:380
19150,0ktimersoftd/10-21swapper/111:10:131
19150,0ktimersoftd/10-21swapper/108:58:161
179950,0migration/127045-21munin-node07:04:441
179950,0migration/120060-21kthreadcore07:49:411
169950,0watchdog/132590-21ssh11:52:551
169950,0watchdog/117262-21kthreadcore09:39:431
169950,0watchdog/115128-21cut11:19:431
149950,0migration/09904-21kthreadcore08:29:410
149950,0migration/06408-21kthreadcore10:34:420
149950,0migration/06181-21kthreadcore08:04:400
149950,0migration/05772-21sh09:51:450
149950,0migration/04338-21kthreadcore11:14:450
149950,0migration/03965-21sh10:34:220
149950,0migration/03890-21kthreadcore11:34:410
149950,0migration/03596-21aten_r1power_en07:39:350
149950,0migration/0325-21taskset11:32:360
149950,0migration/032277-21sh11:52:040
149950,0migration/031480-21pmu-power09:04:430
149950,0migration/029809-21taskset10:08:350
149950,0migration/028439-21kthreadcore08:44:430
149950,0migration/028101-21kthreadcore09:24:450
149950,0migration/027741-21sh10:25:530
149950,0migration/026975-21sh11:09:360
149950,0migration/024158-21kthreadcore06:39:440
149950,0migration/023643-21aten_r1power_en07:54:350
149950,0migration/022666-21kthreadcore08:39:430
149950,0migration/022015-21kthreadcore09:19:440
149950,0migration/020274-21sh08:59:440
149950,0migration/019300-21kthreadcore09:39:440
149950,0migration/018360-21kthreadcore07:49:400
149950,0migration/017103-21unixbench-2d08:09:470
149950,0migration/017081-21kthreadcore08:34:400
149950,0migration/016316-21sh12:02:260
149950,0migration/015745-21irqcore09:59:420
149950,0migration/013179-21kthreadcore11:39:430
149950,0migration/011096-21latency10:14:450
8628243,0sleep36445-21snmpd07:15:063
339940,0migration/3536-21kthreadcore07:34:423
339940,0migration/35243-21kthreadcore09:09:423
339940,0migration/328003-21kthreadcore08:19:433
339940,0migration/322357-21diskmemload09:40:533
339940,0migration/314573-21kthreadcore08:09:423
339940,0migration/311001-21latency_hist10:59:253
28415040,0irq/27-eth0-tx-22563-21munin-node07:29:340
28415040,0irq/27-eth0-tx-0-21swapper/211:24:442
28415040,0irq/27-eth0-tx-0-21swapper/210:54:512
28415040,0irq/27-eth0-tx-0-21swapper/007:02:090
28415040,0irq/27-eth0-tx-0-21swapper/006:54:380
28415040,0irq/27-eth0-tx-0-21swapper/006:47:070
28405040,0irq/26-eth0-rx-0-21swapper/311:15:243
28405040,0irq/26-eth0-rx-0-21swapper/310:48:503
259940,0migration/25291-21kthreadcore07:14:412
259940,0migration/2466-21sendmail_mailqu06:44:472
259940,0migration/224911-21df_abs08:19:392
259940,0migration/222141-21fschecks_count07:04:382
259940,0migration/217241-21kthreadcore10:39:422
249940,0watchdog/29097-21open_files07:39:442
249940,0watchdog/22356-21idleruntime06:49:402
249940,0watchdog/217951-21irqcore07:49:402
249940,0watchdog/20-21swapper/208:28:442
204549943,0cyclictest6445-21snmpd08:09:062
204549943,0cyclictest6445-21snmpd07:26:512
204549943,0cyclictest6445-21snmpd07:21:062
204549943,0cyclictest6445-21snmpd07:12:362
204549943,0cyclictest6445-21snmpd06:43:052
204549943,0cyclictest30112-21awk07:34:392
204549943,0cyclictest18782-21awk08:14:432
204539944,0cyclictest19006-21kworker/1:208:42:291
204539943,0cyclictest7758-21awk09:54:411
204539943,0cyclictest6445-21snmpd11:44:381
204539943,0cyclictest3425-21lldpd11:54:361
204539943,0cyclictest3425-21lldpd11:15:291
204539943,0cyclictest3425-21lldpd09:11:301
204539943,0cyclictest3425-21lldpd08:17:471
204539943,0cyclictest3425-21lldpd07:54:381
204539943,0cyclictest3425-21lldpd07:27:271
204539943,0cyclictest3425-21lldpd07:19:271
204539943,0cyclictest3425-21lldpd07:10:521
204539943,0cyclictest3425-21lldpd06:53:511
204539943,0cyclictest31238-21awk10:09:411
204539943,0cyclictest19006-21kworker/1:212:08:461
204539943,0cyclictest19006-21kworker/1:211:06:191
204539943,0cyclictest19006-21kworker/1:210:35:331
204539943,0cyclictest19006-21kworker/1:210:20:231
204539943,0cyclictest19006-21kworker/1:210:18:371
204539943,0cyclictest19006-21kworker/1:209:38:271
204539943,0cyclictest19006-21kworker/1:209:19:271
204539943,0cyclictest18966-21awk08:59:431
204539943,0cyclictest12134-21awk08:09:391
204539942,0cyclictest19006-21kworker/1:209:50:031
204529943,0cyclictest6445-21snmpd11:54:530
204529943,0cyclictest6445-21snmpd08:25:210
204529943,0cyclictest6445-21snmpd08:18:510
204529943,0cyclictest4133-21awk10:54:420
204529943,0cyclictest2467-21awk06:49:400
204529943,0cyclictest22547-21awk07:04:400
204529943,0cyclictest18136-21awk11:44:400
204529943,0cyclictest14504-21awk10:19:390
204529943,0cyclictest10427-21awk09:14:400
204529940,0cyclictest0-21swapper/011:23:010
204529940,0cyclictest0-21swapper/011:00:000
204529940,0cyclictest0-21swapper/010:45:010
179940,0migration/18931-21missed_timers07:39:421
179940,0migration/17548-21kthreadcore08:04:411
179940,0migration/128503-21kthreadcore08:19:441
179940,0migration/123595-21kthreadcore10:44:431
179940,0migration/121238-21taskset07:02:521
179940,0migration/118725-21sh10:41:211
179940,0migration/11762-21kthreadcore07:34:421
179940,0migration/115059-21proc_pri09:14:451
179940,0migration/114788-21aten_r1power_po09:59:361
179940,0migration/11277-21taskset06:49:111
169940,0watchdog/19505-21ssh10:55:111
169940,0watchdog/124819-21kthreadcore06:39:441
169940,0watchdog/123868-21kthreadcore07:29:401
169940,0watchdog/111427-21grep06:54:441
169940,0watchdog/10-21swapper/111:59:351
169940,0watchdog/10-21swapper/111:39:391
169940,0watchdog/10-21swapper/111:24:431
169940,0watchdog/10-21swapper/110:51:191
169940,0watchdog/10-21swapper/110:29:431
169940,0watchdog/10-21swapper/109:44:431
169940,0watchdog/10-21swapper/109:29:431
169940,0watchdog/10-21swapper/108:44:401
169940,0watchdog/10-21swapper/108:34:401
169940,0watchdog/10-21swapper/108:29:441
169940,0watchdog/10-21swapper/107:44:441
149940,0migration/09687-21runrttasks08:54:000
149940,0migration/06616-21kthreadcore07:14:420
149940,0migration/04615-21sh10:11:190
149940,0migration/014675-21nfs4_client07:19:470
149940,0migration/013355-21sh09:55:560
28405030,0irq/26-eth0-rx-0-21swapper/307:59:363
28405030,0irq/26-eth0-rx-0-21swapper/307:28:413
28405030,0irq/26-eth0-rx-0-21swapper/307:09:383
28405030,0irq/26-eth0-rx-0-21swapper/306:59:123
28405030,0irq/26-eth0-rx-0-21swapper/306:49:423
204549933,0cyclictest0-21swapper/207:44:372
204549932,0cyclictest12532-21ps08:09:412
204539932,0cyclictest6122-21kthreadcore11:34:421
204529933,0cyclictest3425-21lldpd11:08:510
204529932,0cyclictest8931-21tune2fs09:34:400
204529932,0cyclictest6445-21snmpd12:07:230
204529932,0cyclictest28275-21munin-node07:09:470
204529932,0cyclictest15655-21ntp_kernel_pll_08:54:450
204529930,0cyclictest12548-21diskstats10:39:370
169930,0watchdog/10-21swapper/108:49:321
169930,0watchdog/10-21swapper/108:24:401
169930,0watchdog/10-21swapper/107:59:321
169930,0watchdog/10-21swapper/107:14:281
28405020,0irq/26-eth0-rx-0-21swapper/307:54:393
204529920,0cyclictest1177-21latency_hist09:09:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional