You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 01:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Wed Dec 03, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350770,0irq/25-eth10-21swapper/306:35:313
291350770,0irq/25-eth10-21swapper/306:35:313
2092023723,0sleep00-21swapper/006:36:390
2092023723,0sleep00-21swapper/006:36:390
2113223627,0sleep10-21swapper/106:39:041
2113223627,0sleep10-21swapper/106:39:041
2097522925,0sleep20-21swapper/206:37:172
2097522925,0sleep20-21swapper/206:37:172
249970,0watchdog/224346-21cut11:09:302
212159970,0cyclictest0-21swapper/110:49:311
169970,0watchdog/130153-21gltestperf10:54:301
35160,0ktimersoftd/30-21swapper/311:49:203
339960,0migration/37849-21sh09:56:563
339960,0migration/34870-21sh08:53:483
339960,0migration/34802-21sh10:35:263
339960,0migration/330868-21sh10:11:043
339960,0migration/328003-21taskset08:44:393
339960,0migration/310880-21ntp_offset08:54:323
28415060,0irq/27-eth0-tx-19639-21munin-node08:19:270
28415060,0irq/27-eth0-tx-18599-21munin-node07:54:320
28415060,0irq/27-eth0-tx-16957-21ssh09:41:502
28415060,0irq/27-eth0-tx-11180-21munin-node11:24:302
28415060,0irq/27-eth0-tx-0-21swapper/212:04:252
28415060,0irq/27-eth0-tx-0-21swapper/211:50:502
28415060,0irq/27-eth0-tx-0-21swapper/211:37:322
28415060,0irq/27-eth0-tx-0-21swapper/211:32:442
28415060,0irq/27-eth0-tx-0-21swapper/211:23:382
28415060,0irq/27-eth0-tx-0-21swapper/211:04:262
28415060,0irq/27-eth0-tx-0-21swapper/210:55:322
28415060,0irq/27-eth0-tx-0-21swapper/210:37:162
28415060,0irq/27-eth0-tx-0-21swapper/210:33:332
28415060,0irq/27-eth0-tx-0-21swapper/210:13:032
28415060,0irq/27-eth0-tx-0-21swapper/209:57:132
28415060,0irq/27-eth0-tx-0-21swapper/209:32:262
28415060,0irq/27-eth0-tx-0-21swapper/209:05:312
28415060,0irq/27-eth0-tx-0-21swapper/208:59:172
28415060,0irq/27-eth0-tx-0-21swapper/208:54:232
28415060,0irq/27-eth0-tx-0-21swapper/208:52:102
28415060,0irq/27-eth0-tx-0-21swapper/008:00:550
28415060,0irq/27-eth0-tx-0-21swapper/007:39:230
28405060,0irq/26-eth0-rx-26076-21munin-run08:24:113
249960,0watchdog/20-21swapper/211:59:342
249960,0watchdog/20-21swapper/208:44:022
212159965,0cyclictest17862-21ps10:04:291
169960,0watchdog/122892-21load11:49:301
149960,0migration/08511-21unixbench_singl06:49:360
149960,0migration/06257-21sh11:59:320
149960,0migration/024490-21sh11:52:320
149960,0migration/018097-21missed_timers11:04:300
149960,0migration/015081-21latency_hist10:24:120
149960,0migration/011776-21sh11:01:590
4150,0ktimersoftd/00-21swapper/011:24:390
4150,0ktimersoftd/00-21swapper/009:10:190
339950,0migration/38634-21kthreadcore11:19:293
339950,0migration/3795-21munin-node08:04:283
339950,0migration/36061-21kthreadcore09:14:293
339950,0migration/331999-21kthreadcore07:14:293
339950,0migration/331317-21munin-node07:39:253
339950,0migration/327810-21taskset06:43:353
339950,0migration/326486-21latency_hist11:34:123
339950,0migration/323902-21fschecks_time07:09:263
339950,0migration/322318-21switchtime10:04:363
339950,0migration/321603-21fschecks_count09:04:283
339950,0migration/319862-21irqcore11:29:273
339950,0migration/317992-21munin-node07:29:283
339950,0migration/315180-21taskset12:06:363
339950,0migration/315124-21latency_hist10:24:123
28415050,0irq/27-eth0-tx-8534-21ssh11:40:502
28415050,0irq/27-eth0-tx-7386-21munin-node08:34:220
28415050,0irq/27-eth0-tx-20271-21ssh08:39:580
28415050,0irq/27-eth0-tx-16947-21ssh11:48:112
28415050,0irq/27-eth0-tx-0-21swapper/211:15:202
28415050,0irq/27-eth0-tx-0-21swapper/210:59:462
28415050,0irq/27-eth0-tx-0-21swapper/210:44:332
28415050,0irq/27-eth0-tx-0-21swapper/210:42:282
28415050,0irq/27-eth0-tx-0-21swapper/210:14:212
28415050,0irq/27-eth0-tx-0-21swapper/210:01:502
28415050,0irq/27-eth0-tx-0-21swapper/209:34:222
28415050,0irq/27-eth0-tx-0-21swapper/209:29:012
28415050,0irq/27-eth0-tx-0-21swapper/208:46:142
28415050,0irq/27-eth0-tx-0-21swapper/007:47:260
28415050,0irq/27-eth0-tx-0-21swapper/007:29:260
28415050,0irq/27-eth0-tx-0-21swapper/007:10:450
28405050,0irq/26-eth0-rx-609-21python09:29:363
28405050,0irq/26-eth0-rx-3903-21ssh10:56:533
28405050,0irq/26-eth0-rx-25494-21ssh09:49:143
28405050,0irq/26-eth0-rx-18418-21sendmail_mailqu11:04:333
28405050,0irq/26-eth0-rx-11444-21cpuspeed_turbos11:24:243
28405050,0irq/26-eth0-rx-0-21swapper/312:02:573
28405050,0irq/26-eth0-rx-0-21swapper/310:19:293
28405050,0irq/26-eth0-rx-0-21swapper/309:59:363
28405050,0irq/26-eth0-rx-0-21swapper/308:39:283
28405050,0irq/26-eth0-rx-0-21swapper/308:29:433
28405050,0irq/26-eth0-rx-0-21swapper/308:24:423
28405050,0irq/26-eth0-rx-0-21swapper/306:44:403
27150,0ktimersoftd/20-21swapper/209:10:192
259950,0migration/29777-21sh09:14:352
259950,0migration/25374-21switchtime08:04:342
259950,0migration/223993-21munin-node09:44:402
259950,0migration/217738-21timerandwakeup07:49:362
259950,0migration/216949-21munin-node07:04:282
259950,0migration/213825-21sh10:20:062
259950,0migration/211917-21cpuspeed_turbos09:19:242
249950,0watchdog/216051-21idleruntime10:24:262
249950,0watchdog/20-21swapper/210:49:262
212169955,0cyclictest0-21swapper/209:52:042
212159955,0cyclictest13797-21diskmemload09:23:451
212159955,0cyclictest0-21swapper/110:15:001
212159955,0cyclictest0-21swapper/110:11:221
212159955,0cyclictest0-21swapper/109:14:371
212159953,0cyclictest21844-21kworker/1:209:11:261
212149955,0cyclictest0-21swapper/010:34:310
212149955,0cyclictest0-21swapper/009:42:100
212149954,0cyclictest25168-21awk10:09:280
212149950,0cyclictest0-21swapper/010:58:380
212149950,0cyclictest0-21swapper/010:28:310
212149950,0cyclictest0-21swapper/010:02:020
212149950,0cyclictest0-21swapper/008:52:120
179950,0migration/18132-21taskset12:03:541
179950,0migration/17542-21kthreadcore10:59:291
179950,0migration/1522-21munin-node08:29:231
179950,0migration/14190-21kthreadcore09:34:301
179950,0migration/13343-21kthreadcore10:34:311
179950,0migration/128279-21kthreadcore08:24:411
179950,0migration/127612-21sh11:12:581
179950,0migration/127076-21kthreadcore08:44:321
179950,0migration/125176-21munin-node08:19:331
179950,0migration/119500-21kthreadcore07:04:321
179950,0migration/11788-21timerwakeupswit11:14:401
179950,0migration/110812-21pmu-power07:44:331
169950,0watchdog/132365-21rm09:51:421
169950,0watchdog/129029-21if_eth006:44:391
169950,0watchdog/1238532sleep107:32:111
169950,0watchdog/116694-21kthreadcore08:39:301
169950,0watchdog/116052-21idleruntime10:24:261
169950,0watchdog/11407-21ssh09:31:101
169950,0watchdog/113733-21tr09:59:301
169950,0watchdog/11280-21ssh11:38:101
169950,0watchdog/110242-21df_abs11:44:261
169950,0watchdog/10-21swapper/109:27:221
169950,0watchdog/10-21swapper/108:55:101
149950,0migration/09398-21taskset06:56:250
149950,0migration/08445-21sh12:04:130
149950,0migration/06580-21kthreadcore11:19:280
149950,0migration/06393-21kthreadcore09:14:290
149950,0migration/028917-21fschecks_time11:14:300
149950,0migration/026356-21kthreadcore09:04:320
149950,0migration/023855-21kthreadcore08:44:310
149950,0migration/023335-21kthreadcore11:29:290
149950,0migration/021841-21munin-node06:39:260
149950,0migration/021767-21kthreadcore09:24:320
149950,0migration/019686-21kthreadcore09:44:340
149950,0migration/01967-21irqcore09:54:280
149950,0migration/019268-21kthreadcore07:04:320
149950,0migration/017237-21taskset07:27:510
149950,0migration/014252-21irqcore08:59:290
149950,0migration/013982-21sh08:14:280
149950,0migration/013370-21kthreadcore09:19:280
149950,0migration/012538-21fschecks_time07:49:270
149950,0migration/010813-21munin-node11:44:290
149950,0migration/010294-21taskset07:20:070
8225040,0irq/24-0000:00:27875-21latency_hist06:44:122
339940,0migration/38982-21kthreadcore07:44:293
339940,0migration/32929-21kthreadcore10:14:313
339940,0migration/319014-21forks07:54:253
339940,0migration/310784-21fschecks_count09:39:263
28405040,0irq/26-eth0-rx-0-21swapper/311:54:323
28405040,0irq/26-eth0-rx-0-21swapper/311:45:383
28405040,0irq/26-eth0-rx-0-21swapper/311:39:323
28405040,0irq/26-eth0-rx-0-21swapper/311:18:183
28405040,0irq/26-eth0-rx-0-21swapper/311:09:503
28405040,0irq/26-eth0-rx-0-21swapper/310:59:233
28405040,0irq/26-eth0-rx-0-21swapper/310:49:313
28405040,0irq/26-eth0-rx-0-21swapper/310:46:223
28405040,0irq/26-eth0-rx-0-21swapper/310:42:383
28405040,0irq/26-eth0-rx-0-21swapper/310:32:503
28405040,0irq/26-eth0-rx-0-21swapper/309:44:343
28405040,0irq/26-eth0-rx-0-21swapper/309:34:233
28405040,0irq/26-eth0-rx-0-21swapper/309:24:533
28405040,0irq/26-eth0-rx-0-21swapper/309:20:213
28405040,0irq/26-eth0-rx-0-21swapper/309:09:443
28405040,0irq/26-eth0-rx-0-21swapper/308:59:193
28405040,0irq/26-eth0-rx-0-21swapper/307:19:323
259940,0migration/25274-21irqcore07:19:272
259940,0migration/24667-21kthreadcore06:49:292
259940,0migration/228528-21kthreadcore08:24:412
259940,0migration/226280-21munin-node11:54:302
259940,0migration/225180-21latency_hist07:59:122
259940,0migration/211862-21kthreadcore08:34:292
249940,0watchdog/226728-21sed06:39:312
249940,0watchdog/20-21swapper/210:04:262
212169944,0cyclictest0-21swapper/207:54:472
212169943,0cyclictest3425-21lldpd07:17:562
212169943,0cyclictest18528-21awk07:29:302
212169943,0cyclictest1482-21awk08:29:272
212159943,0cyclictest9404-21awk12:04:281
212159943,0cyclictest6445-21snmpd09:08:201
212159943,0cyclictest6445-21snmpd07:59:201
212159943,0cyclictest4272-21awk11:19:261
212149944,0cyclictest0-21swapper/010:05:210
212149944,0cyclictest0-21swapper/007:14:270
212149943,0cyclictest6445-21snmpd09:51:500
212149943,0cyclictest6445-21snmpd09:37:050
212149943,0cyclictest3425-21lldpd07:36:490
212149943,0cyclictest10738-21awk06:59:270
179940,0migration/15144-21kthreadcore06:49:301
179940,0migration/14592-21taskset07:19:251
179940,0migration/14359-21timerandwakeup07:39:351
179940,0migration/132570-21kthreadcore07:14:291
179940,0migration/129991-21cpu08:49:251
179940,0migration/123563-21fschecks_time10:29:261
179940,0migration/121897-21fschecks_count06:39:261
179940,0migration/119854-21irqcore11:29:271
179940,0migration/118434-21ntp_offset08:14:321
179940,0migration/117989-21taskset07:49:501
179940,0migration/117860-21kthreadcore10:44:311
179940,0migration/117593-21sh11:25:421
179940,0migration/110885-21taskset07:24:201
169940,0watchdog/118496-21switchtime11:04:341
169940,0watchdog/113359-21kthreadcore09:39:301
169940,0watchdog/10-21swapper/109:44:341
169940,0watchdog/10-21swapper/108:59:261
169940,0watchdog/10-21swapper/107:10:311
149940,0migration/06097-21taskset10:15:180
149940,0migration/029691-21kthreadcore11:54:320
149940,0migration/028880-21sh10:53:000
149940,0migration/028812-21switchtime10:29:350
149940,0migration/013124-21runrttasks10:42:100
339930,0migration/325217-21latency_hist07:59:123
28405030,0irq/26-eth0-rx-0-21swapper/308:14:233
28405030,0irq/26-eth0-rx-0-21swapper/307:53:483
28405030,0irq/26-eth0-rx-0-21swapper/307:35:283
28405030,0irq/26-eth0-rx-0-21swapper/307:24:163
28405030,0irq/26-eth0-rx-0-21swapper/307:05:453
28405030,0irq/26-eth0-rx-0-21swapper/306:59:513
28405030,0irq/26-eth0-rx-0-21swapper/306:58:033
28405030,0irq/26-eth0-rx-0-21swapper/306:50:493
259930,0migration/212994-21kthreadcore07:24:322
249930,0watchdog/20-21swapper/208:14:262
249930,0watchdog/20-21swapper/208:14:022
249930,0watchdog/20-21swapper/207:34:352
212169932,0cyclictest6136-21awk07:44:272
212169930,0cyclictest0-21swapper/207:12:312
212169930,0cyclictest0-21swapper/206:59:292
212159933,0cyclictest21844-21kworker/1:211:39:351
212159933,0cyclictest14803-21kworker/1:008:09:161
212159933,0cyclictest14803-21kworker/1:008:04:181
212159933,0cyclictest14803-21kworker/1:007:54:141
212159933,0cyclictest14803-21kworker/1:007:34:201
212159933,0cyclictest14803-21kworker/1:006:59:221
212159933,0cyclictest14803-21kworker/1:006:54:221
212159932,0cyclictest1877-21awk09:54:271
212149933,0cyclictest0-21swapper/011:09:500
212149932,0cyclictest7214-21awk08:09:270
212149932,0cyclictest6445-21snmpd11:35:060
212149932,0cyclictest5558-21grep08:54:240
212149932,0cyclictest27159-21awk08:24:390
212149932,0cyclictest14778-21grep10:44:280
212149932,0cyclictest1458-21awk08:29:270
212149930,0cyclictest31966-21munin-node08:04:300
212149930,0cyclictest0-21swapper/006:44:310
179930,0migration/18195-21fschecks_time10:19:251
169930,0watchdog/10-21swapper/111:55:221
169930,0watchdog/10-21swapper/110:39:181
169930,0watchdog/10-21swapper/108:34:221
35120,0ktimersoftd/311242-21kthreadcore08:09:303
28405020,0irq/26-eth0-rx-0-21swapper/308:34:333
249920,0watchdog/20-21swapper/208:19:142
212169921,0cyclictest9179-21latency_hist06:54:122
212169920,0cyclictest30874-21latency_hist07:39:122
212149921,0cyclictest0-21swapper/009:29:130
212149920,0cyclictest0-21swapper/011:39:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional