You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-12 - 13:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Wed Nov 12, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350970,0irq/25-eth10-21swapper/318:36:043
2323023422,0sleep00-21swapper/018:36:270
2335323127,0sleep10-21swapper/118:37:481
2321223026,0sleep20-21swapper/218:36:132
28415070,0irq/27-eth0-tx-0-21swapper/018:50:100
236089976,0cyclictest12611-21ssh21:11:480
35160,0ktimersoftd/30-21swapper/323:20:043
339960,0migration/39500-21sh23:12:163
339960,0migration/33751-21taskset20:45:243
339960,0migration/313639-21munin-node21:55:153
28415060,0irq/27-eth0-tx-6826-21munin-node19:40:230
28415060,0irq/27-eth0-tx-30140-21munin-node20:45:142
28415060,0irq/27-eth0-tx-0-21swapper/223:30:172
28415060,0irq/27-eth0-tx-0-21swapper/223:25:422
28415060,0irq/27-eth0-tx-0-21swapper/223:20:002
28415060,0irq/27-eth0-tx-0-21swapper/223:11:532
28415060,0irq/27-eth0-tx-0-21swapper/223:08:092
28415060,0irq/27-eth0-tx-0-21swapper/223:00:282
28415060,0irq/27-eth0-tx-0-21swapper/222:46:112
28415060,0irq/27-eth0-tx-0-21swapper/222:42:472
28415060,0irq/27-eth0-tx-0-21swapper/222:35:572
28415060,0irq/27-eth0-tx-0-21swapper/222:12:292
28415060,0irq/27-eth0-tx-0-21swapper/222:07:412
28415060,0irq/27-eth0-tx-0-21swapper/222:01:182
28415060,0irq/27-eth0-tx-0-21swapper/221:49:462
28415060,0irq/27-eth0-tx-0-21swapper/221:37:142
28415060,0irq/27-eth0-tx-0-21swapper/221:27:302
28415060,0irq/27-eth0-tx-0-21swapper/020:40:110
28415060,0irq/27-eth0-tx-0-21swapper/019:35:180
28415060,0irq/27-eth0-tx-0-21swapper/019:00:100
28405060,0irq/26-eth0-rx-9159-21nfs4_client22:30:193
28405060,0irq/26-eth0-rx-791-21python22:45:213
28405060,0irq/26-eth0-rx-27235-21sed21:40:223
28405060,0irq/26-eth0-rx-22603-21gltestperf21:20:133
249960,0watchdog/227271-21tr18:40:182
236119960,0cyclictest21892-21diskmemload20:42:212
236089960,0cyclictest0-21swapper/020:46:530
149960,0migration/07027-21unixbench_multi20:00:230
8225050,0irq/24-0000:00:1804-21latency_hist20:24:582
4150,0ktimersoftd/00-21swapper/023:54:030
4150,0ktimersoftd/00-21swapper/023:43:470
4150,0ktimersoftd/00-21swapper/022:47:110
339950,0migration/39911-21kthreadcore19:40:173
339950,0migration/37189-21kthreadcore21:30:163
339950,0migration/37129-21kthreadcore23:30:173
339950,0migration/36142-21munin-node21:10:133
339950,0migration/34284-21irqcore22:50:183
339950,0migration/328761-21kthreadcore23:45:163
339950,0migration/328430-21latency_hist22:04:593
339950,0migration/328084-21kthreadcore23:25:153
339950,0migration/326137-21kthreadcore18:40:173
339950,0migration/325892-21switchtime22:20:223
339950,0migration/325456-21diskmemload23:01:103
339950,0migration/321026-21munin-node20:35:243
339950,0migration/319786-21fschecks_time19:25:143
339950,0migration/317114-21kthreadcore21:15:173
339950,0migration/315169-21kthreadcore18:55:163
339950,0migration/314336-21latency_hist20:10:003
339950,0migration/313113-21munin-node20:05:213
28415050,0irq/27-eth0-tx-9991-21ssh22:31:032
28415050,0irq/27-eth0-tx-6547-21runrttasks18:45:110
28415050,0irq/27-eth0-tx-4812-21ssh21:07:332
28415050,0irq/27-eth0-tx-2627-21ssh22:28:012
28415050,0irq/27-eth0-tx-23822-21munin-node18:40:200
28415050,0irq/27-eth0-tx-15092-21cut22:15:182
28415050,0irq/27-eth0-tx-0-21swapper/223:55:102
28415050,0irq/27-eth0-tx-0-21swapper/223:50:152
28415050,0irq/27-eth0-tx-0-21swapper/223:45:582
28415050,0irq/27-eth0-tx-0-21swapper/223:44:272
28415050,0irq/27-eth0-tx-0-21swapper/223:36:302
28415050,0irq/27-eth0-tx-0-21swapper/223:16:132
28415050,0irq/27-eth0-tx-0-21swapper/222:56:232
28415050,0irq/27-eth0-tx-0-21swapper/222:50:132
28415050,0irq/27-eth0-tx-0-21swapper/221:55:022
28415050,0irq/27-eth0-tx-0-21swapper/221:50:152
28415050,0irq/27-eth0-tx-0-21swapper/221:41:042
28415050,0irq/27-eth0-tx-0-21swapper/221:33:182
28415050,0irq/27-eth0-tx-0-21swapper/221:24:062
28415050,0irq/27-eth0-tx-0-21swapper/221:15:292
28415050,0irq/27-eth0-tx-0-21swapper/221:09:592
28415050,0irq/27-eth0-tx-0-21swapper/220:58:382
28415050,0irq/27-eth0-tx-0-21swapper/220:50:202
28415050,0irq/27-eth0-tx-0-21swapper/020:34:480
28405050,0irq/26-eth0-rx-9713-21open_files22:10:193
28405050,0irq/26-eth0-rx-29961-21kthreadcore19:30:213
28405050,0irq/26-eth0-rx-21688-21sed22:00:163
28405050,0irq/26-eth0-rx-16940-21load22:35:183
28405050,0irq/26-eth0-rx-15513-21ssh23:55:583
28405050,0irq/26-eth0-rx-13820-21kthreadcore23:15:183
28405050,0irq/26-eth0-rx-10869-21kthreadcore20:50:183
28405050,0irq/26-eth0-rx-0-21swapper/323:51:243
28405050,0irq/26-eth0-rx-0-21swapper/322:40:203
28405050,0irq/26-eth0-rx-0-21swapper/321:50:173
28405050,0irq/26-eth0-rx-0-21swapper/321:45:173
28405050,0irq/26-eth0-rx-0-21swapper/321:26:263
28405050,0irq/26-eth0-rx-0-21swapper/321:05:223
28405050,0irq/26-eth0-rx-0-21swapper/321:00:053
28405050,0irq/26-eth0-rx-0-21swapper/320:25:163
28405050,0irq/26-eth0-rx-0-21swapper/319:49:533
28405050,0irq/26-eth0-rx-0-21swapper/300:05:183
259950,0migration/230518-21switchtime19:05:262
259950,0migration/218935-21idleruntime-cro19:24:592
259950,0migration/215355-21latency_hist20:34:592
249950,0watchdog/224003-21sh00:03:132
236119955,0cyclictest0-21swapper/221:02:502
236119955,0cyclictest0-21swapper/200:06:212
236119950,0cyclictest0-21swapper/219:30:272
236109955,0cyclictest0-21swapper/121:00:141
236109950,0cyclictest0-21swapper/123:15:121
236109950,0cyclictest0-21swapper/121:36:391
236109950,0cyclictest0-21swapper/120:52:121
236089955,0cyclictest0-21swapper/023:16:590
236089955,0cyclictest0-21swapper/000:00:110
236089950,0cyclictest0-21swapper/022:51:080
236089950,0cyclictest0-21swapper/021:49:380
19150,0ktimersoftd/10-21swapper/123:54:031
169950,0watchdog/132090-21kthreadcore23:05:171
169950,0watchdog/118558-21entropy23:40:131
169950,0watchdog/10-21swapper/123:55:131
169950,0watchdog/10-21swapper/123:01:571
169950,0watchdog/10-21swapper/122:44:331
169950,0watchdog/10-21swapper/122:33:211
169950,0watchdog/10-21swapper/121:26:141
149950,0migration/09980-21kthreadcore20:50:170
149950,0migration/09712-21unixbench_singl22:30:240
149950,0migration/09075-21runrttasks23:32:270
149950,0migration/08089-21latency23:10:170
149950,0migration/029382-21kthreadcore23:05:160
149950,0migration/028436-21kthreadcore19:55:160
149950,0migration/027640-21kthreadcore23:45:160
149950,0migration/027038-21kthreadcore00:05:170
149950,0migration/024061-21kthreadcore22:40:170
149950,0migration/022322-21kthreadcore19:50:170
149950,0migration/021932-21kthreadcore23:20:170
149950,0migration/021424-21kthreadcore19:25:170
149950,0migration/019024-21kthreadcore20:10:230
149950,0migration/016644-21kthreadcore21:55:170
149950,0migration/013003-21latency20:05:200
149950,0migration/012475-21latency_hist21:54:590
339940,0migration/35028-21cpuspeed_turbos18:50:123
339940,0migration/332035-21fschecks_time19:10:143
339940,0migration/330949-21kthreadcore22:25:163
339940,0migration/328730-21runrttasks20:20:193
339940,0migration/326554-21taskset19:54:573
339940,0migration/325701-21runrttasks19:05:183
339940,0migration/323090-21unixbench_singl00:00:243
339940,0migration/321041-21kthreadcore19:00:163
339940,0migration/316813-21runrttasks22:55:233
339940,0migration/315061-21kthreadcore22:15:183
339940,0migration/311651-21kthreadcore23:35:173
28415040,0irq/27-eth0-tx-20029-21munin-node22:20:142
28405040,0irq/26-eth0-rx-0-21swapper/323:40:053
28405040,0irq/26-eth0-rx-0-21swapper/323:05:063
28405040,0irq/26-eth0-rx-0-21swapper/321:35:143
28405040,0irq/26-eth0-rx-0-21swapper/320:55:513
28405040,0irq/26-eth0-rx-0-21swapper/320:40:153
28405040,0irq/26-eth0-rx-0-21swapper/320:01:203
28405040,0irq/26-eth0-rx-0-21swapper/319:20:223
259940,0migration/222880-21kthreadcore19:25:182
259940,0migration/217199-21kthreadcore20:10:222
259940,0migration/214262-21hwlatdetect19:45:132
249940,0watchdog/223350-21kthreadcore19:50:182
236119943,0cyclictest905-21awk19:35:162
236119943,0cyclictest8683-21awk20:05:172
236119943,0cyclictest31871-21fschecks_count19:10:142
236119943,0cyclictest3110-21irqbalance20:37:232
236119943,0cyclictest3110-21irqbalance20:27:032
236119943,0cyclictest3110-21irqbalance20:15:132
236119943,0cyclictest3110-21irqbalance20:04:032
236119943,0cyclictest3110-21irqbalance19:55:232
236119943,0cyclictest3110-21irqbalance19:40:132
236119943,0cyclictest3110-21irqbalance19:17:442
236119943,0cyclictest3110-21irqbalance19:00:342
236119943,0cyclictest3110-21irqbalance18:50:042
236119943,0cyclictest3110-21irqbalance18:45:042
236119943,0cyclictest12450-21awk18:55:142
236109944,0cyclictest0-21swapper/120:44:281
236109944,0cyclictest0-21swapper/119:45:121
236109943,0cyclictest6445-21snmpd18:57:311
236109943,0cyclictest3425-21lldpd19:56:231
236109943,0cyclictest32317-21awk19:10:161
236109943,0cyclictest31213-21sadc19:09:591
236109943,0cyclictest2988-21awk20:25:151
236109943,0cyclictest26129-21grep00:05:171
236109943,0cyclictest25555-21sadc19:29:591
236109943,0cyclictest21032-21awk22:20:151
236109943,0cyclictest20886-21kworker/1:023:30:161
236109943,0cyclictest18965-21awk19:00:141
236109943,0cyclictest15390-21awk20:10:211
236109943,0cyclictest12951-21awk22:15:161
236109943,0cyclictest11205-21awk23:35:161
236089944,0cyclictest0-21swapper/023:29:550
236089944,0cyclictest0-21swapper/022:35:120
236089944,0cyclictest0-21swapper/021:44:360
236089944,0cyclictest0-21swapper/021:23:000
236089944,0cyclictest0-21swapper/021:16:130
236089943,0cyclictest6445-21snmpd22:18:320
236089943,0cyclictest6445-21snmpd20:17:470
236089943,0cyclictest6445-21snmpd19:07:460
236089943,0cyclictest32357-21awk19:10:160
236089943,0cyclictest26660-21awk19:30:180
236089943,0cyclictest23070-21awk21:00:150
236089943,0cyclictest21003-21awk22:20:150
236089943,0cyclictest17867-21sadc23:39:590
236089943,0cyclictest11942-21awk22:55:150
179940,0migration/18919-21irqcore20:05:181
179940,0migration/18744-21kthreadcore19:40:171
179940,0migration/1727-21munin-node20:20:221
179940,0migration/16502-21gltestperf19:15:161
179940,0migration/14323-21cpu22:10:111
179940,0migration/13221-21kthreadcore20:00:161
179940,0migration/130462-21aten_r1power_po21:05:141
179940,0migration/129438-21kthreadcore19:30:211
179940,0migration/128678-21aten_r1power_vo22:05:141
179940,0migration/118595-21timerwakeupswit21:55:231
169940,0watchdog/18957-21kthreadcore21:50:171
169940,0watchdog/18886-21ssh23:10:291
169940,0watchdog/15053-21ssh20:48:221
169940,0watchdog/129753-21fschecks_time21:45:131
169940,0watchdog/123915-21kthreadcore20:15:181
169940,0watchdog/115761-21tr22:35:171
169940,0watchdog/111777-21kthreadcore20:30:181
169940,0watchdog/111471-21df_inode22:55:131
169940,0watchdog/10-21swapper/123:45:091
169940,0watchdog/10-21swapper/123:20:131
169940,0watchdog/10-21swapper/122:53:531
169940,0watchdog/10-21swapper/122:46:091
169940,0watchdog/10-21swapper/122:26:451
169940,0watchdog/10-21swapper/122:00:171
169940,0watchdog/10-21swapper/121:40:051
169940,0watchdog/10-21swapper/121:30:141
169940,0watchdog/10-21swapper/121:20:261
169940,0watchdog/10-21swapper/121:17:581
169940,0watchdog/10-21swapper/119:38:021
169940,0watchdog/10-21swapper/118:45:221
169940,0watchdog/10-21swapper/118:40:181
169940,0watchdog/10-21swapper/100:00:531
149940,0migration/09688-21ntp_kernel_pll_22:10:180
149940,0migration/05704-21latency_hist19:14:590
149940,0migration/031690-21kthreadcore22:05:190
149940,0migration/030576-21cpuspeed_turbos21:05:150
149940,0migration/030017-21kthreadcore20:20:200
149940,0migration/021442-21kthreadcore23:00:160
339930,0migration/33347-21kthreadcore19:35:183
28415030,0irq/27-eth0-tx-0-21swapper/020:25:130
28405030,0irq/26-eth0-rx-0-21swapper/320:30:123
28405030,0irq/26-eth0-rx-0-21swapper/319:55:113
28405030,0irq/26-eth0-rx-0-21swapper/319:15:143
28405030,0irq/26-eth0-rx-0-21swapper/318:45:183
236109933,0cyclictest3425-21lldpd18:50:221
236089933,0cyclictest0-21swapper/023:56:140
236089932,0cyclictest16135-21fschecks_time20:35:170
236089930,0cyclictest13139-21perl19:20:120
236089930,0cyclictest0-21swapper/019:45:220
236089930,0cyclictest0-21swapper/018:55:150
169930,0watchdog/116299-21awk20:35:181
169930,0watchdog/112444-21munin-node19:20:101
169930,0watchdog/10-21swapper/123:25:091
169930,0watchdog/10-21swapper/121:10:181
169930,0watchdog/10-21swapper/120:55:141
169930,0watchdog/10-21swapper/119:50:141
149930,0migration/011790-21sh21:31:090
28405020,0irq/26-eth0-rx-0-21swapper/320:15:173
236089921,0cyclictest21892-21diskmemload21:24:590
236089921,0cyclictest20514-21ls21:59:590
236089920,0cyclictest13059-21latency_hist21:34:590
236089920,0cyclictest0-21swapper/022:24:590
236089920,0cyclictest0-21swapper/020:54:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional