You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 10:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Nov 13, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
979027166,0sleep10-21swapper/118:35:191
291350480,0irq/25-eth10-21swapper/318:35:163
1037824130,0sleep20-21swapper/218:35:322
1043723826,0sleep00-21swapper/018:36:100
339970,0migration/37691-21sh21:59:253
28415070,0irq/27-eth0-tx-12986-21munin-node19:30:180
339960,0migration/324856-21sh21:28:203
339960,0migration/322125-21sh22:27:543
339960,0migration/314727-21gltestperf19:55:153
28415060,0irq/27-eth0-tx-9179-21diskmemload21:27:072
28415060,0irq/27-eth0-tx-11044-21munin-node18:40:190
28415060,0irq/27-eth0-tx-0-21swapper/223:53:102
28415060,0irq/27-eth0-tx-0-21swapper/223:37:002
28415060,0irq/27-eth0-tx-0-21swapper/223:30:212
28415060,0irq/27-eth0-tx-0-21swapper/222:57:042
28415060,0irq/27-eth0-tx-0-21swapper/222:46:142
28415060,0irq/27-eth0-tx-0-21swapper/222:35:092
28415060,0irq/27-eth0-tx-0-21swapper/222:22:482
28415060,0irq/27-eth0-tx-0-21swapper/222:10:172
28415060,0irq/27-eth0-tx-0-21swapper/221:52:162
28415060,0irq/27-eth0-tx-0-21swapper/221:42:062
28415060,0irq/27-eth0-tx-0-21swapper/221:39:082
28415060,0irq/27-eth0-tx-0-21swapper/221:22:452
28415060,0irq/27-eth0-tx-0-21swapper/220:54:502
28415060,0irq/27-eth0-tx-0-21swapper/220:48:142
28415060,0irq/27-eth0-tx-0-21swapper/200:03:412
28415060,0irq/27-eth0-tx-0-21swapper/019:22:560
28415060,0irq/27-eth0-tx-0-21swapper/018:58:530
28405060,0irq/26-eth0-rx-25822-21fschecks_count19:15:133
28405060,0irq/26-eth0-rx-24009-21users21:05:223
28405060,0irq/26-eth0-rx-0-21swapper/323:20:153
28405060,0irq/26-eth0-rx-0-21swapper/320:56:213
249960,0watchdog/20-21swapper/223:42:352
249960,0watchdog/20-21swapper/200:09:392
249960,0watchdog/20-21swapper/200:09:392
179960,0migration/123881-21sh20:46:141
169960,0watchdog/10-21swapper/123:31:351
149960,0migration/07244-21latency_hist22:19:570
149960,0migration/020713-21gltestperf23:50:120
108279965,0cyclictest3420-21kthreadcore22:15:182
108279960,0cyclictest0-21swapper/221:55:172
108259966,0cyclictest9179-21diskmemload21:47:241
108259960,0cyclictest0-21swapper/123:56:311
108259960,0cyclictest0-21swapper/122:45:341
108259960,0cyclictest0-21swapper/119:35:171
8225050,0irq/24-0000:00:13753-21awk19:54:572
339950,0migration/39415-21munin-node20:15:133
339950,0migration/3435-21latency_hist19:44:573
339950,0migration/34176-21sh23:18:273
339950,0migration/332624-21fschecks_time19:20:113
339950,0migration/331604-21kthreadcore22:55:183
339950,0migration/328008-21taskset20:28:253
339950,0migration/325944-21latency_hist21:09:573
339950,0migration/325018-21munin-node19:35:173
339950,0migration/324884-21sh21:50:093
339950,0migration/323504-21kthreadcore19:10:173
339950,0migration/320765-21munin-node18:45:123
339950,0migration/318246-21latency19:30:153
339950,0migration/315618-21sendmail_mailtr21:20:193
339950,0migration/315022-21munin-node20:40:193
339950,0migration/314531-21missed_timers21:40:173
339950,0migration/313895-21diskmemload22:21:403
339950,0migration/313007-21taskset22:41:243
339950,0migration/312782-21hddtemp_smartct19:05:123
339950,0migration/312581-21df_abs23:45:093
339950,0migration/311562-21taskset19:02:323
339950,0migration/310687-21kthreadcore23:00:183
28415050,0irq/27-eth0-tx-32376-21ssh21:32:092
28415050,0irq/27-eth0-tx-2683-21munin-node20:35:190
28415050,0irq/27-eth0-tx-1637-21munin-node20:10:120
28415050,0irq/27-eth0-tx-13933-21munin-node22:00:222
28415050,0irq/27-eth0-tx-0-21swapper/223:57:172
28415050,0irq/27-eth0-tx-0-21swapper/223:45:202
28415050,0irq/27-eth0-tx-0-21swapper/223:29:012
28415050,0irq/27-eth0-tx-0-21swapper/223:21:592
28415050,0irq/27-eth0-tx-0-21swapper/223:10:002
28415050,0irq/27-eth0-tx-0-21swapper/223:07:182
28415050,0irq/27-eth0-tx-0-21swapper/223:02:292
28415050,0irq/27-eth0-tx-0-21swapper/222:40:292
28415050,0irq/27-eth0-tx-0-21swapper/222:30:152
28415050,0irq/27-eth0-tx-0-21swapper/221:13:322
28415050,0irq/27-eth0-tx-0-21swapper/221:07:332
28415050,0irq/27-eth0-tx-0-21swapper/221:00:592
28415050,0irq/27-eth0-tx-0-21swapper/220:56:512
28415050,0irq/27-eth0-tx-0-21swapper/020:25:210
28415050,0irq/27-eth0-tx-0-21swapper/018:54:380
28405050,0irq/26-eth0-rx-31094-21kthreadcore23:55:153
28405050,0irq/26-eth0-rx-26808-21chrt23:52:343
28405050,0irq/26-eth0-rx-22854-21fschecks_time22:50:113
28405050,0irq/26-eth0-rx-15405-21munin-node22:45:133
28405050,0irq/26-eth0-rx-12288-21irqrtprio00:05:143
28405050,0irq/26-eth0-rx-12288-21irqrtprio00:05:143
28405050,0irq/26-eth0-rx-0-21swapper/323:35:203
28405050,0irq/26-eth0-rx-0-21swapper/323:30:143
28405050,0irq/26-eth0-rx-0-21swapper/323:25:103
28405050,0irq/26-eth0-rx-0-21swapper/323:05:033
28405050,0irq/26-eth0-rx-0-21swapper/322:38:593
28405050,0irq/26-eth0-rx-0-21swapper/322:15:213
28405050,0irq/26-eth0-rx-0-21swapper/322:00:173
28405050,0irq/26-eth0-rx-0-21swapper/321:36:183
28405050,0irq/26-eth0-rx-0-21swapper/321:30:243
28405050,0irq/26-eth0-rx-0-21swapper/320:00:183
28405050,0irq/26-eth0-rx-0-21swapper/319:25:123
28405050,0irq/26-eth0-rx-0-21swapper/300:00:513
259950,0migration/229106-21taskset22:52:442
259950,0migration/219474-21kthreadcore21:45:152
259950,0migration/217622-21sendmail_mailqu19:05:192
259950,0migration/212181-21irqcore18:40:142
249950,0watchdog/23707-21kthreadcore21:15:152
249950,0watchdog/222148-21entropy20:25:112
249950,0watchdog/20-21swapper/222:25:152
249950,0watchdog/20-21swapper/219:25:112
179950,0migration/14122-21open_files18:55:211
179950,0migration/127042-21munin-node19:40:121
179950,0migration/121774-21aten_r1power_cu20:25:091
179950,0migration/11923-21munin-node21:35:121
179950,0migration/117790-21kthreadcore22:25:151
179950,0migration/116903-21kthreadcore23:05:161
179950,0migration/111578-21fschecks_count00:05:111
179950,0migration/111578-21fschecks_count00:05:111
169950,0watchdog/16643-21cat23:00:151
169950,0watchdog/14374-21ssh22:56:471
169950,0watchdog/111967-21kthreadcore21:00:191
169950,0watchdog/10-21swapper/123:49:111
169950,0watchdog/10-21swapper/121:31:191
169950,0watchdog/10-21swapper/121:15:151
169950,0watchdog/10-21swapper/120:40:071
149950,0migration/09609-21kthreadcore23:20:140
149950,0migration/07736-21timerandwakeup21:15:210
149950,0migration/032720-21kthreadcore22:35:140
149950,0migration/03205-21taskset23:37:340
149950,0migration/031083-21kthreadcore21:10:150
149950,0migration/030095-21fschecks_time23:15:100
149950,0migration/030073-21kthreadcore20:30:140
149950,0migration/028949-21kthreadcore22:10:150
149950,0migration/028578-21runrttasks23:13:230
149950,0migration/026213-21if_eth019:15:150
149950,0migration/025173-21latency_hist19:14:570
149950,0migration/024698-21sh21:27:520
149950,0migration/024425-21taskset21:49:440
149950,0migration/023540-21kthreadcore19:35:150
149950,0migration/023349-21sh22:09:170
149950,0migration/02169-21kthreadcore21:55:160
149950,0migration/020527-21kthreadcore18:45:120
149950,0migration/019504-21kthreadcore20:45:120
149950,0migration/018843-21timerandwakeup23:25:220
149950,0migration/015724-21kthreadcore23:45:140
149950,0migration/015170-21aten_r1power_fr20:20:090
149950,0migration/014280-21cpuspeed_turbos19:55:140
149950,0migration/014093-21kthreadcore20:40:160
149950,0migration/012520-21fschecks_count19:05:120
149950,0migration/011984-21munin-node19:25:160
149950,0migration/011895-21kthreadcore21:40:150
149950,0migration/010577-21kthreadcore22:00:170
108279954,0cyclictest6445-21snmpd20:34:402
108279954,0cyclictest6445-21snmpd20:07:402
108259955,0cyclictest0-21swapper/122:35:101
108259954,0cyclictest3110-21irqbalance20:37:311
108259954,0cyclictest3110-21irqbalance20:33:111
108259950,0cyclictest0-21swapper/122:17:211
108259950,0cyclictest0-21swapper/120:56:101
108249954,0cyclictest2299-21awk20:55:140
339940,0migration/38271-21sh21:16:093
339940,0migration/3723-21kthreadcore18:55:183
339940,0migration/330589-21runrttasks18:53:043
339940,0migration/329986-21kthreadcore20:50:153
339940,0migration/328181-21hwlatdetect20:05:153
339940,0migration/327017-21gltestperf19:40:123
339940,0migration/313273-21kthreadcore18:40:153
339940,0migration/312560-21latency19:50:153
28415040,0irq/27-eth0-tx-0-21swapper/222:06:182
28415040,0irq/27-eth0-tx-0-21swapper/019:50:080
28405040,0irq/26-eth0-rx-0-21swapper/323:40:013
28405040,0irq/26-eth0-rx-0-21swapper/323:14:503
28405040,0irq/26-eth0-rx-0-21swapper/322:30:003
28405040,0irq/26-eth0-rx-0-21swapper/322:10:433
28405040,0irq/26-eth0-rx-0-21swapper/322:05:413
28405040,0irq/26-eth0-rx-0-21swapper/321:48:093
28405040,0irq/26-eth0-rx-0-21swapper/321:00:203
28405040,0irq/26-eth0-rx-0-21swapper/320:45:523
259940,0migration/221465-21kthreadcore19:35:142
259940,0migration/220926-21kthreadcore18:45:122
259940,0migration/21793-21aten_r1power_vo20:10:092
259940,0migration/215213-21kthreadcore19:30:142
259940,0migration/212624-21kthreadcore20:15:142
259940,0migration/210690-21missed_timers19:00:152
179940,0migration/17577-21timerwakeupswit20:10:201
179940,0migration/131908-21sendmail_mailst21:10:191
179940,0migration/129782-21kthreadcore20:05:171
179940,0migration/127940-21kthreadcore20:50:141
179940,0migration/121693-21kthreadcore23:50:151
179940,0migration/120414-21kthreadcore19:10:161
179940,0migration/117894-21kthreadcore20:20:141
179940,0migration/113035-21diskmemload22:41:331
179940,0migration/110929-21kthreadcore21:40:151
179940,0migration/110121-21kthreadcore19:00:141
169940,0watchdog/128583-21ssh22:51:031
169940,0watchdog/125955-21cut22:30:151
169940,0watchdog/114222-21irqstats23:25:151
169940,0watchdog/113360-21cut21:20:151
169940,0watchdog/10-21swapper/123:35:151
169940,0watchdog/10-21swapper/122:20:111
169940,0watchdog/10-21swapper/122:10:151
169940,0watchdog/10-21swapper/122:05:431
169940,0watchdog/10-21swapper/121:55:391
169940,0watchdog/10-21swapper/121:05:031
169940,0watchdog/10-21swapper/119:15:121
169940,0watchdog/10-21swapper/119:05:121
169940,0watchdog/10-21swapper/100:00:151
149940,0migration/028059-21kthreadcore21:30:160
149940,0migration/027270-21sh23:31:210
149940,0migration/024254-21sh21:05:580
149940,0migration/011600-21kthreadcore20:15:140
149940,0migration/010004-21kthreadcore23:00:180
108279943,0cyclictest6445-21snmpd20:20:402
108279943,0cyclictest6445-21snmpd20:03:402
108279943,0cyclictest6445-21snmpd19:48:252
108279943,0cyclictest6445-21snmpd19:16:392
108279943,0cyclictest6445-21snmpd19:10:092
108279943,0cyclictest6445-21snmpd18:58:092
108279943,0cyclictest6445-21snmpd18:53:092
108279943,0cyclictest3425-21lldpd19:42:022
108279943,0cyclictest30308-21awk23:15:122
108279943,0cyclictest10323-21awk20:40:132
108259943,0cyclictest9367-21awk20:15:121
108259943,0cyclictest6445-21snmpd22:02:401
108259943,0cyclictest6445-21snmpd19:30:401
108259943,0cyclictest6445-21snmpd19:29:551
108259943,0cyclictest6445-21snmpd18:51:541
108259943,0cyclictest6445-21snmpd18:43:241
108259943,0cyclictest356-21awk19:20:121
108259943,0cyclictest25788-21awk21:50:131
108259943,0cyclictest25580-21kworker/1:223:23:421
108259943,0cyclictest25580-21kworker/1:223:15:501
108259943,0cyclictest25580-21kworker/1:221:25:021
108259943,0cyclictest21517-21awk20:00:121
108259943,0cyclictest14872-21awk19:55:161
108249944,0cyclictest0-21swapper/022:55:180
108249944,0cyclictest0-21swapper/022:40:190
108249944,0cyclictest0-21swapper/021:03:030
108249944,0cyclictest0-21swapper/020:53:440
108249943,0cyclictest28757-21awk23:55:130
108249943,0cyclictest28328-21awk20:05:160
108249943,0cyclictest27153-21awk19:40:130
108249943,0cyclictest14546-21awk23:05:140
108249940,0cyclictest0-21swapper/022:46:190
108249940,0cyclictest0-21swapper/021:54:550
339930,0migration/317445-21kthreadcore20:20:143
28415030,0irq/27-eth0-tx-0-21swapper/019:45:120
28405030,0irq/26-eth0-rx-0-21swapper/320:35:083
28405030,0irq/26-eth0-rx-0-21swapper/320:11:093
249930,0watchdog/20-21swapper/219:55:232
169930,0watchdog/10-21swapper/119:50:111
169930,0watchdog/10-21swapper/118:45:161
108279933,0cyclictest3425-21lldpd19:20:022
108279930,0cyclictest3510-21hwlatdetect20:35:112
108259933,0cyclictest25580-21kworker/1:223:09:581
108259933,0cyclictest25580-21kworker/1:219:45:111
108259932,0cyclictest5074-21munin-node23:40:111
108249933,0cyclictest16082-21awk22:25:120
108249932,0cyclictest7757-21kthreadcore19:00:140
108249932,0cyclictest4950-21sh23:40:100
108249932,0cyclictest4175-21awk00:00:130
108249932,0cyclictest25285-21grep22:30:150
108249932,0cyclictest23135-21awk22:50:120
108249932,0cyclictest21619-21awk20:00:130
108249932,0cyclictest12993-21kthreadcore21:20:150
108249932,0cyclictest1144-21basename21:35:090
108249930,0cyclictest32419-21perl22:15:150
28405020,0irq/26-eth0-rx-0-21swapper/320:30:223
108249920,0cyclictest0-21swapper/000:04:570
108249920,0cyclictest0-21swapper/000:04:570
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional