You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-08 - 00:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Sat Mar 07, 2026 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1545327877,0sleep00-21swapper/007:00:060
304450350,0irq/25-eth10-21swapper/306:58:253
1531523526,0sleep20-21swapper/206:58:312
1525423324,0sleep10-21swapper/106:57:501
15696992323,0cyclictest3514-21snmpd10:29:281
339960,0migration/38714-21aten_r1power_fr11:32:143
339960,0migration/38313-21munin-node10:07:193
339960,0migration/34281-21sh12:08:273
339960,0migration/313640-21kthreadcore07:22:213
29715060,0irq/26-eth0-rx-0-21swapper/311:17:233
156959965,0cyclictest24459-21grep12:22:210
8215050,0irq/24-0000:00:4851-21latency_hist07:42:042
8215050,0irq/24-0000:00:22472-21latency_hist09:07:042
8215050,0irq/24-0000:00:0-21swapper/207:27:042
339950,0migration/37646-21kthreadcore07:42:193
339950,0migration/37239-21munin-node11:07:283
339950,0migration/33699-21munin-node07:37:243
339950,0migration/331363-21missed_timers10:42:223
339950,0migration/331156-21ntp_offset11:02:233
339950,0migration/329156-21taskset09:11:103
339950,0migration/326492-21kthreadcore07:32:213
339950,0migration/325074-21aten_r1power_cu11:22:133
339950,0migration/324422-21aten_r1power_po09:32:183
339950,0migration/320143-21kthreadcore10:57:253
339950,0migration/318983-21munin-node07:52:173
339950,0migration/318523-21taskset09:48:043
339950,0migration/318149-21munin-node07:27:193
339950,0migration/317347-21munin-node11:37:173
339950,0migration/316329-21munin-node09:22:253
339950,0migration/315923-21kthreadcore07:47:233
339950,0migration/315395-21proc_pri10:52:253
339950,0migration/311460-21kthreadcore11:12:213
339950,0migration/311002-21latency12:12:203
29725050,0irq/27-eth0-tx-7466-21munin-node11:52:172
29725050,0irq/27-eth0-tx-6588-21ssh11:49:552
29725050,0irq/27-eth0-tx-6189-21ntpq11:27:232
29725050,0irq/27-eth0-tx-30847-21perl12:07:162
29725050,0irq/27-eth0-tx-28051-21munin-node08:47:280
29725050,0irq/27-eth0-tx-21298-21munin-node08:42:230
29725050,0irq/27-eth0-tx-13584-21munin-node12:17:272
29725050,0irq/27-eth0-tx-0-21swapper/211:39:342
29725050,0irq/27-eth0-tx-0-21swapper/210:44:182
29725050,0irq/27-eth0-tx-0-21swapper/010:12:180
29725050,0irq/27-eth0-tx-0-21swapper/009:42:270
29725050,0irq/27-eth0-tx-0-21swapper/009:07:220
29725050,0irq/27-eth0-tx-0-21swapper/009:02:180
29715050,0irq/26-eth0-rx-31634-21tr08:47:223
29715050,0irq/26-eth0-rx-2800-21sendmail_mailst12:27:253
29715050,0irq/26-eth0-rx-21467-21hwlatdetect10:17:193
29715050,0irq/26-eth0-rx-19712-21awk12:17:273
29715050,0irq/26-eth0-rx-18241-21df_abs10:37:263
29715050,0irq/26-eth0-rx-16739-21if_err_eth007:02:183
29715050,0irq/26-eth0-rx-0-21swapper/312:22:213
29715050,0irq/26-eth0-rx-0-21swapper/312:02:183
29715050,0irq/26-eth0-rx-0-21swapper/311:28:513
29715050,0irq/26-eth0-rx-0-21swapper/310:29:313
259950,0migration/2948-21latency_hist11:07:032
259950,0migration/27575-21kthreadcore09:17:222
259950,0migration/232721-21kthreadcore07:12:222
259950,0migration/232078-21missed_timers08:22:202
259950,0migration/229380-21latency_hist09:12:042
259950,0migration/227720-21runrttasks12:22:392
259950,0migration/224053-21irqrtprio12:02:202
259950,0migration/222865-21sendmail_mailst11:17:272
259950,0migration/21728-21fschecks_time11:07:182
259950,0migration/215425-21kthreadcore08:12:222
259950,0migration/211274-21kthreadcore10:52:212
249950,0watchdog/216502-21rt-features09:22:262
179950,0migration/132363-21sh11:26:051
179950,0migration/124538-21kthreadcore08:42:221
156969955,0cyclictest15415-21ssh11:34:321
156969955,0cyclictest0-21swapper/112:26:531
156969955,0cyclictest0-21swapper/112:14:161
156969955,0cyclictest0-21swapper/108:37:101
156969950,0cyclictest0-21swapper/110:56:231
156969950,0cyclictest0-21swapper/110:44:051
156969950,0cyclictest0-21swapper/110:38:441
156959955,0cyclictest0-21swapper/012:31:110
156959955,0cyclictest0-21swapper/011:18:020
156959955,0cyclictest0-21swapper/011:07:000
156959955,0cyclictest0-21swapper/010:52:200
156959950,0cyclictest0-21swapper/011:55:330
149950,0migration/06865-21munin-node08:27:260
149950,0migration/06603-21kthreadcore09:17:220
149950,0migration/030768-21kthreadcore08:22:190
149950,0migration/026929-21kthreadcore07:32:210
149950,0migration/021239-21kthreadcore09:27:220
149950,0migration/019019-21kthreadcore07:27:200
149950,0migration/017925-21aten_r1power_fr10:37:180
149950,0migration/015613-21sh11:35:070
8215040,0irq/24-0000:00:0-21swapper/208:27:172
339940,0migration/37394-21taskset08:30:043
339940,0migration/322818-21kthreadcore08:17:223
339940,0migration/322638-21taskset07:07:143
339940,0migration/32172-21kthreadcore11:47:213
339940,0migration/320471-21kthreadcore09:02:223
339940,0migration/315540-21kthreadcore08:12:223
339940,0migration/312568-21kthreadcore08:57:223
29725040,0irq/27-eth0-tx-1642-21ssh10:47:162
29725040,0irq/27-eth0-tx-0-21swapper/212:27:242
29725040,0irq/27-eth0-tx-0-21swapper/211:32:162
29725040,0irq/27-eth0-tx-0-21swapper/211:22:042
29725040,0irq/27-eth0-tx-0-21swapper/211:12:052
29725040,0irq/27-eth0-tx-0-21swapper/210:37:152
29725040,0irq/27-eth0-tx-0-21swapper/210:32:332
29725040,0irq/27-eth0-tx-0-21swapper/007:52:160
29725040,0irq/27-eth0-tx-0-21swapper/007:12:260
29725040,0irq/27-eth0-tx-0-21swapper/007:02:190
29715040,0irq/26-eth0-rx-3751-21kthreadcore08:02:213
29715040,0irq/26-eth0-rx-30781-21ps09:12:203
29715040,0irq/26-eth0-rx-13869-21latency_hist10:12:043
29715040,0irq/26-eth0-rx-0-21swapper/312:00:283
29715040,0irq/26-eth0-rx-0-21swapper/311:55:513
29715040,0irq/26-eth0-rx-0-21swapper/311:42:423
29715040,0irq/26-eth0-rx-0-21swapper/310:47:263
29715040,0irq/26-eth0-rx-0-21swapper/310:35:513
29715040,0irq/26-eth0-rx-0-21swapper/310:22:313
29715040,0irq/26-eth0-rx-0-21swapper/309:17:163
29715040,0irq/26-eth0-rx-0-21swapper/307:57:213
259940,0migration/29283-21kthreadcore07:42:202
259940,0migration/28352-21kthreadcore12:12:202
259940,0migration/27286-21idleruntime08:07:172
259940,0migration/26271-21kthreadcore10:27:222
259940,0migration/232349-21kthreadcore08:47:222
259940,0migration/230217-21sendmail_mailqu11:42:232
259940,0migration/225949-21kthreadcore08:42:222
259940,0migration/225377-21kthreadcore07:32:202
259940,0migration/221290-21kthreadcore07:52:212
259940,0migration/218114-21df_abs10:57:172
259940,0migration/217960-21kthreadcore07:02:212
259940,0migration/214200-21kthreadcore09:47:202
259940,0migration/213823-21kthreadcore07:47:222
259940,0migration/21173-21kthreadcore09:37:192
259940,0migration/210472-21kthreadcore08:32:202
249940,0watchdog/2633-21diskstats10:02:182
249940,0watchdog/230261-21hwlatdetect09:12:182
249940,0watchdog/223162-21tr10:17:222
249940,0watchdog/219426-21netstat10:12:222
179940,0migration/17981-21fschecks_count11:52:151
179940,0migration/17564-21kthreadcore07:17:221
179940,0migration/15470-21kthreadcore10:02:261
179940,0migration/130665-21kernelversion07:12:221
179940,0migration/130365-21aten_r1power_po12:07:141
179940,0migration/12917-21kthreadcore08:27:201
179940,0migration/127353-21kthreadcore07:57:211
179940,0migration/127296-21kthreadcore12:02:211
179940,0migration/126517-21kthreadcore09:32:231
179940,0migration/126223-21fschecks_time11:02:161
179940,0migration/125533-21kthreadcore09:07:221
179940,0migration/123861-21taskset11:19:201
179940,0migration/120768-21aten_r1power_fr10:17:161
179940,0migration/119443-21kthreadcore07:27:211
179940,0migration/118544-21kthreadcore07:02:211
179940,0migration/118311-21fschecks_time10:57:211
179940,0migration/11729-21aten_r1power_fr10:47:161
179940,0migration/116419-21fschecks_count09:02:181
179940,0migration/113014-21kthreadcore09:22:221
179940,0migration/11274-21aten2.4-expect11:07:161
179940,0migration/112417-21aten2.4-expect09:47:151
179940,0migration/112110-21kthreadcore08:57:211
179940,0migration/111506-21timerwakeupswit09:42:281
179940,0migration/110085-21kthreadcore08:07:201
169940,0watchdog/13827-21ssh12:29:131
169940,0watchdog/128373-21if_eth010:22:261
169940,0watchdog/127060-21df_inode08:22:141
169940,0watchdog/118882-21cut12:17:211
169940,0watchdog/11431-21if_err_eth1.211:27:181
169940,0watchdog/113660-21fschecks_count08:12:181
169940,0watchdog/10-21swapper/111:47:211
169940,0watchdog/10-21swapper/109:52:181
156979943,0cyclictest6378-21awk09:42:202
156979943,0cyclictest26208-21awk07:57:192
156979943,0cyclictest20802-21awk08:17:192
156979943,0cyclictest16161-21awk11:57:202
156979943,0cyclictest10169-21awk08:57:202
156979940,0cyclictest13682-21latency_hist10:12:042
156969944,0cyclictest0-21swapper/111:14:221
156969944,0cyclictest0-21swapper/109:57:061
156969943,0cyclictest8022-21awk10:07:181
156969943,0cyclictest5931-21awk07:42:171
156969943,0cyclictest32014-21awk09:37:171
156969943,0cyclictest31630-21awk07:37:211
156969943,0cyclictest28983-21awk08:47:201
156969943,0cyclictest26437-21sadc08:22:041
156969943,0cyclictest16015-21awk11:57:191
156969943,0cyclictest14838-21awk10:12:181
156969943,0cyclictest12827-21awk07:47:211
156959944,0cyclictest0-21swapper/010:17:200
156959943,0cyclictest3556-21lldpd12:06:580
156959943,0cyclictest3241-21irqbalance11:48:320
156959943,0cyclictest3241-21irqbalance11:46:220
156959943,0cyclictest3241-21irqbalance11:12:120
156959943,0cyclictest3241-21irqbalance11:00:120
156959943,0cyclictest3241-21irqbalance10:07:320
156959943,0cyclictest3241-21irqbalance09:57:220
156959943,0cyclictest3241-21irqbalance09:54:020
156959943,0cyclictest3241-21irqbalance08:37:120
156959943,0cyclictest3241-21irqbalance08:21:220
156959943,0cyclictest29476-21munin-node09:12:200
156959943,0cyclictest28452-21awk10:22:260
156959943,0cyclictest25187-21awk09:32:210
156959943,0cyclictest23548-21awk07:07:190
156959943,0cyclictest2094-21awk11:07:200
156959943,0cyclictest17442-21awk11:37:180
156959943,0cyclictest1423-21awk11:27:170
156959943,0cyclictest14088-21awk08:12:200
156959943,0cyclictest13886-21grep08:32:260
156959943,0cyclictest13231-21awk09:47:190
156959943,0cyclictest1174-21awk10:02:230
156959943,0cyclictest11668-21awk09:22:210
149940,0migration/09355-21cpu08:57:160
149940,0migration/06782-21irqcore12:12:190
149940,0migration/06499-21kthreadcore10:27:220
149940,0migration/06302-21latency_hist08:07:040
149940,0migration/05028-21kthreadcore08:52:220
149940,0migration/04674-21timerwakeupswit09:37:250
149940,0migration/03904-21aten2.4-expect07:17:160
149940,0migration/027824-21kthreadcore11:22:200
149940,0migration/026882-21kthreadcore07:57:210
149940,0migration/014063-21kthreadcore07:47:220
149940,0migration/013158-21kthreadcore10:32:200
339930,0migration/329475-21aten_r1power_fr07:12:153
29725030,0irq/27-eth0-tx-0-21swapper/007:22:150
29715030,0irq/26-eth0-rx-0-21swapper/310:02:173
29715030,0irq/26-eth0-rx-0-21swapper/309:57:153
29715030,0irq/26-eth0-rx-0-21swapper/309:54:033
29715030,0irq/26-eth0-rx-0-21swapper/309:37:133
29715030,0irq/26-eth0-rx-0-21swapper/309:27:163
29715030,0irq/26-eth0-rx-0-21swapper/308:52:253
29715030,0irq/26-eth0-rx-0-21swapper/308:45:523
29715030,0irq/26-eth0-rx-0-21swapper/308:32:153
29715030,0irq/26-eth0-rx-0-21swapper/308:22:133
29715030,0irq/26-eth0-rx-0-21swapper/307:17:163
259930,0migration/214810-21cpuspeed_turbos08:37:172
249930,0watchdog/20-21swapper/210:22:262
249930,0watchdog/20-21swapper/209:52:182
249930,0watchdog/20-21swapper/208:52:222
249930,0watchdog/20-21swapper/207:27:222
249930,0watchdog/20-21swapper/207:17:232
179930,0migration/128178-21kthreadcore07:32:211
169930,0watchdog/17569-21kthreadcore09:17:221
169930,0watchdog/10-21swapper/111:37:171
169930,0watchdog/10-21swapper/109:12:181
169930,0watchdog/10-21swapper/107:07:191
156979932,0cyclictest548-21awk08:02:192
156979932,0cyclictest26834-21awk09:57:192
156979932,0cyclictest25274-21awk09:32:222
156979932,0cyclictest25238-21kthreadcore07:07:212
156979932,0cyclictest24084-21sadc09:32:042
156969932,0cyclictest461-21awk08:02:181
156969932,0cyclictest3556-21lldpd11:42:581
156969932,0cyclictest3220-21awk08:52:191
156969932,0cyclictest18392-21awk09:27:201
156969932,0cyclictest10518-21awk10:32:181
156969931,0cyclictest0-21swapper/108:32:161
156969930,0cyclictest0-21swapper/107:54:081
156959932,0cyclictest5946-21awk07:42:170
156959932,0cyclictest3556-21lldpd11:59:580
156959932,0cyclictest31582-21awk07:37:210
156959932,0cyclictest31259-21awk12:07:180
156959932,0cyclictest2524-21awk10:47:260
156959932,0cyclictest16866-21tr12:17:210
156959930,0cyclictest29688-21kthreadcore10:42:210
156959930,0cyclictest0-21swapper/008:07:160
35120,0ktimersoftd/38304-21kthreadcore08:07:203
29715020,0irq/26-eth0-rx-0-21swapper/309:42:083
29715020,0irq/26-eth0-rx-0-21swapper/308:37:443
169920,0watchdog/10-21swapper/107:22:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional