You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 23:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Wed Dec 10, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
436024332,0sleep00-21swapper/006:37:030
446623727,0sleep30-21swapper/306:38:183
434123531,0sleep10-21swapper/106:36:481
417623127,0sleep20-21swapper/206:34:572
459499130,0cyclictest0-21swapper/008:41:410
4595991010,0cyclictest16361-21sensors10:54:191
28415070,0irq/27-eth0-tx-6878-21munin-node07:29:280
28415070,0irq/27-eth0-tx-0-21swapper/007:42:210
8225060,0irq/24-0000:00:16901-21sleep11:18:562
8225060,0irq/24-0000:00:15515-21munin-run11:38:562
45979960,0cyclictest0-21swapper/208:42:112
45959960,0cyclictest0-21swapper/108:34:131
339960,0migration/39993-21sh09:05:133
339960,0migration/32413-21sh09:01:103
339960,0migration/313514-21munin-node07:09:133
339960,0migration/310808-21sh10:30:123
28415060,0irq/27-eth0-tx-30273-21ssh10:00:462
28415060,0irq/27-eth0-tx-27384-21munin-node10:44:112
28415060,0irq/27-eth0-tx-26479-21ssh08:54:252
28415060,0irq/27-eth0-tx-2107-21ssh09:00:082
28415060,0irq/27-eth0-tx-10030-21munin-node09:29:182
28415060,0irq/27-eth0-tx-0-21swapper/211:42:232
28415060,0irq/27-eth0-tx-0-21swapper/211:29:062
28415060,0irq/27-eth0-tx-0-21swapper/211:25:572
28415060,0irq/27-eth0-tx-0-21swapper/211:20:392
28415060,0irq/27-eth0-tx-0-21swapper/211:11:542
28415060,0irq/27-eth0-tx-0-21swapper/210:53:262
28415060,0irq/27-eth0-tx-0-21swapper/210:40:282
28415060,0irq/27-eth0-tx-0-21swapper/210:37:112
28415060,0irq/27-eth0-tx-0-21swapper/210:24:192
28415060,0irq/27-eth0-tx-0-21swapper/209:43:162
28415060,0irq/27-eth0-tx-0-21swapper/209:17:272
28415060,0irq/27-eth0-tx-0-21swapper/209:05:582
28415060,0irq/27-eth0-tx-0-21swapper/007:20:100
28405060,0irq/26-eth0-rx-22543-21tr09:14:153
179960,0migration/126546-21kthreadcore11:24:151
169960,0watchdog/10-21swapper/111:11:151
149960,0migration/09498-21sh10:51:150
149960,0migration/02925-21gltestperf08:14:130
149960,0migration/02925-21gltestperf08:14:130
149960,0migration/025517-21fschecks_time06:54:110
149960,0migration/017419-21turbostat09:33:560
149960,0migration/012847-21munin-node06:44:150
8225050,0irq/24-0000:00:22831-21perf08:33:572
8225050,0irq/24-0000:00:1128-21cat07:48:572
45979950,0cyclictest0-21swapper/210:19:372
45959955,0cyclictest0-21swapper/109:30:491
45959955,0cyclictest0-21swapper/106:58:591
45959950,0cyclictest0-21swapper/108:09:281
35150,0ktimersoftd/30-21swapper/308:41:383
339950,0migration/3838-21kthreadcore10:04:143
339950,0migration/36001-21aten_r1power_vo07:04:103
339950,0migration/35251-21kthreadcore09:24:163
339950,0migration/33887-21irqcore10:49:133
339950,0migration/332244-21munin-node07:44:173
339950,0migration/329994-21kthreadcore11:24:163
339950,0migration/326604-21kthreadcore10:19:173
339950,0migration/324469-21diskmemload11:23:303
339950,0migration/324463-21kthreadcore12:04:193
339950,0migration/323604-21kthreadcore10:39:133
339950,0migration/322077-21idleruntime08:04:113
339950,0migration/320804-21sh10:15:203
339950,0migration/320582-21unixbench-2d11:59:203
339950,0migration/315702-21latency_hist11:38:563
339950,0migration/314996-21munin-node07:59:093
339950,0migration/314109-21fschecks_count07:34:133
339950,0migration/312977-21vmstat07:29:283
339950,0migration/31261-21munin-node11:29:143
339950,0migration/312484-21fschecks_count10:34:103
339950,0migration/311586-21kthreadcore09:49:163
339950,0migration/310581-21irqrtprio11:14:143
28415050,0irq/27-eth0-tx-23440-21munin-node11:44:132
28415050,0irq/27-eth0-tx-2232-21munin-node09:24:122
28415050,0irq/27-eth0-tx-19476-21ssh08:51:512
28415050,0irq/27-eth0-tx-18366-21munin-node10:59:092
28415050,0irq/27-eth0-tx-1449-21kthreadcore10:04:142
28415050,0irq/27-eth0-tx-0-21swapper/212:05:002
28415050,0irq/27-eth0-tx-0-21swapper/211:49:152
28415050,0irq/27-eth0-tx-0-21swapper/211:06:052
28415050,0irq/27-eth0-tx-0-21swapper/210:54:522
28415050,0irq/27-eth0-tx-0-21swapper/210:30:472
28415050,0irq/27-eth0-tx-0-21swapper/210:10:472
28415050,0irq/27-eth0-tx-0-21swapper/209:57:152
28415050,0irq/27-eth0-tx-0-21swapper/209:50:582
28415050,0irq/27-eth0-tx-0-21swapper/209:35:072
28415050,0irq/27-eth0-tx-0-21swapper/209:22:132
28415050,0irq/27-eth0-tx-0-21swapper/209:10:412
28415050,0irq/27-eth0-tx-0-21swapper/008:34:100
28415050,0irq/27-eth0-tx-0-21swapper/008:10:320
28415050,0irq/27-eth0-tx-0-21swapper/007:34:120
28415050,0irq/27-eth0-tx-0-21swapper/007:26:580
28415050,0irq/27-eth0-tx-0-21swapper/006:49:100
28415050,0irq/27-eth0-tx-0-21swapper/006:39:090
28405050,0irq/26-eth0-rx-4600-21processes11:49:173
28405050,0irq/26-eth0-rx-454-21ssh11:06:293
28405050,0irq/26-eth0-rx-24213-21hddtemp_smartct11:44:113
28405050,0irq/26-eth0-rx-21647-21kthreadcore08:54:123
28405050,0irq/26-eth0-rx-0-21swapper/311:34:103
28405050,0irq/26-eth0-rx-0-21swapper/310:59:143
28405050,0irq/26-eth0-rx-0-21swapper/309:58:583
28405050,0irq/26-eth0-rx-0-21swapper/309:42:193
28405050,0irq/26-eth0-rx-0-21swapper/309:37:343
28405050,0irq/26-eth0-rx-0-21swapper/309:29:143
28405050,0irq/26-eth0-rx-0-21swapper/309:18:593
259950,0migration/27297-21fschecks_time11:54:092
259950,0migration/231257-21needreboot07:19:182
259950,0migration/221167-21latency_hist08:03:572
259950,0migration/211561-21aten_r1power_cu06:44:092
249950,0watchdog/25253-21munin-node08:44:112
249950,0watchdog/219626-21cut11:59:152
249950,0watchdog/216293-21kthreadcore07:09:152
19150,0ktimersoftd/10-21swapper/109:08:171
179950,0migration/19926-21kthreadcore09:49:161
179950,0migration/19798-21kthreadcore07:54:131
179950,0migration/18181-21aten_r1power_vo11:34:091
179950,0migration/124524-21munin-node09:59:121
179950,0migration/123275-21munin-node11:19:231
179950,0migration/115038-21fschecks_count11:59:101
169950,0watchdog/136262sleep110:26:591
169950,0watchdog/130085-21grep06:54:151
169950,0watchdog/12651-21kthreadcore10:04:151
169950,0watchdog/125770-21diskstats09:39:111
169950,0watchdog/119849-21perl09:14:111
169950,0watchdog/10-21swapper/111:52:391
169950,0watchdog/10-21swapper/110:42:551
169950,0watchdog/10-21swapper/109:46:351
149950,0migration/09929-21kthreadcore11:34:140
149950,0migration/07376-21fschecks_time11:54:090
149950,0migration/0551-21sh09:44:040
149950,0migration/05461-21taskset07:02:180
149950,0migration/0400-21taskset11:06:200
149950,0migration/028323-21kthreadcore09:39:150
149950,0migration/027744-21latency_hist08:08:570
149950,0migration/026484-21taskset08:54:260
149950,0migration/023277-21munin-node11:19:230
149950,0migration/02326-21hwlatdetect11:09:100
149950,0migration/02281-21kthreadcore11:29:150
149950,0migration/020840-21taskset08:01:460
149950,0migration/020777-21kthreadcore09:14:140
149950,0migration/019991-21sh08:53:460
149950,0migration/018101-21sh09:10:590
149950,0migration/017599-21irqrtprio08:29:140
149950,0migration/016339-21sendmail_mailqu10:54:190
149950,0migration/016003-21taskset08:27:160
149950,0migration/014517-21kthreadcore10:34:140
149950,0migration/01365-21cpu07:49:090
149950,0migration/012498-21latency_hist07:08:570
45979943,0cyclictest25638-21sadc07:18:572
45979943,0cyclictest20038-21sadc07:38:572
45959944,0cyclictest0-21swapper/109:54:561
45959943,0cyclictest7822-21awk07:29:201
45959943,0cyclictest7610-21awk11:54:111
45959943,0cyclictest7525-21awk10:09:141
45959943,0cyclictest3425-21lldpd08:44:321
45959943,0cyclictest3425-21lldpd08:42:021
45959943,0cyclictest3425-21lldpd08:24:261
45959943,0cyclictest22850-21awk10:19:151
45959943,0cyclictest21206-21awk07:39:131
45959943,0cyclictest19358-21awk10:59:121
45959943,0cyclictest13491-21awk07:09:131
45959940,0cyclictest0-21swapper/107:23:041
45949944,0cyclictest0-21swapper/010:59:070
45949943,0cyclictest9886-21awk08:19:120
45949943,0cyclictest6445-21snmpd10:19:120
45949943,0cyclictest6445-21snmpd07:04:410
45949943,0cyclictest32293-21awk10:04:130
45949943,0cyclictest3161-21awk09:24:140
45949943,0cyclictest27841-21awk07:44:140
45949943,0cyclictest10226-21awk11:14:120
339940,0migration/328873-21kthreadcore10:44:143
339940,0migration/31996-21kthreadcore10:24:173
339940,0migration/319750-21kthreadcore09:54:143
339940,0migration/319629-21kthreadcore08:29:153
339940,0migration/315926-21taskset08:26:233
339940,0migration/315395-21idleruntime-cro08:23:573
339940,0migration/31491-21sh11:09:013
339940,0migration/313942-21kthreadcore06:44:153
339940,0migration/313382-21kthreadcore09:09:153
339940,0migration/310515-21latency_hist10:53:563
28415040,0irq/27-eth0-tx-0-21swapper/209:44:092
28405040,0irq/26-eth0-rx-6802-21aten2.4-expect11:54:073
28405040,0irq/26-eth0-rx-20649-21kthreadcore07:14:163
28405040,0irq/26-eth0-rx-1283-21fschecks_time09:44:133
28405040,0irq/26-eth0-rx-0-21swapper/310:09:173
28405040,0irq/26-eth0-rx-0-21swapper/308:52:403
28405040,0irq/26-eth0-rx-0-21swapper/308:44:103
28405040,0irq/26-eth0-rx-0-21swapper/307:49:103
259940,0migration/27742-21open_inodes08:14:192
259940,0migration/27742-21open_inodes08:14:192
259940,0migration/26862-21munin-node07:04:142
259940,0migration/228714-21munin-node08:09:152
259940,0migration/227126-21unixbench-2d08:04:202
259940,0migration/220431-21kthreadcore06:49:172
259940,0migration/218797-21kthreadcore10:14:172
259940,0migration/210004-21irqcore08:19:132
249940,0watchdog/27297-21kthreadcore06:39:152
249940,0watchdog/2618-21fschecks_time07:24:112
249940,0watchdog/20-21swapper/208:34:152
249940,0watchdog/20-21swapper/206:54:152
179940,0migration/1950-21runrttasks07:48:551
179940,0migration/125266-21kthreadcore11:44:141
179940,0migration/120493-21kernelversion07:14:151
179940,0migration/11739-21fschecks_time07:49:101
179940,0migration/115791-21kthreadcore10:14:161
179940,0migration/115141-21kthreadcore07:34:161
179940,0migration/111793-21sh09:09:101
169940,0watchdog/126154-21ssh11:04:071
169940,0watchdog/124608-21ssh09:37:191
169940,0watchdog/120331-21munin-node08:54:111
169940,0watchdog/119813-21kthreadcore08:29:151
169940,0watchdog/113354-21kthreadcore11:14:151
169940,0watchdog/10-21swapper/112:04:191
169940,0watchdog/10-21swapper/111:42:231
169940,0watchdog/10-21swapper/111:29:111
169940,0watchdog/10-21swapper/110:47:151
169940,0watchdog/10-21swapper/110:34:111
169940,0watchdog/10-21swapper/109:24:151
169940,0watchdog/10-21swapper/108:51:511
169940,0watchdog/10-21swapper/108:14:151
169940,0watchdog/10-21swapper/108:14:151
169940,0watchdog/10-21swapper/108:04:111
169940,0watchdog/10-21swapper/106:44:151
149940,0migration/0930-21timerwakeupswit10:44:210
149940,0migration/07119-21kthreadcore10:29:130
149940,0migration/026289-21kthreadcore11:24:150
45979932,0cyclictest7758-21awk07:29:192
45979932,0cyclictest4855-21sensors06:59:212
45959933,0cyclictest5906-21kworker/1:208:19:041
45959933,0cyclictest5906-21kworker/1:207:58:581
45959933,0cyclictest5906-21kworker/1:207:23:581
45959933,0cyclictest5906-21kworker/1:206:49:041
45959933,0cyclictest29503-21kworker/1:109:19:101
45959933,0cyclictest29503-21kworker/1:108:59:001
45959933,0cyclictest21858-21kworker/1:210:28:581
45959932,0cyclictest3015-21tr10:49:091
45949933,0cyclictest3425-21lldpd09:06:580
45949932,0cyclictest7588-21awk10:09:150
45949932,0cyclictest6445-21snmpd11:40:420
45949932,0cyclictest3425-21lldpd09:54:140
45949932,0cyclictest29146-21awk08:59:120
45949932,0cyclictest24293-21awk11:44:120
45949932,0cyclictest2222-21cut11:49:130
45949932,0cyclictest20536-21ethtool10:39:110
45949932,0cyclictest15464-21awk11:59:120
45949931,0cyclictest27877-21sed12:04:210
45949930,0cyclictest6445-21snmpd07:53:580
45949930,0cyclictest1693-40yum-updatesd-he09:23:030
28415030,0irq/27-eth0-tx-0-21swapper/007:14:120
28405030,0irq/26-eth0-rx-0-21swapper/308:34:373
28405030,0irq/26-eth0-rx-0-21swapper/308:14:103
28405030,0irq/26-eth0-rx-0-21swapper/308:14:103
28405030,0irq/26-eth0-rx-0-21swapper/308:09:123
28405030,0irq/26-eth0-rx-0-21swapper/307:54:083
28405030,0irq/26-eth0-rx-0-21swapper/307:19:403
28405030,0irq/26-eth0-rx-0-21swapper/307:01:263
28405030,0irq/26-eth0-rx-0-21swapper/306:57:343
28405030,0irq/26-eth0-rx-0-21swapper/306:52:563
28405030,0irq/26-eth0-rx-0-21swapper/306:42:243
249930,0watchdog/20-21swapper/207:39:152
169930,0watchdog/10-21swapper/107:04:151
169930,0watchdog/10-21swapper/106:38:591
149930,0migration/029895-21runrttasks09:59:300
149930,0migration/014993-21sh09:50:530
45979921,0cyclictest0-21swapper/207:53:572
45949920,0cyclictest17553-21latency_hist09:33:560
45949920,0cyclictest14357-21latency_hist10:13:560
45949920,0cyclictest0-21swapper/010:23:570
45949920,0cyclictest0-21swapper/008:43:570
28405020,0irq/26-eth0-rx-0-21swapper/307:38:573
28405020,0irq/26-eth0-rx-0-21swapper/307:24:103
249920,0watchdog/20-21swapper/208:24:032
249920,0watchdog/20-21swapper/207:43:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional