You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 09:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Wed Dec 31, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450600,0irq/25-eth10-21swapper/319:03:373
1531125644,0sleep10-21swapper/119:04:561
1521123623,0sleep00-21swapper/019:03:470
1549023220,0sleep20-21swapper/219:07:012
1566999100,0cyclictest9297-21ssh22:39:010
156699983,0cyclictest32038-21awk21:18:340
156699983,0cyclictest30666-21awk23:33:350
156699983,0cyclictest11094-21awk19:28:350
156699982,0cyclictest408-21awk00:13:360
156729970,0cyclictest0-21swapper/219:28:372
156699973,0cyclictest3556-21lldpd22:33:200
156699973,0cyclictest3556-21lldpd22:19:500
156699973,0cyclictest3556-21lldpd21:49:500
156699972,0cyclictest3556-21lldpd00:07:200
156699972,0cyclictest3514-21snmpd22:07:510
156699972,0cyclictest32181-21wc20:53:440
156699972,0cyclictest27046-21ssh22:52:000
156699972,0cyclictest26349-21awk20:28:360
156699972,0cyclictest23446-21awk21:13:330
156699972,0cyclictest21357-21awk23:08:320
156699972,0cyclictest20669-21awk20:48:360
156699972,0cyclictest13373-21cut19:53:390
156699971,0cyclictest769-21cut23:13:360
156699971,0cyclictest18942-21awk22:08:370
156699970,0cyclictest7474-21kthreadcore22:58:370
156699970,0cyclictest31644-21sh21:36:110
156699970,0cyclictest30513-21df_abs19:43:330
156699970,0cyclictest29062-21ssh00:08:380
156699970,0cyclictest29018-21munin-run19:18:190
156699970,0cyclictest25986-21kthreadcore23:48:360
156699970,0cyclictest25856-21sh22:31:020
156699970,0cyclictest2477-21kthreadcore00:33:360
156699970,0cyclictest23236-21sh00:26:040
156699970,0cyclictest22051-21ssh21:12:530
156699970,0cyclictest2025-21gltestperf21:58:340
156699970,0cyclictest17759-21port_sshpriv20:18:390
156699970,0cyclictest16345-21sh21:28:330
156699970,0cyclictest13491-21diskstats23:23:320
156699970,0cyclictest13018-21ssh23:58:450
156699970,0cyclictest11874-21munin-node22:23:360
8215060,0irq/24-0000:00:18503-21idleruntime-cro22:28:192
339960,0migration/36186-21sh21:21:273
339960,0migration/329017-21irqrtprio22:53:373
339960,0migration/322797-21sh00:07:173
339960,0migration/319603-21sh23:06:503
339960,0migration/316618-21sh22:05:363
29715060,0irq/26-eth0-rx-3793-21ssh22:58:263
29715060,0irq/26-eth0-rx-0-21swapper/300:18:383
156729960,0cyclictest0-21swapper/220:23:382
156709966,0cyclictest16836-21fschecks_time00:23:341
156699962,0cyclictest1470-21kthreadcore20:33:360
156699961,0cyclictest6859-21df_abs20:38:330
156699961,0cyclictest25166-21latency_hist21:53:190
156699961,0cyclictest24641-21sh20:03:310
156699961,0cyclictest24104-21fschecks_time19:38:360
156699961,0cyclictest20896-21kthreadcore19:58:370
156699960,0cyclictest9023-21sh21:43:310
156699960,0cyclictest7806-21rm00:18:250
156699960,0cyclictest7734-21cpuspeed_turbos21:03:320
156699960,0cyclictest7657-21grep21:23:320
156699960,0cyclictest5685-21sh23:38:220
156699960,0cyclictest5465-21aten_r1power_cu20:13:320
156699960,0cyclictest4741-21ssh23:18:210
156699960,0cyclictest4420-21grep19:48:190
156699960,0cyclictest424-21rm21:38:260
156699960,0cyclictest3804-21grep19:23:320
156699960,0cyclictest31671-21tune2fs20:08:330
156699960,0cyclictest30748-21sh23:53:220
156699960,0cyclictest29974-21gltestperf19:18:350
156699960,0cyclictest27865-21tail22:53:310
156699960,0cyclictest26231-21sh22:13:200
156699960,0cyclictest24436-21sh00:28:240
156699960,0cyclictest24436-21sh00:28:240
156699960,0cyclictest18852-21cat20:23:190
156699960,0cyclictest16872-21cat19:33:190
156699960,0cyclictest15884-21aten_r1power_cu19:08:310
156699960,0cyclictest1452-21gltestperf20:58:330
156699960,0cyclictest14316-21egrep23:43:320
156699960,0cyclictest13199-21aten_r1power_cu20:43:330
156699960,0cyclictest12146-21sh23:03:240
156699960,0cyclictest11234-21sh22:43:240
156699960,0cyclictest0-21swapper/023:28:340
8215050,0irq/24-0000:00:29007-21latency_hist19:18:192
8215050,0irq/24-0000:00:0-21swapper/219:48:192
339950,0migration/37819-21munin-node21:03:323
339950,0migration/37459-21runrttasks21:40:143
339950,0migration/37072-21fschecks_time20:38:343
339950,0migration/37006-21kthreadcore20:13:373
339950,0migration/36085-21taskset20:36:283
339950,0migration/35913-21kthreadcore22:18:363
339950,0migration/35798-21kthreadcore19:48:373
339950,0migration/35640-21kthreadcore21:58:383
339950,0migration/35066-21taskset23:54:533
339950,0migration/34726-21kthreadcore00:13:383
339950,0migration/33566-21aten_r1power_en19:23:313
339950,0migration/332131-21munin-node20:08:353
339950,0migration/332120-21runrttasks00:32:253
339950,0migration/332120-21runrttasks00:32:253
339950,0migration/331709-21kthreadcore19:18:383
339950,0migration/328749-21taskset23:13:213
339950,0migration/326595-21kthreadcore20:03:363
339950,0migration/324022-21kthreadcore19:13:353
339950,0migration/322868-21kthreadcore22:28:363
339950,0migration/320792-21munin-node20:48:363
339950,0migration/319412-21taskset20:46:173
339950,0migration/319064-21munin-node22:08:383
339950,0migration/318048-21munin-node21:48:323
339950,0migration/317908-21munin-node19:33:353
339950,0migration/317838-21kthreadcore23:43:373
339950,0migration/314950-21timerwakeupswit21:43:443
339950,0migration/310555-21munin-node19:28:333
29725050,0irq/27-eth0-tx-8657-21kthreadcore23:58:372
29725050,0irq/27-eth0-tx-5740-21munin-node23:38:312
29725050,0irq/27-eth0-tx-3187-21kthreadcore21:38:362
29725050,0irq/27-eth0-tx-29708-21munin-node23:33:382
29725050,0irq/27-eth0-tx-28801-21munin-node23:13:332
29725050,0irq/27-eth0-tx-24534-21aten_r1power_en00:28:332
29725050,0irq/27-eth0-tx-24534-21aten_r1power_en00:28:332
29725050,0irq/27-eth0-tx-24483-21munin-node20:03:322
29725050,0irq/27-eth0-tx-24333-21munin-node21:33:372
29725050,0irq/27-eth0-tx-2072-21smartctl21:58:342
29725050,0irq/27-eth0-tx-17954-21grep21:08:372
29725050,0irq/27-eth0-tx-16903-21munin-node19:33:372
29725050,0irq/27-eth0-tx-15926-21ssh21:45:392
29725050,0irq/27-eth0-tx-12521-21grep21:23:382
29725050,0irq/27-eth0-tx-12195-21munin-node23:03:372
29725050,0irq/27-eth0-tx-0-21swapper/223:57:592
29725050,0irq/27-eth0-tx-0-21swapper/222:37:422
29725050,0irq/27-eth0-tx-0-21swapper/221:13:302
29715050,0irq/26-eth0-rx-29656-21taskset23:33:223
29715050,0irq/26-eth0-rx-28957-21ssh21:14:213
29715050,0irq/26-eth0-rx-26840-21cut23:48:373
29715050,0irq/26-eth0-rx-26712-21diskstats22:13:333
29715050,0irq/26-eth0-rx-25869-21ssh21:53:343
29715050,0irq/26-eth0-rx-2475-21kthreadcore00:33:363
29715050,0irq/26-eth0-rx-24436-21aten_r1power_po21:33:313
29715050,0irq/26-eth0-rx-23120-21memory19:58:383
29715050,0irq/26-eth0-rx-22515-21kthreadcore23:08:353
29715050,0irq/26-eth0-rx-19804-21expr22:48:323
29715050,0irq/26-eth0-rx-18590-21nfs4_client23:23:383
29715050,0irq/26-eth0-rx-17996-21ssh22:27:063
29715050,0irq/26-eth0-rx-16957-21smartctl00:23:343
29715050,0irq/26-eth0-rx-12544-21uname23:58:423
29715050,0irq/26-eth0-rx-0-21swapper/323:18:333
29715050,0irq/26-eth0-rx-0-21swapper/322:43:363
29715050,0irq/26-eth0-rx-0-21swapper/322:33:333
29715050,0irq/26-eth0-rx-0-21swapper/321:23:353
29715050,0irq/26-eth0-rx-0-21swapper/321:10:343
29715050,0irq/26-eth0-rx-0-21swapper/300:08:363
259950,0migration/26456-21kthreadcore22:58:362
259950,0migration/26206-21sh21:21:302
259950,0migration/25373-21fschecks_time23:18:312
259950,0migration/2499-21gltestperf20:33:332
259950,0migration/229595-21unixbench_multi00:08:422
259950,0migration/221407-21sh23:08:332
259950,0migration/218677-21latency_hist22:28:192
259950,0migration/218502-21fschecks_count22:08:352
259950,0migration/210026-21cpuspeed_turbos22:03:322
249950,0watchdog/26999-21tune2fs20:38:342
249950,0watchdog/20-21swapper/221:28:372
179950,0migration/16021-21munin-node00:33:391
179950,0migration/111248-21switchtime20:13:441
169950,0watchdog/132139-21kthreadcore19:18:381
169950,0watchdog/128051-21kthreadcore00:08:371
156729955,0cyclictest0-21swapper/223:43:422
156729955,0cyclictest0-21swapper/221:53:592
156709955,0cyclictest0-21swapper/123:53:361
156709954,0cyclictest31482-21grep21:53:441
156709954,0cyclictest1563-21sed21:38:351
156709950,0cyclictest0-21swapper/123:09:461
156709950,0cyclictest0-21swapper/121:32:581
156709950,0cyclictest0-21swapper/100:22:461
339940,0migration/329142-21kthreadcore20:28:393
339940,0migration/32658-21kthreadcore20:58:363
339940,0migration/322919-21kthreadcore23:28:363
339940,0migration/318627-21kthreadcore19:08:373
339940,0migration/317094-21unixbench_singl19:53:453
29725040,0irq/27-eth0-tx-0-21swapper/223:48:332
29725040,0irq/27-eth0-tx-0-21swapper/223:28:202
29725040,0irq/27-eth0-tx-0-21swapper/223:23:322
29725040,0irq/27-eth0-tx-0-21swapper/222:48:302
29725040,0irq/27-eth0-tx-0-21swapper/222:43:432
29725040,0irq/27-eth0-tx-0-21swapper/222:38:412
29725040,0irq/27-eth0-tx-0-21swapper/221:48:492
29725040,0irq/27-eth0-tx-0-21swapper/200:23:292
29725040,0irq/27-eth0-tx-0-21swapper/200:13:252
29715040,0irq/26-eth0-rx-0-21swapper/323:39:053
29715040,0irq/26-eth0-rx-0-21swapper/322:38:523
29715040,0irq/26-eth0-rx-0-21swapper/321:28:303
259940,0migration/232675-21kthreadcore20:08:362
259940,0migration/232049-21sendmail_mailtr22:13:422
259940,0migration/22417-21kthreadcore20:58:362
259940,0migration/22269-21kthreadcore00:33:362
259940,0migration/220873-21proc_pri00:03:422
259940,0migration/211589-21kthreadcore21:03:372
249940,0watchdog/28424-21kthreadcore19:48:382
249940,0watchdog/2318362sleep220:31:462
249940,0watchdog/228975-21ntp_kernel_err19:38:422
249940,0watchdog/20-21swapper/222:53:372
179940,0migration/17489-21taskset21:03:291
179940,0migration/17105-21latency_hist21:23:191
179940,0migration/16983-21irqrtprio23:38:351
179940,0migration/130602-21taskset20:04:531
179940,0migration/130003-21latency_hist19:43:191
179940,0migration/128718-21taskset19:15:051
179940,0migration/127990-21kthreadcore20:28:381
179940,0migration/12460-21kthreadcore00:13:381
179940,0migration/123411-21sh21:13:331
179940,0migration/121932-21fschecks_time23:28:311
179940,0migration/120945-21sh21:10:341
179940,0migration/120878-21kthreadcore19:33:371
179940,0migration/12076-21sh21:58:341
179940,0migration/116644-21hddtemp_smartct19:08:341
179940,0migration/115305-21kthreadcore19:28:381
179940,0migration/115111-21kthreadcore20:43:391
179940,0migration/113353-21unixbench-2d21:23:431
179940,0migration/11331-21kthreadcore20:33:361
179940,0migration/112970-21kthreadcore19:53:391
169940,0watchdog/17001-21fschecks_count20:38:341
169940,0watchdog/128027-21ssh22:33:331
169940,0watchdog/127786-21irqstats20:53:381
169940,0watchdog/126504-21ssh22:32:411
169940,0watchdog/120878-21ssh23:27:491
169940,0watchdog/115155-21kthreadcore22:43:371
169940,0watchdog/114089-21munin-node23:43:331
169940,0watchdog/10-21swapper/100:03:371
156729943,0cyclictest8703-21awk00:18:352
156729943,0cyclictest3556-21lldpd19:24:492
156729943,0cyclictest3514-21snmpd20:55:592
156729943,0cyclictest3051-21awk22:18:332
156729943,0cyclictest23411-21awk19:13:332
156729943,0cyclictest11921-21sadc20:18:192
156709944,0cyclictest0-21swapper/123:49:401
156709944,0cyclictest0-21swapper/123:19:051
156709944,0cyclictest0-21swapper/122:57:571
156709944,0cyclictest0-21swapper/121:46:131
156709944,0cyclictest0-21swapper/120:49:461
156709944,0cyclictest0-21swapper/119:48:461
156709943,0cyclictest7666-21awk23:58:351
156709943,0cyclictest4721-21awk22:58:341
156709943,0cyclictest3514-21snmpd23:03:291
156709943,0cyclictest3514-21snmpd19:25:141
156709943,0cyclictest3514-21snmpd00:30:591
156709943,0cyclictest3514-21snmpd00:30:591
156709943,0cyclictest32091-21awk20:08:351
156709943,0cyclictest31053-21awk19:43:361
156709943,0cyclictest30640-21awk23:33:351
156709943,0cyclictest29785-21awk23:13:341
156709943,0cyclictest27176-21awk22:13:351
156709943,0cyclictest25273-21sadc20:28:191
156709943,0cyclictest20398-21awk22:48:341
156709943,0cyclictest13001-21awk20:18:341
29725030,0irq/27-eth0-tx-0-21swapper/219:58:322
29715030,0irq/26-eth0-rx-0-21swapper/320:25:533
29715030,0irq/26-eth0-rx-0-21swapper/320:18:393
29715030,0irq/26-eth0-rx-0-21swapper/319:43:513
29715030,0irq/26-eth0-rx-0-21swapper/319:38:343
249930,0watchdog/20-21swapper/220:43:462
19130,0ktimersoftd/19633-21latency_hist22:03:191
169930,0watchdog/10-21swapper/122:38:211
169930,0watchdog/10-21swapper/121:48:331
169930,0watchdog/10-21swapper/121:33:331
169930,0watchdog/10-21swapper/120:58:341
156729933,0cyclictest3556-21lldpd20:21:502
156729932,0cyclictest16719-21awk19:08:352
156729932,0cyclictest12032-21awk19:53:372
156729930,0cyclictest0-21swapper/220:48:352
156709933,0cyclictest2614-21cut22:18:321
156709932,0cyclictest3556-21lldpd19:58:501
156709932,0cyclictest19985-21tr22:08:391
156709932,0cyclictest14102-21kthreadcore22:23:361
29715020,0irq/26-eth0-rx-0-21swapper/320:53:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional