You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 07:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Mon Oct 20, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
337550899754749,4cyclictest3421114-21kworker/1:1+events22:06:271
337550899754749,4cyclictest3421114-21kworker/1:1+events22:06:261
337550899754748,5cyclictest3421114-21kworker/1:1+events23:09:511
337551499749743,5cyclictest3775948-21turbostat00:34:553
337550899748738,5cyclictest3421114-21kworker/1:1+events22:34:271
337550899746741,4cyclictest3421114-21kworker/1:1+events19:46:331
337550899744739,4cyclictest3421114-21kworker/1:1+events00:34:011
337550899744738,5cyclictest3421114-21kworker/1:1+events23:26:011
337551499743737,5cyclictest3521353-21turbostat21:09:553
337550899743738,4cyclictest3421114-21kworker/1:1+events20:56:031
337550899743733,4cyclictest3421114-21kworker/1:1+events19:57:151
337551099742738,3cyclictest3299687-21kworker/2:1+events19:30:212
337550899742736,5cyclictest3421114-21kworker/1:1+events23:16:031
337550899741737,3cyclictest3421114-21kworker/1:1+events20:11:341
337550899741737,3cyclictest3421114-21kworker/1:1+events20:00:321
337550899741737,3cyclictest3421114-21kworker/1:1+events19:51:151
337550899740736,3cyclictest3421114-21kworker/1:1+events00:19:231
337550899740736,3cyclictest3421114-21kworker/1:1+events00:19:221
337550899740736,3cyclictest3421114-21kworker/1:1+events00:13:251
337550899740736,3cyclictest3312199-21kworker/1:0+events19:40:311
337551099739734,4cyclictest3389923-21turbostat19:24:552
337550899739734,4cyclictest3421114-21kworker/1:1+events21:08:101
337551499738733,4cyclictest3446577-21kworker/3:1+events20:15:213
337550899738734,3cyclictest3421114-21kworker/1:1+events22:26:361
337550899738733,4cyclictest3421114-21kworker/1:1+events22:19:081
337550899738733,4cyclictest3421114-21kworker/1:1+events00:27:301
337551499737732,4cyclictest3633223-21turbostat22:39:563
337550899736732,3cyclictest3421114-21kworker/1:1+events22:22:181
337550899735730,4cyclictest3421114-21kworker/1:1+events23:12:071
337550899735729,5cyclictest3421114-21kworker/1:1+events21:00:321
337551499734729,4cyclictest3459309-21kworker/3:2+events21:39:323
337550899734730,3cyclictest3421114-21kworker/1:1+events22:02:141
337550899734730,3cyclictest3421114-21kworker/1:1+events21:56:241
337550899733722,7cyclictest3421114-21kworker/1:1+events22:13:251
337550899732728,3cyclictest3421114-21kworker/1:1+events00:21:581
337551099730727,2cyclictest3496558-21kworker/2:2+events23:45:212
337550899730725,4cyclictest3421114-21kworker/1:1+events23:23:261
337550899729725,3cyclictest3421114-21kworker/1:1+events22:38:341
337551499717709,6cyclictest3459309-21kworker/3:2+events21:25:223
337551499716710,4cyclictest3459309-21kworker/3:2+events20:35:223
337551099714711,2cyclictest3707501-21turbostat23:39:552
337550899714711,2cyclictest3421114-21kworker/1:1+events20:08:241
337550899714711,2cyclictest3421114-21kworker/1:1+events20:08:231
337551099713708,4cyclictest3299687-21kworker/2:1+events20:25:212
337551499711707,3cyclictest272381-21kworker/3:0+events19:22:333
337551099710706,3cyclictest3299687-21kworker/2:1+events20:20:232
337551499708704,3cyclictest3459309-21kworker/3:2+events23:00:213
337550899708703,4cyclictest3409026-21turbostat19:39:541
337551499707701,5cyclictest3459309-21kworker/3:2+events21:50:223
337551099705701,3cyclictest3299687-21kworker/2:1+events20:33:052
337551499704701,2cyclictest3459309-21kworker/3:2+events22:54:043
337551099704698,5cyclictest3299687-21kworker/2:1+events19:25:222
337551499703698,4cyclictest3459309-21kworker/3:2+events21:14:203
337551499702699,2cyclictest3459309-21kworker/3:2+events22:56:363
337551499700695,4cyclictest3459309-21kworker/3:2+events21:19:113
337550899700695,4cyclictest3664159-21turbostat23:04:551
337551499699696,2cyclictest3459309-21kworker/3:2+events00:39:263
337551499697692,4cyclictest3490047-21turbostat20:44:563
337550599697694,2cyclictest3375261-21kworker/0:2+events19:18:580
337551499696693,2cyclictest3570804-21turbostat21:49:553
337551499696691,4cyclictest3459309-21kworker/3:2+events21:42:413
337551099696692,3cyclictest3299687-21kworker/2:1+events19:35:202
337551499695691,3cyclictest3459309-21kworker/3:2+events23:30:223
337551499694690,3cyclictest3459309-21kworker/3:2+events20:46:473
337551499691687,3cyclictest3459309-21kworker/3:2+events21:23:193
337551499690687,2cyclictest3459309-21kworker/3:2+events23:50:313
337551499690686,3cyclictest3459309-21kworker/3:2+events22:47:323
337551499689685,3cyclictest272381-21kworker/3:0+events19:11:473
337551499689684,4cyclictest3459309-21kworker/3:2+events20:53:573
337551499688685,2cyclictest3459309-21kworker/3:2+events23:35:433
337551499688685,2cyclictest3459309-21kworker/3:2+events00:05:203
337551499688684,3cyclictest3459309-21kworker/3:2+events00:00:213
337551499687684,2cyclictest3459309-21kworker/3:2+events22:41:073
337551499687683,3cyclictest3459309-21kworker/3:2+events23:56:273
337551499687682,4cyclictest3459309-21kworker/3:2+events21:30:213
337551099678674,3cyclictest3496558-21kworker/2:2+events23:41:062
337551099674671,2cyclictest3459067-21turbostat20:19:562
337550899667664,2cyclictest3744710-21turbostat00:09:541
337551499650646,3cyclictest3477645-21turbostat20:34:563
337550899650647,2cyclictest3576972-21turbostat21:54:551
337550599649646,2cyclictest3375373-21turbostat19:14:550
337551499644640,3cyclictest3381637-21turbostat19:19:553
337551499631626,4cyclictest3695108-21turbostat23:29:543
337551499623620,2cyclictest3452885-21turbostat20:14:553
337550899621617,3cyclictest3502753-21turbostat20:54:551
337551499601597,3cyclictest3719915-21turbostat23:49:553
3375514993516,13cyclictest523-21gdbus20:25:013
337551499329,10cyclictest493-21dbus-daemon23:20:213
3375505992822,4cyclictest3497119-21cut20:45:150
3375508992614,10cyclictest3783371-21irqcore00:35:171
3375508992514,10cyclictest0-21swapper/121:20:001
3375505992414,8cyclictest0-21swapper/022:40:000
3375514992317,5cyclictest3515884-21irqcore21:00:173
3375508992314,8cyclictest3557513-21needreboot21:30:201
3375508992311,2cyclictest0-21swapper/119:30:001
3375505992317,5cyclictest0-21swapper/022:45:010
3375505992315,6cyclictest0-21swapper/023:59:590
3375505992311,5cyclictest3763191-21systemd-journal19:40:000
3375510992216,1cyclictest191rcu_preempt21:50:002
3375510992212,5cyclictest0-21swapper/222:00:192
3375508992210,11cyclictest0-21swapper/121:40:221
3375505992210,2cyclictest3445486-21grep20:00:200
3375505992210,10cyclictest0-21swapper/023:15:010
3375514992117,2cyclictest3402138-21unixbench_singl19:25:253
3375510992112,4cyclictest0-21swapper/221:30:192
337550899210,12cyclictest0-21swapper/119:15:231
3375505992118,2cyclictest0-21swapper/020:00:000
3375505992111,8cyclictest3658339-21cpuspeed_turbos22:55:140
3375505992111,1cyclictest0-21swapper/022:04:590
3375505992111,1cyclictest0-21swapper/020:05:220
3375505992111,1cyclictest0-21swapper/020:05:210
337551499209,9cyclictest0-21swapper/322:30:153
3375514992012,7cyclictest3683438-21irqcore23:15:183
3375510992016,1cyclictest191rcu_preempt23:55:002
337550899208,3cyclictest3658300-21cpuspeed_turbos22:55:141
337550899208,10cyclictest0-21swapper/123:40:151
3375505992016,2cyclictest3558471-21idleruntime-cro21:35:000
3375505992016,2cyclictest3491048-21irqcore20:40:160
3375505992012,7cyclictest0-21swapper/022:30:010
37020192196,2sleep0191rcu_preempt23:30:160
337551499196,1cyclictest0-21swapper/320:05:003
3375514991915,2cyclictest3751931-21irqcore00:10:183
337551099195,4cyclictest3683559-21cut23:15:182
3375510991916,2cyclictest3415998-21irqcore19:40:182
337550899197,2cyclictest0-21swapper/122:45:231
3375508991910,2cyclictest3657564-21unixbench_singl22:50:241
337550599199,2cyclictest0-21swapper/021:25:150
337550599196,4cyclictest523-21gdbus21:45:000
337550599196,4cyclictest3763191-21systemd-journal00:25:390
3375505991916,2cyclictest3539909-21latency_hist21:20:000
3375505991910,2cyclictest3695524-21cpuspeed_turbos23:25:120
337550599190,4cyclictest3607888-21idleruntime-cro22:14:590
337551499186,9cyclictest0-21swapper/323:10:153
337551499186,5cyclictest3602785-21sed22:10:163
337551499186,10cyclictest0-21swapper/322:00:173
3375514991815,2cyclictest3763585-21latency_hist00:20:003
3375514991815,2cyclictest3763585-21latency_hist00:19:593
3375514991814,2cyclictest3509686-21irqcore20:55:193
3375514991814,2cyclictest0-21swapper/319:45:003
3375514991812,4cyclictest3769781-21sh00:25:003
3375514991811,6cyclictest3406097-21kthreadcore19:30:183
337551099183,4cyclictest3677249-21iwlist23:10:162
3375510991816,1cyclictest0-21swapper/221:05:012
3375508991815,2cyclictest3381638-21idleruntime-cro19:14:591
3375508991815,2cyclictest0-21swapper/123:59:591
337550599189,2cyclictest3664540-21cpuspeed_turbos23:00:130
337550599189,1cyclictest0-21swapper/021:10:170
337550599189,1cyclictest0-21swapper/000:30:150
337550599188,2cyclictest37828782sleep000:35:140
337550599188,1cyclictest0-21swapper/022:25:000
337550599187,4cyclictest3763324-21cron00:17:010
337550599187,4cyclictest3763324-21cron00:17:000
337550599186,4cyclictest3713724-21basename23:40:000
337550599183,1cyclictest295798rtkit-daemon20:32:410
337550599181,8cyclictest3515122-21latency_hist21:00:000
337550599180,4cyclictest3683084-21cut23:15:140
337550599180,2cyclictest3402554-21ls19:29:590
337551499179,7cyclictest0-21swapper/319:45:183
337551499175,9cyclictest3433784-21latency_hist19:55:003
337551499175,4cyclictest523-21gdbus19:55:013
337551499175,4cyclictest3614436-21expr22:20:153
337551499172,5cyclictest493-21dbus-daemon00:30:003
337551499171,8cyclictest3584373-21grep21:55:163
337551499171,7cyclictest3674858-21kthreadcore23:05:183
337551099179,1cyclictest0-21swapper/200:35:152
337551099176,3cyclictest0-21swapper/220:40:152
337551099172,4cyclictest3779783-21cut00:30:182
337551099172,4cyclictest3776285-21latency_hist00:30:012
337551099172,4cyclictest3551444-21ntpq21:25:212
3375510991714,2cyclictest3626786-21sh22:30:012
3375510991714,2cyclictest0-21swapper/221:15:002
3375510991710,2cyclictest0-21swapper/220:10:172
337551099170,5cyclictest3764295-21if_eno100:20:172
337550599177,3cyclictest3432802-21needreboot19:50:190
337550599171,9cyclictest3375369-21latency_hist19:10:010
337550599171,8cyclictest3657964-21munin-run22:55:010
3375505991713,2cyclictest0-21swapper/022:20:010
3375505991712,3cyclictest3546106-21cron21:24:590
337551499165,4cyclictest3613309-21awk22:15:243
337551499165,4cyclictest3596407-21sed22:05:183
337551499165,4cyclictest3596407-21sed22:05:173
337551499165,3cyclictest3446921-21gdbus20:05:013
337551499165,3cyclictest3446921-21gdbus20:05:013
337551499164,4cyclictest3412345-21cut19:35:183
337551499163,4cyclictest3625534-21tr22:25:203
337551499162,6cyclictest3714700-21irqcore23:40:173
337551499161,4cyclictest3473145-21irqstats20:25:163
337551099168,7cyclictest3564638-21awk21:40:002
337551099163,4cyclictest3605872-21cut22:10:192
337551099163,3cyclictest0-21swapper/223:30:152
337551099162,3cyclictest3695356-21sort23:25:102
337551099161,5cyclictest3376934-21cut19:10:192
3375510991613,2cyclictest3540404-21fschecks_count21:20:132
3375510991613,2cyclictest0-21swapper/221:45:002
3375510991613,2cyclictest0-21swapper/200:00:002
3375510991612,2cyclictest0-21swapper/223:10:012
337551099160,6cyclictest3439414-21ps19:55:212
337551099160,3cyclictest41-21ksoftirqd/219:50:012
337550899168,1cyclictest0-21swapper/120:35:131
3375508991612,3cyclictest3726532-21fschecks_count23:50:181
337550599161,2cyclictest3527513-21latency_hist21:10:000
3375505991611,4cyclictest3763191-21systemd-journal23:42:270
3375505991611,2cyclictest3657685-21kworker/0:0+kacpid23:20:070
337550599160,9cyclictest3583219-21cat21:55:010
337550599160,4cyclictest3769817-21unin-run00:25:010
337551099159,4cyclictest523-21gdbus22:25:002
337551099159,4cyclictest0-21swapper/222:55:182
337551099159,2cyclictest191rcu_preempt20:35:182
337551099159,1cyclictest191rcu_preempt22:17:342
337551099156,5cyclictest0-21swapper/221:05:182
337551099154,3cyclictest0-21swapper/220:00:172
337551099153,3cyclictest3597683-21cut22:05:192
337551099153,3cyclictest3597683-21cut22:05:182
337551099153,3cyclictest0-21swapper/221:15:262
337551099151,6cyclictest3508731-21runrttasks20:52:132
337551099151,5cyclictest3750120-21wc00:05:192
337551099151,4cyclictest3742820-21cut00:00:192
337551099151,3cyclictest3645939-21grep22:45:122
337551099151,3cyclictest3645017-21wc22:40:242
3375510991510,4cyclictest401ktimers/223:00:192
337550899154,1cyclictest295798rtkit-daemon20:34:351
3375508991512,2cyclictest3741778-21kthreadcore00:00:191
3375508991512,2cyclictest3571218-21cpuspeed_turbos21:45:111
3375508991511,3cyclictest3546666-21gltestperf21:25:151
337550599157,2cyclictest21750irq/159-ahci[0000:00:17.0]19:25:010
3375505991512,2cyclictest3732122-21kworker/0:1+kacpid00:06:510
3375505991512,2cyclictest3657685-21kworker/0:0+kacpid23:52:450
3375505991512,2cyclictest3657685-21kworker/0:0+kacpid23:08:110
3375505991512,2cyclictest3645276-21kworker/0:1+kacpid22:45:250
3375505991512,2cyclictest3477340-21kworker/0:2+kacpid21:00:160
3375505991512,2cyclictest3477340-21kworker/0:2+kacpid20:35:240
3375505991512,2cyclictest3464888-21kworker/0:1+kacpid20:22:560
3375505991512,2cyclictest3396142-21kworker/0:1+kacpid19:45:090
3375505991512,2cyclictest3396142-21kworker/0:1+kacpid19:40:020
3375505991511,2cyclictest3446412-21kworker/0:2+kacpid20:15:070
337550599150,2cyclictest3709-21packagekitd21:45:180
337551099149,4cyclictest3653020-21cut22:50:162
337551099149,4cyclictest3577327-21cpuspeed_turbos21:50:132
337551099142,3cyclictest3433229-21ls19:50:232
337551099141,5cyclictest41-21ksoftirqd/223:20:112
337551099141,3cyclictest3632916-21runrttasks22:31:152
3375510991412,1cyclictest0-21swapper/222:35:192
3375510991412,1cyclictest0-21swapper/221:55:222
3375510991410,3cyclictest3501885-21ntp_offset20:45:212
337551099140,3cyclictest3757852-21cat00:15:152
337551099140,3cyclictest3757852-21cat00:15:142
337550899149,3cyclictest3639687-21find22:40:131
3375508991411,2cyclictest0-21swapper/123:35:131
3375508991410,3cyclictest3720465-21gltestperf23:45:151
3375505991412,1cyclictest3744617-21kworker/0:0+kacpid00:11:350
3375505991412,1cyclictest3732122-21kworker/0:1+kacpid00:00:150
3375505991412,1cyclictest3589590-21kworker/0:0+kacpid22:05:070
3375505991412,1cyclictest3589590-21kworker/0:0+kacpid22:05:070
3375505991412,1cyclictest3570667-21kworker/0:2+kacpid21:55:180
3375505991412,1cyclictest3558099-21kworker/0:1+kacpid21:35:000
3375505991412,1cyclictest3489768-21kworker/0:0+kacpid20:50:170
3375505991412,1cyclictest3464888-21kworker/0:1+kacpid20:25:480
3375505991412,1cyclictest3446412-21kworker/0:2+kacpid20:12:080
3375505991411,2cyclictest3657685-21kworker/0:0+kacpid23:45:530
3375505991411,2cyclictest3613733-21kworker/0:0+kacpid22:30:090
3375505991411,2cyclictest3396142-21kworker/0:1+kacpid19:30:370
33694612142,7sleep20-21swapper/219:05:142
3375514991310,2cyclictest3375579-21latency_hist19:10:023
337551099132,3cyclictest3752145-21cut00:10:182
337551099130,5cyclictest3452665-21runrttasks20:07:212
337551099130,5cyclictest3452665-21runrttasks20:07:202
337551099130,4cyclictest3382206-21cat19:15:032
337550899138,4cyclictest3496511-21cron20:45:001
337550899131,6cyclictest34609452cat20:15:201
3375508991310,2cyclictest3701813-21fschecks_time23:30:151
337551099128,1cyclictest191rcu_preempt20:55:162
337550899128,3cyclictest3403165-21cpuspeed_turbos19:30:141
337550899128,2cyclictest0-21swapper/120:20:131
337550899122,9cyclictest3476592-21missed_timers20:25:191
337550899120,1cyclictest0-21swapper/120:45:201
337550899118,2cyclictest6950irq/9-acpi19:22:211
337550899116,4cyclictest523-21gdbus21:10:251
3375508991110,1cyclictest0-21swapper/121:35:231
337550899110,1cyclictest0-21swapper/121:22:391
33697672101,2sleep10-21swapper/119:05:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional