You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 15:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Tue Nov 25, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
949263993834,3cyclictest1026420-21kworker/u8:2+flush-259:009:39:590
949274993227,3cyclictest949466-21kworker/u8:1+flush-259:008:15:012
949274993224,5cyclictest1037198-21latency_hist08:35:002
949269993228,3cyclictest1245613-21kworker/u8:0+flush-259:011:59:591
949269993225,5cyclictest1062436-21ntpq08:55:201
949269993125,4cyclictest1177752-21gpgv10:50:001
949274993025,3cyclictest964943-21/usr/sbin/munin07:25:242
949263993025,3cyclictest990469-21apt-get07:50:020
949263993023,5cyclictest2698871-21systemd-journal08:00:000
949278992922,5cyclictest1042375-21latency_hist08:40:013
949274992921,5cyclictest1214175-21latency_hist11:25:012
949274992824,3cyclictest906517-21kworker/u8:2+flush-259:007:24:592
949274992821,5cyclictest1266162-21kworker/u8:1+flush-259:012:35:172
949274992821,5cyclictest1266162-21kworker/u8:1+flush-259:012:35:172
949269992822,4cyclictest985312-21latency_hist07:45:011
949263992822,4cyclictest1088616-21ntpq09:20:200
949278992721,5cyclictest1057959-21cp08:55:013
949263992721,5cyclictest1187527-21ntpq10:55:220
94926399271,25cyclictest1-21systemd07:41:300
949274992619,5cyclictest1218434-21ntpq11:25:222
949274992618,5cyclictest1271307-21latency_hist12:20:002
949269992622,3cyclictest1104960-21kworker/u8:3+flush-259:010:00:001
949269992621,3cyclictest2698871-21systemd-journal10:30:301
949269992618,5cyclictest1240480-21latency_hist11:50:001
949278992522,2cyclictest1084113-21kworker/u8:0+flush-259:009:20:173
949278992521,3cyclictest1037216-21kworker/u8:1+flush-259:009:00:183
949278992520,4cyclictest989586-21ntpq07:45:203
949274992521,3cyclictest1089554-21kworker/u8:1+events_unbound09:55:142
949274992520,3cyclictest949466-21kworker/u8:1+events_unbound08:20:192
949274992519,4cyclictest906517-21kworker/u8:2+flush-259:007:55:182
949263992519,4cyclictest1183267-21latency_hist10:55:010
949278992421,2cyclictest1026420-21kworker/u8:2+flush-259:010:05:003
949274992420,3cyclictest1178067-21gpgv10:50:012
949274992417,5cyclictest954419-21latency_hist07:15:002
949278992318,4cyclictest980170-21latency_hist07:40:013
949274992319,3cyclictest980191-21kworker/u8:3+flush-259:007:54:592
949274992318,3cyclictest980191-21kworker/u8:3+flush-259:007:45:002
949274992317,4cyclictest1126017-21diskstats10:00:152
949269992318,4cyclictest1057764-21apt-get08:55:011
949278992219,2cyclictest1177658-21kworker/u8:0+flush-259:010:55:243
949274992218,3cyclictest1219546-21kworker/u8:3+events_unbound12:30:152
949274992217,4cyclictest1026420-21kworker/u8:2+events_unbound11:00:012
949274992216,4cyclictest980191-21kworker/u8:3+flush-259:008:05:192
949274992216,4cyclictest1136178-21/usr/sbin/munin10:10:232
949274992214,6cyclictest1032051-21latency_hist08:30:012
949269992217,4cyclictest1166935-21latency_hist10:40:011
949269992216,4cyclictest1209019-21latency_hist11:20:011
949263992218,3cyclictest954690-21apt-get07:15:150
949278992118,2cyclictest949466-21kworker/u8:1+flush-259:008:10:173
949274992114,5cyclictest1026420-21kworker/u8:2+flush-259:009:30:172
949269992116,3cyclictest1089554-21kworker/u8:1+flush-259:009:35:171
949269992115,4cyclictest1240719-21/usr/sbin/munin11:50:141
949269992114,5cyclictest1083487-21ntpq09:15:211
949278992017,2cyclictest1178290-21kworker/u8:1+flush-259:011:54:593
949274992017,2cyclictest1219359-21kworker/u8:0+events_unbound11:35:132
949274992015,4cyclictest1104960-21kworker/u8:3+events_unbound10:05:252
949274992015,3cyclictest1245613-21kworker/u8:0+flush-259:012:05:002
949274992015,3cyclictest1178290-21kworker/u8:1+flush-259:011:50:242
949274992012,6cyclictest1006339-21latency_hist08:05:012
949269992016,3cyclictest964968-21apt-get07:25:121
949269992013,5cyclictest2698871-21systemd-journal12:04:591
949263992018,1cyclictest1016660-21latency_hist08:15:000
949278991916,2cyclictest1026420-21kworker/u8:2+events_unbound11:00:233
949278991915,2cyclictest1110321-21kworker/u8:4+flush-259:010:35:173
949274991917,1cyclictest1240488-21idleruntime-cro11:49:592
949274991914,3cyclictest1110321-21kworker/u8:4+events_unbound10:35:152
949274991914,3cyclictest1089554-21kworker/u8:1+flush-259:009:35:192
949274991914,3cyclictest1016890-21kworker/u8:3+flush-259:008:35:212
949274991913,4cyclictest2698871-21systemd-journal09:00:012
949269991916,2cyclictest1261713-21irqcore12:10:171
949269991911,6cyclictest991090-21cat07:50:011
949269991910,7cyclictest1189687-21kthreadcore11:00:181
949269991910,7cyclictest1037433-21/usr/sbin/munin08:35:161
94926999190,3cyclictest1001092-21chrt07:59:031
94926999190,2cyclictest1151213-21seq10:22:021
949263991916,2cyclictest3000061-21in:imuxsock11:25:000
949263991916,2cyclictest1219546-21kworker/u8:3+flush-259:012:30:160
949274991815,2cyclictest1234374-21ntp_kernel_pll_11:40:222
949274991815,2cyclictest1026420-21kworker/u8:2+flush-259:009:40:222
949274991814,3cyclictest1078808-21apt-get09:15:012
949274991814,3cyclictest1016890-21kworker/u8:3+flush-259:008:16:592
949274991812,4cyclictest3000063-21rs:main2
949269991813,3cyclictest1026420-21kworker/u8:2+flush-259:010:20:001
94926999180,3cyclictest1120393-21latency_hist09:54:591
949263991815,2cyclictest1110077-21idleruntime-cro09:45:000
949263991813,4cyclictest1115250-21latency_hist09:50:000
949263991812,5cyclictest1135058-21ntpq10:05:220
949263991812,4cyclictest1026920-21latency_hist08:25:010
949263991811,5cyclictest1145385-21ntpq10:15:210
949278991715,1cyclictest1214863-21irqcore11:25:173
949278991714,2cyclictest1198938-21kworker/u8:4+flush-259:011:15:173
949278991714,2cyclictest1026420-21kworker/u8:2+flush-259:011:05:253
949278991713,3cyclictest1261272-21/usr/sbin/munin12:10:243
949278991713,3cyclictest1110332-21/usr/sbin/munin09:45:153
949278991713,3cyclictest1079244-21cat09:15:073
949274991714,1cyclictest0-21swapper/212:15:012
949274991713,3cyclictest1219546-21kworker/u8:3+flush-259:012:25:152
949274991713,3cyclictest1219546-21kworker/u8:3+flush-259:012:04:592
949274991713,3cyclictest1026420-21kworker/u8:2+events_unbound09:20:252
949274991712,4cyclictest1151487-21kworker/u8:0+events_unbound10:25:122
949274991712,3cyclictest1156860-21kworker/u8:1+flush-259:010:30:182
949269991714,2cyclictest1219546-21kworker/u8:3+flush-259:011:30:151
949269991713,3cyclictest1219546-21kworker/u8:3+flush-259:012:25:181
949269991713,3cyclictest1026420-21kworker/u8:2+events_unbound10:40:191
94926999170,16cyclictest3000063-21rs:main1
949263991712,3cyclictest1089960-21diskstats09:25:150
94927499169,5cyclictest2698871-21systemd-journal08:50:002
949274991612,3cyclictest1084113-21kworker/u8:0+events_unbound09:25:172
949274991611,4cyclictest1115519-21apt-get09:50:132
94926999169,6cyclictest0-21swapper/108:15:241
94926999165,9cyclictest1047824-21cpuspeed_turbos08:45:131
94926999164,10cyclictest0-21swapper/107:20:011
94926999161,4cyclictest1141131-21cut10:15:001
94926999161,3cyclictest1130768-21apt10:05:001
949269991610,4cyclictest2698871-21systemd-journal08:08:291
949269991610,4cyclictest1266403-21apt-get12:15:121
94926999160,5cyclictest1198701-21basename11:10:001
94926999160,4cyclictest990485-21basename07:49:591
94926999160,4cyclictest1047496-21munin-run08:45:011
94926999160,4cyclictest1022484-21irqcore08:20:191
94926399169,5cyclictest1193589-21latency_hist11:05:010
949263991614,1cyclictest1156635-21latency_hist10:30:010
949263991613,2cyclictest61450irq/134-enp0s29f1:tx-011:40:240
949263991613,2cyclictest60650irq/133-enp0s29f1:rx-011:32:310
949263991613,2cyclictest60650irq/133-enp0s29f1:rx-007:10:160
949263991613,2cyclictest1225308-21cpuspeed_turbos11:35:130
949263991613,2cyclictest1104960-21kworker/u8:3+flush-259:010:20:240
949263991612,3cyclictest1016890-21kworker/u8:3+flush-259:008:30:170
949263991610,5cyclictest1255019-21ntpq12:00:230
949278991512,2cyclictest1209379-21cpuspeed_turbos11:20:143
949278991512,2cyclictest1036329-21ntp_kernel_pll_08:30:223
949278991510,4cyclictest2698871-21systemd-journal09:54:293
949278991510,4cyclictest1229251-21ntpq11:35:213
94927499157,5cyclictest2698871-21systemd-journal08:40:292
94927499156,3cyclictest1084380-21idleruntime-cro09:20:012
949274991512,2cyclictest1246274-21irqcore11:55:182
949274991511,3cyclictest949466-21kworker/u8:1+flush-259:007:20:002
949274991511,3cyclictest1026420-21kworker/u8:2+flush-259:011:05:222
94926999159,4cyclictest1063764-21diskstats09:00:161
94926999155,9cyclictest1230460-21cut11:40:131
94926999153,9cyclictest949683-21grep07:10:141
94926999152,2cyclictest970536-21irqcore07:30:181
94926999151,6cyclictest2698871-21systemd-journal09:45:001
94926999151,3cyclictest1199511-21irqcore11:10:161
94926999151,2cyclictest979300-21ntpq07:35:221
94926999151,2cyclictest1134966-21head10:05:201
94926999151,1cyclictest1182350-21ntpq10:50:211
94926999151,1cyclictest1072710-21ntpq09:05:221
94926999150,4cyclictest1286706-21idleruntime-cro12:35:001
94926999150,4cyclictest1219315-21basename11:30:001
94926999150,4cyclictest1089061-21grep09:20:251
94926999150,4cyclictest1037206-21munin-run08:35:001
94926999150,3cyclictest1291868-21idleruntime-cro12:40:011
94926999150,3cyclictest1291868-21idleruntime-cro12:40:001
94926999150,3cyclictest1188416-21munin-run11:00:001
94926999150,3cyclictest1090566-21cut09:25:181
94926999150,3cyclictest1032059-21munin-run08:30:001
94926999150,3cyclictest1011879-21tr08:10:131
949263991513,2cyclictest60650irq/133-enp0s29f1:rx-011:05:240
949263991513,1cyclictest60650irq/133-enp0s29f1:rx-012:20:300
949263991513,1cyclictest60650irq/133-enp0s29f1:rx-009:10:240
949263991513,1cyclictest60650irq/133-enp0s29f1:rx-009:05:150
949263991513,1cyclictest0-21swapper/008:55:010
949263991512,2cyclictest60650irq/133-enp0s29f1:rx-011:15:170
949263991512,2cyclictest60650irq/133-enp0s29f1:rx-007:20:110
949278991412,1cyclictest1240475-21idleruntime-cro11:49:593
949278991412,1cyclictest0-21swapper/312:05:003
949278991412,1cyclictest0-21swapper/309:40:003
949278991412,1cyclictest0-21swapper/309:05:173
94927499147,5cyclictest1068475-21awk09:05:012
949274991411,2cyclictest1182811-21unixbench_multi10:50:252
949274991410,3cyclictest990472-21dump-pmu-power07:50:002
94926999149,4cyclictest1209279-21apt-get11:20:131
94926999142,4cyclictest1001761-21grep08:00:181
949269991411,2cyclictest1257182-21kthreadcore12:05:161
94926999140,5cyclictest1229213-21ntpq11:35:201
94926999140,4cyclictest961312-21cut07:20:161
94926999140,3cyclictest1110865-21cut09:45:171
94926999140,3cyclictest1098953-21ntp_kernel_pll_09:30:221
94926999140,3cyclictest1073866-21taskset09:10:131
949263991413,1cyclictest60650irq/133-enp0s29f1:rx-009:54:250
949263991413,1cyclictest60650irq/133-enp0s29f1:rx-009:34:160
949263991413,1cyclictest60650irq/133-enp0s29f1:rx-009:16:070
949263991413,1cyclictest60650irq/133-enp0s29f1:rx-008:27:280
949263991412,1cyclictest60650irq/133-enp0s29f1:rx-012:35:150
949263991412,1cyclictest60650irq/133-enp0s29f1:rx-012:35:150
949263991412,1cyclictest60650irq/133-enp0s29f1:rx-011:45:370
949263991412,1cyclictest60650irq/133-enp0s29f1:rx-011:10:130
949263991412,1cyclictest60650irq/133-enp0s29f1:rx-010:45:240
949263991412,1cyclictest1250783-21turbostat11:59:590
949263991412,1cyclictest0-21swapper/009:00:010
949263991411,2cyclictest985791-21fschecks_time07:45:120
949263991411,2cyclictest1166919-21idleruntime-cro10:40:000
949263991411,2cyclictest1130310-21unixbench_multi10:00:260
949263991411,2cyclictest1064024-21irqcore09:00:190
949278991311,1cyclictest1178290-21kworker/u8:1-events_unbound12:00:003
949278991311,1cyclictest1154040-21kthreadcore10:25:173
949278991311,1cyclictest0-21swapper/310:25:003
949278991310,2cyclictest1281904-21cpuspeed_turbos12:30:133
949278991310,2cyclictest1219554-21/usr/sbin/munin11:30:233
949278991310,2cyclictest1141457-21cpuspeed_turbos10:15:153
949278991310,2cyclictest1006665-21cpuspeed_turbos08:05:133
94927499139,3cyclictest1271997-21tr12:20:192
94927499139,3cyclictest1198938-21kworker/u8:4+flush-259:011:15:162
94927499139,3cyclictest1141814-21irqcore10:15:182
94927499133,8cyclictest1073601-21awk09:10:012
94926399139,3cyclictest2698871-21systemd-journal08:04:430
94926399139,2cyclictest975002-21turbostat07:34:590
949263991312,1cyclictest60650irq/133-enp0s29f1:rx-012:25:170
949263991312,1cyclictest60650irq/133-enp0s29f1:rx-012:10:220
949263991312,1cyclictest60650irq/133-enp0s29f1:rx-012:08:390
949263991311,1cyclictest60650irq/133-enp0s29f1:rx-008:05:180
949263991311,1cyclictest0-21swapper/010:45:000
949263991310,2cyclictest1124578-21needreboot09:55:190
949263991310,2cyclictest1110321-21kworker/u8:4+flush-259:010:35:000
949263991310,2cyclictest1016990-21cpuspeed_turbos08:15:130
94927899128,3cyclictest2698871-21systemd-journal07:56:303
949278991210,1cyclictest0-21swapper/310:50:003
94927899120,11cyclictest492-21dbus-daemon08:30:013
949274991210,1cyclictest0-21swapper/210:20:242
949263991211,1cyclictest60650irq/133-enp0s29f1:rx-011:25:170
949263991211,1cyclictest60650irq/133-enp0s29f1:rx-008:49:220
949263991211,1cyclictest60650irq/133-enp0s29f1:rx-007:37:190
949263991210,1cyclictest60650irq/133-enp0s29f1:rx-011:50:190
949263991210,1cyclictest60650irq/133-enp0s29f1:rx-008:40:440
94927899119,1cyclictest1203919-21systemd-run11:15:003
94927899119,1cyclictest0-21swapper/307:20:133
94927899118,2cyclictest1156860-21kworker/u8:1+events_unbound10:30:143
94927899118,2cyclictest0-21swapper/312:40:013
94927899118,2cyclictest0-21swapper/312:40:003
94927499118,1cyclictest0-21swapper/207:35:212
94927499110,3cyclictest1193593-21munin-plugin-st11:05:002
94926399119,1cyclictest60650irq/133-enp0s29f1:rx-010:10:160
949263991110,1cyclictest60650irq/133-enp0s29f1:rx-007:25:240
94927899107,2cyclictest1276776-21cpuspeed_turbos12:25:153
94927899107,2cyclictest1047824-21cpuspeed_turbos08:45:123
94927899107,2cyclictest0-21swapper/310:40:153
94927499105,3cyclictest1111013-21sed09:45:182
94926399109,1cyclictest60650irq/133-enp0s29f1:rx-012:15:140
9492789997,1cyclictest0-21swapper/311:40:163
9492789997,1cyclictest0-21swapper/308:55:163
9492789996,2cyclictest1256303-21cpuspeed_turbos12:05:123
9492789995,2cyclictest968949-21ntpq07:25:203
9492789986,1cyclictest0-21swapper/307:30:183
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional