You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-25 - 04:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Sat Oct 25, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2839030994340,2cyclictest3021143-21latency_hist22:05:003
2839025994137,3cyclictest3084065-21kworker/u8:1+flush-259:000:04:592
2839025993833,3cyclictest2864919-21gpgv19:35:002
2839030993533,2cyclictest3016242-21kworker/u8:0+events_unbound23:15:193
2839030993530,4cyclictest3109948-21latency_hist23:30:023
2839019993428,4cyclictest2698871-21systemd-journal19:49:591
2839019993326,5cyclictest2698871-21systemd-journal22:40:221
283901699331,31cyclictest837-21snmpd19:50:150
2839019993226,4cyclictest3084065-21kworker/u8:1+flush-259:023:35:001
2839019993216,14cyclictest2698871-21systemd-journal20:28:221
2839016993226,4cyclictest2896108-21latency_hist20:05:010
2839030993128,2cyclictest3021375-21kworker/u8:4+flush-259:023:25:003
2839030993128,2cyclictest2906457-21kworker/u8:4+flush-259:020:55:233
2839016993128,2cyclictest2823720-21kworker/u8:2+flush-259:019:15:270
2839016993125,4cyclictest2698871-21systemd-journal20:11:220
2839030993026,3cyclictest3021375-21kworker/u8:4+flush-259:000:05:253
2839019993023,5cyclictest2698871-21systemd-journal19:15:221
2839030992926,2cyclictest2937622-21kworker/u8:1+flush-259:021:00:163
2839016992925,3cyclictest3125742-21apt-get23:45:130
2839030992825,2cyclictest3021375-21kworker/u8:4+flush-259:023:40:233
2839030992825,2cyclictest2963658-21kworker/u8:1+flush-259:021:10:183
2839030992824,3cyclictest3021375-21kworker/u8:4+flush-259:022:55:213
2839025992824,3cyclictest3021375-21kworker/u8:4+flush-259:022:55:162
2839019992824,3cyclictest2885809-21kworker/u8:2+flush-259:020:00:001
2839016992824,3cyclictest2698871-21systemd-journal20:57:070
2839030992724,2cyclictest2994840-21apt-get21:40:123
2839030992724,2cyclictest2906694-21apt-get20:15:123
2839030992724,2cyclictest2906457-21kworker/u8:4+flush-259:021:15:203
2839025992723,3cyclictest3109971-21kworker/u8:2+events_unbound23:55:222
2839019992724,2cyclictest2906457-21kworker/u8:4+flush-259:020:40:271
2839019992721,5cyclictest3021375-21kworker/u8:4+flush-259:022:30:171
2839019992721,4cyclictest2999689-21latency_hist21:45:011
2839016992723,3cyclictest3084065-21kworker/u8:1+flush-259:000:10:180
2839030992623,2cyclictest3021375-21kworker/u8:4+events_unbound23:05:143
2839030992623,2cyclictest3000348-21kworker/u8:0+events_unbound21:45:193
2839030992623,2cyclictest2978898-21kworker/u8:3+events_unbound21:30:133
2839030992623,2cyclictest2906457-21kworker/u8:4+events_unbound21:22:233
2839030992623,2cyclictest2823720-21kworker/u8:2+events_unbound19:10:233
2839025992622,3cyclictest3089231-21apt-get23:10:132
2839025992622,3cyclictest2823720-21kworker/u8:2+events_unbound19:15:192
2839019992620,4cyclictest3119406-21ntpq23:35:201
2839019992620,4cyclictest2891206-21/usr/sbin/munin20:00:141
2839019992616,9cyclictest3002-21cron19:45:001
2839016992623,2cyclictest3016242-21kworker/u8:0+events_unbound23:05:230
2839030992523,2cyclictest3052689-21kworker/u8:3+events_unbound22:40:143
2839030992522,3cyclictest2978684-21kworker/u8:1+events_unbound21:26:233
2839030992522,2cyclictest2906457-21kworker/u8:4+events_unbound21:05:233
2839030992522,2cyclictest2885809-21kworker/u8:2+events_unbound19:55:123
2839030992522,2cyclictest2854455-21kworker/u8:3+flush-259:019:35:223
2839030992521,3cyclictest3021375-21kworker/u8:4+flush-259:023:00:233
2839025992522,2cyclictest3021375-21kworker/u8:4+flush-259:000:20:002
2839025992522,2cyclictest3021375-21kworker/u8:4+flush-259:000:05:282
2839025992522,2cyclictest3016242-21kworker/u8:0+flush-259:022:50:252
283902599250,23cyclictest3000063-21rs:main2
2839016992522,2cyclictest3021375-21kworker/u8:4+flush-259:000:35:240
2839016992522,2cyclictest2978898-21kworker/u8:3+events_unbound21:50:150
2839016992519,4cyclictest3114250-21ntpq23:30:200
2839030992421,2cyclictest3109971-21kworker/u8:2+events_unbound00:25:223
2839030992420,3cyclictest3094127-21apt-get23:15:013
2839030992420,3cyclictest3016242-21kworker/u8:0+flush-259:022:55:003
2839025992421,2cyclictest2989221-21kworker/u8:2+flush-259:021:35:152
2839025992419,3cyclictest2698871-21systemd-journal19:52:222
2839019992420,3cyclictest2885809-21kworker/u8:2+flush-259:020:15:001
2839019992419,3cyclictest3099961-21apt-get23:20:161
2839019992418,4cyclictest2698871-21systemd-journal21:03:231
2839019992418,4cyclictest2698871-21systemd-journal00:10:221
2839019992417,5cyclictest3172204-21/usr/sbin/munin00:30:151
2839019992417,5cyclictest3172204-21/usr/sbin/munin00:30:141
283901999241,20cyclictest1-21systemd23:50:001
2839016992421,2cyclictest2875459-21kworker/u8:0+events_unbound19:45:230
2839030992320,2cyclictest3084065-21kworker/u8:1+events_unbound23:35:143
2839030992320,2cyclictest2901267-21kworker/u8:3+flush-259:020:45:263
2839030992320,2cyclictest2901267-21kworker/u8:3+flush-259:020:25:153
2839030992320,2cyclictest2838997-21kworker/u8:0+events_unbound19:30:203
2839030992318,3cyclictest3084065-21kworker/u8:1+events_unbound00:15:223
2839025992320,2cyclictest3010673-21kworker/u8:2+events_unbound21:55:262
2839025992320,2cyclictest2963658-21kworker/u8:1+flush-259:021:10:272
2839025992320,2cyclictest2942784-21kworker/u8:2+events_unbound20:52:232
2839025992317,4cyclictest3020276-21ntpq22:00:222
2839019992317,4cyclictest2885809-21kworker/u8:2+flush-259:020:10:001
2839019992317,4cyclictest2885809-21kworker/u8:2+flush-259:020:10:001
2839016992320,2cyclictest3109971-21kworker/u8:2+events_unbound00:30:130
2839016992320,2cyclictest3109971-21kworker/u8:2+events_unbound00:30:130
2839016992320,2cyclictest3021375-21kworker/u8:4+events_unbound23:00:220
2839016992320,2cyclictest2978898-21kworker/u8:3+events_unbound21:32:120
2839016992318,3cyclictest2983849-21latency_hist21:30:000
2839016992317,5cyclictest3041304-21ntpq22:20:230
2839030992219,2cyclictest3109971-21kworker/u8:2+events_unbound23:45:213
2839030992219,2cyclictest2901267-21kworker/u8:3+events_unbound20:30:233
2839030992218,3cyclictest2994186-21gpgv21:40:003
283902599226,8cyclictest2937379-21idleruntime-cro20:45:002
2839025992219,2cyclictest3084065-21kworker/u8:1+events_unbound23:30:182
2839025992219,2cyclictest3021375-21kworker/u8:4+events_unbound22:20:172
2839025992219,2cyclictest3016242-21kworker/u8:0+flush-259:022:35:192
2839025992219,2cyclictest3000348-21kworker/u8:0+flush-259:021:45:142
2839025992217,4cyclictest2698871-21systemd-journal22:20:002
2839025992217,4cyclictest2698871-21systemd-journal22:20:002
2839019992218,3cyclictest3015616-21gpgv22:00:001
2839019992218,3cyclictest2921881-21kworker/u8:0+flush-259:021:25:011
2839019992216,4cyclictest3042426-21apt-get22:25:141
2839019992216,4cyclictest2823720-21kworker/u8:2+flush-259:019:20:171
2839016992219,2cyclictest3052689-21kworker/u8:3+events_unbound22:40:230
2839030992118,2cyclictest3109971-21kworker/u8:2+events_unbound00:20:233
2839030992118,2cyclictest3062137-21kworker/u8:1+events_unbound22:45:203
2839030992118,2cyclictest3021375-21kworker/u8:4+flush-259:022:30:253
2839030992118,2cyclictest3021375-21kworker/u8:4+events_unbound00:00:223
2839030992118,2cyclictest3016242-21kworker/u8:0+events_unbound22:35:183
2839030992118,2cyclictest2885809-21kworker/u8:2+events_unbound20:20:143
2839030992118,2cyclictest2875459-21kworker/u8:0+flush-259:020:05:243
2839030992118,2cyclictest2875459-21kworker/u8:0+flush-259:020:05:243
2839030992118,2cyclictest2854455-21kworker/u8:3+events_unbound19:29:223
2839030992117,3cyclictest3084065-21kworker/u8:1+events_unbound23:30:203
2839030992117,3cyclictest3021375-21kworker/u8:4+events_unbound23:50:023
2839030992117,3cyclictest2978898-21kworker/u8:3+events_unbound21:57:243
2839030992115,4cyclictest3021385-21/usr/sbin/munin22:05:203
2839025992118,2cyclictest3021375-21kworker/u8:4+flush-259:023:35:222
2839025992118,2cyclictest3021375-21kworker/u8:4+flush-259:000:25:302
2839025992118,2cyclictest3016242-21kworker/u8:0+flush-259:022:05:192
2839025992118,2cyclictest2823720-21kworker/u8:2+events_unbound19:20:212
2839019992115,4cyclictest3136018-21/usr/sbin/munin23:55:151
2839019992114,5cyclictest2698871-21systemd-journal21:00:001
2839016992117,3cyclictest3016242-21kworker/u8:0+flush-259:022:35:150
2839016992115,4cyclictest3099694-21latency_hist23:20:010
2839016992114,5cyclictest3020265-21ntpq22:00:220
2839030992017,2cyclictest3016242-21kworker/u8:0+flush-259:022:25:253
2839030992017,2cyclictest2875459-21kworker/u8:0+flush-259:019:50:213
2839030992017,2cyclictest2875459-21kworker/u8:0+events_unbound19:49:563
2839030992017,2cyclictest2823720-21kworker/u8:2+flush-259:019:15:263
2839030992014,3cyclictest3026752-21base6422:10:023
2839025992017,2cyclictest3052689-21kworker/u8:3+flush-259:022:40:202
2839025992017,2cyclictest2978898-21kworker/u8:3+events_unbound21:40:222
2839025992017,2cyclictest2978684-21kworker/u8:1+events_unbound21:25:152
2839025992016,3cyclictest3021375-21kworker/u8:4+flush-259:023:05:182
2839025992014,4cyclictest3009563-21ntpq21:50:212
2839019992016,3cyclictest2823720-21kworker/u8:2+flush-259:019:30:261
2839019992015,3cyclictest2958320-21/usr/sbin/munin21:05:211
2839019992014,4cyclictest3016242-21kworker/u8:0+flush-259:023:20:001
2839019992014,4cyclictest3016242-21kworker/u8:0+flush-259:022:50:171
2839019992011,6cyclictest2988996-21latency_hist21:35:001
2839016992017,2cyclictest3021375-21kworker/u8:4+events_unbound00:00:210
2839016992017,2cyclictest3010673-21kworker/u8:2+flush-259:022:10:200
2839016992017,2cyclictest3010673-21kworker/u8:2+events_unbound22:10:000
2839016992016,3cyclictest3016242-21kworker/u8:0+flush-259:023:10:160
2839016992013,5cyclictest3124625-21ntpq23:40:230
2839030991916,2cyclictest3021375-21kworker/u8:4+events_unbound00:30:243
2839030991916,2cyclictest3021375-21kworker/u8:4+events_unbound00:30:243
2839030991916,2cyclictest2875459-21kworker/u8:0+events_unbound20:01:573
2839030991915,2cyclictest2906457-21kworker/u8:4+flush-259:020:40:213
2839030991913,4cyclictest3140986-21cp23:59:593
2839025991916,3cyclictest3084065-21kworker/u8:1+events_unbound00:20:022
2839025991916,2cyclictest3109971-21kworker/u8:2+flush-259:000:10:252
2839025991916,2cyclictest3021375-21kworker/u8:4+flush-259:000:30:212
2839025991916,2cyclictest3021375-21kworker/u8:4+flush-259:000:30:202
2839025991916,2cyclictest3016242-21kworker/u8:0+flush-259:023:20:312
2839025991916,2cyclictest2937622-21kworker/u8:1+flush-259:020:50:002
2839025991916,2cyclictest2937622-21kworker/u8:1+events_unbound21:05:012
2839025991913,4cyclictest3134938-21ntpq23:50:232
2839019991915,3cyclictest3000063-21rs:main1
2839019991912,6cyclictest2942566-21idleruntime-cro20:50:001
2839019991912,5cyclictest3109971-21kworker/u8:2+flush-259:000:25:201
2839016991917,1cyclictest211rcuc/000:20:000
2839016991917,1cyclictest0-21swapper/023:40:000
2839016991916,2cyclictest3016242-21kworker/u8:0+events_unbound23:20:280
2839016991916,2cyclictest2901267-21kworker/u8:3+flush-259:020:30:160
2839016991916,2cyclictest2901267-21kworker/u8:3+events_unbound20:45:180
2839016991915,3cyclictest3000348-21kworker/u8:0+events_unbound21:47:170
2839016991915,3cyclictest2957701-21kworker/u8:2+events_unbound21:05:130
2839016991915,3cyclictest2906457-21kworker/u8:4+events_unbound20:40:160
2839030991815,2cyclictest3109971-21kworker/u8:2+events_unbound00:37:223
2839030991815,2cyclictest2875459-21kworker/u8:0+events_unbound20:14:233
2839030991815,2cyclictest2854455-21kworker/u8:3+events_unbound19:40:013
2839030991814,3cyclictest2921881-21kworker/u8:0+events_unbound20:50:143
2839025991815,2cyclictest3021375-21kworker/u8:4+events_unbound00:35:192
2839025991814,3cyclictest2854455-21kworker/u8:3+events_unbound19:40:232
2839019991812,4cyclictest3026653-21base6422:10:001
2839016991815,2cyclictest3021375-21kworker/u8:4+flush-259:023:50:150
2839016991815,2cyclictest2978898-21kworker/u8:3+events_unbound21:37:230
2839016991814,3cyclictest3016242-21kworker/u8:0+events_unbound22:27:230
2839030991714,2cyclictest2906457-21kworker/u8:4+flush-259:021:50:243
2839030991710,5cyclictest3042155-21latency_hist22:25:003
2839025991714,2cyclictest2978684-21kworker/u8:1+flush-259:021:30:242
2839025991714,2cyclictest2906457-21kworker/u8:4+events_unbound20:55:252
2839025991714,2cyclictest2901267-21kworker/u8:3+events_unbound20:30:192
2839025991713,3cyclictest3021375-21kworker/u8:4+events_unbound23:17:222
2839019991713,2cyclictest0-21swapper/121:50:001
2839016991715,1cyclictest3021375-21kworker/u8:4+flush-259:022:30:260
2839016991714,2cyclictest3021375-21kworker/u8:4+events_unbound22:54:230
2839016991714,2cyclictest2885809-21kworker/u8:2+events_unbound20:20:260
2839016991711,4cyclictest2958080-21latency_hist21:05:010
2839016991710,5cyclictest2886205-21diskstats19:55:100
2839030991614,1cyclictest0-21swapper/319:25:003
2839025991613,2cyclictest2885809-21kworker/u8:2+events_unbound19:55:162
2839019991613,2cyclictest0-21swapper/119:30:001
2839019991612,2cyclictest0-21swapper/123:10:001
2839019991612,2cyclictest0-21swapper/100:20:001
2839016991614,1cyclictest60650irq/133-enp0s29f1:rx-019:42:170
2839016991613,3cyclictest2901267-21kworker/u8:3+events_unbound20:29:220
2839016991611,3cyclictest3078556-21base6423:00:010
2839030991511,3cyclictest2921881-21kworker/u8:0+events_unbound20:35:173
2839025991512,2cyclictest3109971-21kworker/u8:2+flush-259:023:45:252
2839025991512,2cyclictest2921881-21kworker/u8:0+events_unbound21:00:032
2839025991512,2cyclictest2885809-21kworker/u8:2+flush-259:020:20:162
2839025991512,2cyclictest2875459-21kworker/u8:0+events_unbound19:45:192
2839025991511,3cyclictest2906457-21kworker/u8:4+events_unbound21:15:162
2839025991511,3cyclictest2875459-21kworker/u8:0+events_unbound20:05:132
2839025991511,3cyclictest2875459-21kworker/u8:0+events_unbound20:05:132
2839016991514,1cyclictest60650irq/133-enp0s29f1:rx-023:55:160
2839016991513,1cyclictest60650irq/133-enp0s29f1:rx-021:56:020
2839016991513,1cyclictest60650irq/133-enp0s29f1:rx-020:51:470
2839016991513,1cyclictest60650irq/133-enp0s29f1:rx-019:25:240
2839016991512,2cyclictest2901267-21kworker/u8:3+flush-259:020:15:050
2839016991512,2cyclictest2823720-21kworker/u8:2+flush-259:019:10:240
2839016991511,3cyclictest2823720-21kworker/u8:2+events_unbound19:30:130
283903099148,4cyclictest3037052-21latency_hist22:20:003
283903099148,4cyclictest3037052-21latency_hist22:20:003
2839025991411,2cyclictest3021375-21kworker/u8:4+flush-259:023:40:162
2839025991411,2cyclictest3021375-21kworker/u8:4+events_unbound22:32:242
2839025991411,2cyclictest3016242-21kworker/u8:0+flush-259:023:25:162
2839025991411,2cyclictest2854455-21kworker/u8:3+events_unbound19:35:252
283901999147,5cyclictest3089206-21/usr/sbin/munin23:10:201
2839019991411,2cyclictest3005931-21irqcore21:50:161
2839016991412,1cyclictest60650irq/133-enp0s29f1:rx-021:19:300
2839016991412,1cyclictest60650irq/133-enp0s29f1:rx-020:35:250
2839016991412,1cyclictest60650irq/133-enp0s29f1:rx-000:25:230
2839016991412,1cyclictest60650irq/133-enp0s29f1:rx-000:20:210
2839016991411,2cyclictest3109952-21turbostat23:30:000
2839016991411,2cyclictest2865361-21kworker/u8:4+events_unbound19:35:150
283903099139,3cyclictest3109971-21kworker/u8:2+events_unbound00:10:413
283902599133,2cyclictest28400912irqcore19:10:182
2839025991310,3cyclictest2901267-21kworker/u8:3+events_unbound20:38:452
2839025991310,2cyclictest2972971-21kworker/u8:2+events_unbound21:20:252
283901999139,3cyclictest3010673-21kworker/u8:2+flush-259:022:08:531
283901999138,4cyclictest2978898-21kworker/u8:3+flush-259:021:25:261
283901999131,3cyclictest3016242-21kworker/u8:0+flush-259:022:25:001
2839019991310,2cyclictest2910674-21ntp_kernel_err20:15:211
283902599129,2cyclictest2901267-21kworker/u8:3+events_unbound20:15:242
283902599128,3cyclictest3016242-21kworker/u8:0+events_unbound22:25:212
283901999129,2cyclictest2989322-21cpuspeed_turbos21:35:131
283901999128,3cyclictest2881294-21sed19:50:171
283901999128,2cyclictest0-21swapper/121:15:181
283901999127,4cyclictest2921881-21kworker/u8:0+flush-259:020:50:191
283901999126,4cyclictest2698871-21systemd-journal23:00:011
283901999126,4cyclictest2698871-21systemd-journal00:05:001
283901999126,3cyclictest3166826-21basename00:24:591
283901999124,6cyclictest3021375-21kworker/u8:4+flush-259:023:50:261
2839016991210,1cyclictest60650irq/133-enp0s29f1:rx-021:10:270
283901999119,1cyclictest0-21swapper/122:55:261
283901999118,1cyclictest0-21swapper/100:05:141
283901999117,3cyclictest2906457-21kworker/u8:4+flush-259:021:13:531
283901999117,2cyclictest3181346-21ntpq00:35:201
283901999117,2cyclictest0-21swapper/122:50:001
283901999110,4cyclictest2927081-21awk20:35:001
283901999108,1cyclictest0-21swapper/123:25:101
283901999106,3cyclictest2865623-21uniq19:35:151
283901999105,4cyclictest2840274-21tr19:10:191
283901999100,3cyclictest2916731-21cron20:24:591
28390199995,2cyclictest3124658-21ntp_offset23:40:241
28390199995,2cyclictest3056762-21ntpq22:35:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional