You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-12 - 06:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Thu Dec 12, 2024 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3549871994542,2cyclictest75-21kswapd023:40:021
3549868994138,2cyclictest75-21kswapd022:37:590
3549868994138,2cyclictest75-21kswapd022:37:580
3549879993935,3cyclictest3694485-21kworker/u8:3+events_unbound21:30:143
3549876993936,2cyclictest75-21kswapd022:37:592
3549876993936,2cyclictest75-21kswapd022:37:582
3549879993832,4cyclictest1117191-21systemd-journal19:55:013
3549871993835,2cyclictest3653296-21kworker/u8:2+flush-259:020:55:211
3549871993834,3cyclictest3668723-21kworker/u8:0+events_unbound22:40:201
3549868993833,3cyclictest75-21kswapd023:40:030
3549868993831,5cyclictest1117191-21systemd-journal22:09:590
3549879993731,4cyclictest3850854-21cp00:00:013
3549871993734,2cyclictest3565500-21kworker/u8:2+events_unbound19:35:131
3549879993633,2cyclictest3710306-21kworker/u8:1+flush-259:022:05:153
3549871993632,3cyclictest3637142-21kworker/u8:3+flush-259:020:50:001
3549868993529,4cyclictest3799000-21/usr/sbin/munin23:10:140
3549879993431,2cyclictest3637142-21kworker/u8:3+flush-259:020:40:213
3549876993327,4cyclictest3741561-21latency_hist22:15:002
3549871993330,2cyclictest3861443-21kworker/u8:3+flush-259:000:35:191
3549871993329,3cyclictest3668723-21kworker/u8:0+flush-259:022:40:001
3549871993329,3cyclictest3668723-21kworker/u8:0+flush-259:022:39:591
3549871993327,5cyclictest1117191-21systemd-journal00:30:001
3549868993327,4cyclictest3560106-21latency_hist19:20:000
3549879993128,2cyclictest3627105-21kworker/u8:1+flush-259:020:35:013
3549879993128,2cyclictest3601303-21kworker/u8:0+flush-259:020:35:153
3549876993124,5cyclictest1117191-21systemd-journal23:20:002
3549879993027,2cyclictest3668723-21kworker/u8:0+flush-259:021:10:173
3549879993026,3cyclictest3861443-21kworker/u8:3+events_unbound00:35:193
3549879993025,3cyclictest3757688-21/usr/sbin/munin22:30:253
3549876993024,4cyclictest3793604-21latency_hist23:05:002
3549876993023,5cyclictest3757444-21latency_hist22:30:002
3549876993023,5cyclictest1117191-21systemd-journal23:00:012
3549871993027,2cyclictest3653296-21kworker/u8:2+flush-259:021:15:201
3549871993027,2cyclictest3627105-21kworker/u8:1+flush-259:020:30:011
3549879992926,2cyclictest3486427-21kworker/u8:3+flush-259:019:40:273
3549871992926,2cyclictest3668723-21kworker/u8:0+events_unbound22:00:211
3549871992926,2cyclictest3663542-21kworker/u8:1+flush-259:021:05:211
3549871992925,3cyclictest3668723-21kworker/u8:0+events_unbound22:10:211
3549879992825,2cyclictest3668723-21kworker/u8:0+flush-259:021:20:213
3549879992825,2cyclictest3668723-21kworker/u8:0+events_unbound23:20:253
3549879992825,2cyclictest3637142-21kworker/u8:3+events_unbound20:45:263
3549879992822,4cyclictest3549784-21latency_hist19:10:013
3549879992724,2cyclictest3772234-21kworker/u8:1+events_unbound22:45:233
3549871992723,3cyclictest509-21rs:main1
3549871992723,3cyclictest3565500-21kworker/u8:2+flush-259:019:45:261
3549871992721,4cyclictest1117191-21systemd-journal21:40:001
3549868992713,5cyclictest3668701-21latency_hist21:05:000
3549879992623,2cyclictest3793636-21kworker/u8:1+flush-259:023:45:153
3549879992623,2cyclictest3668723-21kworker/u8:0+events_unbound23:00:213
3549879992623,2cyclictest3560152-21kworker/u8:1+flush-259:020:05:193
3549871992620,4cyclictest3798759-21latency_hist23:10:011
354986899260,24cyclictest3689377-21munin-run21:25:010
3549879992522,2cyclictest3851113-21kworker/u8:4+flush-259:000:29:593
3549879992522,2cyclictest3741588-21kworker/u8:2+flush-259:022:20:193
3549876992522,2cyclictest3793636-21kworker/u8:1+flush-259:023:25:012
3549876992519,4cyclictest3846105-21diskstats23:55:152
3549871992522,2cyclictest3851113-21kworker/u8:4+events_unbound00:05:201
3549871992522,2cyclictest3793636-21kworker/u8:1+events_unbound23:35:241
3549871992522,2cyclictest3668723-21kworker/u8:0+flush-259:023:50:161
3549879992422,1cyclictest3606553-21kworker/u8:2+events_unbound20:20:193
3549879992422,1cyclictest3601303-21kworker/u8:0+flush-259:020:10:243
3549879992421,2cyclictest3606553-21kworker/u8:2+flush-259:020:15:253
3549871992421,2cyclictest3793636-21kworker/u8:1+flush-259:023:15:241
3549871992421,2cyclictest3606553-21kworker/u8:2+flush-259:020:15:251
3549871992421,2cyclictest3486427-21kworker/u8:3+flush-259:019:15:211
3549871992417,3cyclictest3601303-21kworker/u8:0+events_unbound20:10:211
3549868992417,5cyclictest3866415-21latency_hist00:15:000
3549879992320,2cyclictest3851113-21kworker/u8:4+flush-259:000:30:213
3549879992320,2cyclictest3668723-21kworker/u8:0+flush-259:022:10:253
3549871992320,2cyclictest3668723-21kworker/u8:0+flush-259:021:25:151
3549871992319,3cyclictest3668723-21kworker/u8:0+flush-259:023:55:251
3549879992219,2cyclictest3704696-21kworker/u8:2+flush-259:021:40:193
3549879992219,2cyclictest3601303-21kworker/u8:0+flush-259:020:00:203
3549879992219,2cyclictest3560152-21kworker/u8:1+events_unbound19:55:253
3549879992219,2cyclictest3545945-21kworker/u8:0+events_unbound19:20:213
3549879992219,2cyclictest3486427-21kworker/u8:3+flush-259:019:10:253
3549879992218,2cyclictest3663542-21kworker/u8:1+events_unbound21:05:183
3549879992217,4cyclictest3871620-21latency_hist00:20:003
3549879992217,4cyclictest3844814-21ntpq23:50:223
3549871992219,2cyclictest3741588-21kworker/u8:2+events_unbound22:25:251
3549871992219,2cyclictest3694485-21kworker/u8:3+flush-259:021:30:131
3549868992216,4cyclictest3829826-21cp23:40:010
3549868992216,4cyclictest3725250-21ntpq21:55:230
3549879992118,2cyclictest3663542-21kworker/u8:1+flush-259:021:35:253
3549879992118,2cyclictest3663542-21kworker/u8:1+events_unbound21:00:223
3549879992118,2cyclictest3601303-21kworker/u8:0+flush-259:020:50:213
3549879992118,2cyclictest3565500-21kworker/u8:2+events_unbound19:25:213
3549876992118,2cyclictest3627105-21kworker/u8:1+flush-259:020:30:252
3549876992117,3cyclictest3762571-21cron22:35:002
3549876992116,3cyclictest3871844-21kworker/u8:1+flush-259:000:20:182
3549876992116,3cyclictest3642628-21systemd-run20:40:002
3549876992114,5cyclictest3731258-21latency_hist22:05:012
3549871992118,2cyclictest3637142-21kworker/u8:3+events_unbound20:40:251
3549871992118,2cyclictest3617058-21kworker/u8:3+flush-259:020:20:271
3549879992018,2cyclictest3704696-21kworker/u8:2+flush-259:021:50:243
3549879992018,2cyclictest3668723-21kworker/u8:0+flush-259:022:50:213
3549879992017,2cyclictest3814212-21kworker/u8:3+events_unbound23:25:193
3549879992017,2cyclictest3668723-21kworker/u8:0+flush-259:021:45:173
3549879992017,2cyclictest3668723-21kworker/u8:0+events_unbound23:10:223
3549879992017,2cyclictest3668723-21kworker/u8:0+events_unbound21:25:153
3549879992017,2cyclictest3653296-21kworker/u8:2+flush-259:020:55:193
3549879992017,2cyclictest3627105-21kworker/u8:1+events_unbound20:25:193
3549876992017,2cyclictest3658406-21cron20:55:002
3549876992017,2cyclictest3486427-21kworker/u8:3+events_unbound19:45:002
3549876992014,4cyclictest3606553-21kworker/u8:2+flush-259:020:20:202
3549871992017,2cyclictest3668723-21kworker/u8:0+events_unbound00:00:081
3549871992017,2cyclictest3627105-21kworker/u8:1+events_unbound20:31:201
3549871992017,2cyclictest3545945-21kworker/u8:0+events_unbound19:20:221
3549871992017,2cyclictest3486427-21kworker/u8:3+flush-259:019:10:231
3549871992016,3cyclictest3565500-21kworker/u8:2+events_unbound19:25:271
3549871992015,4cyclictest3887665-21latency_hist00:35:001
3549871992014,4cyclictest1117191-21systemd-journal21:55:041
3549868992016,3cyclictest3830376-21kworker/u8:2+flush-259:000:09:590
3549868992016,3cyclictest3632521-21/usr/sbin/munin20:30:140
3549879991916,2cyclictest3668723-21kworker/u8:0+events_unbound22:00:183
3549876991915,3cyclictest3887928-21apt-get00:35:122
3549876991915,3cyclictest3668723-21kworker/u8:0+flush-259:021:25:162
3549876991915,3cyclictest3560152-21kworker/u8:1+events_unbound20:05:182
3549871991917,2cyclictest3601303-21kworker/u8:0+flush-259:020:35:201
3549871991916,2cyclictest3814212-21kworker/u8:3+flush-259:023:25:101
3549871991916,2cyclictest3793636-21kworker/u8:1+events_unbound23:23:571
3549871991916,2cyclictest3710306-21kworker/u8:1+events_unbound22:05:241
3549871991916,2cyclictest3668723-21kworker/u8:0+events_unbound21:45:121
3549871991916,2cyclictest3560152-21kworker/u8:1+flush-259:019:55:001
3549871991916,2cyclictest3486427-21kworker/u8:3+events_unbound19:31:211
3549871991915,2cyclictest3565500-21kworker/u8:2+events_unbound19:40:021
3549868991916,2cyclictest3845679-21sh23:55:010
3549868991914,4cyclictest3783213-21kworker/u8:3+flush-259:022:55:150
3549868991913,4cyclictest3591047-21latency_hist19:50:000
3549879991815,2cyclictest3830376-21kworker/u8:2+events_unbound00:05:123
3549879991815,2cyclictest3814212-21kworker/u8:3+events_unbound23:30:153
3549879991814,3cyclictest3851311-21diskstats00:00:173
354987699182,5cyclictest3648169-21munin-run20:45:012
3549876991812,4cyclictest3668723-21kworker/u8:0+flush-259:021:20:062
3549876991812,4cyclictest3662679-21ntpq20:55:212
3549876991811,5cyclictest3565273-21latency_hist19:25:002
3549871991815,2cyclictest3772234-21kworker/u8:1+events_unbound22:45:151
3549871991815,2cyclictest3668723-21kworker/u8:0+events_unbound23:11:201
3549871991815,2cyclictest3668723-21kworker/u8:0+events_unbound21:20:561
3549871991815,2cyclictest3601303-21kworker/u8:0+flush-259:020:00:261
3549871991815,2cyclictest3565500-21kworker/u8:2+events_unbound19:55:151
3549871991814,3cyclictest3772234-21kworker/u8:1+events_unbound22:54:571
3549868991814,3cyclictest3642622-21sh20:40:010
3549868991813,4cyclictest3752520-21kworker/u8:1+flush-259:022:30:160
3549879991714,2cyclictest3741588-21kworker/u8:2+events_unbound22:19:553
3549879991714,2cyclictest3668723-21kworker/u8:0+events_unbound23:35:143
3549879991714,2cyclictest3668723-21kworker/u8:0+events_unbound22:42:233
3549876991713,3cyclictest3877397-21base6400:25:012
3549871991714,2cyclictest3814212-21kworker/u8:3+flush-259:023:30:211
3549871991714,2cyclictest3783213-21kworker/u8:3+events_unbound23:00:161
3549871991714,2cyclictest3741588-21kworker/u8:2+events_unbound22:20:161
3549871991713,3cyclictest3788420-21latency_hist23:00:011
3549868991714,2cyclictest60650irq/133-enp0s29f1:rx-020:45:240
3549868991714,2cyclictest3612342-21irqcore20:10:170
3549868991713,3cyclictest3741545-21cron22:15:000
3549879991614,1cyclictest0-21swapper/319:34:593
3549879991613,3cyclictest3668723-21kworker/u8:0+events_unbound23:44:373
3549879991613,2cyclictest3861443-21kworker/u8:3+events_unbound00:10:293
3549879991613,2cyclictest3741588-21kworker/u8:2+events_unbound22:28:213
3549879991613,2cyclictest3486427-21kworker/u8:3+flush-259:019:15:193
3549879991612,3cyclictest3783213-21kworker/u8:3+events_unbound23:08:073
3549879991612,2cyclictest0-21swapper/300:25:003
3549876991611,3cyclictest3668723-21kworker/u8:0+flush-259:000:15:252
3549871991614,2cyclictest3668723-21kworker/u8:0+events_unbound21:41:211
3549871991613,2cyclictest3836024-21irqcore23:45:171
3549871991613,2cyclictest3762592-21idleruntime-cro22:35:001
3549871991613,2cyclictest3668723-21kworker/u8:0+flush-259:021:10:211
3549871991613,2cyclictest3668723-21kworker/u8:0+flush-259:000:15:201
3549871991613,2cyclictest3668723-21kworker/u8:0+events_unbound00:14:271
3549871991612,3cyclictest3871844-21kworker/u8:1+events_unbound00:24:551
3549871991612,3cyclictest3601303-21kworker/u8:0+events_unbound20:08:551
354986899168,6cyclictest1117191-21systemd-journal23:46:560
3549868991614,1cyclictest60650irq/133-enp0s29f1:rx-022:41:240
3549868991614,1cyclictest60650irq/133-enp0s29f1:rx-000:15:190
3549868991613,2cyclictest61450irq/134-enp0s29f1:tx-021:45:190
3549868991613,2cyclictest60650irq/133-enp0s29f1:rx-022:50:120
3549868991613,2cyclictest60650irq/133-enp0s29f1:rx-020:05:230
3549868991613,2cyclictest60650irq/133-enp0s29f1:rx-019:35:210
3549868991610,4cyclictest509-21rs:main0
354987699159,4cyclictest509-21rs:main2
3549876991512,2cyclictest3674540-21irqcore21:10:192
3549876991511,3cyclictest3560152-21kworker/u8:1+flush-259:019:45:252
3549871991512,2cyclictest3668723-21kworker/u8:0+events_unbound21:50:021
3549868991514,1cyclictest60650irq/133-enp0s29f1:rx-023:20:540
3549868991513,2cyclictest61450irq/134-enp0s29f1:tx-023:30:120
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-022:15:210
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-020:25:150
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-020:20:280
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-020:01:200
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-019:59:360
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-019:50:000
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-000:32:090
3549868991513,1cyclictest60650irq/133-enp0s29f1:rx-000:21:160
3549868991512,2cyclictest60650irq/133-enp0s29f1:rx-023:25:180
3549868991512,2cyclictest60650irq/133-enp0s29f1:rx-022:48:200
3549868991512,2cyclictest60650irq/133-enp0s29f1:rx-021:10:310
3549868991510,3cyclictest509-21rs:main0
3549879991412,2cyclictest3721215-21kworker/u8:3+events_unbound21:55:273
3549879991412,1cyclictest3560152-21kworker/u8:1+events_unbound19:45:243
3549879991411,2cyclictest3783213-21kworker/u8:3+flush-259:022:55:113
3549879991411,2cyclictest3761949-21kworker/u8:3+events_unbound22:35:543
3549879991411,2cyclictest3761949-21kworker/u8:3+events_unbound22:35:533
354987699149,4cyclictest3668723-21kworker/u8:0+flush-259:021:49:572
354987699149,3cyclictest3741588-21kworker/u8:2+events_unbound22:15:202
3549876991411,2cyclictest3793636-21kworker/u8:1+flush-259:023:40:262
3549876991411,2cyclictest3668723-21kworker/u8:0+flush-259:000:00:252
3549876991410,3cyclictest3668723-21kworker/u8:0+flush-259:023:50:012
3549871991412,1cyclictest0-21swapper/120:50:211
3549871991411,2cyclictest3663542-21kworker/u8:1+flush-259:021:00:131
3549868991413,1cyclictest60650irq/133-enp0s29f1:rx-021:07:300
3549868991413,1cyclictest60650irq/133-enp0s29f1:rx-020:17:300
3549868991413,1cyclictest60650irq/133-enp0s29f1:rx-019:13:240
3549868991412,2cyclictest60650irq/133-enp0s29f1:rx-020:50:430
3549868991412,1cyclictest60650irq/133-enp0s29f1:rx-023:00:230
3549868991412,1cyclictest60650irq/133-enp0s29f1:rx-021:30:180
3549868991412,1cyclictest60650irq/133-enp0s29f1:rx-021:15:120
3549868991412,1cyclictest60650irq/133-enp0s29f1:rx-019:40:200
3549868991411,2cyclictest60650irq/133-enp0s29f1:rx-000:25:160
354987999139,3cyclictest3803933-21kworker/u8:2+events_unbound23:16:223
3549879991311,2cyclictest3653296-21kworker/u8:2+flush-259:021:15:163
354987699139,3cyclictest3570433-21sh19:29:592
354987699138,3cyclictest3560152-21kworker/u8:1+flush-259:019:55:172
354987699137,4cyclictest1117191-21systemd-journal23:10:012
3549876991310,2cyclictest3672854-21needreboot21:05:212
3549876991310,2cyclictest0-21swapper/219:14:592
354986899139,3cyclictest60650irq/133-enp0s29f1:rx-000:35:120
354986899139,2cyclictest3715836-21tr21:50:120
354986899138,4cyclictest60650irq/133-enp0s29f1:rx-022:00:110
3549868991312,1cyclictest60650irq/133-enp0s29f1:rx-021:40:200
3549868991312,1cyclictest60650irq/133-enp0s29f1:rx-019:33:010
3549868991311,1cyclictest60650irq/133-enp0s29f1:rx-023:55:180
3549868991311,1cyclictest60650irq/133-enp0s29f1:rx-000:00:170
3549868991310,2cyclictest60650irq/133-enp0s29f1:rx-023:05:250
3549868991310,2cyclictest60650irq/133-enp0s29f1:rx-019:20:200
3549879991210,1cyclictest0-21swapper/319:35:153
354987699128,2cyclictest3777284-21ntpq22:45:202
354987699125,5cyclictest3845682-21latency_hist23:55:012
354987699125,5cyclictest3721191-21cp21:55:002
354987699124,6cyclictest3783289-21modprobe22:55:002
354986899129,2cyclictest60650irq/133-enp0s29f1:rx-022:25:140
354986899129,2cyclictest0-21swapper/019:29:590
3549868991210,1cyclictest60650irq/133-enp0s29f1:rx-020:57:180
354987699118,2cyclictest3770367-21kthreadcore22:40:202
354987699118,1cyclictest0-21swapper/221:00:242
354987699118,1cyclictest0-21swapper/219:10:012
354987699116,3cyclictest0-21swapper/223:35:172
354987699111,9cyclictest3819398-21awk23:30:012
354987699111,9cyclictest0-21swapper/219:35:102
354987699111,8cyclictest3700012-21cron21:35:002
354986899119,1cyclictest60650irq/133-enp0s29f1:rx-020:40:180
354987699107,2cyclictest3882734-21cpuspeed_turbos00:30:092
354987699107,2cyclictest3861544-21cpuspeed_turbos00:10:142
354987699104,4cyclictest3606532-21apt20:05:012
354986899105,4cyclictest0-21swapper/023:15:110
35498769994,3cyclictest3617066-21/usr/sbin/munin20:15:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional