You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-14 - 16:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Sat Sep 14, 2024 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
345257099746739,5cyclictest3433841-21kworker/2:0+events12:37:222
345257099745740,4cyclictest3433841-21kworker/2:0+events08:40:432
345257099745740,4cyclictest3433841-21kworker/2:0+events08:30:442
345257099744737,6cyclictest3433841-21kworker/2:0+events08:47:392
345257099743738,4cyclictest3433841-21kworker/2:0+events08:25:212
345257099743736,6cyclictest3433841-21kworker/2:0+events08:36:042
345257499742738,3cyclictest3839199-21kworker/3:2+events12:31:153
345256899742738,3cyclictest1106546-21kworker/1:2+events08:56:471
345256899742738,3cyclictest1106546-21kworker/1:2+events08:51:391
345256899742737,4cyclictest1106546-21kworker/1:2+events09:03:021
345256899742736,4cyclictest1106546-21kworker/1:2+events09:07:421
345257499741737,3cyclictest3727264-21kworker/3:0+events11:57:313
345257099741736,4cyclictest3433841-21kworker/2:0+events11:37:462
345256899739734,4cyclictest3727429-21kworker/1:1+events11:31:071
345256899739734,4cyclictest3572577-21turbostat08:49:551
345257499738736,1cyclictest1562450-21kworker/3:0+events09:24:313
345257499735731,3cyclictest1562450-21kworker/3:0+events09:48:403
345257099735729,5cyclictest3851707-21turbostat12:34:552
345257499731725,5cyclictest1562450-21kworker/3:0+events09:32:103
345257499729724,4cyclictest3652745-21kworker/3:1+events10:30:233
345257499727723,3cyclictest3652745-21kworker/3:1+events10:21:283
345257499725722,2cyclictest3727264-21kworker/3:0+events12:16:503
345257499725721,3cyclictest3652745-21kworker/3:1+events10:36:583
345257499723719,3cyclictest3727264-21kworker/3:0+events11:00:513
345257499721717,3cyclictest3652745-21kworker/3:1+events10:18:543
345257499720716,3cyclictest1562450-21kworker/3:0+events07:41:373
345257499718713,4cyclictest3783451-21turbostat11:39:553
345257499713711,1cyclictest3727264-21kworker/3:0+events11:15:203
345257499713709,3cyclictest3727264-21kworker/3:0+events11:11:143
345257499712708,3cyclictest3727264-21kworker/3:0+events11:08:003
345257499711708,2cyclictest1562450-21kworker/3:0+events07:53:253
345257499711708,2cyclictest1562450-21kworker/3:0+events07:19:283
345257499711707,3cyclictest3727264-21kworker/3:0+events11:53:553
345257499710706,3cyclictest3727264-21kworker/3:0+events11:41:273
345257499710705,4cyclictest1562450-21kworker/3:0+events07:57:113
345257499706704,1cyclictest1562450-21kworker/3:0+events09:17:283
345257499706703,2cyclictest1562450-21kworker/3:0+events08:08:053
345257499706702,3cyclictest1562450-21kworker/3:0+events07:36:583
345257499705700,4cyclictest1562450-21kworker/3:0+events08:10:233
345257499705699,5cyclictest1562450-21kworker/3:0+events09:50:233
345257499701696,4cyclictest1562450-21kworker/3:0+events09:25:233
345257499700697,2cyclictest1562450-21kworker/3:0+events07:24:333
345257499700696,3cyclictest3652745-21kworker/3:1+events10:03:133
345257499700695,4cyclictest3652745-21kworker/3:1+events10:40:223
345257499698694,3cyclictest3727264-21kworker/3:0+events10:55:433
345257499697694,2cyclictest3727264-21kworker/3:0+events11:22:283
345257499697694,2cyclictest1562450-21kworker/3:0+events09:37:223
345257499696693,2cyclictest3727264-21kworker/3:0+events11:27:503
345257499696692,3cyclictest3727264-21kworker/3:0+events12:03:063
345257499692687,4cyclictest3652745-21kworker/3:1+events10:07:533
345257499690687,2cyclictest3839199-21kworker/3:2+events12:28:133
345257499689686,2cyclictest3727264-21kworker/3:0+events12:10:213
345257499689684,4cyclictest1562450-21kworker/3:0+events09:40:233
345257099689686,2cyclictest3777293-21turbostat11:34:552
345257499687683,3cyclictest1562450-21kworker/3:0+events09:55:233
345257499687683,3cyclictest1562450-21kworker/3:0+events08:20:233
345257499686683,2cyclictest3727264-21kworker/3:0+events10:51:043
345257499685681,3cyclictest1562450-21kworker/3:0+events07:34:403
345257499685680,4cyclictest1562450-21kworker/3:0+events09:10:233
345257499684680,3cyclictest3652745-21kworker/3:1+events10:49:383
345257499684680,3cyclictest3652745-21kworker/3:1+events10:25:233
345257499682676,5cyclictest3727264-21kworker/3:0+events12:09:493
345257499681678,2cyclictest1562450-21kworker/3:0+events08:04:183
345257499677674,2cyclictest3727264-21kworker/3:0+events12:21:373
345257499677674,2cyclictest3727264-21kworker/3:0+events11:45:233
345257499677674,2cyclictest3498028-21turbostat07:49:553
345257499675671,3cyclictest1562450-21kworker/3:0+events08:18:343
345257499672668,3cyclictest3652745-21kworker/3:1+events10:11:073
345257499671665,5cyclictest1562450-21kworker/3:0+events07:29:183
345257499671665,5cyclictest1562450-21kworker/3:0+events07:29:183
345257499668665,2cyclictest1562450-21kworker/3:0+events07:11:033
345257499649644,4cyclictest3597246-21turbostat09:09:553
345257099643640,2cyclictest3541729-21turbostat08:24:562
345256899640637,2cyclictest3771140-21turbostat11:29:551
34527602430,1chrt3452758-21apt-get07:10:101
35862382310,1sleep30-21swapper/308:55:183
3452574992320,2cyclictest499-21polkitd09:05:003
3452574992317,1cyclictest191rcu_preempt08:25:173
3452570992218,3cyclictest506-21in:imuxsock10:05:002
3452570992118,2cyclictest491-21avahi-daemon09:45:002
345257099210,11cyclictest3510892-21idleruntime-cro07:55:002
3452564992116,3cyclictest3721257-21turbostat10:45:000
3452574992011,4cyclictest0-21swapper/308:45:173
3452574992011,4cyclictest0-21swapper/308:40:163
3452574992010,4cyclictest0-21swapper/308:30:183
3452570992017,2cyclictest3758799-21idleruntime-cro11:15:002
3452568992017,2cyclictest2873-21cron08:45:001
3452568992016,2cyclictest3851690-21cron12:30:001
3452570991916,2cyclictest3802249-21sh11:50:002
3452568991916,2cyclictest3764948-21idleruntime-cro11:20:001
3452568991916,2cyclictest3604141-21irqcore09:10:151
3452568991916,2cyclictest312-21jbd2/sda2-810:45:001
3452568991915,2cyclictest3665482-21systemctl10:00:001
3452568991815,2cyclictest3452555-21cyclictest12:15:001
3452568991814,3cyclictest3473798-21gltestperf07:25:151
3452568991814,3cyclictest3473798-21gltestperf07:25:141
3452570991714,2cyclictest3839334-21sh12:20:002
3452570991714,2cyclictest3734854-21irqcore10:55:162
3452570991713,3cyclictest3485981-21fschecks_count07:35:132
3452570991713,3cyclictest0-21swapper/211:20:002
3452568991715,1cyclictest3739050-21kthreadcore10:55:191
3452568991713,3cyclictest0-21swapper/110:20:001
345256899170,8cyclictest493-21dbus-daemon07:25:001
3452564991712,3cyclictest3764984-21sendmail11:20:000
345257499164,4cyclictest493-21dbus-daemon12:40:003
3452570991612,3cyclictest0-21swapper/208:20:012
345257099160,3cyclictest3597252-21idleruntime-cro09:05:002
3452568991613,2cyclictest3784090-21gltestperf11:35:141
3452568991613,2cyclictest3461393-21gltestperf07:15:141
3452568991612,3cyclictest506-21in:imuxsock10:50:001
3452568991611,3cyclictest3535532-21cron08:15:011
345256499169,5cyclictest493-21dbus-daemon09:45:000
345256499161,5cyclictest493-21dbus-daemon11:30:130
3452564991614,1cyclictest3579380-21gltestperf08:50:160
3452564991613,2cyclictest3864323-21idleruntime-cro12:40:000
3452564991613,2cyclictest3847233-21kthreadcore12:25:150
3452564991613,2cyclictest3777003-21kworker/0:1+kacpid11:42:520
3452564991613,2cyclictest3510872-21sh07:55:000
34475372161,3sleep20-21swapper/207:05:152
3452574991512,2cyclictest0-21swapper/307:10:013
3452574991511,3cyclictest3579132-21cpuspeed_turbos08:50:143
3452574991511,3cyclictest3560810-21gltestperf08:35:143
3452574991510,1cyclictest191rcu_preempt11:30:153
3452570991513,1cyclictest3634332-21munin-run09:35:002
3452570991512,2cyclictest3640496-21ls09:40:002
3452570991512,2cyclictest3579262-21fschecks_count08:50:152
3452570991512,2cyclictest3485525-21cron07:35:002
3452570991512,2cyclictest0-21swapper/210:35:012
3452570991512,2cyclictest0-21swapper/208:00:002
345256899157,2cyclictest191rcu_preempt10:10:131
345256899157,1cyclictest0-21swapper/108:00:141
3452568991513,1cyclictest0-21swapper/108:30:011
3452568991512,2cyclictest3636048-21kthreadcore09:35:181
3452568991512,2cyclictest0-21swapper/112:40:001
3452568991511,3cyclictest6950irq/9-acpi10:50:131
3452568991510,4cyclictest3658734-21sshd09:51:141
345256899150,5cyclictest3646930-21acpi09:45:121
345256899150,4cyclictest3628136-21/usr/sbin/munin09:30:001
345256499159,4cyclictest1-21systemd11:05:000
345256499151,3cyclictest3614154-21systemd-journal11:25:000
3452564991512,2cyclictest3839353-21sendmail12:20:000
3452564991512,2cyclictest3603886-21fschecks_count09:10:130
3452564991512,2cyclictest3591690-21gltestperf09:00:150
3452564991512,2cyclictest0-21swapper/007:15:010
3452564991511,3cyclictest3824104-21kthreadcore12:05:190
3452564991511,3cyclictest3673154-21tr10:05:160
3452564991511,3cyclictest3659720-21cpuspeed_turbos09:55:130
3452564991510,4cyclictest3614154-21systemd-journal11:50:000
345257099149,4cyclictest3776976-21sshd11:26:132
345257099141,7cyclictest3696570-21munin-run10:25:012
3452570991411,2cyclictest3597826-21gltestperf09:05:132
3452570991411,2cyclictest0-21swapper/208:55:242
3452570991410,2cyclictest3684233-21sh10:15:002
3452570991410,2cyclictest0-21swapper/210:50:002
345256899148,4cyclictest3614154-21systemd-journal08:20:131
345256899146,1cyclictest191rcu_preempt11:45:141
345256899142,3cyclictest3609979-21tr09:15:121
3452568991412,1cyclictest3452555-21cyclictest10:35:191
3452568991411,2cyclictest3833588-21tr12:15:121
3452568991411,2cyclictest3814890-21/usr/sbin/munin12:00:181
3452568991411,2cyclictest3754378-21kthreadcore11:10:171
3452568991411,2cyclictest3640877-21cpuspeed_turbos09:40:131
3452568991411,2cyclictest3479654-21cpuspeed_turbos07:30:121
3452568991411,2cyclictest3452555-21cyclictest11:20:191
3452568991411,2cyclictest0-21swapper/110:05:011
3452568991410,2cyclictest0-21swapper/107:35:141
345256899140,4cyclictest3808012-21sed11:50:241
345256899140,3cyclictest3501452-21tr07:45:171
345256899140,3cyclictest0-21swapper/110:20:111
345256899140,2cyclictest3708468-21grep10:30:251
345256899140,2cyclictest3677362-21python310:05:231
345256499149,4cyclictest3690389-21cron10:20:000
345256499149,4cyclictest3659075-21systemd-run09:55:000
345256499147,4cyclictest3504813-21cat07:50:010
3452564991412,1cyclictest3708660-21kworker/0:0+kacpid11:13:160
3452564991412,1cyclictest3708660-21kworker/0:0+kacpid11:07:500
3452564991411,2cyclictest3827596-21gltestperf12:10:140
3452564991411,2cyclictest3785178-21tr11:35:160
3452564991411,2cyclictest3708660-21kworker/0:0+kacpid10:38:400
3452564991411,2cyclictest3693666-21kthreadcore10:20:170
3452564991411,2cyclictest3665735-21/usr/sbin/munin10:00:170
3452564991411,2cyclictest3596940-21kworker/0:2+kacpid09:32:260
3452564991411,2cyclictest3596940-21kworker/0:2+kacpid09:09:260
3452564991411,2cyclictest3578785-21idleruntime-cro08:50:000
3452564991411,2cyclictest3572499-21kworker/0:1+kacpid08:55:140
3452564991411,2cyclictest3491630-21kworker/0:2+kacpid08:34:390
3452564991411,2cyclictest0-21swapper/011:00:000
3452564991410,3cyclictest3473509-21cpuspeed_turbos07:25:120
3452564991410,3cyclictest3473509-21cpuspeed_turbos07:25:120
3452564991410,2cyclictest3808787-21cpuspeed_turbos11:55:130
345257099139,3cyclictest3524433-21irqrtprio08:05:162
345257099139,3cyclictest0-21swapper/210:30:012
345257099139,2cyclictest0-21swapper/212:30:012
345257099138,4cyclictest506-21in:imuxsock10:10:002
345257099138,3cyclictest3826153-21ntpq12:05:222
345257099137,4cyclictest3665482-21systemctl10:00:002
3452570991310,2cyclictest3734127-21latency_hist10:55:002
3452570991310,2cyclictest3603445-21kworker/u8:2-ext4-rsv-conversion10:40:002
3452570991310,2cyclictest0-21swapper/209:20:172
345256899139,3cyclictest3751615-21memory11:05:201
345256899139,2cyclictest0-21swapper/111:55:161
345256899136,1cyclictest191rcu_preempt07:55:171
345256899135,2cyclictest191rcu_preempt08:30:181
345256899132,3cyclictest1106546-21kworker/1:2-events10:30:011
345256899131,1cyclictest191rcu_preempt08:05:171
3452568991311,1cyclictest0-21swapper/112:10:011
3452568991310,2cyclictest3740665-21cpuspeed_turbos11:00:121
3452568991310,2cyclictest3616138-21cpuspeed_turbos09:20:131
345256899130,3cyclictest3852462-21fschecks_time12:30:131
345256899130,3cyclictest0-21swapper/108:15:221
345256899130,3cyclictest0-21swapper/107:50:021
345256899130,3cyclictest0-21swapper/107:40:131
345256499139,3cyclictest62850irq/126-eno1:rx-007:33:520
345256499139,3cyclictest3596940-21kworker/0:2+kacpi_notify09:20:490
345256499139,2cyclictest0-21swapper/008:05:160
345256499137,5cyclictest3464219-21tr07:15:170
345256499136,5cyclictest3614154-21systemd-journal11:30:000
345256499133,3cyclictest3560123-21kworker/0:0+kacpid08:40:160
345256499132,2cyclictest3492115-21apt-get07:40:120
3452564991311,1cyclictest3845226-21kworker/0:0+kacpid12:30:290
3452564991310,2cyclictest3684218-21kworker/0:1+kacpid10:25:130
3452564991310,2cyclictest3596940-21kworker/0:2+kacpid09:27:070
3452564991310,2cyclictest3491630-21kworker/0:2+kacpid07:38:260
3452564991310,2cyclictest0-21swapper/010:55:000
345257099129,2cyclictest3478551-21systemctl07:25:222
345257099129,2cyclictest3478551-21systemctl07:25:222
345257099129,2cyclictest3461490-21grep07:15:162
345257099129,2cyclictest0-21swapper/207:45:002
345257099120,2cyclictest35208522cut08:00:182
345256899125,1cyclictest0-21swapper/108:35:151
345256899120,4cyclictest0-21swapper/112:20:151
345256899120,3cyclictest0-21swapper/111:40:141
345256899120,3cyclictest0-21swapper/109:32:491
345256499129,2cyclictest3684218-21kworker/0:1+kacpi_notify10:30:200
345256499129,2cyclictest3535883-21cpuspeed_turbos08:15:110
345256499129,2cyclictest3511924-21irqcore07:55:150
345256499129,2cyclictest3458554-21kworker/0:0+kacpid07:20:130
345256499128,2cyclictest0-21swapper/010:45:170
345256499121,9cyclictest3566373-21munin-run08:40:000
345256499121,3cyclictest753-21runrttasks11:50:120
3452564991210,1cyclictest3491630-21kworker/0:2+kacpid08:10:500
34521262123,2sleep10-21swapper/107:06:181
345257099119,1cyclictest0-21swapper/211:20:262
345257099119,1cyclictest0-21swapper/208:10:182
345257099118,2cyclictest3827359-21cpuspeed_turbos12:10:122
345257099118,2cyclictest3790313-21cpuspeed_turbos11:40:132
345257099118,2cyclictest3740734-21cpuspeed_turbos11:00:122
345257099118,2cyclictest3498844-21gltestperf07:45:142
345257099118,2cyclictest3453060-21gltestperf07:10:132
345257099117,3cyclictest0-21swapper/210:15:142
345257099116,4cyclictest3652207-21sed09:45:242
345257099116,4cyclictest3623395-21sed09:25:162
345257099116,4cyclictest3609629-21ls09:15:002
3452570991110,1cyclictest0-21swapper/212:23:142
3452570991110,1cyclictest0-21swapper/210:42:112
345257099110,3cyclictest0-21swapper/212:00:202
345257099110,1cyclictest0-21swapper/209:54:542
345257099110,1cyclictest0-21swapper/207:22:152
345256499119,1cyclictest3678040-21kworker/0:2+kacpid10:13:580
345256499119,1cyclictest3613742-21kthreadcore09:15:170
345256499116,4cyclictest3647314-21head09:45:150
345256499114,5cyclictest493-21dbus-daemon12:25:010
345256499111,9cyclictest3548178-21apt-get08:25:110
345256499111,9cyclictest0-21swapper/012:00:130
345256499110,1cyclictest0-21swapper/008:20:480
345256499110,1cyclictest0-21swapper/008:02:060
345257099107,2cyclictest401ktimers/209:15:122
345257099107,2cyclictest3746837-21cpuspeed_turbos11:05:142
345257099104,4cyclictest3813764-21ntpq11:55:212
345256499105,3cyclictest3634602-21/usr/sbin/munin09:35:130
34523512102,2sleep00-21swapper/007:09:130
34525709997,1cyclictest0-21swapper/211:50:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional