You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-10 - 08:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Sun Dec 10, 2023 00:44:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311681199740737,2cyclictest3163706-21turbostat19:54:563
311681199734731,2cyclictest3444561-21turbostat00:04:543
311681199732726,5cyclictest3461727-21turbostat00:19:553
311681199730728,1cyclictest3236717-21kworker/3:1+pm22:15:113
311681199729727,1cyclictest3236717-21kworker/3:1+pm22:00:103
311680499727720,6cyclictest3484376-21turbostat00:39:551
311680799726724,1cyclictest3332112-21turbostat22:24:542
311681199721718,2cyclictest3478782-21turbostat00:34:553
311680499719716,2cyclictest3242312-21turbostat21:04:551
311681199715712,2cyclictest3203407-21turbostat20:29:563
311681199714711,2cyclictest3456111-21turbostat00:14:563
311680799713710,2cyclictest3116613-21turbostat19:14:552
311680499713710,2cyclictest3433414-21turbostat23:54:541
311681199712709,2cyclictest3473238-21turbostat00:29:553
311680499712709,2cyclictest3349186-21turbostat22:39:561
311681199711708,2cyclictest3275800-21turbostat21:34:553
311681199710703,6cyclictest3303642-21turbostat21:59:563
311680499709707,1cyclictest3158160-21turbostat19:49:561
311681199708705,2cyclictest3388472-21turbostat23:14:553
311681199708705,2cyclictest3192243-21turbostat20:19:553
311680799708705,2cyclictest3343273-21turbostat22:34:552
311681199706700,5cyclictest3175235-21turbostat20:04:553
311680499705702,2cyclictest3122299-21turbostat19:19:551
311681199704701,2cyclictest3337707-21turbostat22:29:563
311681199701694,6cyclictest3129941-21turbostat19:24:553
311681199700697,2cyclictest3366244-21turbostat22:54:563
311681199700697,2cyclictest3366244-21turbostat22:54:553
311681199699697,1cyclictest3405502-21turbostat23:29:553
311681199698695,2cyclictest3438989-21turbostat23:59:553
311681199697695,1cyclictest3320692-21turbostat22:14:563
311680199697694,2cyclictest3411089-21turbostat23:34:560
311681199696694,1cyclictest3360359-21turbostat22:49:553
311680499696693,2cyclictest3314792-21turbostat22:09:561
311681199695692,2cyclictest3371832-21turbostat22:59:553
311681199695692,2cyclictest3197822-21turbostat20:24:553
311680199695693,1cyclictest3450247-21turbostat00:09:550
311681199694691,2cyclictest3394025-21turbostat23:19:553
311681199689687,1cyclictest3247902-21turbostat21:09:553
311681199689686,2cyclictest3186683-21turbostat20:14:553
311680499688686,1cyclictest3422259-21turbostat23:44:551
311681199687681,5cyclictest3281375-21turbostat21:39:553
311680499687684,2cyclictest3309212-21turbostat22:04:551
311680499687684,2cyclictest3236752-21turbostat20:59:551
311680799686684,1cyclictest3208930-21turbostat20:34:552
311681199684681,2cyclictest3135548-21turbostat19:29:563
311680499684681,2cyclictest3416659-21turbostat23:39:551
311680199684676,5cyclictest3286940-21turbostat21:44:550
311680799683680,2cyclictest3382934-21turbostat23:09:562
311681199682680,1cyclictest3264664-21turbostat21:24:553
311680799681679,1cyclictest3152574-21turbostat19:44:562
311680799681679,1cyclictest3152574-21turbostat19:44:552
311680499681678,2cyclictest3377371-21turbostat23:04:561
311680499679677,1cyclictest3298071-21turbostat21:54:561
311680499679677,1cyclictest3231195-21turbostat20:54:561
311680799678675,2cyclictest3253518-21turbostat21:14:552
311680799675667,7cyclictest3220048-21turbostat20:44:542
311681199672669,2cyclictest3141086-21turbostat19:34:553
311680799671669,1cyclictest3225630-21turbostat20:49:552
311681199670668,1cyclictest3214500-21turbostat20:39:553
311681199670664,5cyclictest3146666-21turbostat19:39:553
311680499668666,1cyclictest3270228-21turbostat21:29:551
311680499665663,1cyclictest3427843-21turbostat23:49:551
311681199664661,2cyclictest3354760-21turbostat22:44:563
311681199661658,2cyclictest3169613-21turbostat19:59:553
311680799657654,2cyclictest3292508-21turbostat21:49:552
311680499657655,1cyclictest3259087-21turbostat21:19:551
311681199654651,2cyclictest3467277-21turbostat00:24:563
311681199654651,2cyclictest3467277-21turbostat00:24:553
311680499654651,2cyclictest3399600-21turbostat23:24:551
311681199646643,2cyclictest3181138-21turbostat20:09:553
311681199635633,1cyclictest3209149-21kworker/3:0+pm20:30:123
311680799626624,1cyclictest3208951-21kworker/2:0+pm20:50:132
311680499608606,1cyclictest3198198-21kworker/1:2+pm21:30:111
311681199607605,1cyclictest3236717-21kworker/3:1+pm22:30:123
311680499607604,2cyclictest3314759-21kworker/1:1+pm23:25:121
311681199605600,4cyclictest3209149-21kworker/3:0+pm20:40:123
311680499605602,2cyclictest3198198-21kworker/1:2+pm21:55:131
311681199599594,4cyclictest2493704-21kworker/3:1+pm19:10:133
311680199579577,1cyclictest3405723-21kworker/0:0+pm23:35:100
311681199559557,1cyclictest3236717-21kworker/3:1+pm21:40:113
311680799556554,1cyclictest3247931-21kworker/2:3+pm22:35:122
311680799340338,1cyclictest3208951-21kworker/2:0+pm20:35:132
311680499310307,2cyclictest3314759-21kworker/1:1+pm22:40:121
311681199300297,2cyclictest3236717-21kworker/3:1+pm23:20:133
311681199289286,2cyclictest3236717-21kworker/3:1+pm21:10:133
311680499272270,1cyclictest3314759-21kworker/1:1+pm22:10:131
311680799271268,2cyclictest3366311-21kworker/2:1+pm23:10:122
311681199269265,2cyclictest3236717-21kworker/3:1+pm22:20:123
311680199261258,2cyclictest3450213-21kworker/0:0+pm00:10:110
311680799251248,2cyclictest3247931-21kworker/2:3+pm22:25:102
311680499251248,2cyclictest3314759-21kworker/1:1+pm23:05:111
311681199243241,1cyclictest3236717-21kworker/3:1+pm00:05:123
311680799236233,2cyclictest901458-21kworker/2:2+pm19:45:122
311680499234231,2cyclictest3110875-21kworker/1:1+pm19:50:121
311680499233230,2cyclictest3198198-21kworker/1:2+pm21:05:131
311680799229227,1cyclictest3082310-21kworker/2:0+pm19:15:102
311680799229226,2cyclictest3247931-21kworker/2:3+pm21:15:102
311681199227224,2cyclictest3479001-21kworker/3:0+pm00:35:113
311681199225222,2cyclictest3236717-21kworker/3:1+pm23:00:123
311680499224222,1cyclictest3314759-21kworker/1:1+pm23:55:121
311680499221218,2cyclictest3198198-21kworker/1:2+pm21:20:111
311681199216213,2cyclictest3236717-21kworker/3:1+pm21:25:133
311681199211209,1cyclictest3236717-21kworker/3:1+pm23:30:113
311680199208205,2cyclictest3214468-21kworker/0:0+pm21:45:110
311680799205202,2cyclictest3247931-21kworker/2:3+pm21:50:132
311680499202200,1cyclictest3110875-21kworker/1:1+pm19:20:131
311681199192190,1cyclictest3141067-21kworker/3:2+pm19:40:113
311681199192190,1cyclictest3141067-21kworker/3:2+pm19:40:103
3116804997571,3cyclictest3438693-21kworker/u8:0+events_unbound00:05:421
3116804997268,3cyclictest3337680-21kworker/u8:1+events_unbound22:25:371
3116804997167,3cyclictest3354732-21kworker/u8:2+events_unbound22:55:251
3116804996763,3cyclictest3456086-21kworker/u8:1+events_unbound00:20:391
3116804996763,3cyclictest3456086-21kworker/u8:1+events_unbound00:20:381
3116804996763,3cyclictest3354732-21kworker/u8:2+events_unbound23:15:251
3116804996561,3cyclictest3427881-21kworker/u8:3+events_unbound00:10:391
3116804996360,2cyclictest3456086-21kworker/u8:1+events_unbound00:15:231
3116807996256,4cyclictest3208918-21latency_hist20:30:012
3116804996259,2cyclictest3276023-21kworker/u8:3+events_unbound22:15:241
3116804996258,3cyclictest3236718-21kworker/u8:3+events_unbound21:10:331
3116807996053,4cyclictest3314783-21latency_hist22:05:012
3116801996052,5cyclictest3320947-21apt-get22:10:190
3116804995955,3cyclictest3393988-21kworker/u8:3+events_unbound23:30:401
3116801995953,4cyclictest3309193-21latency_hist22:00:010
3116804995854,3cyclictest3337680-21kworker/u8:1+events_unbound22:30:231
3116804995854,3cyclictest3169639-21kworker/u8:3+events_unbound20:15:371
3116807995652,2cyclictest3400082-21gpgv23:20:022
3116804995552,2cyclictest3427881-21kworker/u8:3+events_unbound00:30:231
3116804995552,2cyclictest3169639-21kworker/u8:3+events_unbound20:40:211
3116807995448,4cyclictest3485066-21diskstats00:35:252
3116807995448,4cyclictest3484375-21apt-get00:35:002
3116807995349,3cyclictest3427881-21kworker/u8:3+flush-8:023:55:242
3116807995347,4cyclictest3399580-21latency_hist23:20:012
3116807995249,2cyclictest3076740-21kworker/u8:2+events_unbound19:30:262
3116807995247,4cyclictest3974348-21systemd-journal22:55:002
3116807995247,4cyclictest3974348-21systemd-journal22:55:002
3116804995248,3cyclictest3146693-21kworker/u8:4+events_unbound19:40:291
3116804995248,3cyclictest3146693-21kworker/u8:4+events_unbound19:40:291
3116804995247,3cyclictest3110872-21kworker/u8:1+events_unbound19:10:221
3116804995148,2cyclictest3169639-21kworker/u8:3+events_unbound20:35:031
3116807995046,2cyclictest3281595-21kworker/u8:0+events_unbound21:35:322
3116804995047,2cyclictest3320728-21kworker/u8:0+events_unbound22:54:401
3116804995047,2cyclictest3320728-21kworker/u8:0+events_unbound22:54:391
3116807994943,3cyclictest3315266-21gpgv22:05:022
3116804994946,2cyclictest3281595-21kworker/u8:0+events_unbound21:35:401
3116804994945,3cyclictest3276023-21kworker/u8:3+events_unbound22:20:221
3116804994945,3cyclictest3129980-21kworker/u8:1+events_unbound20:24:401
3116804994845,2cyclictest3438693-21kworker/u8:0+events_unbound00:04:381
3116804994845,2cyclictest3169639-21kworker/u8:3+events_unbound20:14:391
3116804994744,2cyclictest3337680-21kworker/u8:1+events_unbound22:49:381
3116804994743,3cyclictest3427881-21kworker/u8:3+events_unbound00:26:391
3116807994539,4cyclictest3175812-21appstreamcli20:00:012
3116804994441,2cyclictest3129980-21kworker/u8:1+events_unbound20:00:311
3116801994439,3cyclictest3175219-21latency_hist20:00:010
3116811994340,2cyclictest3354732-21kworker/u8:2+flush-8:023:05:353
3116807994338,3cyclictest3974348-21systemd-journal22:10:012
3116804994339,3cyclictest3129980-21kworker/u8:1+events_unbound20:06:391
3116801994337,4cyclictest3974348-21systemd-journal23:22:380
3116801994236,4cyclictest3473227-21latency_hist00:25:000
3116807994138,2cyclictest3456086-21kworker/u8:1+flush-8:000:20:412
3116807994138,2cyclictest3456086-21kworker/u8:1+flush-8:000:20:402
3116801994035,3cyclictest3974348-21systemd-journal23:04:390
3116801994034,4cyclictest3135539-21latency_hist19:25:010
3116811993934,3cyclictest3416884-21kworker/u8:2+flush-8:023:40:003
3116807993936,2cyclictest3337680-21kworker/u8:1+events_unbound22:40:272
3116801993934,3cyclictest3383190-21apt-get23:05:210
3116811993831,5cyclictest3242305-21latency_hist21:00:013
3116807993834,3cyclictest3416633-21kworker/u8:1+flush-8:023:35:422
3116807993834,2cyclictest3427881-21kworker/u8:3+events_unbound23:55:012
3116811993732,3cyclictest482-21rs:main3
3116807993734,2cyclictest3259058-21kworker/u8:0+flush-8:021:20:242
3116807993733,3cyclictest3270198-21kworker/u8:1+events_unbound21:25:222
3116807993733,3cyclictest3247944-21kworker/u8:2+flush-8:021:40:352
3116807993733,3cyclictest3076740-21kworker/u8:2+events_unbound19:20:252
3116801993734,2cyclictest3225596-21kworker/u8:0+flush-8:020:50:250
3116801993734,2cyclictest3129980-21kworker/u8:1+flush-8:020:00:350
3116801993733,3cyclictest3169639-21kworker/u8:3+flush-8:020:20:310
3116801993731,4cyclictest3259078-21latency_hist21:15:000
3116811993632,3cyclictest3276023-21kworker/u8:3+flush-8:021:55:003
3116804993633,2cyclictest3281595-21kworker/u8:0+events_unbound21:40:011
3116804993632,3cyclictest3129980-21kworker/u8:1+events_unbound20:45:291
3116801993630,4cyclictest3388730-21apt-get23:10:200
3116811993533,1cyclictest3276023-21kworker/u8:3+events_unbound21:45:263
3116811993530,3cyclictest3152815-21kworker/u8:0+flush-8:019:45:353
3116807993531,3cyclictest3236718-21kworker/u8:3+events_unbound21:00:212
3116807993531,2cyclictest3416633-21kworker/u8:1+events_unbound23:40:272
3116807993528,5cyclictest3147226-21appstreamcli19:35:012
3116804993532,2cyclictest3076740-21kworker/u8:2+flush-8:019:30:001
3116804993530,2cyclictest3320728-21kworker/u8:0+events_unbound23:10:021
3116811993430,3cyclictest3416884-21kworker/u8:2+flush-8:023:45:343
3116804993428,4cyclictest3974348-21systemd-journal20:00:001
3116801993426,5cyclictest3286927-21latency_hist21:40:010
3116807993330,2cyclictest3236718-21kworker/u8:3+events_unbound21:05:262
3116801993329,3cyclictest3129980-21kworker/u8:1+flush-8:020:10:350
3116807993229,2cyclictest3163674-21kworker/u8:2+events_unbound19:55:342
3116807993229,2cyclictest3110872-21kworker/u8:1+events_unbound19:10:002
3116807993228,3cyclictest3450636-21gpgv00:05:012
3116801993229,2cyclictest3405467-21kworker/u8:4+flush-8:023:30:000
3116807993128,2cyclictest3270198-21kworker/u8:1+events_unbound21:30:352
3116807993128,2cyclictest3129980-21kworker/u8:1+flush-8:020:55:282
3116807993128,2cyclictest3121724-21kworker/u8:3+flush-8:019:25:342
3116804993128,2cyclictest3121724-21kworker/u8:3+flush-8:019:30:461
3116801993128,2cyclictest3129980-21kworker/u8:1+flush-8:020:35:020
3116801993127,3cyclictest482-21rs:main0
3116801993127,2cyclictest3456086-21kworker/u8:1+events_unbound00:30:000
3116801993125,4cyclictest3427828-21latency_hist23:45:010
3116811993028,1cyclictest3416633-21kworker/u8:1+events_unbound23:40:453
3116811993027,2cyclictest3225596-21kworker/u8:0+events_unbound20:50:003
3116811993026,3cyclictest3416884-21kworker/u8:2+events_unbound23:50:423
3116807993027,2cyclictest3427881-21kworker/u8:3+events_unbound00:10:452
3116807993026,3cyclictest3354732-21kworker/u8:2+events_unbound22:55:232
3116807993026,3cyclictest3169639-21kworker/u8:3+events_unbound20:15:352
3116811992922,5cyclictest3264291-21python321:15:353
3116807992926,2cyclictest3163674-21kworker/u8:2+events_unbound19:50:282
3116807992925,3cyclictest3354732-21kworker/u8:2+flush-8:023:00:252
3116811992823,3cyclictest3129926-21latency_hist19:20:003
3116801992825,2cyclictest3129980-21kworker/u8:1+events_unbound20:45:280
3116801992824,3cyclictest3146693-21kworker/u8:4+events_unbound19:40:010
3116807992724,2cyclictest3438693-21kworker/u8:0+events_unbound00:00:442
3116807992724,2cyclictest3169639-21kworker/u8:3+events_unbound20:20:252
3116801992721,4cyclictest3264645-21latency_hist21:20:010
3116811992623,2cyclictest3129980-21kworker/u8:1+events_unbound21:00:333
3116807992623,2cyclictest3276023-21kworker/u8:3+flush-8:022:15:262
3116807992622,3cyclictest3456086-21kworker/u8:1+events_unbound00:25:422
3116804992620,3cyclictest3247944-21kworker/u8:2+events_unbound21:45:401
311680799252,20cyclictest3428096-21apt-get23:45:222
3116801992521,3cyclictest3484415-21apt-config00:35:010
3116801992520,4cyclictest3444576-40updatedb.mlocat00:00:000
311680199251,22cyclictest3303627-21sh21:55:000
311680199251,22cyclictest3270212-21cron21:25:000
311680199250,4cyclictest3220048-21turbostat20:39:590
3116807992421,2cyclictest3405137-21kworker/u8:1+events_unbound23:25:282
3116801992418,4cyclictest3186669-21latency_hist20:10:010
311680199240,22cyclictest3343261-21sh22:30:000
3116811992321,1cyclictest3225596-21kworker/u8:0+events_unbound20:50:243
3116807992320,2cyclictest3427881-21kworker/u8:3+events_unbound00:15:412
3116807992320,2cyclictest3129980-21kworker/u8:1+events_unbound20:10:232
3116807992319,3cyclictest3169639-21kworker/u8:3+events_unbound20:05:252
3116804992321,2cyclictest3129980-21kworker/u8:1+events_unbound20:31:391
311680199231,20cyclictest3247898-21munin-run21:05:010
311680199231,20cyclictest3158417-21apt-get19:45:230
311680799220,2cyclictest0-21swapper/222:00:002
3116801992219,2cyclictest3105496-21kworker/u8:0+flush-8:019:15:000
3116801992218,3cyclictest3450250-21apt-get00:05:010
311680199221,2cyclictest0-21swapper/022:00:200
311680199221,2cyclictest0-21swapper/019:40:210
311680199221,2cyclictest0-21swapper/019:40:210
311680199221,20cyclictest0-21swapper/023:15:220
311680199221,1cyclictest0-21swapper/022:20:210
311680199220,2cyclictest3242302-21cron21:00:000
3116804992116,3cyclictest3146664-21apt-get19:35:031
3116801992118,2cyclictest3281360-21latency_hist21:35:000
3116801992113,6cyclictest3467796-21cat00:20:010
311680199210,2cyclictest3253517-21munin-run21:10:000
311680199210,20cyclictest0-21swapper/022:55:220
3116807992018,1cyclictest0-21swapper/223:30:212
3116801992018,1cyclictest0-21swapper/022:50:220
3116801992018,1cyclictest0-21swapper/022:50:210
3116801992018,1cyclictest0-21swapper/022:40:210
3116801992018,1cyclictest0-21swapper/019:33:580
3116801992017,2cyclictest3225654-21unin-run20:45:000
3116801992015,4cyclictest3974348-21systemd-journal19:53:380
311680199200,1cyclictest0-21swapper/022:35:220
311680199200,19cyclictest0-21swapper/020:15:200
311680199200,19cyclictest0-21swapper/019:15:240
3116811991915,3cyclictest3354735-21cron22:40:003
3116801991916,2cyclictest3484375-21apt-get00:35:020
3116801991913,4cyclictest3433671-21apt-get23:50:220
3116801991816,1cyclictest0-21swapper/022:45:020
311680199176,3cyclictest3316295-21ps22:05:330
311680199171,14cyclictest451-21dbus-daemon22:20:010
3116804991615,1cyclictest0-21swapper/120:27:151
3116801991612,2cyclictest3136215-21ps19:25:280
3116801991510,3cyclictest0-21swapper/021:30:000
311680199132,3cyclictest14-21ksoftirqd/022:30:000
311680199129,2cyclictest3450213-21kworker/0:0+kacpi_notify00:15:310
3116801991210,1cyclictest3405723-21kworker/0:0+kacpi_notify23:45:120
311680799116,3cyclictest3337680-21kworker/u8:1+flush-8:022:45:272
311680199104,4cyclictest3116806-21cat19:10:000
31168049980,2cyclictest3116614-21latency_hist19:10:011
31168119962,2cyclictest3116845-21cat19:10:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional