You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-20 - 20:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jan 20, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
237402234163,23sleep20-21swapper/207:08:392
218832234164,57sleep30-21swapper/307:05:053
218812203166,24sleep10-21swapper/107:05:041
235712201164,24sleep00-21swapper/007:06:270
23948991092,105cyclictest0-21swapper/210:03:432
239489910197,3cyclictest0-21swapper/208:46:532
23948999490,3cyclictest0-21swapper/209:52:462
23948996662,3cyclictest0-21swapper/211:34:302
212412580,4sleep121-21rcuc/111:15:141
216672570,4sleep12394299cyclictest08:20:111
23948995249,2cyclictest0-21swapper/211:18:532
2395599361,21cyclictest836-21runrttasks08:24:593
2393899367,13cyclictest14655-21idleruntime-cro12:25:020
2393899359,12cyclictest21005-21idleruntime-cro11:15:020
2393899358,12cyclictest17044-21awk08:10:010
55752330,4sleep230-21rcuc/207:40:012
2395599338,7cyclictest581-21systemd-journal07:20:013
2395599331,22cyclictest0-21swapper/312:16:333
2395599331,12cyclictest0-21swapper/311:46:433
23955993224,3cyclictest0-21swapper/308:12:253
2395599321,12cyclictest0-21swapper/310:00:173
2393899327,12cyclictest0-21swapper/008:05:020
2393899321,25cyclictest1235-21kworker/0:107:50:200
23955993114,3cyclictest1933-21cut10:20:113
2393899313,13cyclictest8070-21munin-run12:10:020
2395599301,20cyclictest0-21swapper/308:46:433
2395599301,20cyclictest0-21swapper/308:01:053
2395599301,15cyclictest0-21swapper/309:40:473
2395599301,13cyclictest0-21swapper/312:30:213
2395599300,12cyclictest0-21swapper/308:19:293
2395599296,3cyclictest0-21swapper/311:30:033
2395599291,19cyclictest0-21swapper/307:50:113
2394299291,22cyclictest0-21swapper/109:09:071
2394299291,17cyclictest7817-21netstat10:35:221
2394299291,17cyclictest7817-21netstat10:35:221
2394299291,15cyclictest23760-21diskstats11:20:141
23955992814,8cyclictest0-21swapper/307:32:473
2395599281,19cyclictest0-21swapper/308:43:113
2395599281,19cyclictest0-21swapper/308:09:193
2395599281,13cyclictest0-21swapper/311:19:333
23955992810,4cyclictest0-21swapper/309:15:153
2394299281,24cyclictest14634-21snmpd09:45:021
2394299280,23cyclictest0-21swapper/108:44:041
2394299280,23cyclictest0-21swapper/108:39:471
23938992815,3cyclictest86750irq/128-enp1s0-11:15:280
2395599271,17cyclictest0-21swapper/308:25:013
2395599271,17cyclictest0-21swapper/308:25:013
2395599271,14cyclictest0-21swapper/307:12:483
2395599271,12cyclictest0-21swapper/311:57:373
2395599270,14cyclictest0-21swapper/312:10:253
2395599270,11cyclictest0-21swapper/310:10:193
2394899270,5cyclictest88050irq/133-enp2s0-10:45:492
2394299272,21cyclictest25808-21ntp_states09:55:231
2394299270,25cyclictest0-21swapper/111:42:171
2394299270,25cyclictest0-21swapper/111:42:171
2393899271,9cyclictest25975-21kworker/0:307:15:030
2395599268,10cyclictest0-21swapper/310:38:273
2395599268,10cyclictest0-21swapper/310:38:273
2395599261,12cyclictest0-21swapper/310:25:233
2394299261,22cyclictest16512-21cat08:05:021
2394299261,22cyclictest0-21swapper/109:50:091
2394299261,13cyclictest0-21swapper/112:10:221
2395599259,3cyclictest0-21swapper/312:03:353
2395599258,4cyclictest0-21swapper/309:57:393
2395599256,3cyclictest0-21swapper/309:25:213
2395599251,12cyclictest0-21swapper/309:05:013
2395599251,12cyclictest0-21swapper/307:36:093
2395599251,11cyclictest0-21swapper/308:52:233
2395599251,11cyclictest0-21swapper/308:31:013
2395599250,12cyclictest0-21swapper/309:54:153
2394899251,11cyclictest0-21swapper/207:40:172
23942992516,3cyclictest0-21swapper/107:57:251
2394299251,15cyclictest0-21swapper/111:00:071
2394299251,15cyclictest0-21swapper/109:37:391
2394299251,15cyclictest0-21swapper/108:55:551
2394299251,12cyclictest0-21swapper/112:15:171
2394299251,11cyclictest26739-21hddtemp_smartct08:30:191
2394299251,11cyclictest25983-21kworker/u8:011:31:201
2394299251,11cyclictest20497-21ntpq08:15:211
2394299251,11cyclictest14806-21kworker/u8:211:07:341
23942992510,3cyclictest32763-21irqstats10:15:181
23942992510,3cyclictest2596-21irqstats10:20:171
2393899251,7cyclictest0-21swapper/010:03:110
2393899251,6cyclictest0-21swapper/011:21:150
2393899251,13cyclictest31209-21awk07:25:010
2393899251,13cyclictest31209-21awk07:25:010
2395599247,3cyclictest0-21swapper/309:34:593
2395599247,3cyclictest0-21swapper/309:34:593
2395599247,3cyclictest0-21swapper/308:55:273
2395599243,7cyclictest30076-21seq11:35:593
2395599241,15cyclictest0-21swapper/311:40:333
2395599241,15cyclictest0-21swapper/311:40:333
2395599241,13cyclictest0-21swapper/309:05:433
23955992410,3cyclictest0-21swapper/307:40:093
2395599240,8cyclictest0-21swapper/311:51:313
2395599240,14cyclictest0-21swapper/307:25:153
2394299249,3cyclictest0-21swapper/112:08:401
2394299247,14cyclictest836-21runrttasks07:52:081
2394299242,10cyclictest19285-21users12:35:271
2394299241,7cyclictest0-21swapper/112:02:021
23942992415,3cyclictest0-21swapper/109:30:231
23942992415,3cyclictest0-21swapper/109:30:231
23942992414,2cyclictest0-21swapper/111:14:281
2394299241,21cyclictest0-21swapper/111:52:021
2394299241,20cyclictest300532chrt11:35:411
2394299241,20cyclictest27377-21diskstats07:15:141
2394299241,20cyclictest0-21swapper/109:15:171
2394299241,15cyclictest0-21swapper/110:14:521
2394299241,15cyclictest0-21swapper/110:09:261
2394299241,15cyclictest0-21swapper/109:01:541
2394299241,15cyclictest0-21swapper/108:25:341
2394299241,15cyclictest0-21swapper/108:25:341
2394299241,12cyclictest0-21swapper/109:26:461
2394299241,10cyclictest14806-21kworker/u8:210:58:511
23942992410,3cyclictest0-21swapper/111:45:401
2394299240,22cyclictest0-21swapper/110:42:501
2393899244,4cyclictest0-21swapper/009:43:330
2393899241,12cyclictest0-21swapper/008:30:010
2393899241,12cyclictest0-21swapper/008:30:010
2395599239,3cyclictest0-21swapper/307:55:053
2395599239,3cyclictest0-21swapper/307:20:473
2395599239,3cyclictest0-21swapper/307:20:473
2395599236,8cyclictest0-21swapper/309:11:213
2395599235,3cyclictest0-21swapper/312:07:193
2395599231,9cyclictest0-21swapper/309:38:173
23955992312,3cyclictest0-21swapper/308:36:353
2395599231,14cyclictest13755-21ntp_states12:20:213
2395599231,10cyclictest0-21swapper/312:35:073
2395599231,10cyclictest0-21swapper/311:08:333
23955992310,3cyclictest0-21swapper/307:46:493
2395599230,22cyclictest0-21swapper/310:43:193
2395599230,10cyclictest0-21swapper/311:26:113
2394299239,3cyclictest0-21swapper/111:26:021
2394299237,4cyclictest25051-21load07:10:191
2394299233,17cyclictest16755-21smart_sda12:30:201
2394299231,19cyclictest5241-21latency_hist12:00:021
2394299231,19cyclictest31759-21df07:25:141
2394299231,14cyclictest0-21swapper/109:23:361
2394299231,10cyclictest836-21runrttasks08:10:511
23942992310,3cyclictest86750irq/128-enp1s0-09:46:591
2394299230,15cyclictest0-21swapper/110:28:081
2394299230,15cyclictest0-21swapper/110:00:211
2394299230,15cyclictest0-21swapper/109:12:131
2394299230,15cyclictest0-21swapper/108:07:021
2393899231,4cyclictest0-21swapper/009:14:330
2395599229,3cyclictest0-21swapper/310:31:053
2395599229,3cyclictest0-21swapper/309:49:033
2395599225,3cyclictest0-21swapper/312:26:493
2395599221,9cyclictest0-21swapper/311:22:473
2395599221,11cyclictest0-21swapper/311:12:133
2394899225,9cyclictest25415-21grep08:25:262
2394899225,9cyclictest25415-21grep08:25:262
2394899221,7cyclictest11828-21tail09:15:272
2394899221,13cyclictest0-21swapper/212:20:252
2394899221,11cyclictest0-21swapper/209:46:242
2394299221,3cyclictest14634-21snmpd07:33:071
2394299221,19cyclictest0-21swapper/112:23:151
2394299221,19cyclictest0-21swapper/107:43:281
2394299221,19cyclictest0-21swapper/107:35:311
2394299221,19cyclictest0-21swapper/107:21:341
2394299221,19cyclictest0-21swapper/107:21:331
2394299221,14cyclictest0-21swapper/110:32:321
2394299221,11cyclictest7937-21kworker/u8:012:28:441
2394299221,10cyclictest13146-21timerandwakeup10:50:211
2394299220,16cyclictest0-21swapper/110:49:001
2393899221,9cyclictest17362-21kworker/0:112:35:160
23938992215,4cyclictest16796-21head11:00:130
2393899221,17cyclictest3055-21munin-run11:55:020
2395599216,3cyclictest0-21swapper/309:20:413
2395599211,13cyclictest0-21swapper/310:17:273
2394899215,9cyclictest12050irq/123-ahci[0007:50:012
2394899215,5cyclictest12711-21missed_timers10:50:142
2394899212,16cyclictest1682-21gltestperf11:50:122
2394899211,6cyclictest0-21swapper/211:03:312
2394899211,5cyclictest0-21swapper/211:42:252
2394899211,5cyclictest0-21swapper/211:42:242
2394899211,12cyclictest0-21swapper/208:52:462
2394299210,20cyclictest0-21swapper/107:46:481
2394299210,14cyclictest0-21swapper/108:47:281
2393899219,2cyclictest24731-21kworker/0:009:55:400
2393899217,2cyclictest23635-21kworker/0:107:12:170
2393899214,8cyclictest0-21swapper/007:49:170
2393899213,15cyclictest17643-21grep09:35:120
2393899212,9cyclictest4719-21kworker/0:207:36:440
2395599201,10cyclictest0-21swapper/310:50:073
2394899206,7cyclictest15932-21fschecks_count12:30:112
2394899205,8cyclictest86550irq/126-enp1s0-08:35:122
2394899205,3cyclictest12050irq/123-ahci[0008:05:012
2394899204,8cyclictest86850irq/129-enp1s0-08:10:222
2394899204,8cyclictest86550irq/126-enp1s0-10:30:082
2394899202,12cyclictest1241-21kworker/2:011:46:142
2394899202,10cyclictest26061-21/usr/sbin/munin08:30:292
2394899201,8cyclictest0-21swapper/209:28:282
2394899201,7cyclictest0-21swapper/212:38:122
2394899201,7cyclictest0-21swapper/211:36:052
23948992017,2cyclictest0-21swapper/212:10:332
2394899201,6cyclictest9378-21sensors_temp09:10:222
2394899201,6cyclictest6102-21irqstats09:00:162
2394899201,6cyclictest18511-21needreboot09:35:192
2394899201,5cyclictest0-21swapper/212:15:422
2394899201,5cyclictest0-21swapper/211:11:032
2394899201,4cyclictest15403-21cpuspeed_turbos09:30:092
2394899201,4cyclictest15403-21cpuspeed_turbos09:30:082
2394899201,12cyclictest0-21swapper/208:20:162
23948992010,2cyclictest4264-21strings11:55:172
2394299201,10cyclictest0-21swapper/108:50:181
2393899209,3cyclictest31423-21/usr/sbin/munin07:25:170
2393899204,4cyclictest11-21rcuc/009:20:250
2393899202,4cyclictest576-21munin-run08:50:020
2393899202,10cyclictest26229-21/usr/sbin/munin11:30:130
2393899201,13cyclictest0-21swapper/011:29:560
2393899201,11cyclictest13995-21/usr/sbin/munin10:55:140
2393899200,9cyclictest0-21swapper/009:51:360
308152191,2sleep00-21swapper/010:10:180
2395599197,7cyclictest86550irq/126-enp1s0-10:55:303
2395599194,3cyclictest0-21swapper/310:05:353
2395599191,10cyclictest0-21swapper/311:00:593
2394899195,8cyclictest18730-21kworker/u8:011:09:302
2394899195,3cyclictest86550irq/126-enp1s0-09:20:192
2394899195,3cyclictest12050irq/123-ahci[0010:25:012
2394899194,4cyclictest12050irq/123-ahci[0008:20:012
2394899192,4cyclictest10890-40dnf10:44:172
2394899192,10cyclictest0-21swapper/212:04:222
2394899191,9cyclictest7409-21grep10:35:172
2394899191,9cyclictest7409-21grep10:35:172
2394899191,7cyclictest0-21swapper/212:07:542
2394899191,6cyclictest25784-21sshd11:27:162
2394899191,6cyclictest13995-21/usr/sbin/munin10:55:212
2394899191,6cyclictest0-21swapper/211:20:202
2394899191,5cyclictest30822-21ntp_states10:10:172
2394899191,4cyclictest0-21swapper/209:57:482
2394899191,12cyclictest0-21swapper/207:32:502
2394899191,12cyclictest0-21swapper/207:15:522
2394899191,11cyclictest32-21ksoftirqd/210:15:142
2394899191,11cyclictest0-21swapper/207:12:532
2393899198,3cyclictest31950-21/usr/sbin/munin10:15:220
2393899197,2cyclictest14952-21kworker/0:212:31:540
2393899193,10cyclictest0-21swapper/011:55:140
2393899192,4cyclictest814-21sshd10:43:180
2393899192,14cyclictest26211-21grep08:30:150
2393899191,8cyclictest0-21swapper/011:42:290
2393899191,8cyclictest0-21swapper/011:42:290
2393899191,7cyclictest17421-1kworker/u9:209:32:400
2393899191,7cyclictest17421-1kworker/u9:209:32:400
2393899191,5cyclictest0-21swapper/009:48:360
2393899191,14cyclictest8446-21tr12:10:150
2393899191,10cyclictest4296-21sshd10:29:170
2393899191,10cyclictest10318-21/usr/sbin/munin09:15:140
67872181,3sleep00-21swapper/010:35:110
67872181,3sleep00-21swapper/010:35:110
308612181,3sleep20-21swapper/207:20:512
308612181,3sleep20-21swapper/207:20:512
2394899184,3cyclictest86550irq/126-enp1s0-07:25:162
2394899181,5cyclictest13502-21smart_sda07:55:232
2394899181,3cyclictest86550irq/126-enp1s0-12:26:072
2394899181,11cyclictest0-21swapper/210:28:342
2394899181,11cyclictest0-21swapper/209:43:382
2394899181,10cyclictest14634-21snmpd08:56:072
2394899181,10cyclictest0-21swapper/208:40:242
2393899185,3cyclictest16813-21kworker/0:008:15:380
2393899183,7cyclictest88050irq/133-enp2s0-07:55:240
2393899183,5cyclictest780-21rngd08:42:280
2393899182,4cyclictest7-21ksoftirqd/011:35:340
2393899181,8cyclictest0-21swapper/010:07:140
2393899181,7cyclictest0-21swapper/010:33:350
2393899181,7cyclictest0-21swapper/008:10:230
2393899181,6cyclictest0-21swapper/010:21:420
2393899181,5cyclictest0-21swapper/011:09:470
2393899181,4cyclictest7459-21munin-run12:05:020
23938991812,3cyclictest11994-21cat10:50:090
23938991811,4cyclictest23095-21sort08:20:240
2393899181,10cyclictest2645-21sshd08:52:180
114292181,3sleep00-21swapper/010:48:200
2395599171,10cyclictest0-21swapper/310:45:283
2394899171,9cyclictest0-21swapper/209:08:452
2394899171,5cyclictest0-21swapper/210:05:342
2394899171,15cyclictest0-21swapper/207:52:232
2394899171,10cyclictest0-21swapper/208:06:352
2393899176,3cyclictest14634-21snmpd09:03:080
2393899171,7cyclictest0-21swapper/009:28:370
23938991713,3cyclictest0-21swapper/007:33:110
2393899171,15cyclictest0-21swapper/012:25:340
2393899171,14cyclictest0-21swapper/008:55:070
23938991711,3cyclictest6314-21gltestperf07:40:170
23938991711,3cyclictest10609-21expr12:15:140
2393899163,6cyclictest87950irq/132-enp2s0-08:36:080
2393899161,9cyclictest0-21swapper/011:45:100
2393899159,3cyclictest7889-21latency_hist09:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional