You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-18 - 18:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 18, 2024 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218642243165,25sleep10-21swapper/107:08:141
218572242164,24sleep00-21swapper/007:08:080
219732240164,24sleep30-21swapper/307:09:393
217072239162,25sleep20-21swapper/207:06:132
440421550,3sleep22210199cyclictest12:35:132
224922620,10sleep3401ktimersoftd/310:50:023
106802590,1sleep30-21swapper/307:55:123
201272540,4sleep07-21ksoftirqd/009:30:140
41172530,4sleep339-21rcuc/311:20:163
67822470,1sleep00-21swapper/009:00:120
22101994137,3cyclictest0-21swapper/209:06:242
147512410,1sleep30-21swapper/311:45:133
22101993830,2cyclictest0-21swapper/208:16:042
276072370,4sleep22210199cyclictest09:45:222
2209799327,16cyclictest15060-21taskset09:18:321
2209799321,4cyclictest780-21rngd07:14:321
22101993111,9cyclictest0-21swapper/212:31:262
2209299312,14cyclictest22448-21munin-run10:50:010
2209299302,13cyclictest7971-21munin-run07:50:010
42332290,6sleep339-21rcuc/307:40:143
22092992911,15cyclictest29368-21awk07:25:010
2209799285,15cyclictest29859-21mailstats09:50:261
2209799284,21cyclictest23566-21grep09:35:271
2210199272,10cyclictest0-21swapper/211:05:562
2210199271,12cyclictest15295-21kworker/2:109:20:252
2209799274,9cyclictest27554-21fschecks_time11:00:141
22097992712,13cyclictest31021-21kworker/u8:011:31:151
2209799271,18cyclictest0-21swapper/108:15:101
22097992711,14cyclictest23507-21kworker/u8:109:24:111
2209299279,15cyclictest5488-21perf11:25:010
2209799264,19cyclictest28258-21cut12:15:171
2209799264,14cyclictest2845-21/usr/sbin/munin10:05:151
2209799264,12cyclictest30329-21grep11:05:201
2209799261,11cyclictest0-21swapper/111:45:241
2209799260,11cyclictest0-21swapper/112:00:241
2209299260,13cyclictest7-21ksoftirqd/011:00:020
2210199256,3cyclictest0-21swapper/210:24:302
2209799253,10cyclictest30987-21cstates09:55:131
2209799252,19cyclictest88250irq/135-enp2s0-11:42:581
2209799251,18cyclictest0-21swapper/111:19:131
2209299257,15cyclictest14572-21munin-run08:05:010
2209299255,16cyclictest3900-21munin-run12:35:010
2209299251,13cyclictest11438-21munin-run10:25:010
2210199244,3cyclictest0-21swapper/211:20:142
2210199242,20cyclictest15295-21kworker/2:110:10:302
2210199241,10cyclictest15295-21kworker/2:111:32:172
2209799243,17cyclictest1069-21turbostat11:15:011
2209799242,18cyclictest29358-21munin-run07:25:001
2209799241,8cyclictest0-21swapper/107:48:181
2209799241,20cyclictest24265-21cat12:05:201
2209799241,17cyclictest0-21swapper/108:44:131
2209799241,11cyclictest0-21swapper/110:53:401
2209799241,10cyclictest0-21swapper/112:28:131
2209799240,6cyclictest0-21swapper/110:55:091
2209299240,11cyclictest3293-21basename11:20:010
2210799237,5cyclictest3759-21memory10:05:183
2210199236,9cyclictest7592-21diskstats10:15:142
2210199236,9cyclictest7592-21diskstats10:15:142
2210199231,7cyclictest18062-21kworker/2:212:25:062
2209799239,13cyclictest23507-21kworker/u8:108:06:121
2209799238,4cyclictest780-21rngd11:50:241
2209799231,9cyclictest0-21swapper/109:46:241
2209799231,4cyclictest0-21swapper/110:31:151
22097992313,2cyclictest86750irq/128-enp1s0-10:35:241
22097992312,3cyclictest0-21swapper/109:06:101
2209799231,20cyclictest0-21swapper/109:00:191
22097992311,6cyclictest2215-21kworker/u8:009:43:121
2210799225,4cyclictest13167-21unixbench_multi10:25:283
2210799222,7cyclictest18051-21munin-run10:40:013
22107992216,3cyclictest13244-21grep08:00:163
2210199227,3cyclictest19564-21hddtemp_smartct11:55:182
2210199227,2cyclictest0-21swapper/208:25:022
22101992216,3cyclictest683-21grep10:00:142
2210199221,3cyclictest32-21ksoftirqd/209:00:272
22101992213,4cyclictest15802-21switchtime11:45:262
2209799227,5cyclictest12876-21awk10:25:251
2209799225,9cyclictest27766-21crond08:35:011
2209799223,10cyclictest0-21swapper/112:11:361
2209799222,17cyclictest221ktimersoftd/108:15:011
22097992218,3cyclictest0-21swapper/111:58:531
22097992216,3cyclictest10817-21basename11:35:191
2209799221,16cyclictest0-21swapper/108:53:421
2209799221,16cyclictest0-21swapper/107:56:521
2209799221,11cyclictest0-21swapper/108:39:081
22097992210,3cyclictest4759-21df_inode08:55:121
22097992210,3cyclictest24971-21diskstats07:15:041
2209799220,11cyclictest14650-21cat08:05:011
2209299220,11cyclictest7-21ksoftirqd/012:10:010
180572222,3sleep018072-21ntp_states08:10:230
2210799218,3cyclictest0-21swapper/311:40:113
22107992116,3cyclictest1840-21cat12:30:013
2210799211,5cyclictest0-21swapper/309:15:353
2210799211,5cyclictest0-21swapper/307:39:463
2210199219,2cyclictest0-21swapper/210:50:222
2210199218,3cyclictest17133-21ntpq10:35:262
2210199215,3cyclictest14867-21strings10:30:222
2210199212,11cyclictest23975-21hddtemp_smartct12:05:162
2210199211,7cyclictest0-21swapper/212:04:582
2209799215,7cyclictest4471-21diskstats12:35:161
22097992115,3cyclictest22511-21ls08:20:241
2209799211,15cyclictest0-21swapper/112:24:071
2209799211,12cyclictest0-21swapper/110:50:021
2209799211,11cyclictest583-21smart_sda07:30:231
2209799211,11cyclictest0-21swapper/109:11:481
2209799211,11cyclictest0-21swapper/109:11:481
2209799211,10cyclictest0-21swapper/109:27:081
2209299219,3cyclictest836-21runrttasks09:22:420
2209299212,7cyclictest863-40dnf11:12:550
2209299212,16cyclictest25717-21grep09:40:280
2209299212,11cyclictest18270-21/usr/sbin/munin10:40:170
2209299212,11cyclictest17613-21/usr/sbin/munin09:25:170
22092992115,3cyclictest10790-21diskmemload10:05:060
2209299211,4cyclictest14170-21kworker/u8:112:04:180
2209299211,11cyclictest0-21swapper/011:58:580
6292201,3sleep30-21swapper/311:10:343
294622201,2sleep30-21swapper/308:35:313
2210799209,3cyclictest11171-21ntp_states11:35:253
2210799207,3cyclictest0-21swapper/311:57:593
2210799204,5cyclictest0-21swapper/309:35:133
2210799204,2cyclictest86550irq/126-enp1s0-09:32:153
2210799201,9cyclictest26537-21/usr/sbin/munin12:10:233
2210799201,11cyclictest4123-21/usr/sbin/munin12:35:273
2210799201,11cyclictest26659-21sshd10:57:503
2210799201,11cyclictest0-21swapper/311:15:223
2210799201,10cyclictest27277-21grep08:30:273
2210799201,10cyclictest0-21swapper/310:22:043
2210799201,10cyclictest0-21swapper/310:15:153
2210799201,10cyclictest0-21swapper/310:15:153
2210199207,3cyclictest0-21swapper/209:16:532
2210199205,3cyclictest0-21swapper/207:12:222
22101992016,3cyclictest0-21swapper/212:17:462
2210199201,3cyclictest32-21ksoftirqd/208:30:572
22101992013,6cyclictest23507-21kworker/u8:109:14:112
22101992013,6cyclictest23507-21kworker/u8:109:14:112
2209799201,9cyclictest10186-21if_enp2s010:20:181
22097992016,3cyclictest0-21swapper/110:14:251
22097992012,2cyclictest0-21swapper/108:45:231
2209799201,18cyclictest0-21swapper/110:40:101
2209799201,17cyclictest0-21swapper/111:22:171
2209799201,14cyclictest0-21swapper/110:01:271
2209799200,18cyclictest0-21swapper/110:17:331
2209799200,18cyclictest0-21swapper/110:17:331
2209799200,11cyclictest2508-21perl12:30:151
2209799200,10cyclictest5698-21/usr/sbin/munin11:25:181
2209299203,9cyclictest780-21rngd09:58:300
2209299203,11cyclictest496-21jbd2/dm-0-809:09:560
2209299201,12cyclictest0-21swapper/010:53:210
2209299201,12cyclictest0-21swapper/008:40:280
324892190,1sleep332491-21grep08:45:133
316472191,3sleep00-21swapper/012:22:200
254592190,3sleep00-21swapper/008:28:460
2210799198,3cyclictest8198-21/usr/sbin/munin07:50:153
2210799197,4cyclictest0-21swapper/312:20:023
2210799194,12cyclictest41-21ksoftirqd/310:10:263
2210799194,12cyclictest41-21ksoftirqd/307:25:193
2210799194,12cyclictest41-21ksoftirqd/307:10:273
2210799193,9cyclictest39-21rcuc/312:05:143
2210799193,3cyclictest0-21swapper/309:21:083
2210799191,8cyclictest22682-21cpu10:50:093
2210799191,8cyclictest0-21swapper/309:49:573
22107991914,3cyclictest24713-21ls08:25:283
2210799191,17cyclictest17740-21kworker/u8:010:40:293
2210799191,15cyclictest15933-21grep08:05:233
2210799191,11cyclictest0-21swapper/311:04:373
2210799191,10cyclictest8821-21/usr/sbin/munin09:05:203
2210799191,10cyclictest5698-21/usr/sbin/munin11:25:123
2210799191,10cyclictest31522-21cat09:55:173
2210799191,10cyclictest29270-21/usr/sbin/munin11:05:263
2210799191,10cyclictest1928-21/usr/sbin/munin12:30:253
2210799191,10cyclictest14634-21snmpd11:30:533
2210799190,8cyclictest0-21swapper/309:14:163
2210799190,8cyclictest0-21swapper/309:14:163
2210199196,3cyclictest0-21swapper/209:28:052
2210199195,3cyclictest0-21swapper/210:27:342
2210199194,13cyclictest18973-21ls08:15:012
2210199193,3cyclictest0-21swapper/208:08:462
2210199192,3cyclictest0-21swapper/208:58:292
2210199191,8cyclictest24253-21gltestperf08:25:192
2210199191,6cyclictest86550irq/126-enp1s0-10:50:022
2210199191,12cyclictest0-21swapper/207:20:152
2210199191,10cyclictest11656-21smart_sda07:55:242
22101991910,2cyclictest0-21swapper/209:40:272
2210199190,3cyclictest15295-21kworker/2:107:40:262
2209799195,3cyclictest0-21swapper/107:30:001
22097991915,3cyclictest0-21swapper/107:51:451
22097991915,3cyclictest0-21swapper/107:40:141
2209799191,17cyclictest0-21swapper/109:32:381
2209799191,17cyclictest0-21swapper/107:38:121
2209299191,5cyclictest0-21swapper/009:11:570
2209299191,5cyclictest0-21swapper/009:11:570
22092991913,3cyclictest29678-21munin-run12:20:010
22092991912,4cyclictest1686-21chrt08:46:110
2209299191,15cyclictest17691-21/usr/sbin/munin11:50:180
2209299191,15cyclictest12853-21fschecks_time11:40:150
2209299191,12cyclictest0-21swapper/011:30:230
2209299191,10cyclictest7244-21/usr/sbin/munin10:15:270
2209299191,10cyclictest7244-21/usr/sbin/munin10:15:270
2209299191,10cyclictest28609-21/usr/sbin/munin09:50:180
69462181,2sleep00-21swapper/010:14:150
294312181,2sleep30-21swapper/312:17:063
272882180,2sleep00-21swapper/012:12:490
2210799185,8cyclictest18687-40dnf08:11:333
2210799183,7cyclictest14634-21snmpd11:50:203
2210799182,12cyclictest24630-21grep07:15:023
22107991813,3cyclictest28952-21sh09:50:143
2210799181,14cyclictest27385-21/usr/sbin/munin07:20:233
2210799181,11cyclictest0-21swapper/310:00:493
2210799181,10cyclictest0-21swapper/309:40:483
2210199186,1cyclictest0-21swapper/207:45:262
2210199184,2cyclictest86650irq/127-enp1s0-11:51:222
2210199183,3cyclictest0-21swapper/211:25:022
2210199183,2cyclictest25505-21/usr/sbin/munin12:10:222
2210199183,2cyclictest15295-21kworker/2:107:50:122
2210199182,3cyclictest0-21swapper/209:58:382
2210199182,2cyclictest0-21swapper/207:35:302
2210199181,7cyclictest31809-21chrt08:40:382
2210199181,3cyclictest27006-21chrt07:18:142
22101991813,3cyclictest2549-21fgrep11:15:292
2210199181,2cyclictest0-21swapper/211:36:102
2210199180,2cyclictest0-21swapper/208:50:212
22097991810,5cyclictest24873-21swap08:25:301
2209299185,10cyclictest0-21swapper/008:56:410
2209299183,14cyclictest7-21ksoftirqd/008:10:010
2209299181,5cyclictest26835-21fschecks_count09:45:130
22092991812,3cyclictest21699-21cat08:20:120
2209299181,10cyclictest28702-21sshd11:00:510
2209299181,10cyclictest0-21swapper/007:35:270
2210799175,4cyclictest21979-21gltestperf08:20:163
2210799174,3cyclictest177692cpuspeed_turbos09:25:113
2210799174,11cyclictest41-21ksoftirqd/310:30:223
2210799173,12cyclictest41-21ksoftirqd/308:15:153
2210799171,4cyclictest7489-21unixbench-2d07:45:283
22107991712,3cyclictest30368-21cat08:40:133
2210799171,15cyclictest0-21swapper/308:57:093
22107991711,3cyclictest7034-21uniq09:00:143
2210199175,3cyclictest0-21swapper/209:30:292
2210199172,3cyclictest0-21swapper/212:23:422
2210199172,2cyclictest88150irq/134-enp2s0-08:39:092
2210199171,7cyclictest25538-21sed10:55:172
22101991712,3cyclictest31774-21/usr/sbin/munin07:30:082
22101991711,3cyclictest32722-21df08:45:132
22101991711,3cyclictest310-21cat11:10:302
2210199171,13cyclictest29053-21grep09:50:152
22101991711,3cyclictest22346-21df09:35:102
2209299176,3cyclictest14634-21snmpd10:33:260
2209299173,9cyclictest0-21swapper/008:30:190
2209299172,12cyclictest9342-21tr07:50:230
2209299172,12cyclictest19851-21head08:15:130
2209299171,9cyclictest0-21swapper/010:35:370
2209299171,6cyclictest0-21swapper/008:50:200
22092991712,3cyclictest546-21/usr/sbin/munin10:00:270
22092991712,3cyclictest24225-21sshd07:14:410
22092991712,3cyclictest22073-21ls09:35:090
22092991712,3cyclictest1371-21munin-run07:35:010
22092991712,3cyclictest13235-21cpu09:15:130
22092991712,3cyclictest11974-21ls10:25:130
2209299171,15cyclictest0-21swapper/007:15:390
22092991711,4cyclictest11564-21unixbench-2d11:35:300
2209299171,10cyclictest30986-21sshd11:06:520
2209299171,10cyclictest0-21swapper/011:45:170
22092991710,4cyclictest4720-21grep12:35:170
2210799163,11cyclictest39-21rcuc/308:50:193
2210799163,10cyclictest21080-21sa112:00:093
2210199167,3cyclictest86550irq/126-enp1s0-10:05:202
2210199161,7cyclictest0-21swapper/211:00:142
22101991611,3cyclictest29901-21cat07:25:112
22101991611,3cyclictest19236-21sed10:40:202
22101991611,3cyclictest13365-21ntpq11:40:212
2210199161,10cyclictest0-21swapper/208:03:322
2209299161,9cyclictest0-21swapper/012:25:080
2209299161,5cyclictest0-21swapper/011:28:250
22092991612,3cyclictest0-21swapper/008:35:050
22092991610,3cyclictest4407-21expr07:40:150
22092991610,3cyclictest10395-21/usr/sbin/munin07:55:200
22107991510,3cyclictest629-21switchtime07:30:243
2209299153,3cyclictest29579-21/usr/sbin/munin07:25:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional