You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-23 - 04:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Feb 23, 2024 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5122240163,25sleep10-21swapper/119:05:081
22542239165,23sleep30-21swapper/319:05:543
18782221192,19sleep00-21swapper/019:05:340
25232207169,25sleep20-21swapper/219:09:232
2683991190,113cyclictest0-21swapper/222:05:572
2683997672,3cyclictest0-21swapper/221:30:092
26122740,2sleep20-21swapper/221:35:212
111432620,4sleep121-21rcuc/119:25:271
136142600,3sleep0267399cyclictest19:30:360
288962570,3sleep011-21rcuc/022:35:010
189222550,4sleep230-21rcuc/220:55:352
100692540,3sleep310072-21unixbench_multi20:35:373
100692540,3sleep310072-21unixbench_multi20:35:373
65272530,1sleep30-21swapper/300:10:303
221832520,1sleep30-21swapper/319:50:293
110582490,3sleep339-21rcuc/323:05:353
190322380,1sleep30-21swapper/300:40:003
2673993624,7cyclictest0-21swapper/022:47:320
2673993624,7cyclictest0-21swapper/022:47:320
267399341,27cyclictest0-21swapper/021:30:310
2677993318,9cyclictest25204-21/usr/sbin/munin23:40:371
267399311,2cyclictest86550irq/126-enp1s0-22:09:170
269099306,7cyclictest7934-21kworker/u8:100:26:163
268399301,6cyclictest8028-21munin-run00:15:022
268399301,6cyclictest0-21swapper/223:27:392
268399301,5cyclictest32168-21kworker/u8:222:30:342
2677993018,3cyclictest0-21swapper/121:34:141
2673993014,10cyclictest0-21swapper/000:07:410
269099292,12cyclictest26550-21munin-run22:30:003
269099291,13cyclictest27164-21munin-run23:45:003
268399291,3cyclictest0-21swapper/221:13:082
2677992916,8cyclictest0-21swapper/100:05:321
132012290,4sleep0267399cyclictest23:10:330
269099283,5cyclictest6330-21unixbench_multi21:40:383
269099283,5cyclictest6330-21unixbench_multi21:40:383
268399281,5cyclictest0-21swapper/222:40:322
268399281,5cyclictest0-21swapper/221:57:052
2683992812,3cyclictest25312-21awk20:00:012
2683992811,3cyclictest86850irq/129-enp1s0-23:57:572
2677992818,3cyclictest0-21swapper/121:25:321
267399281,20cyclictest0-21swapper/022:22:370
269099273,3cyclictest31788-21gltestperf22:40:223
269099271,9cyclictest0-21swapper/321:10:563
268399271,5cyclictest0-21swapper/200:07:042
2677992717,3cyclictest0-21swapper/121:40:591
2677992717,3cyclictest0-21swapper/121:40:591
2677992717,3cyclictest0-21swapper/100:01:471
2677992716,7cyclictest0-21swapper/100:28:111
267799271,13cyclictest0-21swapper/120:14:341
267399274,3cyclictest10633-21irqstats23:05:280
267399271,23cyclictest9476-21taskset23:03:490
267399271,22cyclictest11468-21cat21:55:190
2673992710,7cyclictest19016-21awk00:40:000
269099261,9cyclictest0-21swapper/321:17:573
268399262,4cyclictest0-21swapper/223:37:122
268399261,5cyclictest0-21swapper/221:21:262
267799261,17cyclictest16606-21irqstats22:05:291
2673992616,3cyclictest0-21swapper/023:38:300
2673992615,3cyclictest0-21swapper/023:26:200
269099258,7cyclictest0-21swapper/323:50:323
269099258,7cyclictest0-21swapper/323:50:323
269099257,8cyclictest21149-21diskstats23:30:213
269099256,9cyclictest18558-21taskset19:43:183
269099253,3cyclictest29279-21kworker/u8:200:04:563
269099252,8cyclictest21394-40dnf21:03:323
2690992521,3cyclictest0-21swapper/320:00:333
269099251,7cyclictest0-21swapper/322:10:203
269099251,7cyclictest0-21swapper/322:06:393
269099251,7cyclictest0-21swapper/321:45:133
269099251,7cyclictest0-21swapper/321:20:333
269099251,3cyclictest0-21swapper/323:47:563
269099251,14cyclictest823-21snmpd21:35:123
269099251,13cyclictest823-21snmpd20:25:033
269099251,12cyclictest5761-21/usr/sbin/munin19:15:203
268399256,2cyclictest86850irq/129-enp1s0-23:40:462
2677992515,6cyclictest0-21swapper/122:35:221
2677992515,3cyclictest0-21swapper/100:20:421
267799251,4cyclictest10761-21unixbench_singl21:50:351
267399257,3cyclictest14801-21/usr/sbin/munin23:15:240
269099248,3cyclictest28982-21/usr/sbin/munin22:35:343
269099246,8cyclictest5824-21tune2fs22:55:203
269099246,8cyclictest11625-21kworker/u8:223:13:313
269099242,10cyclictest9766-21perl21:50:233
269099242,10cyclictest22999-21/usr/sbin/munin23:35:263
269099241,4cyclictest0-21swapper/300:21:333
269099241,3cyclictest0-21swapper/323:27:413
269099241,3cyclictest0-21swapper/322:00:203
269099241,12cyclictest986-21/usr/sbin/munin22:45:213
269099241,12cyclictest986-21/usr/sbin/munin22:45:213
2683992417,4cyclictest8441-21timerwakeupswit21:45:402
267799244,12cyclictest28757-21basename22:35:011
267799241,9cyclictest2945-21df_inode21:35:241
2677992418,3cyclictest32125-21df23:55:241
2677992415,3cyclictest0-21swapper/123:35:461
2677992415,2cyclictest0-21swapper/123:24:201
2677992413,3cyclictest21446-21smart_sda22:15:331
267799241,20cyclictest4346-21pmu-power22:50:331
2677992411,2cyclictest0-21swapper/123:27:391
267799240,17cyclictest0-21swapper/122:55:081
267799240,12cyclictest0-21swapper/121:47:301
267399249,3cyclictest14545-21/usr/sbin/munin19:35:200
267399247,5cyclictest14930-21awk20:50:010
267399242,10cyclictest22345-21processes19:50:320
267399241,6cyclictest0-21swapper/023:54:180
267399241,6cyclictest0-21swapper/023:54:180
267399241,6cyclictest0-21swapper/000:25:490
267399241,6cyclictest0-21swapper/000:15:180
267399241,5cyclictest28847-21fschecks_time21:20:230
267399241,4cyclictest0-21swapper/021:47:120
2673992413,3cyclictest0-21swapper/021:13:090
267399241,22cyclictest0-21swapper/019:20:220
267399241,11cyclictest27398-21/usr/sbin/munin23:45:250
267399241,11cyclictest22560-21sshd23:32:340
267399240,2cyclictest29279-21kworker/u8:200:10:270
267399240,22cyclictest0-21swapper/022:55:300
51232230,3sleep081ktimersoftd/021:40:210
51232230,3sleep081ktimersoftd/021:40:210
269099236,8cyclictest3950-21if_enp1s022:50:253
269099235,7cyclictest29279-21kworker/u8:223:55:433
269099232,9cyclictest784-21NetworkManager23:20:283
2690992320,2cyclictest0-21swapper/322:15:283
2690992318,3cyclictest823-21snmpd20:21:593
2690992317,3cyclictest823-21snmpd22:30:203
269099231,4cyclictest0-21swapper/322:22:363
269099231,3cyclictest0-21swapper/321:25:313
269099231,11cyclictest29941-21/usr/sbin/munin20:10:273
269099231,11cyclictest23331-21/usr/sbin/munin19:55:243
269099231,11cyclictest12965-21/usr/sbin/munin20:45:293
269099231,11cyclictest0-21swapper/323:04:013
269099231,11cyclictest0-21swapper/321:55:293
269099231,10cyclictest15159-21/usr/sbin/munin20:50:243
267799235,9cyclictest27237-21ntp_states21:15:311
267799231,5cyclictest814-21sshd22:22:251
267799231,19cyclictest25659-21ntp_states22:25:311
267799231,19cyclictest2470-21grep20:20:241
267799231,15cyclictest19995-21latency_hist22:15:021
267799231,14cyclictest9910-21cpuspeed_turbos23:05:221
2677992311,4cyclictest759-21dbus-daemon22:03:471
267399233,11cyclictest32168-21kworker/u8:222:03:460
267399232,5cyclictest848-21sssd_nss22:15:280
267399231,6cyclictest0-21swapper/022:43:170
267399231,6cyclictest0-21swapper/022:29:150
267399231,5cyclictest7-21ksoftirqd/022:50:270
267399231,5cyclictest0-21swapper/023:55:560
2673992314,3cyclictest0-21swapper/021:25:130
2673992313,3cyclictest0-21swapper/021:50:020
267399231,17cyclictest0-21swapper/023:24:050
267399231,10cyclictest4178-21/usr/sbin/munin20:25:190
267399231,10cyclictest25545-21df_inode23:40:210
267399230,21cyclictest0-21swapper/000:20:240
269099226,8cyclictest86850irq/129-enp1s0-21:30:323
269099226,7cyclictest7957-21/usr/sbin/munin19:20:243
269099226,7cyclictest16633-21cat19:40:013
269099225,8cyclictest21761-21/usr/sbin/munin21:05:263
269099225,7cyclictest823-21snmpd00:15:393
269099221,10cyclictest823-21snmpd20:30:043
269099221,10cyclictest14214-21/usr/sbin/munin23:15:273
268399226,7cyclictest2827-21processes00:00:392
268399221,6cyclictest0-21swapper/222:57:552
268399221,5cyclictest0-21swapper/222:01:052
268399221,5cyclictest0-21swapper/200:24:142
2683992215,5cyclictest13200-21/usr/sbin/munin23:10:342
267799226,4cyclictest836-21runrttasks00:35:391
267799225,9cyclictest24298-21df21:10:201
2677992215,2cyclictest0-21swapper/100:32:021
267799221,4cyclictest0-21swapper/121:57:561
267799221,14cyclictest685-21turbostat22:49:561
267799221,14cyclictest685-21turbostat22:49:561
267799220,20cyclictest0-21swapper/123:54:271
267799220,20cyclictest0-21swapper/123:54:271
267799220,12cyclictest0-21swapper/123:48:501
267399224,2cyclictest0-21swapper/000:32:000
267399221,9cyclictest0-21swapper/000:01:340
2673992216,3cyclictest31900-21ls20:15:010
267399221,5cyclictest0-21swapper/022:37:400
267399221,13cyclictest0-21swapper/021:35:050
267399220,5cyclictest0-21swapper/022:10:430
56522210,3sleep2268399cyclictest21:40:272
56522210,3sleep2268399cyclictest21:40:272
269099217,3cyclictest86550irq/126-enp1s0-20:55:213
269099217,3cyclictest14851-21/usr/sbin/munin00:30:273
269099217,3cyclictest10770-21/usr/sbin/munin20:40:243
269099216,3cyclictest823-21snmpd00:05:373
269099211,9cyclictest12351-21/usr/sbin/munin19:30:183
268399216,5cyclictest31206-21chrt20:10:312
268399215,4cyclictest86550irq/126-enp1s0-20:25:262
268399211,8cyclictest0-21swapper/200:35:252
268399211,7cyclictest581-21systemd-journal23:20:012
268399211,7cyclictest24690-21expr22:25:202
268399211,7cyclictest0-21swapper/222:52:312
268399211,7cyclictest0-21swapper/200:34:452
268399211,6cyclictest0-21swapper/200:18:232
268399211,5cyclictest32168-21kworker/u8:222:47:332
268399211,5cyclictest32168-21kworker/u8:222:47:332
268399211,5cyclictest11396-21chrt19:25:312
2683992115,3cyclictest31555-21grep23:55:012
2683992115,3cyclictest31555-21grep23:55:002
267799218,8cyclictest0-21swapper/120:45:201
267799216,13cyclictest22756-21crond23:35:011
267799213,9cyclictest0-21swapper/123:04:551
2677992117,3cyclictest0-21swapper/119:13:381
2677992116,3cyclictest1730-21crond20:20:011
267799211,19cyclictest0-21swapper/119:50:301
267799211,10cyclictest0-21swapper/121:09:001
267799210,19cyclictest0-21swapper/121:20:341
267799210,19cyclictest0-21swapper/100:10:391
267799210,12cyclictest0-21swapper/120:40:271
267399215,3cyclictest86750irq/128-enp1s0-20:35:290
267399215,3cyclictest86750irq/128-enp1s0-20:35:290
267399212,5cyclictest26450-21memory20:00:260
267399211,11cyclictest86550irq/126-enp1s0-21:15:290
267399210,19cyclictest0-21swapper/019:25:250
269099204,5cyclictest0-21swapper/319:45:533
269099203,5cyclictest11928-21sshd19:27:453
268399203,6cyclictest22514-1kworker/u9:123:24:252
268399202,7cyclictest23727-21diskmemload22:37:032
268399202,7cyclictest22594-21expr22:20:202
268399202,10cyclictest0-21swapper/200:29:342
268399201,8cyclictest28230-21kernelversion23:45:282
268399201,7cyclictest11185-21tune2fs20:40:242
268399201,6cyclictest23736-21perf21:10:012
268399201,6cyclictest0-21swapper/221:50:362
268399201,5cyclictest3949-21munin-run20:25:002
268399201,4cyclictest0-21swapper/222:17:162
2683992012,4cyclictest19492-21unixbench_multi22:10:342
267799201,6cyclictest14968-21if_enp2s023:15:271
267799201,2cyclictest0-21swapper/120:31:021
267799201,18cyclictest0-21swapper/120:03:481
267799201,17cyclictest0-21swapper/122:40:091
267799201,17cyclictest0-21swapper/119:57:521
267799201,17cyclictest0-21swapper/119:20:061
267799201,10cyclictest16256-21sshd19:36:581
267799200,15cyclictest0-21swapper/100:17:511
267399204,14cyclictest11-21rcuc/021:05:260
267399201,9cyclictest0-21swapper/020:51:390
267399201,18cyclictest0-21swapper/020:56:360
267399201,16cyclictest1977-21/usr/sbin/munin20:20:260
86802191,3sleep20-21swapper/219:20:222
269099191,9cyclictest3941-21ntp_states19:10:333
2690991912,4cyclictest32655-21fschecks_time20:15:223
2690991912,4cyclictest28415-21sh20:05:223
268399196,2cyclictest15203-21kworker/2:119:45:532
268399195,6cyclictest0-21swapper/221:15:322
268399194,5cyclictest32136-21/usr/sbin/munin20:15:242
268399191,8cyclictest768-21in:imjournal19:11:462
268399191,7cyclictest0-21swapper/223:07:592
268399191,6cyclictest18085-21grep19:40:352
268399191,6cyclictest0-21swapper/221:25:252
268399191,5cyclictest7612-21/usr/sbin/munin23:00:282
2683991913,3cyclictest20772-21rt-features21:00:312
268399191,10cyclictest0-21swapper/220:04:062
267799192,14cyclictest1-21systemd20:40:011
267799192,14cyclictest1-21systemd20:40:011
267799191,8cyclictest13014-21strings23:10:311
2677991912,4cyclictest5600-21munin-node19:15:041
2677991912,4cyclictest28346-21head20:05:221
267799191,17cyclictest0-21swapper/120:51:311
267799191,17cyclictest0-21swapper/119:41:471
267799191,17cyclictest0-21swapper/119:34:171
267799190,6cyclictest0-21swapper/120:25:461
267799190,17cyclictest0-21swapper/121:00:101
267399197,3cyclictest0-21swapper/020:16:340
267399197,3cyclictest0-21swapper/019:10:200
267399194,6cyclictest7423-21kworker/0:319:16:240
267399190,9cyclictest0-21swapper/020:31:500
267399190,5cyclictest0-21swapper/021:00:270
206072191,3sleep10-21swapper/119:46:281
189732191,3sleep10-21swapper/120:55:481
268399186,3cyclictest86550irq/126-enp1s0-19:35:302
268399186,3cyclictest12050irq/123-ahci[0020:55:012
268399185,3cyclictest20896-21cpuspeed_turbos23:30:192
268399183,6cyclictest0-21swapper/220:45:242
268399182,5cyclictest5667-21diskstats19:15:042
268399181,5cyclictest10658-21cat20:40:012
268399181,5cyclictest10658-21cat20:40:012
2683991813,3cyclictest28410-21tail20:05:222
267399184,3cyclictest18937-21/usr/sbin/munin19:45:340
267399182,4cyclictest16744-21/usr/sbin/munin19:40:200
267399181,2cyclictest0-21swapper/019:55:200
2673991812,3cyclictest28651-21load20:05:240
267399181,14cyclictest12447-21taskset20:41:320
268399172,3cyclictest6378-21/usr/sbin/munin20:30:252
2683991610,3cyclictest22191-21ntp_offset19:50:292
268399151,7cyclictest0-21swapper/219:30:292
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional