You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-18 - 05:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot3s.osadl.org (updated Mon Jun 17, 2024 12:50:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26712600582,12sleep00-21swapper/007:06:060
26712600582,12sleep00-21swapper/007:06:050
30122589568,15sleep40-21swapper/407:09:234
30122589568,15sleep40-21swapper/407:09:234
23972544521,15sleep60-21swapper/607:06:036
23972544521,15sleep60-21swapper/607:06:036
30512515488,19sleep50-21swapper/507:09:555
30512515488,19sleep50-21swapper/507:09:555
21472507485,15sleep10-21swapper/107:05:501
21472507485,15sleep10-21swapper/107:05:501
29372491466,19sleep30-21swapper/307:08:173
29372491466,19sleep30-21swapper/307:08:173
30282490469,14sleep20-21swapper/207:09:362
30282490469,14sleep20-21swapper/207:09:362
21282374352,15sleep70-21swapper/707:05:407
21282374352,15sleep70-21swapper/707:05:407
3500996160,0cyclictest16984-21apt11:00:007
93822590,3sleep0101ktimersoftd/008:25:500
37612590,0sleep10-21swapper/111:39:031
111882580,0sleep70-21swapper/710:45:357
3498995452,1cyclictest16983-21cron11:00:015
3498995249,2cyclictest14807-21needreboot09:45:355
3496995250,1cyclictest26210-21cpuspeed_turbos10:10:173
3499995049,0cyclictest27010-21sendmail_mailqu07:55:516
3499995048,1cyclictest30319-21cron10:20:006
3496995049,0cyclictest11659-21tr09:40:163
3496995048,1cyclictest8119-21cpuspeed_turbos10:40:133
3496994745,1cyclictest25544-21cat12:25:003
3496994443,0cyclictest32748-21grep10:25:013
3495994441,2cyclictest23766-21tr12:20:142
3495994342,0cyclictest15381-21tail12:00:462
3500994240,1cyclictest11615-21tr11:55:177
3499994238,2cyclictest11616-21cpuspeed_turbos11:55:176
3493994241,0cyclictest8571-21sh07:20:010
3493994241,0cyclictest8571-21sh07:20:000
3500994139,1cyclictest14608-21sh10:55:007
3497994139,1cyclictest16980-21sh11:00:004
3497994139,1cyclictest1446-21cron11:35:004
3496994118,1cyclictest11559-21grep11:55:173
3495994139,1cyclictest4523-21grep11:40:142
3495994129,8cyclictest121rcu_sched11:00:002
3494994139,1cyclictest2256-21grep08:11:111
3500994038,1cyclictest30320-21cron10:20:017
3500994038,1cyclictest26202-21grep10:10:187
3500994015,1cyclictest21401-21grep07:45:157
3498994038,1cyclictest0-21swapper/507:55:165
349699400,8cyclictest0-21swapper/310:25:093
3494994038,1cyclictest19938-21cpuspeed_turbos11:05:151
3493994038,1cyclictest3724-21cpuspeed_turbos07:10:150
3500993937,1cyclictest20019-21grep11:05:157
3499993937,1cyclictest1954-21timerandwakeup08:11:086
3499993937,1cyclictest14607-21cron10:55:006
3497993937,1cyclictest6003-21fschecks_time10:35:234
3496993938,0cyclictest10455-21grep08:30:183
349599390,1cyclictest0-21swapper/210:10:482
3494993937,1cyclictest9797-21needreboot09:35:271
349499390,37cyclictest0-21swapper/109:05:161
3500993836,1cyclictest14059-21tr09:45:157
3499993836,1cyclictest19961-21cpuspeed_turbos11:05:166
3498993831,6cyclictest0-21swapper/508:35:015
3497993829,2cyclictest8363-21grep10:40:174
349799380,0cyclictest0-21swapper/407:40:344
3496993836,1cyclictest29306-21needreboot10:15:423
3496993815,14cyclictest0-21swapper/311:39:423
3495993837,0cyclictest20044-21tr11:05:152
3495993836,1cyclictest16226-21fgrep08:40:502
3495993836,1cyclictest10136-21wc08:30:012
3495993831,6cyclictest29416-21cat11:25:012
3493993836,1cyclictest0-21swapper/008:10:380
3500993736,0cyclictest21205-21tr10:00:137
3500993735,1cyclictest9270-21tr07:20:127
3500993735,1cyclictest5497-21wc11:40:437
3500993735,1cyclictest0-21swapper/708:00:157
3499993735,1cyclictest16599-21grep12:05:136
3499993722,14cyclictest11565-21grep07:25:176
3499993722,14cyclictest11565-21grep07:25:176
3498993735,1cyclictest28990-21cat12:30:215
3497993736,0cyclictest7559-21cut07:15:404
3497993736,0cyclictest7559-21cut07:15:404
3497993722,14cyclictest27155-21mailstats10:10:434
349699376,20cyclictest0-21swapper/309:51:233
3495993735,1cyclictest16495-21grep09:50:152
3494993735,1cyclictest9391-21grep08:25:521
3494993735,1cyclictest0-21swapper/111:25:251
3493993736,0cyclictest9798-21strings09:35:280
3493993735,1cyclictest850-21cpuspeed_turbos10:25:170
3500993635,0cyclictest29635-21wc12:30:417
3500993634,1cyclictest3749-21cpuspeed_turbos07:10:147
3500993634,1cyclictest19427-21cat09:55:237
3499993634,1cyclictest5345-21switchtime09:25:446
3499993634,1cyclictest14058-21grep09:45:156
3498993634,1cyclictest18998-21cpuspeed_turbos07:40:145
3497993634,1cyclictest30914-21grep12:35:114
3497993634,1cyclictest26844-21needreboot12:25:284
349799360,1cyclictest280-21systemd-journal09:45:004
349699366,10cyclictest0-21swapper/309:35:193
3496993635,0cyclictest10426-21cpuspeed_turbos10:45:173
3496993634,1cyclictest4665-21unixbench_singl08:15:563
3496993634,1cyclictest23033-21missed_timers08:55:383
3495993634,1cyclictest29232-21grep12:30:272
3495993634,1cyclictest19441-21wc07:40:272
3495993626,6cyclictest111rcu_preempt09:20:402
349599360,12cyclictest280-21systemd-journal11:20:012
3494993634,1cyclictest29633-21cat12:30:411
3494993621,14cyclictest25678-21cat12:25:001
3493993635,0cyclictest12859-21cut08:35:150
3493993634,1cyclictest1945-21tr09:20:120
3493993634,1cyclictest0-21swapper/007:40:530
3500993534,0cyclictest0-21swapper/712:20:007
3500993533,1cyclictest0-21swapper/711:50:007
3500993533,1cyclictest0-21swapper/709:25:417
3499993534,0cyclictest27883-21cron12:30:006
3499993534,0cyclictest14619-21wc12:00:316
3499993533,1cyclictest28489-21grep08:00:156
3499993533,1cyclictest19880-21grep09:55:346
3499993533,1cyclictest0-21swapper/607:20:126
3498993534,0cyclictest1037-21unixbench_multi11:30:425
3498993534,0cyclictest0-21swapper/511:40:265
3498993533,1cyclictest15297-21cat12:00:465
3498993533,1cyclictest0-21swapper/510:50:435
349899350,1cyclictest27885-21sh12:30:005
3497993533,1cyclictest26731-21apt11:20:004
3497993533,1cyclictest12838-21cut10:50:134
3497993533,1cyclictest0-21swapper/407:50:404
349699359,10cyclictest0-21swapper/311:11:053
3496993534,0cyclictest21344-21cpuspeed_turbos07:45:153
3496993533,1cyclictest5732-21cpuspeed_turbos08:20:193
3496993533,1cyclictest21289-21tr10:00:133
3496993533,1cyclictest18908-21cat08:45:493
349699352,22cyclictest0-21swapper/311:22:473
3495993533,1cyclictest28220-21cat09:05:562
3495993533,1cyclictest10486-21cpuspeed_turbos10:45:172
3494993533,1cyclictest0-21swapper/107:15:521
3494993533,1cyclictest0-21swapper/107:15:521
349499350,33cyclictest0-21swapper/112:00:321
3493993533,1cyclictest28330-21grep09:05:570
349399350,34cyclictest0-21swapper/011:00:300
3500993432,1cyclictest23592-21cpuspeed_turbos10:05:157
3500993432,1cyclictest0-21swapper/711:00:437
3499993433,0cyclictest26203-21tr10:10:176
3499993432,1cyclictest3747-21tr07:10:146
3499993432,1cyclictest24138-21wc10:05:286
3498993433,1cyclictest0-21swapper/507:15:005
3498993432,1cyclictest3323-21grep10:30:155
3498993432,1cyclictest32006-21tr09:15:145
3498993432,1cyclictest30869-21tr08:05:165
3498993432,1cyclictest14085-21grep08:35:525
3497993432,1cyclictest0-21swapper/411:20:134
349799340,0cyclictest0-21swapper/412:00:154
349699345,15cyclictest0-21swapper/309:16:193
3496993433,0cyclictest7581-21cut11:45:283
3496993433,0cyclictest2646-21needreboot09:20:293
3496993432,1cyclictest30868-21cpuspeed_turbos12:35:113
349699340,0cyclictest0-21swapper/312:15:133
3495993432,1cyclictest7554-21head07:15:402
3495993432,1cyclictest7554-21head07:15:402
3495993432,1cyclictest0-21swapper/208:40:002
3494993433,0cyclictest0-21swapper/111:40:431
3494993432,1cyclictest32240-21unixbench_singl08:05:531
3493993432,1cyclictest18842-21tr09:55:110
3500993331,1cyclictest9678-21grep09:35:267
3500993331,1cyclictest28061-21wc09:05:497
3500993331,1cyclictest10475-21cpuspeed_turbos08:30:187
3500993331,1cyclictest0-21swapper/710:45:007
3500993331,1cyclictest0-21swapper/708:50:507
3500993331,1cyclictest0-21swapper/708:35:507
3499993331,1cyclictest8776-21cat11:50:006
3499993331,1cyclictest29434-21sendmail_mailqu12:30:396
3499993331,1cyclictest23171-21grep12:20:006
3498993331,1cyclictest29920-21head08:01:105
3498993331,1cyclictest23864-21grep11:10:425
3498993331,1cyclictest19000-21tr12:10:145
3498993317,14cyclictest22052-21grep07:45:355
349899330,1cyclictest0-21swapper/511:56:255
3497993332,0cyclictest9944-21needreboot07:20:364
3497993332,0cyclictest16618-21tr12:05:134
3497993332,0cyclictest0-21swapper/412:15:464
3497993331,1cyclictest2715-21latency_hist10:30:004
3497993331,1cyclictest15412-21grep07:31:054
3497993331,1cyclictest15412-21grep07:31:054
3497993331,1cyclictest10396-21tr08:30:184
3496993331,1cyclictest5344-21grep11:40:403
3496993331,1cyclictest21233-21wc11:05:483
3495993330,1cyclictest11638-21tr07:25:172
3495993330,1cyclictest11638-21tr07:25:162
3495993318,14cyclictest5041-21needreboot09:25:322
3494993331,1cyclictest27979-21cat10:15:001
3494993331,1cyclictest0-21swapper/112:35:291
3494993331,1cyclictest0-21swapper/112:10:011
3493993332,0cyclictest23121-21sendmail12:20:010
349399330,1cyclictest0-21swapper/009:40:490
3500993231,0cyclictest16339-21cpuspeed_turbos07:35:177
3500993231,0cyclictest16339-21cpuspeed_turbos07:35:177
3500993230,1cyclictest0-21swapper/709:00:497
3499993230,1cyclictest21186-21cpuspeed_turbos10:00:146
3499993230,1cyclictest18275-21wc11:00:306
3499993223,7cyclictest7088-21head08:21:016
3498993230,1cyclictest867-21tr10:25:175
3498993230,1cyclictest0-21swapper/509:05:165
3498993227,4cyclictest8026-21tr10:40:135
349899320,31cyclictest0-21swapper/510:15:185
3497993231,0cyclictest4465-21tr11:40:144
3497993230,1cyclictest32546-21wc09:15:294
3497993230,1cyclictest20006-21grep08:50:164
3496993231,0cyclictest17420-21cat07:35:563
3496993231,0cyclictest17420-21cat07:35:563
3496993230,1cyclictest9824-21wc07:20:343
3496993230,1cyclictest8059-21cat07:15:533
3496993230,1cyclictest8059-21cat07:15:533
349699320,31cyclictest0-21swapper/309:10:273
3495993230,1cyclictest499-21grep09:15:412
3495993230,1cyclictest32749-21wc10:25:002
3495993230,1cyclictest2118-21tr11:35:142
3495993230,1cyclictest18103-21turbostat07:40:002
3495993230,1cyclictest18103-21turbostat07:40:002
3495993222,7cyclictest121rcu_sched08:20:592
3494993231,0cyclictest1574-21wc10:25:361
3494993231,0cyclictest13868-21grep10:50:411
3494993230,1cyclictest26683-21missed_timers07:55:391
3494993230,1cyclictest17760-21grep07:36:001
3494993230,1cyclictest17760-21grep07:36:001
3493993230,1cyclictest6801-21cpuspeed_turbos11:45:150
3493993230,1cyclictest5249-21sendmail_mailqu09:25:410
3493993230,1cyclictest32268-21grep12:35:430
3493993230,1cyclictest26734-21sendmail11:20:000
3500993129,1cyclictest3301-21tr08:15:157
3500993129,1cyclictest18849-21cat08:45:497
3500993129,1cyclictest0-21swapper/709:25:017
3499993130,0cyclictest30863-21tr08:05:156
3499993129,1cyclictest9842-21sh10:45:006
3499993129,1cyclictest28059-21grep09:05:496
3499993129,1cyclictest2727-21wc11:35:306
3499993129,1cyclictest20069-21cpuspeed_turbos08:50:176
3499993129,1cyclictest13914-21cat08:35:506
3499993129,1cyclictest0-21swapper/609:35:266
3498993129,1cyclictest821-21tr08:10:195
3498993129,1cyclictest6807-21cpuspeed_turbos11:45:155
3498993129,1cyclictest26381-21unixbench_singl11:15:455
3498993129,1cyclictest25479-21latency_hist07:55:015
3498993129,1cyclictest15115-21grep07:31:015
3498993129,1cyclictest15115-21grep07:31:015
3498993129,1cyclictest11553-21grep07:25:165
3498993129,1cyclictest11553-21grep07:25:165
3498993122,8cyclictest30324-21ls11:25:225
3498993112,1cyclictest0-21swapper/507:23:235
3497993129,1cyclictest21404-21cpuspeed_turbos07:45:154
3497993117,13cyclictest11584-21cpuspeed_turbos07:25:164
3497993117,13cyclictest11584-21cpuspeed_turbos07:25:164
3496993129,1cyclictest30472-21cut11:25:273
3495993129,1cyclictest6632-21grep09:30:152
3495993129,1cyclictest3802-21latency10:30:262
3495993129,1cyclictest3242-21tr08:15:152
3495993129,1cyclictest29921-21unixbench_singl08:01:092
3495993129,1cyclictest0-21swapper/211:05:002
3495993129,1cyclictest0-21swapper/210:05:162
3495993128,2cyclictest13379-21latency10:50:262
349599310,30cyclictest0-21swapper/211:50:152
3494993129,1cyclictest24958-21cpuspeed_turbos11:15:151
3494993129,1cyclictest22956-21cat08:55:341
3494993129,1cyclictest10083-21ps07:20:451
3494993129,1cyclictest0-21swapper/110:45:001
3494993129,1cyclictest0-21swapper/108:40:561
3493993130,0cyclictest25661-21wc10:10:000
3493993130,0cyclictest0-21swapper/008:55:560
3493993129,1cyclictest3829-21cron11:40:000
3493993129,1cyclictest28386-21switchtime11:20:410
3493993129,1cyclictest26607-21grep07:55:320
3493993129,1cyclictest25030-21cat12:20:400
3493993129,1cyclictest0-21swapper/012:10:340
3500993029,0cyclictest790-21grep08:10:187
3500993029,0cyclictest27148-21cat12:25:417
3500993028,1cyclictest5672-21cpuspeed_turbos08:20:197
3500993028,1cyclictest31413-21cat12:35:247
3500993028,1cyclictest29542-21tr09:10:147
3500993028,1cyclictest24675-21fgrep12:20:367
3500993028,1cyclictest23495-21wc08:55:567
3500993028,1cyclictest18599-21grep12:10:007
3499993029,0cyclictest11763-21cpuspeed_turbos09:40:166
3499993028,1cyclictest3284-21tr08:15:156
3499993028,1cyclictest26097-21wc09:00:506
3499993028,1cyclictest16990-21latency_hist11:00:006
3499993028,1cyclictest0-21swapper/608:45:496
3498993029,0cyclictest0-21swapper/509:20:125
3498993028,1cyclictest9082-21cut09:35:135
3498993028,1cyclictest21965-21cat12:15:335
3498993028,1cyclictest16528-21cut09:50:165
3498993028,1cyclictest0-21swapper/508:25:155
3497993030,0cyclictest0-21swapper/410:05:154
3497993030,0cyclictest0-21swapper/407:10:144
3497993029,0cyclictest28141-21grep08:00:004
3497993029,0cyclictest2130-21tr11:35:144
3497993028,1cyclictest30642-21cat10:20:004
3497993028,1cyclictest25509-21sh12:25:004
3497993028,1cyclictest22460-21cpuspeed_turbos08:55:184
349799302,14cyclictest0-21swapper/408:27:544
3497993014,15cyclictest521ktimersoftd/411:00:354
349699304,11cyclictest0-21swapper/307:40:013
3496993028,1cyclictest9180-21cut11:50:153
3496993028,1cyclictest26170-21cpuspeed_turbos12:25:133
3496993028,1cyclictest24909-21tr11:15:153
3495993028,1cyclictest17673-21switchtime12:05:412
3495993028,1cyclictest0-21swapper/212:10:262
3495993024,5cyclictest27115-21mailstats12:25:392
3495993024,5cyclictest0-21swapper/210:15:182
3494993028,1cyclictest26116-21cpuspeed_turbos12:25:131
3494993028,1cyclictest26095-21cat09:00:491
3494993028,1cyclictest20117-21cut08:50:161
3494993028,1cyclictest17673-21cpuspeed_turbos08:45:161
3494993028,1cyclictest16989-21turbostat11:00:011
3494993028,1cyclictest15386-21grep07:31:051
3494993028,1cyclictest15386-21grep07:31:051
3494993028,1cyclictest0-21swapper/109:55:121
3494993028,1cyclictest0-21swapper/109:30:481
349399304,11cyclictest0-21swapper/011:40:230
3493993029,0cyclictest0-21swapper/008:50:160
3493993028,1cyclictest17629-21timerandwakeup09:50:430
3493993028,1cyclictest1282-21sh09:20:000
3493993028,1cyclictest12510-21grep11:56:160
3500992928,0cyclictest0-21swapper/709:55:007
3500992927,1cyclictest19559-21missed_timers07:40:327
3500992927,1cyclictest0-21swapper/709:30:337
3499992927,1cyclictest3697-21apt09:25:006
3499992927,1cyclictest20374-21head12:10:406
3499992927,1cyclictest10430-21tr10:45:176
3499992927,1cyclictest0-21swapper/612:35:246
3499992926,1cyclictest29544-21cpuspeed_turbos09:10:146
3498992927,1cyclictest6796-21tr07:15:155
3498992927,1cyclictest6796-21tr07:15:145
3498992927,1cyclictest25741-21find09:00:435
3498992927,1cyclictest2129-21tr11:35:145
3498992927,1cyclictest0-21swapper/510:05:155
3498992926,1cyclictest31016-21grep10:20:165
3497992928,0cyclictest0-21swapper/411:05:154
3497992927,1cyclictest0-21swapper/412:10:414
3496992929,0cyclictest0-21swapper/310:35:003
3496992928,0cyclictest29913-21unixbench_singl12:30:463
3496992927,1cyclictest12807-21tr10:50:133
3495992927,1cyclictest31139-21sed11:25:382
3495992927,1cyclictest0-21swapper/209:35:132
3495992927,1cyclictest0-21swapper/208:50:552
3495992927,1cyclictest0-21swapper/207:10:572
349499295,22cyclictest31952-21tr09:15:141
3494992928,0cyclictest27394-21tr11:20:131
3494992927,1cyclictest3747-21latency08:15:301
3494992927,1cyclictest1964-21cpuspeed_turbos09:20:121
3493992928,0cyclictest0-21swapper/009:10:280
3493992927,1cyclictest18095-21sh07:40:000
3493992927,1cyclictest18095-21sh07:40:000
3493992927,1cyclictest16088-21ps10:55:380
3500992826,1cyclictest6876-21cpuspeed_turbos07:15:147
3500992826,1cyclictest6876-21cpuspeed_turbos07:15:147
3500992826,1cyclictest2081-21tr11:35:157
3500992826,1cyclictest16712-21grep08:40:577
3499992826,1cyclictest16515-21tr09:50:156
3499992826,1cyclictest0-21swapper/612:20:356
349899288,17cyclictest0-21swapper/511:23:025
3498992826,1cyclictest9263-21cpuspeed_turbos11:50:155
3498992825,1cyclictest18967-21grep11:00:445
349899280,26cyclictest5664-21cpuspeed_turbos10:35:165
3497992827,0cyclictest17279-21fgrep07:35:524
3497992827,0cyclictest17279-21fgrep07:35:514
3497992826,1cyclictest29234-21needreboot12:30:274
3497992826,1cyclictest0-21swapper/408:35:154
3497992815,0cyclictest6436-21kworker/4:009:59:344
349799280,26cyclictest17184-21needreboot09:50:314
3496992827,0cyclictest964-21cat11:30:433
3496992827,0cyclictest2063-21cat08:11:093
3496992827,0cyclictest0-21swapper/307:10:533
3496992826,1cyclictest0-21swapper/309:10:013
349599285,1cyclictest25545-21wc07:55:002
3495992826,1cyclictest24822-21cpuspeed_turbos09:00:162
3495992826,1cyclictest0-21swapper/209:40:482
3495992825,1cyclictest31352-21cat09:15:002
3495992822,5cyclictest0-21swapper/207:45:152
3495992813,1cyclictest0-21swapper/207:21:222
3494992826,1cyclictest28450-21cpuspeed_turbos08:00:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional