You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-24 - 17:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Sat Feb 24, 2024 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2889911000,90ptp4l0-21swapper/107:06:361
284412980,0sleep60-21swapper/607:40:238
288126352,9sleep50-21swapper/507:09:427
281026251,9sleep40-21swapper/407:08:426
264825846,10sleep70-21swapper/707:06:339
275225644,10sleep110-21swapper/1107:08:013
275125544,9sleep100-21swapper/1007:08:002
107442550,1sleep810747-21unixbench_singl09:20:3610
284925443,9sleep00-21swapper/007:09:140
259325443,9sleep60-21swapper/607:05:438
288991240,0ptp4l0-21swapper/107:30:271
327099232,17cyclictest0-21swapper/410:15:006
288991230,4ptp4l0-21swapper/112:15:301
288991230,4ptp4l0-21swapper/112:15:291
327099223,14cyclictest0-21swapper/411:20:006
327099221,16cyclictest0-21swapper/409:30:006
326099203,10cyclictest0-21swapper/209:29:594
326099202,11cyclictest0-21swapper/210:35:004
288991200,0ptp4l0-21swapper/109:20:001
3275991916,2cyclictest14920-21turbostat09:34:557
325699193,9cyclictest0-21swapper/111:50:011
288991190,1ptp4l0-21swapper/107:55:361
272291190,0getstats0-21swapper/209:41:174
329799182,2cyclictest0-21swapper/1012:35:282
329799182,2cyclictest0-21swapper/1012:35:282
329799180,17cyclictest0-21swapper/1011:15:432
328499184,7cyclictest0-21swapper/710:10:009
326099183,10cyclictest0-21swapper/209:25:004
268762180,0sleep90-21swapper/907:35:3711
328899173,9cyclictest0-21swapper/807:35:0110
327999173,9cyclictest0-21swapper/610:50:008
327999173,9cyclictest0-21swapper/610:30:008
327999170,15cyclictest0-21swapper/610:10:298
326099174,8cyclictest0-21swapper/212:05:014
326099173,9cyclictest0-21swapper/212:25:014
325699172,8cyclictest0-21swapper/110:45:001
288991170,1ptp4l21-21ksoftirqd/112:35:571
288991170,1ptp4l21-21ksoftirqd/112:35:561
288991170,1ptp4l21-21ksoftirqd/112:05:331
288991170,1ptp4l21-21ksoftirqd/112:05:331
288991170,1ptp4l21-21ksoftirqd/112:00:261
288991170,1ptp4l0-21swapper/107:15:011
288991170,0ptp4l0-21swapper/109:25:291
288991170,0ptp4l0-21swapper/108:05:271
328899162,9cyclictest0-21swapper/811:20:0010
328499164,8cyclictest0-21swapper/710:35:009
328499162,9cyclictest0-21swapper/710:24:599
327999163,7cyclictest0-21swapper/609:45:008
326099163,8cyclictest0-21swapper/212:00:004
326099162,8cyclictest0-21swapper/208:25:364
325699163,8cyclictest0-21swapper/107:30:011
325199160,16cyclictest0-21swapper/009:45:390
288991160,1ptp4l0-21swapper/112:10:381
288991160,0ptp4l0-21swapper/108:45:301
288991160,0ptp4l0-21swapper/107:45:271
272291160,0getstats0-21swapper/210:11:164
272291160,0getstats0-21swapper/209:36:174
328899153,8cyclictest0-21swapper/812:35:0110
328899153,8cyclictest0-21swapper/812:35:0110
328899152,8cyclictest0-21swapper/811:35:0110
327999153,7cyclictest0-21swapper/608:55:018
327999152,8cyclictest0-21swapper/610:40:008
327999151,9cyclictest0-21swapper/609:20:008
327099150,2cyclictest0-21swapper/409:55:256
326099153,8cyclictest0-21swapper/208:35:014
326099153,8cyclictest0-21swapper/208:20:414
325699153,8cyclictest0-21swapper/111:25:011
288991150,6ptp4l0-21swapper/111:10:351
288991150,1ptp4l21-21ksoftirqd/112:20:311
288991150,0ptp4l0-21swapper/111:45:341
288991150,0ptp4l0-21swapper/111:30:381
288991150,0ptp4l0-21swapper/111:15:261
288991150,0ptp4l0-21swapper/109:55:331
288991150,0ptp4l0-21swapper/109:50:231
288991150,0ptp4l0-21swapper/109:30:231
288991150,0ptp4l0-21swapper/109:23:311
288991150,0ptp4l0-21swapper/108:25:361
288991150,0ptp4l0-21swapper/107:15:021
28012157,6sleep80-21swapper/807:08:3410
328899142,7cyclictest0-21swapper/809:05:0010
328499142,9cyclictest2690-21kworker/u24:210:20:009
327999142,8cyclictest0-21swapper/609:00:008
327999142,8cyclictest0-21swapper/607:15:018
327999142,7cyclictest0-21swapper/609:40:008
327099140,2cyclictest0-21swapper/407:40:276
326099142,8cyclictest0-21swapper/211:15:014
326099142,8cyclictest0-21swapper/208:35:424
325699142,7cyclictest0-21swapper/108:45:011
325199143,6cyclictest0-21swapper/011:30:000
325199142,7cyclictest0-21swapper/008:45:010
288991140,1ptp4l21-21ksoftirqd/112:25:421
288991140,1ptp4l21-21ksoftirqd/112:25:411
288991140,1ptp4l0-21swapper/109:05:381
288991140,0ptp4l0-21swapper/110:35:381
288991140,0ptp4l0-21swapper/109:40:351
288991140,0ptp4l0-21swapper/108:50:291
288991140,0ptp4l0-21swapper/107:20:271
272291140,0getstats0-21swapper/211:31:164
330299130,13cyclictest0-21swapper/1111:50:233
329799132,7cyclictest0-21swapper/1011:00:002
328499130,8cyclictest0-21swapper/711:00:009
328499130,12cyclictest0-21swapper/709:40:409
327999132,7cyclictest0-21swapper/611:10:008
327999130,8cyclictest0-21swapper/611:30:008
325199133,7cyclictest0-21swapper/012:10:010
325199132,7cyclictest0-21swapper/012:25:400
325199132,7cyclictest0-21swapper/012:25:400
3251991311,1cyclictest0-21swapper/009:10:310
288991130,3ptp4l0-21swapper/108:20:281
288991130,1ptp4l21-21ksoftirqd/110:30:221
288991130,0ptp4l0-21swapper/111:44:221
288991130,0ptp4l0-21swapper/111:20:311
288991130,0ptp4l0-21swapper/110:47:281
288991130,0ptp4l0-21swapper/110:10:251
288991130,0ptp4l0-21swapper/108:12:351
284972130,0sleep110-21swapper/1109:05:253
272291130,0getstats0-21swapper/211:26:174
272291130,0getstats0-21swapper/209:46:174
3302991210,1cyclictest0-21swapper/1109:34:553
330299120,11cyclictest19183-21cstates08:10:263
329399123,8cyclictest6186-21unixbench_singl07:50:3911
329399120,11cyclictest0-21swapper/911:35:3711
328499122,6cyclictest0-21swapper/712:10:009
328499122,6cyclictest0-21swapper/712:10:009
328499122,5cyclictest0-21swapper/710:00:009
328499120,5cyclictest0-21swapper/711:40:599
328499120,12cyclictest0-21swapper/711:50:229
3279991210,1cyclictest0-21swapper/608:30:018
327999120,11cyclictest9929-21diskstats12:10:248
327099123,6cyclictest0-21swapper/407:30:346
327099122,7cyclictest0-21swapper/408:15:346
326099121,9cyclictest0-21swapper/210:03:484
326099120,2cyclictest0-21swapper/209:00:254
3256991210,1cyclictest0-21swapper/110:00:311
325199128,3cyclictest19584-21hddtemp_smartct11:40:250
325199128,3cyclictest10092-21aten2_r3power_e10:05:180
325199122,6cyclictest0-21swapper/007:25:010
325199122,5cyclictest0-21swapper/011:35:000
3251991211,1cyclictest0-21swapper/008:13:410
296672120,0sleep80-21swapper/809:45:3810
288991120,3ptp4l31078-21latency_hist07:45:011
288991120,1ptp4l0-21swapper/111:55:321
288991120,1ptp4l0-21swapper/110:20:221
288991120,0ptp4l0-21swapper/111:05:541
288991120,0ptp4l0-21swapper/111:00:281
288991120,0ptp4l0-21swapper/110:15:211
288991120,0ptp4l0-21swapper/109:37:081
288991120,0ptp4l0-21swapper/108:03:461
19342120,0sleep10-21swapper/108:30:231
138152120,0sleep70-21swapper/712:15:309
138152120,0sleep70-21swapper/712:15:309
94480110,0rtkit-daemon0-21swapper/1012:34:232
94480110,0rtkit-daemon0-21swapper/1012:34:232
329799117,3cyclictest32276-21ps08:25:312
329799110,1cyclictest0-21swapper/1012:13:272
3293991110,1cyclictest0-21swapper/907:15:3311
329399110,1cyclictest0-21swapper/909:29:3211
329399110,11cyclictest0-21swapper/909:09:3311
329399110,11cyclictest0-21swapper/907:23:5411
328899110,1cyclictest0-21swapper/810:31:2410
328499112,5cyclictest0-21swapper/711:20:009
328499111,7cyclictest0-21swapper/707:50:389
328499111,6cyclictest0-21swapper/708:25:019
328499110,7cyclictest0-21swapper/707:39:399
328499110,1cyclictest1033-21runrttasks09:50:259
328499110,1cyclictest0-21swapper/709:05:479
327999118,2cyclictest29036-21cstates09:05:358
327999110,3cyclictest30822-21sed10:30:298
327999110,10cyclictest0-21swapper/609:50:298
327599110,1cyclictest0-21swapper/509:52:337
327099110,7cyclictest0-21swapper/412:00:586
326699110,1cyclictest0-21swapper/308:05:085
326699110,11cyclictest0-21swapper/311:42:365
326099118,2cyclictest22977-21fschecks_count11:45:264
326099118,2cyclictest0-21swapper/211:10:294
326099118,2cyclictest0-21swapper/210:45:224
326099110,1cyclictest0-21swapper/207:43:494
326099110,10cyclictest13306-21chrt08:00:334
325199119,1cyclictest22656-21latency_hist09:39:590
325199118,2cyclictest29133-21runrttasks10:30:150
325199112,3cyclictest11444-21cpuspeed_turbos08:00:230
3251991110,1cyclictest0-21swapper/010:55:330
325199110,4cyclictest17222-21grep12:20:250
325199110,2cyclictest0-21swapper/012:35:230
325199110,2cyclictest0-21swapper/012:35:230
288991110,1ptp4l6892-21idleruntime08:35:331
288991110,1ptp4l21-21ksoftirqd/108:56:431
288991110,1ptp4l0-21swapper/110:10:001
288991110,0ptp4l0-21swapper/112:30:341
288991110,0ptp4l0-21swapper/112:30:341
288991110,0ptp4l0-21swapper/109:10:341
288991110,0ptp4l0-21swapper/107:35:201
330299109,1cyclictest0-21swapper/1107:40:173
330299107,2cyclictest21989-21perl08:55:263
330299100,9cyclictest29680-21sed12:35:283
330299100,9cyclictest29680-21sed12:35:273
330299100,9cyclictest17602-21sed08:05:363
330299100,9cyclictest12600-21grep09:25:243
330299100,7cyclictest30064-21/usr/sbin/munin11:10:413
330299100,4cyclictest0-21swapper/1109:39:423
330299100,1cyclictest0-21swapper/1107:26:163
330299100,10cyclictest0-21swapper/1109:10:023
330299100,10cyclictest0-21swapper/1108:16:203
329799100,9cyclictest23816-21taskset12:29:252
329799100,9cyclictest23816-21taskset12:29:252
329799100,9cyclictest21906-21munin-run08:15:012
329799100,4cyclictest1138-21sed09:50:362
329799100,3cyclictest17807-21munin-run11:39:592
329799100,1cyclictest0-21swapper/1007:35:352
329799100,10cyclictest0-21swapper/1010:22:092
329399109,1cyclictest0-21swapper/910:37:0211
329399109,1cyclictest0-21swapper/908:17:1711
329399106,4cyclictest0-21swapper/911:56:1411
329399103,6cyclictest6196-21pcscd11:25:0011
329399100,10cyclictest0-21swapper/909:18:2911
328899109,1cyclictest0-21swapper/811:10:4210
328899109,1cyclictest0-21swapper/809:51:3110
328899107,2cyclictest9125-21/usr/sbin/munin12:10:2710
328899106,4cyclictest0-21swapper/807:38:3110
328899106,3cyclictest19351-21sshd07:25:5610
328899103,6cyclictest8370-21phc2sys10:00:3410
328899102,5cyclictest0-21swapper/809:10:0010
328899101,8cyclictest0-21swapper/808:35:3410
328899100,9cyclictest3306-21latency_hist09:15:0010
328899100,9cyclictest32277-21switchtime11:55:3410
328899100,9cyclictest21508-21taskset11:43:2610
328899100,9cyclictest16228-21latency_hist12:20:0010
328899100,9cyclictest16228-21latency_hist12:19:5910
328899100,10cyclictest0-21swapper/812:28:5110
328899100,10cyclictest0-21swapper/812:28:5010
328899100,10cyclictest0-21swapper/812:09:2510
328899100,10cyclictest0-21swapper/812:09:2510
328899100,10cyclictest0-21swapper/810:47:4510
328899100,10cyclictest0-21swapper/808:27:5910
328499106,3cyclictest7146-21sshd07:55:119
328499106,3cyclictest6742-21gltestperf08:35:329
328499106,3cyclictest16710-21idleruntime09:30:299
328499106,3cyclictest11744-21crond10:50:009
328499102,7cyclictest22979-21fschecks_count11:45:269
328499100,9cyclictest30237-21tr11:55:229
328499100,9cyclictest28801-21cstates12:35:229
328499100,9cyclictest28801-21cstates12:35:229
328499100,9cyclictest14454-21memory07:20:339
328499100,9cyclictest1000-21cut09:10:269
328499100,9cyclictest0-21swapper/708:15:229
328499100,4cyclictest20315-21awk08:55:019
328499100,2cyclictest5368-21irqrtprio09:15:339
328499100,2cyclictest24126-21munin-run08:59:599
328499100,10cyclictest0-21swapper/709:27:229
327999109,1cyclictest0-21swapper/610:06:098
327999107,2cyclictest22738-21gltestperf10:20:228
327999106,3cyclictest7206-21timerandwakeup10:40:368
327999106,3cyclictest25967-21/usr/sbin/munin11:50:208
327999106,3cyclictest25851-21sed07:35:308
327999105,4cyclictest7964-21diskstats07:55:238
327999100,9cyclictest5292-21chrt09:56:148
327999100,9cyclictest15571-21taskset07:21:308
327999100,9cyclictest15316-21latency_hist09:30:008
327999100,9cyclictest1449-21proc_pri07:45:308
327999100,9cyclictest11141-21cpuspeed08:00:228
327999100,9cyclictest0-21swapper/608:45:318
327999100,4cyclictest5289-21uname07:50:328
327999100,3cyclictest28783-21expr12:35:228
327999100,3cyclictest28783-21expr12:35:228
327999100,10cyclictest0-21swapper/609:47:478
327599109,1cyclictest0-21swapper/509:06:487
327599109,1cyclictest0-21swapper/507:23:357
327599100,2cyclictest0-21swapper/508:25:267
327599100,1cyclictest0-21swapper/510:58:007
327099105,3cyclictest0-21swapper/410:49:176
327099100,9cyclictest0-21swapper/410:25:006
327099100,10cyclictest0-21swapper/408:44:406
326699109,1cyclictest0-21swapper/312:24:145
326699109,1cyclictest0-21swapper/310:22:455
326699108,2cyclictest29443-21fschecks_count09:05:365
326699106,3cyclictest1592-21swap07:45:335
326699102,1cyclictest0-21swapper/311:59:245
326699100,10cyclictest0-21swapper/307:50:275
326699100,10cyclictest0-21swapper/307:33:005
326099107,2cyclictest29024-21/usr/sbin/munin09:05:344
326099107,2cyclictest0-21swapper/210:55:234
326099100,9cyclictest21611-21grep10:20:194
326099100,9cyclictest0-21swapper/207:45:344
326099100,10cyclictest0-21swapper/209:59:034
325199109,1cyclictest0-21swapper/010:46:230
325199106,3cyclictest17529-21sed08:05:360
325199105,4cyclictest18272-21expr10:15:240
325199105,4cyclictest10492-21/usr/sbin/munin07:15:310
325199100,9cyclictest4835-21taskset08:34:140
325199100,9cyclictest31986-21cpuspeed_turbos07:45:210
325199100,9cyclictest2543-21cat12:00:230
325199100,9cyclictest23360-21tail11:45:270
325199100,9cyclictest22419-21taskset08:15:080
325199100,9cyclictest13940-21expr08:45:290
325199100,6cyclictest0-21swapper/010:44:490
325199100,4cyclictest16799-21unixbench_singl10:10:390
325199100,3cyclictest19389-21switchtime08:50:360
325199100,1cyclictest0-21swapper/007:52:030
325199100,10cyclictest0-21swapper/011:03:480
325199100,10cyclictest0-21swapper/009:05:420
325199100,10cyclictest0-21swapper/008:35:290
288991100,0ptp4l0-21swapper/111:35:221
288991100,0ptp4l0-21swapper/110:55:191
288991100,0ptp4l0-21swapper/110:50:221
288991100,0ptp4l0-21swapper/110:25:281
288991100,0ptp4l0-21swapper/109:45:231
288991100,0ptp4l0-21swapper/109:00:331
288991100,0ptp4l0-21swapper/108:15:231
288991100,0ptp4l0-21swapper/107:50:211
33029998,1cyclictest0-21swapper/1108:40:403
33029997,1cyclictest4280-21missed_timers09:55:283
33029996,2cyclictest27867-21fschecks_time09:45:253
33029995,3cyclictest1857-21basename10:35:303
33029995,1cyclictest0-21swapper/1110:05:293
33029995,1cyclictest0-21swapper/1108:52:173
33029993,6cyclictest0-21swapper/1107:19:573
33029991,7cyclictest24208-21tr11:05:263
33029990,8cyclictest8249-21unixbench_singl12:05:453
33029990,8cyclictest8249-21unixbench_singl12:05:453
33029990,8cyclictest28027-21latency_hist12:35:013
33029990,8cyclictest28027-21latency_hist12:35:013
33029990,8cyclictest22621-21cat11:00:393
33029990,8cyclictest15588-21fschecks_time11:35:243
33029990,8cyclictest13820-21unixbench-2d08:00:353
33029990,7cyclictest16872-21cut12:20:243
33029990,3cyclictest25159-21head07:35:273
32979998,1cyclictest0-21swapper/1010:35:022
32979998,1cyclictest0-21swapper/1010:10:432
32979997,2cyclictest14695-21grep08:45:342
32979996,2cyclictest19439-21fschecks_time11:40:242
32979996,2cyclictest11822-21latency10:05:272
32979992,6cyclictest5232-21cat07:50:332
32979992,6cyclictest12987-21latency_hist08:45:012
32979990,9cyclictest0-21swapper/1009:55:232
32979990,8cyclictest7228-21cpuspeed_turbos11:25:222
32979990,8cyclictest613-21chrt11:59:542
32979990,8cyclictest32625-21perl07:45:242
32979990,8cyclictest27653-21expr09:45:232
32979990,8cyclictest26471-21fw_conntrack10:25:302
32979990,8cyclictest20490-21cpuspeed_turbos07:30:232
32979990,8cyclictest14029-21kernelversion10:50:272
32979990,8cyclictest1380-21if_p34p109:10:302
32979990,8cyclictest12119-21if_err_enp4s011:30:292
32979990,8cyclictest11742-21expr08:00:232
32979990,5cyclictest0-21swapper/1009:30:002
32979990,3cyclictest29277-21cpu10:30:212
32979990,3cyclictest25172-21timerwakeupswit08:15:352
32979990,3cyclictest2105-21tune2fs12:00:222
32979990,3cyclictest1717-21cpuspeed08:30:232
32979990,3cyclictest16304-21fschecks_count09:30:262
32979990,3cyclictest1032-21snmpd09:37:462
32939998,1cyclictest0-21swapper/911:00:2711
32939998,1cyclictest0-21swapper/910:40:5511
32939998,1cyclictest0-21swapper/908:00:3511
32939996,2cyclictest15100-21cpuspeed_turbos08:05:2311
32939996,2cyclictest0-21swapper/909:00:2911
32939996,2cyclictest0-21swapper/908:50:2411
32939995,3cyclictest25887-21latency_hist11:50:0111
32939990,9cyclictest0-21swapper/912:25:3111
32939990,9cyclictest0-21swapper/912:25:3011
32939990,9cyclictest0-21swapper/911:31:4511
32939990,9cyclictest0-21swapper/910:05:3111
32939990,9cyclictest0-21swapper/909:40:0011
32939990,9cyclictest0-21swapper/909:10:4011
32889998,1cyclictest0-21swapper/809:40:2910
32889997,1cyclictest0-21swapper/807:40:2510
32889997,1cyclictest0-21swapper/807:20:2710
32889996,2cyclictest4179-21cpuspeed_turbos09:15:2610
32889991,7cyclictest1032-21snmpd10:23:2010
32889991,5cyclictest0-21swapper/808:15:0110
32889991,5cyclictest0-21swapper/808:05:3610
32889990,9cyclictest0-21swapper/809:35:2310
32889990,8cyclictest9521-21switchtime07:55:3510
32889990,8cyclictest2478-21latency10:35:3410
32889990,8cyclictest24449-21ls09:00:0010
32889990,8cyclictest18624-21/usr/sbin/munin08:10:3910
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional