You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-17 - 02:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sun Jun 16, 2024 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
235902749413,7435sleep320893-21kworker/u8:421:23:453
6034236110,316sleep36101-21fschecks_time20:28:363
10153234943,263sleep310216-21fschecks_time22:23:363
17783234817,311sleep217798-21apt19:23:172
6789232645,149sleep36817-21fschecks_count22:13:393
6789232645,149sleep36817-21fschecks_count22:13:393
5714224335,126sleep35798-21fschecks_time23:53:373
5714224335,126sleep35798-21fschecks_time23:53:373
15086224373,87sleep30-21swapper/322:38:473
32694224018,203sleep3396-21latency_hist21:53:203
5722223813,203sleep30-21swapper/320:28:203
25379222644,92sleep30-21swapper/323:13:193
13792222370,113sleep30-21swapper/319:08:453
28784222028,152sleep328914-21latency_hist19:58:203
18180221743,121sleep30-21swapper/322:48:443
18163220224,156sleep118199-21fschecks_count19:23:371
9314219411,146sleep39373-21switchtime00:03:593
29302219432,127sleep329357-21cat19:58:483
25804219151,56sleep30-21swapper/323:13:403
7735218610,123sleep30-21swapper/320:33:413
2379721778,107sleep323859-21sed23:08:193
4429217277,60sleep34444-21ntp_kernel_err23:48:453
21550217119,108sleep30-21swapper/321:18:193
7266216948,85sleep37304-21fschecks_count23:58:383
15071216776,43sleep315155-21latency_hist20:58:203
2795221668,146sleep31-21systemd21:38:183
11583215637,75sleep311602-21fschecks_count19:04:123
15897215046,67sleep316033-21tail00:23:553
24315214713,48sleep124391-21cat23:08:451
18622214712,119sleep018140-21apt-get21:08:270
8624214525,103sleep38707-21tune2fs22:18:523
23207214428,40sleep323374-21sed21:23:203
13418214012,114sleep113517-21latency_hist00:18:191
31099213925,85sleep30-21swapper/321:48:193
20066213930,70sleep320176-21timerwakeupswit22:53:533
3829213025,95sleep30-21swapper/320:20:003
3829213025,95sleep30-21swapper/320:20:003
2717021307,14sleep30-21swapper/323:18:323
2717021307,14sleep30-21swapper/323:18:323
19710212922,89sleep119757-21fschecks_count19:28:371
18126212912,102sleep318165-21http21:08:183
2624212818,28sleep30-21swapper/321:58:493
26161212816,12sleep30-21swapper/321:30:313
24376212814,34sleep30-21swapper/319:43:383
29978212411,75sleep330023-21unin-run23:28:173
16160212426,27sleep311096-21cyclictest20:58:573
14018212411,24sleep30-21swapper/322:33:543
18297212316,20sleep00-21swapper/022:48:490
14712212127,68sleep11166-21snmpd22:38:321
30392212046,23sleep30-21swapper/320:03:203
13263212026,84sleep31166-21snmpd20:51:063
10482212026,23sleep30-21swapper/300:08:283
735211965,14sleep30-21swapper/321:53:453
735211965,14sleep30-21swapper/321:53:453
21629211646,25sleep30-21swapper/322:58:543
14851211527,77sleep31166-21snmpd00:22:293
28642211436,36sleep30-21swapper/321:38:473
9695211313,85sleep19701-21df18:58:331
20641211314,80sleep020661-21/usr/sbin/munin21:13:480
29031211220,63sleep20-21swapper/219:58:392
18097211229,41sleep30-21swapper/300:33:203
18097211229,41sleep30-21swapper/300:33:203
15020211213,83sleep015049-21awk20:58:170
4613211019,74sleep04651-21if_eth020:23:410
1767921100,8sleep09716-1kworker/0:0H22:48:200
28733210919,75sleep028734-21cron19:58:170
22278210815,11sleep30-21swapper/319:36:033
22278210815,11sleep30-21swapper/319:36:033
1626521075,91sleep09716-1kworker/0:0H20:59:270
13926210515,65sleep113626-21/usr/sbin/munin00:18:401
8458210419,62sleep17989-21apt-get22:18:291
17803210426,44sleep317833-21unin-run19:23:183
11171210212,72sleep30-21swapper/320:43:553
3542210122,36sleep30-21swapper/322:03:253
29766210122,56sleep229784-21apt-get21:43:302
27470210012,73sleep127514-21cpu19:53:331
11591210022,39sleep30-21swapper/322:28:333
10973210031,46sleep311096-21cyclictest19:03:193
955829917,68sleep29583-21cut20:38:492
8402987,11sleep30-21swapper/320:11:443
8402987,11sleep30-21swapper/320:11:443
925429714,70sleep09313-21cat20:38:390
2715229713,69sleep227228-21/usr/sbin/munin19:53:182
51852965,14sleep30-21swapper/322:08:343
309202968,44sleep330243-21/usr/sbin/munin23:28:583
1864629625,48sleep218643-21apt-get21:08:292
1685229619,58sleep216922-21switchtime22:43:552
307522959,73sleep030793-21processes23:28:550
29492957,38sleep30-21swapper/323:43:483
1942429510,61sleep014667-21kworker/u8:119:28:200
1868929530,22sleep318789-21cat19:23:513
1969729411,67sleep30-21swapper/319:28:353
3036529314,63sleep230467-21cut23:28:412
3028129314,61sleep030288-21modprobe20:03:170
13522937,50sleep31255-21/usr/sbin/munin20:13:383
13522937,50sleep31255-21/usr/sbin/munin20:13:383
318372929,17sleep30-21swapper/323:33:303
1326329116,19sleep20-21swapper/219:08:272
2297629011,64sleep023025-21/usr/sbin/munin23:03:470
1210729024,18sleep30-21swapper/300:13:333
297228911,63sleep23034-21fschecks_time20:18:382
297228911,63sleep23034-21fschecks_time20:18:382
275392895,14sleep20-21swapper/223:18:432
275392895,14sleep20-21swapper/223:18:432
262072889,44sleep326242-21ntpq19:48:433
1296528816,20sleep10-21swapper/122:33:201
186762878,45sleep318643-21apt-get21:08:503
222822854,11sleep10-21swapper/119:36:051
222822854,11sleep10-21swapper/119:36:051
163502857,43sleep316392-21fschecks_count22:43:403
1040328522,45sleep110005-21/usr/sbin/munin22:23:431
779828420,48sleep17815-21sed20:33:431
2747228417,51sleep127562-21sed21:33:561
2367928318,44sleep223721-21fschecks_count21:23:522
1733728313,53sleep117352-21open_inodes21:03:521
1230328312,46sleep212082-21/usr/sbin/munin00:13:402
1149228320,43sleep211442-21apt-get19:04:032
1104628315,53sleep211083-21/usr/sbin/munin20:43:502
3275128220,38sleep2379-21timerandwakeup20:08:532
3275128220,38sleep2379-21timerandwakeup20:08:532
2955128226,30sleep00-21swapper/021:43:200
2188528118,16sleep20-21swapper/221:18:392
1752328021,36sleep11-21systemd22:48:181
324327915,45sleep03280-21/usr/sbin/munin20:18:440
324327915,45sleep03280-21/usr/sbin/munin20:18:440
162362797,34sleep30-21swapper/319:18:183
1089127927,32sleep21-21systemd19:03:182
493227813,49sleep15070-21sed22:08:201
3259627817,38sleep132734-21grep23:33:531
2386127812,16sleep00-21swapper/023:08:200
2119927821,37sleep221250-21sort19:33:352
2119927821,37sleep221250-21sort19:33:352
353427713,50sleep23541-21basename22:03:242
2375127721,42sleep223796-21unin-run23:08:182
2355227712,50sleep00-21swapper/021:23:290
730927616,45sleep07387-21perl23:58:390
3104927621,38sleep131077-21munin-run21:48:171
2951827619,42sleep129611-21uptime23:23:511
2678927623,29sleep00-21swapper/021:33:390
2380427620,47sleep30-21swapper/319:40:473
2225327612,48sleep122368-21latency_hist23:03:201
1009427618,41sleep010100-21ntp_states18:58:530
646127520,31sleep00-21swapper/022:13:200
3029127519,40sleep030367-21mailstats21:43:520
2287027514,49sleep00-21swapper/021:21:030
1674727519,33sleep20-21swapper/200:28:302
1674727519,33sleep20-21swapper/200:28:302
1543127519,31sleep00-21swapper/022:38:560
3199927422,39sleep032031-21expr21:48:530
2404227414,42sleep024051-21munin-run19:43:180
119672749,16sleep00-21swapper/022:28:460
94002736,29sleep30-21swapper/320:38:423
31672735,59sleep00-21swapper/022:01:530
1447527323,29sleep00-21swapper/022:38:190
905127217,31sleep19063-21apt-get20:38:281
3161127219,31sleep20-21swapper/221:48:432
1812527218,26sleep00-21swapper/000:33:210
1812527218,26sleep00-21swapper/000:33:210
1767827214,35sleep217753-21grep21:03:572
1679127111,46sleep016802-21irqstats19:18:430
448727014,29sleep10-21swapper/120:23:381
2806327012,28sleep10-21swapper/121:38:201
1254827024,26sleep20-21swapper/220:48:472
91126917,20sleep20-21swapper/223:38:302
480726918,36sleep24837-21ntpq20:23:472
3265126925,20sleep00-21swapper/020:08:510
3265126925,20sleep00-21swapper/020:08:510
2911026912,41sleep129115-21diskstats19:58:421
2624526925,25sleep00-21swapper/019:48:440
2434126925,24sleep10-21swapper/121:24:071
182626915,38sleep11891-21grep20:13:571
182626915,38sleep11891-21grep20:13:571
1531126922,24sleep10-21swapper/120:58:351
1349226920,31sleep113595-21fschecks_time19:08:391
111126919,30sleep10-21swapper/121:53:551
111126919,30sleep10-21swapper/121:53:551
920026812,24sleep00-21swapper/000:03:560
760126810,26sleep20-21swapper/220:33:362
2374726821,27sleep10-21swapper/123:08:171
210326831,27sleep30-21swapper/323:42:063
1721126817,30sleep10-21swapper/119:18:561
848926711,43sleep28530-21grep22:18:352
534126716,32sleep00-21swapper/023:53:180
3248526715,23sleep10-21swapper/120:08:441
3248526715,23sleep10-21swapper/120:08:441
2559926731,19sleep00-21swapper/019:48:190
2112626715,27sleep20-21swapper/222:58:412
203402679,15sleep10-21swapper/121:13:411
2002926719,30sleep120170-21ls21:13:211
1565926712,25sleep00-21swapper/000:23:470
1436026720,29sleep114379-21wc20:53:541
1115826712,26sleep20-21swapper/200:09:032
795826620,28sleep2288-21dbus-daemon22:18:182
795826620,28sleep2288-21dbus-daemon22:18:182
71132667,11sleep00-21swapper/020:32:050
487526617,30sleep00-21swapper/022:08:190
3237526613,28sleep20-21swapper/223:33:502
2517526611,39sleep125199-21df_inode21:28:351
1319426619,27sleep00-21swapper/019:08:250
630226515,30sleep26397-21/usr/sbin/munin23:53:532
630226515,30sleep26397-21/usr/sbin/munin23:53:532
1914726527,21sleep10-21swapper/122:53:191
379526415,20sleep20-21swapper/223:48:192
1265926419,23sleep10-21swapper/120:48:541
1153926410,43sleep00-21swapper/000:10:100
779026321,25sleep10-21swapper/123:58:521
2489726312,35sleep124970-21/usr/sbin/munin19:43:521
85662626,44sleep00-21swapper/020:35:320
305482629,24sleep10-21swapper/120:03:311
2301826218,28sleep223105-21latency19:38:442
1335026221,28sleep113412-21cat22:33:401
78682610,8sleep20-21swapper/223:58:522
63482618,12sleep10-21swapper/122:12:521
2070226111,35sleep120839-21cat22:58:191
96902605,13sleep00-21swapper/022:22:360
310026015,20sleep10-21swapper/120:18:411
310026015,20sleep10-21swapper/120:18:411
2554826016,23sleep217006-1kworker/2:0H19:48:172
1816726017,19sleep00-21swapper/021:08:190
1157226017,23sleep10-21swapper/119:04:101
310812599,29sleep10-21swapper/123:29:001
2807925912,28sleep20-21swapper/219:53:552
2178625913,26sleep021846-21/usr/sbin/munin19:33:530
2178625913,26sleep021846-21/usr/sbin/munin19:33:530
1581225916,23sleep20-21swapper/220:58:522
1578125916,26sleep115822-21/usr/sbin/munin00:23:531
1538525917,19sleep00-21swapper/019:13:460
1237725916,26sleep01166-21snmpd19:04:300
118662597,40sleep00-21swapper/020:48:000
1000425919,23sleep00-21swapper/022:23:250
909725814,31sleep18704-21/usr/sbin/munin00:03:491
477925813,16sleep10-21swapper/123:48:551
2716825815,23sleep10-21swapper/123:18:301
2716825815,23sleep10-21swapper/123:18:301
1999525813,26sleep20-21swapper/222:53:522
741825717,18sleep10-21swapper/122:13:571
741825717,18sleep10-21swapper/122:13:571
3131225716,25sleep131331-21apt-get21:48:301
3078325715,19sleep00-21swapper/020:03:420
2882225714,26sleep20-21swapper/221:38:542
210125716,19sleep10-21swapper/121:58:351
128325714,26sleep01276-21apt-get20:13:350
128325714,26sleep01276-21apt-get20:13:350
923625620,19sleep20-21swapper/200:03:582
3183925614,19sleep00-21swapper/023:33:310
2880825620,20sleep20-21swapper/223:23:362
2405225617,17sleep10-21swapper/119:43:191
1968925613,18sleep20-21swapper/219:28:342
1280625614,18sleep00-21swapper/000:13:540
122325616,18sleep20-21swapper/221:53:562
122325616,18sleep20-21swapper/221:53:562
695925512,28sleep17098-21cut23:58:201
695925512,28sleep17098-21cut23:58:201
2586325513,25sleep025892-21irqstats23:13:410
256425512,26sleep22623-21sensors_temp21:58:482
2403825513,25sleep224031-21apt-get23:08:342
212102559,16sleep00-21swapper/022:58:430
1316625515,24sleep213159-21apt-get22:33:332
1111125517,18sleep10-21swapper/120:43:531
804025413,15sleep00-21swapper/022:18:200
804025413,15sleep00-21swapper/022:18:200
2714725414,23sleep027165-21apt-get23:18:290
2714725414,23sleep027165-21apt-get23:18:290
2571825412,24sleep225803-21perl23:13:392
1981625412,16sleep00-21swapper/022:53:460
1628825416,21sleep116403-21sed19:18:201
727225313,23sleep17286-21munin-run20:33:191
3188825316,16sleep20-21swapper/220:08:202
2024125311,25sleep220237-21apt-get21:13:342
165525315,22sleep21255-21/usr/sbin/munin20:13:492
165525315,22sleep21255-21/usr/sbin/munin20:13:492
1582325321,17sleep20-21swapper/200:23:542
1374925313,24sleep213746-21apt-get20:53:312
1007525312,24sleep210099-21df22:23:312
1383625215,17sleep20-21swapper/200:18:362
60702514,13sleep20-21swapper/222:09:262
595325111,24sleep05992-21/usr/sbin/munin23:53:420
595325111,24sleep05992-21/usr/sbin/munin23:53:420
260825113,20sleep12668-21perl23:43:381
2561525114,22sleep025664-21awk21:28:550
2304625119,18sleep10-21swapper/121:23:181
182672519,17sleep20-21swapper/222:48:472
1051625112,23sleep110580-21/usr/sbin/munin00:08:341
92225012,22sleep0928-21ntp_states21:53:500
92225012,22sleep0928-21ntp_states21:53:500
253225012,21sleep02557-21df23:43:330
2512225014,21sleep225118-21apt-get21:28:322
149502509,19sleep20-21swapper/219:13:282
1159025013,19sleep20-21swapper/222:28:322
728724914,18sleep20-21swapper/220:33:202
52152497,32sleep20-21swapper/223:52:192
2669624911,22sleep226690-21apt-get21:33:332
1814424911,22sleep218153-21diskstats19:23:342
167252499,20sleep116744-21apt-get00:28:291
167252499,20sleep116744-21apt-get00:28:291
138824910,14sleep00-21swapper/023:38:480
11212499,16sleep10-21swapper/123:38:411
2602524810,15sleep10-21swapper/123:13:461
1470424812,21sleep214706-21apt-get22:38:292
2583324611,18sleep10-21swapper/119:48:281
1217724610,18sleep00-21swapper/020:48:320
1363424511,20sleep013646-21apt-get00:18:260
2043224411,17sleep00-21swapper/019:28:530
2879124211,16sleep00-21swapper/023:23:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional