You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 14:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Oct 20, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1347425167,497sleep013521-21ntpq22:40:590
15403236816,330sleep015504-21latency_hist00:30:360
32099234012,200sleep032165-21fschecks_time23:40:540
106232324127,178sleep010754-21grep22:31:070
3582227715,244sleep00-21swapper/022:10:560
20783227323,228sleep220927-21grep21:21:082
9376226814,232sleep29407-21fschecks_count19:06:082
21264226396,149sleep021305-21fschecks_time23:06:100
4727225124,215sleep04225-21/usr/sbin/munin23:56:040
11998225123,213sleep112062-21find22:36:041
1453722405,10sleep00-21swapper/021:01:160
2517622217,205sleep2195-21jbd2/mmcblk2p2-19:55:212
10533221538,161sleep010538-21modprobe19:10:340
15328221443,149sleep015470-21unixbench_multi19:21:280
2169221312,116sleep01-21systemd20:25:350
2169221312,116sleep01-21systemd20:25:350
4010220840,33sleep03984-40mandb23:55:360
4010220840,33sleep03984-40mandb23:55:360
7105220728,139sleep07195-21mailstats22:21:050
7105220728,139sleep07195-21mailstats22:21:050
20636220614,171sleep020078-21/usr/sbin/munin21:21:060
2764321959,171sleep027709-21date21:45:370
19395219515,159sleep00-21swapper/023:00:530
17516219412,159sleep00-21swapper/022:55:370
30852219118,153sleep030995-21grep20:11:100
16523219120,148sleep030986-1kworker/0:2H21:10:360
28622218597,20sleep00-21swapper/020:05:530
8684218314,154sleep08773-21grep20:45:370
93421809,158sleep01018-21cat22:01:070
93421809,158sleep01018-21cat22:01:070
12497217816,142sleep012554-21awk19:15:350
23885217219,98sleep00-21swapper/021:31:070
2418217111,145sleep02444-21latency_hist23:50:350
24902217017,132sleep124938-21fschecks_time21:35:571
22258216910,136sleep00-21swapper/019:45:380
22904216710,76sleep00-21swapper/023:10:590
6375215349,90sleep06525-21latency_hist22:20:370
10427215323,113sleep010465-21sendmail00:15:340
10427215323,113sleep010465-21sendmail00:15:340
17758215188,48sleep016946-21/usr/sbin/munin21:11:210
1407821519,129sleep014121-21gpgv19:20:350
20748214616,107sleep320959-21apt-config23:05:373
18637214614,102sleep00-21swapper/021:15:530
13369214232,14sleep00-21swapper/021:00:350
13369214232,14sleep00-21swapper/021:00:350
31725214086,31sleep00-21swapper/023:40:340
1746621379,93sleep017512-21wc19:30:360
1746621379,93sleep017512-21wc19:30:360
191121327,13sleep00-21swapper/022:05:510
7334213117,95sleep37352-21/usr/sbin/munin00:05:483
11103213123,87sleep111120-21fschecks_count00:16:201
628021306,112sleep00-21swapper/000:01:050
14383213016,98sleep114523-21sed22:45:371
7345212818,33sleep10-21swapper/120:40:461
18372212611,69sleep00-21swapper/019:31:130
2682221238,82sleep026829-21munin-run20:00:360
4974212230,66sleep35116-21head23:56:073
2625221229,19sleep00-21swapper/023:21:040
15238212131,68sleep315320-21cat19:21:273
32305211621,75sleep332391-21/usr/sbin/munin20:16:053
3219211414,80sleep13361-21latency_hist22:10:371
4739210911,52sleep00-21swapper/020:31:280
9297210713,81sleep19334-21sed20:45:591
19073210739,52sleep019202-21latency_hist23:00:370
32358210632,62sleep00-21swapper/021:58:280
1907210520,42sleep20-21swapper/222:05:492
16045210541,52sleep016092-21ntpq00:31:010
24541210420,63sleep124589-21latency_hist21:35:361
2190210412,25sleep32274-21memory22:05:583
2164210419,67sleep32166-21cron20:25:343
2164210419,67sleep32166-21cron20:25:343
16932210416,69sleep117060-21/usr/sbin/munin19:26:141
16932210416,69sleep117060-21/usr/sbin/munin19:26:141
5826210210,16sleep00-21swapper/020:35:520
30638210214,72sleep130697-21processes20:11:071
2851210217,47sleep00-21swapper/020:25:590
16936210218,66sleep116450-21apt-get21:10:461
9389210113,74sleep19391-21sshd22:27:161
27547210112,72sleep00-21swapper/023:25:560
841421008,16sleep00-21swapper/022:26:140
29516210014,71sleep129553-21fschecks_count21:50:541
29516210014,71sleep129553-21fschecks_count21:50:541
1149521008,57sleep0362-21snmpd20:52:280
16782990,8sleep20-21swapper/222:05:362
16782990,8sleep20-21swapper/222:05:362
82142987,15sleep00-21swapper/000:06:090
5672986,11sleep10-21swapper/123:42:161
425029811,28sleep20-21swapper/223:55:502
252362979,49sleep00-21swapper/021:36:040
2242229714,67sleep122551-21awk21:26:081
2241029716,65sleep20-21swapper/223:10:362
2173829711,46sleep00-21swapper/021:25:540
344829611,68sleep23567-21users23:51:102
344829611,68sleep23567-21users23:51:102
2892429615,29sleep00-21swapper/021:47:510
2519029645,40sleep0362-21snmpd19:55:310
875829536,34sleep28857-21gpgv19:05:362
43412958,24sleep30-21swapper/320:30:553
3092629513,68sleep031081-21gpgv21:55:380
3092629513,68sleep031081-21gpgv21:55:380
2704629514,64sleep327095-21/usr/sbin/munin23:25:353
245529525,40sleep22448-21apt-get20:25:492
2285029442,40sleep022884-21ps19:46:070
1417429418,53sleep214191-21fschecks_count00:25:542
1098129312,38sleep00-21swapper/000:15:500
1997829218,54sleep020108-21grep19:36:180
589229125,35sleep20-21swapper/200:00:552
1470529116,62sleep114725-21fschecks_count22:45:531
1470529116,62sleep114725-21fschecks_count22:45:531
892229012,18sleep20-21swapper/220:45:482
2513929026,28sleep00-21swapper/023:16:080
784628818,56sleep27907-21sendmail_mailqu20:41:032
902328719,49sleep00-21swapper/019:05:380
322132876,42sleep00-21swapper/020:16:020
2822428711,19sleep00-21swapper/020:04:150
1053628716,50sleep310618-21threads22:31:063
101928725,32sleep10-21swapper/122:01:081
101928725,32sleep10-21swapper/122:01:081
293412868,33sleep00-21swapper/023:31:010
2855628510,14sleep10-21swapper/121:46:061
1304428421,34sleep10-21swapper/100:21:041
1044828414,56sleep210535-21mailstats22:31:042
667828228,29sleep10-21swapper/122:20:541
667828228,29sleep10-21swapper/122:20:541
1469828233,33sleep00-21swapper/022:45:520
1469828233,33sleep00-21swapper/022:45:520
2899528124,27sleep10-21swapper/120:06:021
1717328110,57sleep217175-21sshd22:52:162
160942818,16sleep10-21swapper/100:31:021
1278228122,41sleep312805-21munin-run22:40:343
3152028021,43sleep331492-21latency_hist20:15:373
1775428013,53sleep217803-21cpuspeed22:55:512
3175427917,31sleep30-21swapper/321:56:173
2256427923,35sleep321630-21/usr/sbin/munin21:26:103
134582795,41sleep00-21swapper/000:21:160
92422787,32sleep00-21swapper/020:45:580
472327814,17sleep20-21swapper/220:31:262
2485927714,47sleep324866-21diskstats21:35:553
1582527724,30sleep10-21swapper/121:06:091
789227622,32sleep27936-21munin-run22:25:352
789227622,32sleep27936-21munin-run22:25:352
3012766,12sleep10-21swapper/120:16:481
2917327618,28sleep20-21swapper/220:06:072
2070627612,51sleep120782-21grep21:21:071
1329327612,50sleep213351-21/usr/sbin/munin19:16:052
1160927616,40sleep00-21swapper/022:35:550
1572927522,32sleep20-21swapper/200:30:542
2982927426,29sleep30-21swapper/321:51:013
2982927426,29sleep30-21swapper/321:51:013
284112745,12sleep10-21swapper/123:28:161
1981227418,37sleep119818-21modprobe21:20:341
1066427421,37sleep310865-21latency_hist00:15:373
790927321,27sleep30-21swapper/320:41:043
406227324,26sleep20-21swapper/222:11:062
2683027324,28sleep10-21swapper/120:00:371
1472627325,29sleep20-21swapper/222:45:542
1472627325,29sleep20-21swapper/222:45:542
3172827218,30sleep20-21swapper/221:56:162
2063027216,28sleep20-21swapper/223:05:362
1116327218,38sleep211294-21grep20:51:092
776727116,31sleep10-21swapper/100:06:021
591227123,25sleep10-21swapper/120:35:571
1743427124,32sleep1225-21systemd-journal22:55:341
96292708,50sleep30-21swapper/322:30:173
317752709,43sleep20-21swapper/223:40:362
212482706,12sleep30-21swapper/319:41:053
1389627020,22sleep20-21swapper/200:25:372
963026917,38sleep19667-21ps00:11:111
963026917,38sleep19667-21ps00:11:111
2494026914,23sleep20-21swapper/221:35:582
244526922,23sleep10-21swapper/123:50:361
2131526919,31sleep221381-21wc19:41:082
967026823,23sleep20-21swapper/200:11:122
967026823,23sleep20-21swapper/200:11:122
3133726825,20sleep10-21swapper/121:55:551
2979826814,24sleep20-21swapper/221:51:002
2979826814,24sleep20-21swapper/221:51:002
1429726818,34sleep314447-21sed21:01:073
125112687,45sleep30-21swapper/300:20:463
891226719,22sleep20-21swapper/222:26:362
812826721,23sleep10-21swapper/119:01:071
3254526713,24sleep20-21swapper/223:41:042
3246526721,30sleep332471-21/usr/sbin/munin23:41:013
282126715,32sleep32849-21/usr/sbin/munin20:25:583
221842675,50sleep10-21swapper/123:08:211
1466526724,19sleep10-21swapper/100:26:051
11152677,12sleep20-21swapper/220:20:572
11152677,12sleep20-21swapper/220:20:572
3162226612,39sleep331623-21sshd23:39:163
3014826616,30sleep130172-21sh23:35:341
2403826618,20sleep20-21swapper/223:15:372
161772667,15sleep30-21swapper/322:50:513
2272126525,27sleep322728-21sh23:10:543
1780426517,25sleep30-21swapper/322:55:523
1338026518,28sleep10-21swapper/121:00:361
1338026518,28sleep10-21swapper/121:00:361
1263626525,18sleep30-21swapper/320:56:063
1263626525,18sleep30-21swapper/320:56:063
120426518,29sleep20-21swapper/223:45:572
1162926517,29sleep311674-21/usr/sbin/munin19:11:023
2484726414,24sleep10-21swapper/123:16:041
1755926414,23sleep20-21swapper/221:11:192
1562226424,22sleep30-21swapper/322:47:163
1562226424,22sleep30-21swapper/322:47:163
867926320,30sleep28783-21latency_hist00:10:352
711826319,29sleep37235-21latency_hist00:05:363
256292638,11sleep30-21swapper/319:55:553
78262625,12sleep30-21swapper/319:01:013
354426215,31sleep33580-21/usr/sbin/munin22:10:553
2863826220,23sleep30-21swapper/323:30:363
2471326216,31sleep324832-21/usr/sbin/munin19:51:063
2204826222,24sleep20-21swapper/221:26:012
205552627,12sleep10-21swapper/123:05:221
1937926215,20sleep20-21swapper/219:36:032
1788326215,21sleep10-21swapper/119:31:021
1409126217,27sleep314143-21cpuspeed00:25:513
924126113,24sleep30-21swapper/319:05:453
2459026111,32sleep224483-21latency_hist21:35:372
1947326118,28sleep3225-21systemd-journal21:16:173
1821226116,24sleep20-21swapper/219:31:112
1673526116,23sleep30-21swapper/319:26:123
1673526116,23sleep30-21swapper/319:26:123
1595926112,19sleep20-21swapper/221:06:102
1412226117,23sleep114142-21munin-run19:20:361
1390826116,24sleep30-21swapper/322:41:073
531626012,35sleep25408-21load22:15:592
302926016,18sleep10-21swapper/120:26:041
283422609,36sleep228382-21ps21:46:032
232942608,13sleep10-21swapper/123:11:081
1944626016,23sleep20-21swapper/221:16:072
182932606,12sleep30-21swapper/319:31:123
1763226020,22sleep30-21swapper/321:11:203
1647826018,28sleep316489-21munin-run21:10:353
1586426015,30sleep215869-21modprobe19:25:342
1255626012,12sleep20-21swapper/220:56:052
1255626012,12sleep20-21swapper/220:56:052
117002609,32sleep211797-21timerwakeupswit00:16:372
1129726014,26sleep311383-21users20:51:103
3224225915,21sleep20-21swapper/220:16:042
2683625918,18sleep20-21swapper/221:41:022
1644525914,17sleep30-21swapper/300:31:083
659025816,25sleep36586-21apt-get22:20:483
659025816,25sleep36586-21apt-get22:20:483
315592585,14sleep10-21swapper/123:38:291
275125811,28sleep10-21swapper/123:50:561
275125811,28sleep10-21swapper/123:50:561
1169425814,24sleep20-21swapper/219:11:052
943525718,23sleep30-21swapper/300:11:053
919725719,25sleep39236-21latency20:45:573
467925711,31sleep14703-21ntp_states23:56:011
2827925711,13sleep30-21swapper/321:46:003
1634625721,22sleep20-21swapper/219:26:022
1634625721,22sleep20-21swapper/219:26:022
1074525715,21sleep10-21swapper/120:50:591
2759425622,19sleep30-21swapper/321:45:353
2365525614,19sleep30-21swapper/321:31:023
2083525614,25sleep120829-21apt-get19:40:511
187325611,16sleep30-21swapper/323:46:103
1510225617,21sleep10-21swapper/119:21:251
984725516,20sleep10-21swapper/119:06:191
580725512,26sleep15802-21apt-get00:00:491
439425516,20sleep10-21swapper/120:31:171
2578425514,25sleep125777-21apt-get23:20:491
2433025516,19sleep20-21swapper/223:15:532
2423125514,25sleep324228-21apt-get23:15:463
2418125514,25sleep124204-21irqstats19:50:561
1929425514,24sleep119279-21apt-get19:35:581
1323225512,18sleep30-21swapper/319:16:023
674125412,29sleep36743-21sshd20:36:163
2559425419,16sleep20-21swapper/219:55:542
1839525411,26sleep118517-21sed22:56:061
100902549,31sleep110153-21fw_packets22:30:551
513125312,24sleep35226-21perl22:15:553
36682539,20sleep10-21swapper/122:10:571
2894625313,24sleep228962-21df_inode23:30:512
287425317,17sleep30-21swapper/323:50:583
287425317,17sleep30-21swapper/323:50:583
2669625312,18sleep10-21swapper/121:40:571
2578525311,23sleep225852-21df23:20:502
2278525316,21sleep10-21swapper/119:46:041
1152125313,21sleep111595-21memory19:11:001
560025212,23sleep15677-21sed22:16:061
3057225215,18sleep20-21swapper/220:11:032
288912525,12sleep30-21swapper/320:06:003
2706325213,22sleep227058-21apt-get20:00:462
1375225214,16sleep20-21swapper/222:41:062
1361725211,24sleep213613-21apt-get21:00:492
628725112,24sleep26322-21grep20:36:072
2017525111,25sleep320181-21df_inode21:20:533
19412516,34sleep10-21swapper/120:22:451
19412516,34sleep10-21swapper/120:22:451
145962515,36sleep10-21swapper/121:02:021
1118325116,21sleep311247-21idleruntime00:16:263
285982508,15sleep20-21swapper/223:30:352
2264125012,14sleep20-21swapper/219:46:012
1929025010,18sleep30-21swapper/319:35:563
42824913,14sleep30-21swapper/322:00:533
42824913,14sleep30-21swapper/322:00:533
2304624918,17sleep30-21swapper/319:46:103
289632487,17sleep30-21swapper/323:30:523
2106724810,22sleep320593-21apt-get23:05:453
2724224616,16sleep10-21swapper/120:00:531
1193824513,17sleep20-21swapper/222:36:012
276782429,14sleep30-21swapper/320:01:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional