You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 17:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Nov 21, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
202522405836,4008sleep219743-21apt-get10:53:532
202522405836,4008sleep219743-21apt-get10:53:532
80302172317,1693sleep124409-21kworker/1:208:33:251
1607128577,836sleep016118-21unin-run12:23:410
5227261412,187sleep05350-21latency_hist11:48:430
3231258910,561sleep33301-21/usr/sbin/munin09:59:383
26620235851,154sleep026688-21fschecks_time07:49:010
8263229619,234sleep00-21swapper/010:14:130
2473822788,235sleep024674-21latency_hist11:08:450
622224950,48sleep00-21swapper/011:34:010
622224950,48sleep00-21swapper/011:34:010
20798224961,168sleep020940-21head07:29:140
1938224328,138sleep02087-21latency_hist08:13:450
2878223617,202sleep33016-21sed08:14:203
705822347,210sleep07081-21kernelversion08:29:040
23096223435,178sleep023109-21modprobe11:03:400
23834223316,138sleep023960-21timerwakeupswit07:39:120
23834223316,138sleep023960-21timerwakeupswit07:39:120
15499223046,162sleep00-21swapper/007:13:590
524322195,11sleep00-21swapper/010:05:490
524322195,11sleep00-21swapper/010:05:490
8217221732,18sleep00-21swapper/011:58:000
27493221610,55sleep027646-21grep11:14:390
27493221610,55sleep027646-21grep11:14:390
16852221429,96sleep017006-21unixbench_multi08:59:170
31120221033,159sleep031141-21idleruntime-cro11:28:400
31120221033,159sleep031141-21idleruntime-cro11:28:400
26256220939,89sleep00-21swapper/009:29:130
26256220939,89sleep00-21swapper/009:29:130
933220740,153sleep0836-21/usr/sbin/munin09:53:570
15981220714,172sleep00-21swapper/010:39:130
15981220714,172sleep00-21swapper/010:39:130
14014220637,143sleep00-21swapper/007:09:030
19243219831,129sleep019280-21latency_hist12:33:420
330321979,171sleep00-21swapper/009:59:390
31456219110,156sleep00-21swapper/008:03:550
23722217616,34sleep30-21swapper/309:23:433
2709021747,128sleep00-21swapper/009:33:580
2667217015,32sleep22702-21cpu09:59:002
1435021657,126sleep0362-21snmpd12:16:300
1435021657,126sleep0362-21snmpd12:16:300
6552216428,22sleep00-21swapper/008:28:430
2994721639,40sleep029991-21latency_hist09:43:430
3474215975,14sleep00-21swapper/011:43:420
844821556,133sleep00-21swapper/008:34:000
22411215418,18sleep00-21swapper/009:18:550
1230021548,87sleep012342-21ntpq07:04:100
31217215111,126sleep031302-21latency_hist08:03:430
28319214910,77sleep00-21swapper/007:53:540
26272214811,124sleep026331-21cat11:13:420
15652214825,90sleep0362-21snmpd08:55:530
15652214825,90sleep0362-21snmpd08:55:530
18280214611,114sleep018408-21latency_hist07:23:450
445721446,99sleep00-21swapper/008:19:140
445721446,99sleep00-21swapper/008:19:140
29573214216,111sleep21-21systemd11:23:412
13584214113,87sleep013667-21latency_hist10:33:430
13584214113,87sleep013667-21latency_hist10:33:430
10301213112,84sleep010364-21if_eth012:04:020
7480213017,89sleep27547-21sendmail_mailqu11:54:112
21603213010,105sleep021720-21latency_hist10:58:430
21603213010,105sleep021720-21latency_hist10:58:430
24684212815,94sleep00-21swapper/009:24:130
24684212815,94sleep00-21swapper/009:24:130
211312127101,13sleep00-21swapper/009:14:030
2505221267,101sleep00-21swapper/007:44:000
1982821269,73sleep00-21swapper/010:53:420
31499212519,89sleep00-21swapper/009:48:440
28398212428,74sleep128415-21fschecks_count11:19:001
13804212128,64sleep313823-21apt-get07:08:533
1436621208,98sleep014442-21sed08:53:430
17720211934,45sleep30-21swapper/309:03:523
1199421196,12sleep00-21swapper/010:28:420
5694211733,73sleep05729-21latency11:49:040
418321178,18sleep10-21swapper/110:03:521
418321178,18sleep10-21swapper/110:03:521
7347211638,22sleep00-21swapper/010:13:440
12939211611,18sleep20-21swapper/212:13:412
964721149,44sleep09723-21systemctl10:19:130
27900211417,78sleep228041-21dpkg07:53:432
856021139,37sleep00-21swapper/011:58:570
17194211225,29sleep00-21swapper/010:44:050
17194211225,29sleep00-21swapper/010:44:050
19690211027,73sleep0362-21snmpd07:27:360
1523421090,6sleep10-21swapper/107:13:431
20835210710,80sleep120856-21apt-get09:13:521
1892921075,17sleep00-21swapper/009:08:010
13913210628,61sleep313946-21fschecks_count10:34:033
13913210628,61sleep313946-21fschecks_count10:34:033
6424210517,70sleep10-21swapper/110:09:531
27833210518,62sleep127977-21unixbench_singl09:34:151
12340210519,63sleep212430-21fschecks_time10:29:012
12340210519,63sleep212430-21fschecks_time10:29:012
10005210529,63sleep010007-21diskstats08:39:010
1275721047,41sleep012769-21munin-run08:48:420
4686210315,71sleep24752-21sendmail_mailqu10:04:092
4686210315,71sleep24752-21sendmail_mailqu10:04:092
4330210320,63sleep30-21swapper/308:19:133
4330210320,63sleep30-21swapper/308:19:133
15079210224,59sleep115128-21unin-run10:38:411
15079210224,59sleep115128-21unin-run10:38:411
5613210011,68sleep25646-21/usr/sbin/munin08:24:092
4358210017,69sleep14367-21ntpq11:44:321
31875210022,14sleep00-21swapper/011:29:080
31875210022,14sleep00-21swapper/011:29:080
2839129923,14sleep00-21swapper/011:18:590
226929615,66sleep12331-21cat11:39:031
795529514,58sleep17560-21/usr/sbin/munin10:14:051
3182529517,63sleep131867-21ntp_kernel_err08:04:041
238629512,64sleep12420-21/usr/sbin/munin08:14:101
189829534,47sleep31936-21latency_hist08:13:443
2656029421,24sleep20-21swapper/209:31:012
2656029421,24sleep20-21swapper/209:31:012
1208029421,34sleep00-21swapper/012:09:070
697629327,32sleep10-21swapper/111:53:531
1643929316,28sleep30-21swapper/308:59:073
2785229220,54sleep127889-21munin-run07:53:421
1977429215,61sleep319825-21mktemp10:53:413
2502429115,61sleep325029-21entropy11:09:003
3149129018,51sleep331385-21/usr/sbin/munin11:28:593
3149129018,51sleep331385-21/usr/sbin/munin11:28:593
174152905,56sleep00-21swapper/012:26:030
1198429035,33sleep20-21swapper/207:04:042
138292899,38sleep00-21swapper/010:34:000
138292899,38sleep00-21swapper/010:34:000
3230428811,62sleep232370-21switchtime09:49:112
231142880,7sleep10-21swapper/111:03:401
1840528820,40sleep218536-21grep09:04:112
2062328715,55sleep00-21swapper/009:13:420
229462868,41sleep022956-21munin-run07:38:430
1917928621,32sleep20-21swapper/207:24:112
1594628619,51sleep316093-21latency_hist08:58:433
1594628619,51sleep316093-21latency_hist08:58:433
1319428519,49sleep113204-21munin-setwarn07:08:431
2631828410,59sleep326463-21latency_hist07:48:443
252028418,30sleep011660-21cyclictest08:14:130
2298428427,26sleep20-21swapper/207:38:452
1823028423,27sleep10-21swapper/110:48:431
1823028423,27sleep10-21swapper/110:48:431
3117128321,16sleep20-21swapper/211:28:422
3117128321,16sleep20-21swapper/211:28:422
2528928318,44sleep325252-21latency_hist09:28:443
2528928318,44sleep325252-21latency_hist09:28:443
2420128311,61sleep11-21systemd07:40:301
2420128311,61sleep11-21systemd07:40:301
981728226,36sleep30-21swapper/312:03:413
317102827,35sleep00-21swapper/009:48:520
3038028214,51sleep230438-21/usr/sbin/munin07:59:092
2966828228,37sleep129796-21latency_hist07:58:431
2144128217,42sleep221590-21latency_hist07:33:442
2394028017,48sleep124007-21awk11:04:101
1963728017,39sleep319249-21/usr/sbin/munin09:09:053
1897728019,28sleep30-21swapper/307:24:083
1273728028,37sleep3225-21systemd-journal08:48:413
2999327925,37sleep130075-21wc09:43:441
832127817,18sleep30-21swapper/311:58:443
2204027821,44sleep322091-21if_eth010:59:033
2036127826,28sleep320369-21df_inode10:54:023
2036127826,28sleep320369-21df_inode10:54:023
1761727821,40sleep3225-21systemd-journal12:28:413
1192227810,14sleep20-21swapper/208:44:082
1047727818,44sleep310503-21netstat12:04:053
100827827,29sleep10-21swapper/111:34:101
100827827,29sleep10-21swapper/111:34:101
2054427726,35sleep220576-21awk09:13:402
1181627717,16sleep30-21swapper/312:09:023
9432760,5sleep10-21swapper/109:53:571
2684927525,26sleep10-21swapper/109:33:431
2684927525,26sleep10-21swapper/109:33:431
1980927514,44sleep119880-21/usr/sbin/munin09:09:111
1105727524,26sleep00-21swapper/006:59:140
38732748,47sleep20-21swapper/211:43:572
3203027417,29sleep30-21swapper/308:04:113
2550827413,13sleep20-21swapper/207:44:312
1456127411,49sleep214589-21cat12:18:422
1456127411,49sleep214589-21cat12:18:422
338627322,23sleep30-21swapper/311:43:413
2633227316,37sleep126496-21latency_hist11:13:431
1843927312,47sleep218456-21/usr/sbin/munin10:48:522
1428727325,31sleep214305-21sh08:53:412
968227119,26sleep30-21swapper/308:38:443
63427124,26sleep10-21swapper/108:09:011
500227122,22sleep10-21swapper/108:23:431
500227122,22sleep10-21swapper/108:23:431
3028227117,23sleep10-21swapper/109:44:011
233727113,30sleep20-21swapper/209:58:442
1429827117,37sleep214327-21ps10:34:142
1429827117,37sleep214327-21ps10:34:142
589427017,36sleep25958-21ps11:49:102
2880527021,20sleep30-21swapper/311:19:103
2286627026,26sleep30-21swapper/307:38:143
1743027022,18sleep30-21swapper/307:19:073
1480827017,26sleep30-21swapper/312:18:553
1372427015,38sleep313759-21ps12:14:113
1372427015,38sleep313759-21ps12:14:113
987126917,25sleep20-21swapper/212:03:442
63326920,30sleep30-21swapper/309:53:443
610826922,28sleep10-21swapper/111:49:131
283162697,41sleep20-21swapper/207:53:522
2361626910,13sleep20-21swapper/207:39:052
2361626910,13sleep20-21swapper/207:39:052
864826816,21sleep10-21swapper/111:59:021
659626813,19sleep30-21swapper/310:10:003
2547126821,28sleep10-21swapper/107:44:301
2400826825,21sleep20-21swapper/211:04:112
2287426723,27sleep122909-21grep09:19:091
1764526715,35sleep217797-21unixbench_multi07:19:102
1589226723,25sleep10-21swapper/107:14:081
992126629,19sleep10-21swapper/108:38:581
676226523,25sleep36891-21ls11:53:433
504826516,33sleep25190-21cut08:23:442
3073726518,27sleep30-21swapper/309:44:113
2834526513,33sleep228367-21cron09:38:402
1537426515,30sleep115424-21cpuspeed10:38:571
1537426515,30sleep115424-21cpuspeed10:38:571
1507826520,22sleep20-21swapper/212:19:022
1121926513,23sleep20-21swapper/206:59:162
592526415,34sleep16071-21unixbench_multi08:24:161
3013226420,25sleep30-21swapper/311:24:033
3013226420,25sleep30-21swapper/311:24:033
2649726422,18sleep20-21swapper/211:13:442
2649726422,18sleep20-21swapper/211:13:442
2410526417,30sleep323167-21/usr/sbin/munin07:39:153
2410526417,30sleep323167-21/usr/sbin/munin07:39:153
1198526417,29sleep10-21swapper/108:44:111
1096326420,24sleep10-21swapper/110:24:071
65322637,14sleep20-21swapper/210:09:592
521526322,29sleep35234-21apt-get08:23:533
3222126313,26sleep10-21swapper/109:49:101
2295726313,23sleep10-21swapper/107:38:441
2140126314,36sleep121440-21latency_hist07:33:431
2102526315,31sleep221080-21/usr/sbin/munin09:14:012
1909026319,21sleep10-21swapper/107:24:101
1555026314,28sleep10-21swapper/108:54:411
1555026314,28sleep10-21swapper/108:54:411
1352626312,27sleep10-21swapper/112:14:051
1352626312,27sleep10-21swapper/112:14:051
1123226316,32sleep211335-21latency_hist08:43:442
719026214,22sleep30-21swapper/308:29:073
215826217,32sleep22167-21diskstats11:39:002
1428626213,33sleep214292-21ntp_states07:09:092
1078026217,23sleep20-21swapper/210:24:042
1054426213,31sleep110563-21ntp_states12:04:071
80726118,22sleep30-21swapper/311:34:043
80726118,22sleep30-21swapper/311:34:043
239242616,12sleep20-21swapper/209:23:462
239242616,12sleep20-21swapper/209:23:462
212526114,24sleep30-21swapper/309:58:413
1787926118,24sleep117898-21apt12:28:511
1766926126,20sleep10-21swapper/112:28:431
1722826118,25sleep10-21swapper/107:19:011
1671926117,23sleep30-21swapper/307:18:433
151902618,12sleep30-21swapper/310:38:433
151902618,12sleep30-21swapper/310:38:433
1067726117,24sleep10-21swapper/106:59:071
1057726122,24sleep310620-21cat06:59:043
941126017,25sleep10-21swapper/110:19:061
32126018,20sleep20-21swapper/211:33:432
1224126018,27sleep312238-21apt-get10:28:543
1224126018,27sleep312238-21apt-get10:28:543
1086626015,30sleep30-21swapper/310:24:053
106526016,20sleep20-21swapper/211:34:112
106526016,20sleep20-21swapper/211:34:112
951025918,26sleep39544-21ntp_kernel_err10:19:083
3067125917,21sleep20-21swapper/209:44:102
2990425910,22sleep10-21swapper/107:58:561
2336225914,27sleep323381-21apt-get11:03:513
156625910,36sleep2362-21snmpd08:10:322
2384425819,23sleep123923-21date09:23:451
2384425819,23sleep123923-21date09:23:451
1729325819,22sleep20-21swapper/210:44:072
1729325819,22sleep20-21swapper/210:44:072
153782586,11sleep20-21swapper/207:13:452
1312325812,28sleep313213-21/usr/sbin/munin08:49:023
809025715,21sleep20-21swapper/210:14:112
221725719,16sleep20-21swapper/208:14:032
178072575,32sleep10-21swapper/109:03:581
1742925717,20sleep30-21swapper/310:44:123
1742925717,20sleep30-21swapper/310:44:123
103632575,41sleep20-21swapper/210:23:192
425925623,19sleep20-21swapper/208:19:122
420625616,25sleep34202-21apt-get10:03:543
420625616,25sleep34202-21apt-get10:03:543
2716625614,27sleep327175-21diskstats09:34:013
2196325612,19sleep20-21swapper/210:59:022
2093225618,22sleep320939-21df_inode09:13:583
2078225615,19sleep10-21swapper/110:54:361
2078225615,19sleep10-21swapper/110:54:361
2006525616,24sleep320060-21apt-get07:28:573
1790725615,17sleep20-21swapper/212:28:562
1480625514,24sleep114801-21apt-get12:18:531
1227425518,17sleep10-21swapper/112:09:121
38562545,37sleep10-21swapper/110:02:551
3183025416,20sleep231866-21ntp_states11:29:062
3178225411,29sleep131829-21/usr/sbin/munin11:29:051
3178225411,29sleep131829-21/usr/sbin/munin11:29:051
3023125414,21sleep10-21swapper/111:24:051
3023125414,21sleep10-21swapper/111:24:051
3013525411,24sleep330172-21irqstats07:59:023
268452546,18sleep30-21swapper/311:14:013
268452546,18sleep30-21swapper/311:14:013
2676925415,16sleep30-21swapper/309:33:413
2676925415,16sleep30-21swapper/309:33:413
1901525413,19sleep10-21swapper/110:49:111
1630225414,20sleep10-21swapper/108:59:051
1591625416,21sleep215945-21latency_hist08:58:422
1591625416,21sleep215945-21latency_hist08:58:422
1360425419,22sleep10-21swapper/108:49:121
967225310,27sleep29681-21munin-run08:38:432
677225313,24sleep26784-21apt-get08:28:532
2810525310,22sleep10-21swapper/111:18:441
2810525310,22sleep10-21swapper/111:18:441
87662526,10sleep30-21swapper/310:17:433
2949925214,24sleep229537-21users09:39:132
2493925213,21sleep224932-21apt-get11:08:552
1812425216,21sleep318175-21irqstats12:29:013
1683725214,20sleep20-21swapper/212:24:112
2905125111,16sleep30-21swapper/309:39:063
2831225113,22sleep327811-21apt-get07:53:493
845525015,22sleep18485-21cut08:34:011
1638124915,17sleep20-21swapper/208:59:062
1622424911,20sleep216277-21users10:39:152
1622424911,20sleep216277-21users10:39:152
2262824810,16sleep20-21swapper/209:19:022
315282479,15sleep20-21swapper/208:03:572
2680424713,21sleep30-21swapper/307:49:043
1207524713,17sleep30-21swapper/307:04:053
1032424715,21sleep30-21swapper/308:39:083
113432465,30sleep20-21swapper/212:08:202
49302456,31sleep30-21swapper/311:46:043
1224424313,19sleep10-21swapper/110:28:561
1224424313,19sleep10-21swapper/110:28:561
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional