You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 11:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Nov 06, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23306239409,3913sleep021222-21kworker/u8:021:59:510
4092218010,2149sleep2410-21apt-get22:29:462
4092218010,2149sleep2410-21apt-get22:29:462
85327497,727sleep0900-21ntpq22:30:080
85327497,727sleep0900-21ntpq22:30:080
21283272213,689sleep021321-21fschecks_count23:34:530
21649270511,680sleep121661-21ntpq20:10:011
3116247769,348sleep03130-21fschecks_count00:19:550
1864923608,338sleep018723-21cat21:44:570
27357233944,238sleep027423-40mandb23:54:350
27357233944,238sleep027423-40mandb23:54:350
7162232722,286sleep27219-21/usr/sbin/munin19:25:002
7162232722,286sleep27219-21/usr/sbin/munin19:25:002
12614231615,185sleep00-21swapper/021:25:030
12614231615,185sleep00-21swapper/021:25:030
2491423040,6sleep10-21swapper/122:04:521
23018226671,183sleep023041-21cut23:39:580
2041722538,102sleep020485-21systemctl21:50:060
425122299,205sleep04292-21ntpq19:15:000
24430222872,133sleep024445-21fschecks_count20:19:570
23010222622,185sleep323217-21latency_hist21:59:383
27674222558,83sleep00-21swapper/023:54:480
22775222434,129sleep022815-21http21:59:350
31218222127,16sleep00-21swapper/022:24:590
31218222127,16sleep00-21swapper/022:24:590
8865221515,82sleep00-21swapper/019:29:580
8865221515,82sleep00-21swapper/019:29:580
7134220920,40sleep00-21swapper/022:50:070
15172220980,50sleep00-21swapper/019:50:010
1101222085,10sleep00-21swapper/019:35:280
22794220212,74sleep00-21swapper/020:14:530
3398220114,148sleep03478-21xargs20:55:070
31031220013,166sleep030540-21/usr/sbin/munin20:40:040
2542421986,181sleep0362-21snmpd23:47:490
7180219245,132sleep37209-21kworker/u8:021:09:353
4847219136,114sleep04928-21mailstats21:00:070
15992219112,72sleep0225-21systemd-journal23:19:340
15992219112,72sleep0225-21systemd-journal23:19:340
13719218743,122sleep013810-21switchtime23:10:060
8242217632,88sleep00-21swapper/021:10:090
8242217632,88sleep00-21swapper/021:10:090
715221705,109sleep00-21swapper/000:31:340
1777421705,153sleep00-21swapper/021:40:330
17096216414,136sleep117103-21kernelversion21:39:581
1038321648,141sleep010337-21latency_hist21:19:380
30004216329,123sleep0362-21snmpd20:36:050
11139216221,29sleep00-21swapper/021:20:060
11139216221,29sleep00-21swapper/021:20:060
24833215752,39sleep00-21swapper/022:04:480
300921530,7sleep30-21swapper/320:54:563
21411214796,34sleep00-21swapper/020:09:560
6379214551,28sleep06417-21modprobe22:49:340
695321438,86sleep00-21swapper/019:24:540
695321438,86sleep00-21swapper/019:24:540
18196214113,113sleep118266-21grep19:59:561
18196214113,113sleep118266-21grep19:59:561
19624214023,53sleep00-21swapper/023:29:520
17266213532,44sleep017401-21grep23:20:190
29238213427,49sleep029228-21apt-get23:59:520
1766213465,17sleep00-21swapper/019:09:390
1766213465,17sleep00-21swapper/019:09:390
13627213442,76sleep00-21swapper/019:45:050
6772213311,103sleep36895-21cat00:30:083
2777213130,45sleep02791-21munin-run00:19:350
29974213019,92sleep230104-21grep00:00:112
26539212823,39sleep00-21swapper/022:10:050
11749212824,78sleep111780-21fschecks_time19:39:571
1826721270,11sleep20-21swapper/219:59:562
1826721270,11sleep20-21swapper/219:59:562
18075212728,44sleep0362-21snmpd19:59:430
18075212728,44sleep0362-21snmpd19:59:430
14510212515,89sleep114659-21sort19:49:381
2484212325,78sleep12498-21modprobe20:54:351
15426212110,44sleep00-21swapper/021:34:560
459821206,101sleep00-21swapper/022:41:350
459821206,101sleep00-21swapper/022:41:350
27712212043,23sleep00-21swapper/020:29:590
1636521199,94sleep016458-21apt-key19:54:420
1636521199,94sleep016458-21apt-key19:54:420
6711211817,42sleep00-21swapper/019:24:370
32601211415,86sleep032661-21find20:45:050
11106211418,21sleep30-21swapper/323:04:353
11106211418,21sleep30-21swapper/323:04:353
276521129,41sleep00-21swapper/022:35:090
31171211125,62sleep031225-21sort18:59:510
2341210916,30sleep00-21swapper/020:52:440
18069210933,61sleep00-21swapper/023:24:470
7389210719,73sleep17411-21latency_hist00:34:341
29667210630,41sleep029729-21perl22:19:580
20880210621,67sleep220884-21cron23:34:352
1260021058,11sleep00-21swapper/023:06:180
24389210418,67sleep224279-21/usr/sbin/munin23:44:562
30943210327,51sleep231003-21date22:24:492
2824121017,32sleep00-21swapper/022:14:580
15183210118,61sleep10-21swapper/119:50:021
3100210012,21sleep13167-21memory20:54:591
2845429915,59sleep227965-21/usr/sbin/munin22:15:042
2762329817,68sleep227675-21cat20:29:572
268922987,80sleep0362-21snmpd20:26:150
366129724,52sleep13675-21grep22:39:581
366129724,52sleep13675-21grep22:39:581
1596129624,54sleep316106-21unixbench_multi21:35:073
353129513,63sleep23605-21/usr/sbin/munin00:20:062
3064829417,16sleep20-21swapper/220:39:542
148132946,12sleep20-21swapper/223:14:522
148132946,12sleep20-21swapper/223:14:522
824429222,43sleep18241-21apt-get22:54:481
2999829114,22sleep30-21swapper/322:20:063
2330329113,40sleep223296-21apt-get21:59:492
669129014,58sleep05896-21/usr/sbin/munin21:05:080
40729015,57sleep132365-21apt-get22:29:451
40729015,57sleep132365-21apt-get22:29:451
1367629019,56sleep30-21swapper/321:29:463
2482628918,50sleep224886-21grep20:20:072
1180928812,62sleep311829-21fw_packets23:05:163
269772877,13sleep30-21swapper/323:50:273
269772877,13sleep30-21swapper/323:50:273
1338928715,58sleep313446-21awk19:45:003
683328624,24sleep20-21swapper/221:05:102
2120228613,38sleep20-21swapper/220:09:502
434128527,32sleep10-21swapper/100:24:371
3157428518,48sleep10-21swapper/122:25:081
3157428518,48sleep10-21swapper/122:25:081
274128528,34sleep32767-21munin-run00:19:343
1846628517,36sleep10-21swapper/121:44:541
3159528421,46sleep00-21swapper/000:05:090
1808928417,47sleep218117-21grep21:44:352
1018728420,49sleep310257-21fschecks_time19:34:543
997828316,17sleep20-21swapper/222:59:582
997828316,17sleep20-21swapper/222:59:582
2091328313,18sleep30-21swapper/320:09:353
1285428328,34sleep10-21swapper/123:09:341
2290528218,39sleep222796-21latency_hist21:59:372
1783728234,21sleep20-21swapper/223:24:372
621528118,44sleep26282-21fschecks_time00:29:552
2018028111,41sleep10-21swapper/121:49:591
1490628119,30sleep00-21swapper/021:33:510
116302817,34sleep00-21swapper/019:39:520
989128015,29sleep00-21swapper/022:59:560
989128015,29sleep00-21swapper/022:59:560
411328016,50sleep14132-21kernelversion19:14:571
2597728021,41sleep125992-21cut20:24:551
21072805,11sleep20-21swapper/222:34:552
1786828024,34sleep30-21swapper/319:59:373
1786828024,34sleep30-21swapper/319:59:373
324427918,29sleep00-21swapper/019:10:570
2067327919,11sleep00-21swapper/020:06:290
929527828,39sleep0362-21snmpd22:56:450
58912786,39sleep05888-21/usr/sbin/munin00:29:360
58912786,39sleep05888-21/usr/sbin/munin00:29:360
3234127810,22sleep00-21swapper/000:09:540
3234127810,22sleep00-21swapper/000:09:540
287227818,44sleep32951-21wc19:10:073
2067427827,24sleep30-21swapper/321:50:093
1346427815,46sleep113521-21cat21:29:361
1346427815,46sleep113521-21cat21:29:361
2405227718,27sleep20-21swapper/220:19:352
521027621,30sleep30-21swapper/300:25:073
521027621,30sleep30-21swapper/300:25:073
548727525,29sleep30-21swapper/319:19:563
1874927519,35sleep318046-21/usr/sbin/munin23:25:073
1618927418,31sleep316357-21apt-config19:54:413
1616427423,31sleep316380-21dpkg23:19:383
1616427423,31sleep316380-21dpkg23:19:383
3032627314,28sleep10-21swapper/120:39:371
2008727324,28sleep10-21swapper/123:30:051
3178327226,28sleep231925-21sed19:00:052
1276427213,41sleep312861-21timerandwakeup21:25:063
1276427213,41sleep312861-21timerandwakeup21:25:063
1171227224,22sleep30-21swapper/319:39:553
116012725,11sleep20-21swapper/219:39:462
1127927220,26sleep10-21swapper/123:04:371
1127927220,26sleep10-21swapper/123:04:371
310922710,8sleep10-21swapper/120:40:041
2815227111,25sleep20-21swapper/223:55:102
156727117,25sleep20-21swapper/200:14:572
1053127119,29sleep20-21swapper/219:35:012
93527022,32sleep2960-21idleruntime-cro20:49:352
723127021,21sleep10-21swapper/121:09:371
1499027017,35sleep115006-21uname23:14:551
1499027017,35sleep115006-21uname23:14:551
9142699,42sleep3993-21timerandwakeup19:05:063
9142699,42sleep3993-21timerandwakeup19:05:063
812726915,38sleep38240-21/usr/sbin/munin21:10:083
812726915,38sleep38240-21/usr/sbin/munin21:10:083
512026920,27sleep10-21swapper/122:44:511
310542696,44sleep20-21swapper/200:04:592
2667926916,35sleep226691-21/usr/sbin/munin22:10:072
190026920,23sleep20-21swapper/220:50:082
1097226919,31sleep211017-21/usr/sbin/munin21:20:002
1097226919,31sleep211017-21/usr/sbin/munin21:20:002
57726831,19sleep30-21swapper/300:10:093
57726831,19sleep30-21swapper/300:10:093
45526816,22sleep10-21swapper/119:04:561
45526816,22sleep10-21swapper/119:04:561
445626818,21sleep30-21swapper/320:59:583
35412686,17sleep20-21swapper/222:39:432
35412686,17sleep20-21swapper/222:39:432
2753126822,25sleep10-21swapper/120:29:561
2720326824,22sleep30-21swapper/320:29:373
1998426813,40sleep120023-21memory20:05:001
2933826719,25sleep30-21swapper/322:19:373
143322675,52sleep30-21swapper/323:13:323
2785626612,23sleep30-21swapper/320:30:023
210226615,22sleep10-21swapper/119:09:441
179802666,9sleep30-21swapper/321:43:113
1068926618,23sleep10-21swapper/121:19:551
1068926618,23sleep10-21swapper/121:19:551
3258226520,24sleep20-21swapper/222:29:382
3258226520,24sleep20-21swapper/222:29:382
956826425,19sleep30-21swapper/322:59:393
42226423,22sleep10-21swapper/100:10:071
42226423,22sleep10-21swapper/100:10:071
2767326415,29sleep11-21systemd22:14:341
2272526419,21sleep10-21swapper/123:39:471
2126726420,29sleep221207-21latency_hist21:54:362
732026312,37sleep17352-21nfs_client19:25:031
732026312,37sleep17352-21nfs_client19:25:031
506426326,18sleep10-21swapper/100:25:051
506426326,18sleep10-21swapper/100:25:051
360626320,21sleep30-21swapper/300:20:073
2767826327,18sleep30-21swapper/323:54:513
2200726315,23sleep10-21swapper/121:55:041
2157026322,17sleep30-21swapper/320:09:593
722026214,31sleep37240-21kernelversion19:25:013
722026214,31sleep37240-21kernelversion19:25:013
37102624,11sleep20-21swapper/219:14:382
327326215,33sleep13274-21sshd22:39:301
3157226224,14sleep10-21swapper/119:00:001
3124626217,23sleep30-21swapper/300:05:043
2902126221,25sleep229075-21sort20:34:532
2439726213,23sleep30-21swapper/323:44:573
2298326217,20sleep30-21swapper/320:15:003
514326111,24sleep20-21swapper/200:25:062
514326111,24sleep20-21swapper/200:25:062
3163026123,24sleep331146-21/usr/sbin/munin19:00:023
1428426112,19sleep20-21swapper/221:30:062
463726016,20sleep30-21swapper/319:15:083
411026013,23sleep20-21swapper/220:59:372
2991326016,19sleep30-21swapper/320:35:113
2985426025,19sleep10-21swapper/100:00:101
2780526017,25sleep127826-21fschecks_count23:55:011
2609726012,23sleep20-21swapper/223:49:532
2609726012,23sleep20-21swapper/223:49:532
1598826023,23sleep216014-21munin-run19:54:362
665925919,21sleep20-21swapper/219:24:352
643125922,17sleep30-21swapper/322:49:373
3124425921,19sleep30-21swapper/320:40:073
3053725912,33sleep330545-21systemd-run00:04:363
2672925922,17sleep10-21swapper/123:50:251
2672925922,17sleep10-21swapper/123:50:251
256225922,19sleep30-21swapper/320:54:373
256002597,12sleep10-21swapper/120:24:351
195902598,9sleep30-21swapper/321:48:563
1854525914,26sleep20-21swapper/221:44:552
1693025920,21sleep10-21swapper/123:20:121
142932597,43sleep10-21swapper/123:13:011
13832598,10sleep30-21swapper/322:30:323
13832598,10sleep30-21swapper/322:30:323
1035625914,32sleep310379-21latency_hist21:19:373
870625813,27sleep18758-21/usr/sbin/munin19:29:551
870625813,27sleep18758-21/usr/sbin/munin19:29:551
858225818,19sleep30-21swapper/319:29:533
858225818,19sleep30-21swapper/319:29:533
377125817,16sleep30-21swapper/322:40:003
377125817,16sleep30-21swapper/322:40:003
2973025819,22sleep10-21swapper/122:20:001
2844725813,20sleep10-21swapper/122:15:031
2599525817,24sleep226098-21perl20:24:562
2279525817,25sleep122784-21apt-get20:14:541
136072586,11sleep20-21swapper/223:10:042
3129225714,24sleep10-21swapper/100:05:061
2434425714,26sleep324339-21apt-get20:19:523
1665525722,20sleep30-21swapper/323:20:063
1191725715,19sleep30-21swapper/321:24:373
1191725715,19sleep30-21swapper/321:24:373
94162566,15sleep20-21swapper/219:30:132
94162566,15sleep20-21swapper/219:30:132
905625614,28sleep29126-21df21:14:512
3258525615,21sleep30-21swapper/320:45:043
239542566,41sleep30-21swapper/323:43:523
233225615,25sleep32390-21memory22:35:003
2019825618,20sleep30-21swapper/323:30:083
1847625613,25sleep118521-21/usr/sbin/munin23:25:011
1675825617,25sleep216817-21tune2fs19:55:572
1675825617,25sleep216817-21tune2fs19:55:572
165402567,18sleep10-21swapper/119:54:511
165402567,18sleep10-21swapper/119:54:511
540025515,23sleep15395-21apt-get19:19:501
511625514,25sleep25113-21apt-get22:44:482
2494125514,26sleep324951-21forks22:04:553
162412558,9sleep20-21swapper/221:35:472
667325414,25sleep36666-21apt-get22:49:543
2456325412,28sleep324565-21cron22:04:353
2164425415,22sleep30-21swapper/323:35:003
1983525416,23sleep20-21swapper/220:04:572
1420425413,25sleep114283-21systemctl21:30:051
93482535,14sleep20-21swapper/222:57:282
436125313,24sleep14356-21apt-get20:59:521
2687225311,27sleep326878-21ntp_states22:10:123
2369825315,21sleep10-21swapper/122:00:131
13672538,10sleep20-21swapper/219:06:592
13672538,10sleep20-21swapper/219:06:592
1354825314,21sleep20-21swapper/219:45:032
804525216,20sleep20-21swapper/221:10:072
804525216,20sleep20-21swapper/221:10:072
3136925215,16sleep30-21swapper/322:25:023
3136925215,16sleep30-21swapper/322:25:023
2321325216,19sleep20-21swapper/223:40:042
1818225211,17sleep30-21swapper/319:59:543
1818225211,17sleep30-21swapper/319:59:543
914025112,20sleep10-21swapper/121:14:541
2485625111,24sleep224852-21apt-get22:04:502
1155825114,20sleep211086-21apt-get23:04:432
1007325112,20sleep10-21swapper/119:34:481
481525013,21sleep24821-21ntp_states21:00:052
254102505,34sleep10-21swapper/123:47:381
1865625016,19sleep20-21swapper/223:25:062
711924910,16sleep20-21swapper/222:50:052
155924911,22sleep11566-21diskstats00:14:561
1239624812,20sleep212423-21irqstats21:24:572
1239624812,20sleep212423-21irqstats21:24:572
85172476,32sleep10-21swapper/121:11:431
85172476,32sleep10-21swapper/121:11:431
796824711,14sleep10-21swapper/122:54:361
323382479,18sleep20-21swapper/200:09:522
323382479,18sleep20-21swapper/200:09:522
3218924715,17sleep10-21swapper/120:44:541
222492468,24sleep20-21swapper/223:36:532
2166424613,15sleep10-21swapper/123:35:021
1549124411,16sleep10-21swapper/121:34:571
193324312,19sleep21998-21df19:09:412
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional