You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 13:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Nov 13, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27182760015,7572sleep21119-21kworker/2:022:15:452
10634238495,3831sleep12380-21kworker/1:119:16:451
10634238495,3831sleep12380-21kworker/1:119:16:451
3027225756,557sleep030278-21ntpq20:19:440
3027225756,557sleep030278-21ntpq20:19:440
13161246621,428sleep113237-21/usr/sbin/munin19:24:401
13161246621,428sleep113237-21/usr/sbin/munin19:24:401
28903238316,353sleep328913-21ntpq23:39:373
16971237820,340sleep217016-21/usr/sbin/munin21:19:392
8586232723,106sleep00-21swapper/000:19:310
297172321126,132sleep029747-21munin-plugin-st23:44:100
1331322697,250sleep013320-21ntpq22:49:390
23468224247,121sleep023608-21unixbench_multi21:39:440
27939221583,58sleep027968-21munin-run20:14:110
838220130,146sleep1976-21unixbench_multi22:09:541
838220130,146sleep1976-21unixbench_multi22:09:541
29528219813,142sleep029548-21tr21:59:350
27963219853,82sleep00-21swapper/021:54:350
13546219511,133sleep013392-21/usr/sbin/munin21:10:080
6628219413,104sleep06654-21munin-run00:14:100
969321785,118sleep00-21swapper/020:58:000
969321785,118sleep00-21swapper/020:58:000
22492217859,68sleep022558-21wc19:54:400
119521756,131sleep00-21swapper/022:11:270
119521756,131sleep00-21swapper/022:11:270
7124216817,139sleep00-21swapper/019:07:070
12929216711,140sleep012945-21gpgv21:09:110
457216553,64sleep00-21swapper/020:29:130
457216553,64sleep00-21swapper/020:29:130
12775216436,72sleep012852-21fschecks_time19:24:310
12775216436,72sleep012852-21fschecks_time19:24:310
10694216127,114sleep010820-21timerwakeupswit20:59:420
10372215751,58sleep00-21swapper/022:39:450
7504215311,18sleep00-21swapper/020:49:450
18008215335,67sleep00-21swapper/019:39:460
18008215335,67sleep00-21swapper/019:39:460
7640215116,55sleep00-21swapper/019:09:170
7640215116,55sleep00-21swapper/019:09:170
12967214911,123sleep013112-21grep00:34:130
584921459,123sleep05887-21ps20:44:400
12431214419,105sleep212579-21sed19:24:132
1109321426,11sleep00-21swapper/019:19:260
11062214021,44sleep311113-21latency_hist22:44:123
15755213724,92sleep10-21swapper/122:59:131
25334213650,47sleep00-21swapper/023:29:280
25334213650,47sleep00-21swapper/023:29:280
6644213517,98sleep30-21swapper/320:49:113
2538821316,10sleep00-21swapper/021:46:350
2538821316,10sleep00-21swapper/021:46:350
10486212915,95sleep10-21swapper/122:39:461
22101212819,84sleep322181-21fschecks_time19:54:313
2012321285,30sleep00-21swapper/019:48:500
1748521275,29sleep00-21swapper/021:19:540
8240212619,89sleep38257-21kernelversion19:09:323
8240212619,89sleep38257-21kernelversion19:09:323
3211921269,60sleep00-21swapper/022:09:120
172921269,60sleep00-21swapper/022:14:250
1497721267,36sleep00-21swapper/021:14:160
1497721267,36sleep00-21swapper/021:14:160
732921256,11sleep00-21swapper/022:30:080
2887212410,45sleep00-21swapper/020:34:420
4193212317,24sleep00-21swapper/020:39:350
1089221239,53sleep00-21swapper/000:24:440
2497821215,29sleep00-21swapper/023:28:230
12651212149,59sleep012749-21latency_hist22:49:140
12651212149,59sleep012749-21latency_hist22:49:140
953221206,29sleep00-21swapper/019:14:170
953221206,29sleep00-21swapper/019:14:170
335621196,30sleep00-21swapper/000:02:150
3040211817,32sleep33182-21latency_hist22:19:133
2276211810,93sleep12277-21cpu20:34:271
20379211617,80sleep220515-21/usr/sbin/munin21:29:452
11192211415,79sleep311201-21/usr/sbin/munin19:19:333
8663211314,81sleep18718-21/usr/sbin/munin00:19:321
7498211314,79sleep27626-21grep19:09:152
7498211314,79sleep27626-21grep19:09:152
26092211325,23sleep00-21swapper/021:49:220
26092211325,23sleep00-21swapper/021:49:220
24886211312,17sleep00-21swapper/020:04:130
3279211247,48sleep00-21swapper/022:19:260
31766211214,85sleep131796-21tune2fs20:24:501
31766211214,85sleep131796-21tune2fs20:24:501
1579221129,64sleep00-21swapper/019:34:290
25663211019,67sleep325812-21latency_hist21:49:123
25663211019,67sleep325812-21latency_hist21:49:123
2173321109,56sleep00-21swapper/019:54:120
692210913,82sleep2711-21grep23:54:252
19359210914,18sleep00-21swapper/023:09:330
19359210914,18sleep00-21swapper/023:09:330
150321095,13sleep10-21swapper/120:29:471
25523210827,69sleep025567-21ntpq20:04:380
14894210711,79sleep015034-21unixbench_multi19:29:440
30891210613,76sleep330928-21/usr/sbin/munin22:04:313
2130821066,63sleep021432-21timerwakeupswit23:14:420
18589210519,71sleep218651-21find21:24:442
14679210517,72sleep114718-21processes19:29:411
29355210421,63sleep129390-21fschecks_count21:59:321
23386210410,80sleep323464-21find21:39:433
8244210231,57sleep00-21swapper/000:19:110
515221028,78sleep05294-21sed00:09:130
17250210219,59sleep217281-21munin-plugin-st23:04:102
8208210031,49sleep38238-21munin-run00:19:103
2062321006,16sleep00-21swapper/021:30:280
2403429917,68sleep324074-21ps19:59:403
1062229914,70sleep310686-21wc20:59:413
1140429811,73sleep111449-21cat00:29:131
382429721,44sleep13821-21apt-get20:39:241
2169829723,45sleep321722-21awk19:54:103
78429648,14sleep00-21swapper/020:29:330
3031929610,72sleep230407-21grep20:19:462
3031929610,72sleep230407-21grep20:19:462
2901329620,19sleep10-21swapper/121:59:121
2446329527,47sleep324595-21unixbench_multi23:24:473
90492948,14sleep10-21swapper/120:54:421
90492948,14sleep10-21swapper/120:54:421
2951929443,17sleep029566-21systemd-run20:19:120
2644429416,62sleep326447-21sshd23:32:223
2644429416,62sleep326447-21sshd23:32:223
1811529412,69sleep118117-21grep21:24:311
3106529325,45sleep30-21swapper/320:24:113
3106529325,45sleep30-21swapper/320:24:113
196529217,59sleep32011-21munin-run23:59:113
164812917,69sleep016560-21grep22:59:440
1004629117,44sleep110065-21apt-get20:59:231
897229023,43sleep29111-21grep22:35:112
2699029021,47sleep327054-21sort23:34:283
2090628912,62sleep220947-21cut19:49:412
1643628914,53sleep216287-21latency_hist21:19:152
1643628914,53sleep216287-21latency_hist21:19:152
120128931,41sleep21277-21awk20:29:442
3155828819,11sleep00-21swapper/023:49:270
181132888,67sleep018198-21cat23:04:410
181132888,67sleep018198-21cat23:04:410
312152875,71sleep00-21swapper/023:48:300
2383428714,52sleep123550-21/usr/sbin/munin19:59:341
832628511,57sleep18174-21/usr/sbin/munin22:34:581
730628515,54sleep17342-21ntpq20:49:371
52022858,19sleep05238-21ntpq22:24:390
2638228529,27sleep10-21swapper/120:09:111
1957628417,47sleep219708-21tail19:44:492
1838328422,46sleep00-21swapper/021:24:380
822328319,10sleep00-21swapper/022:34:400
3182928322,25sleep10-21swapper/123:49:361
3095228312,59sleep130953-21sshd20:22:441
3095228312,59sleep130953-21sshd20:22:441
2903628323,26sleep20-21swapper/221:59:132
2782028319,11sleep00-21swapper/023:34:450
551528119,29sleep30-21swapper/322:24:463
2086028114,28sleep20-21swapper/223:14:322
1411228117,43sleep314123-21modprobe22:54:103
3027928018,41sleep330371-21/usr/sbin/munin23:44:333
2972128014,45sleep229763-21wc21:59:412
2956728010,56sleep129618-21latency_hist20:19:131
69027920,12sleep00-21swapper/023:54:230
3224327917,42sleep232387-21tail23:49:452
2123727911,48sleep00-21swapper/021:34:270
768427819,17sleep20-21swapper/220:49:472
2260227822,23sleep30-21swapper/321:39:153
953927712,50sleep29558-21apt-get19:14:232
953927712,50sleep29558-21apt-get19:14:232
2162527715,21sleep10-21swapper/121:34:371
1683327713,51sleep316880-21latency21:19:363
1035327718,21sleep30-21swapper/319:14:453
1035327718,21sleep30-21swapper/319:14:453
52952760,7sleep10-21swapper/100:09:131
2348627619,40sleep323517-21awk23:24:103
2336527622,29sleep10-21swapper/119:59:131
1289127628,32sleep312904-21apt-get21:09:103
464627522,29sleep10-21swapper/118:59:331
1880127517,41sleep10-21swapper/123:09:101
1880127517,41sleep10-21swapper/123:09:101
1752427512,49sleep117568-21idleruntime19:39:351
1752427512,49sleep117568-21idleruntime19:39:351
1339027518,27sleep10-21swapper/121:09:201
2353927424,32sleep123640-21latency_hist23:24:121
443927325,20sleep00-21swapper/018:59:280
232882736,57sleep0362-21snmpd23:21:410
2099227324,23sleep10-21swapper/121:34:121
148932737,18sleep014927-21ps22:54:390
1294627321,28sleep10-21swapper/121:09:121
589227223,20sleep20-21swapper/200:09:402
408727221,21sleep20-21swapper/220:39:332
3047527218,26sleep10-21swapper/123:44:391
2254827225,23sleep20-21swapper/221:39:142
203527223,28sleep30-21swapper/320:34:133
735627119,24sleep10-21swapper/119:09:141
668827111,14sleep10-21swapper/100:14:111
505027131,19sleep20-21swapper/218:59:422
239727115,40sleep32461-21/usr/sbin/munin23:59:313
1631327122,34sleep316335-21latency_hist21:19:123
1631327122,34sleep316335-21latency_hist21:19:123
1537627114,42sleep215378-21modprobe22:55:102
391027021,28sleep30-21swapper/320:39:303
2177827020,23sleep10-21swapper/119:54:131
1627527016,15sleep10-21swapper/119:34:421
124227021,21sleep30-21swapper/322:12:023
124227021,21sleep30-21swapper/322:12:023
234926916,31sleep30-21swapper/320:34:293
359726823,18sleep10-21swapper/100:04:141
2694126813,41sleep226961-21sed20:09:362
1347626821,26sleep20-21swapper/221:10:062
1274726818,22sleep30-21swapper/319:24:303
1274726818,22sleep30-21swapper/319:24:303
1073226829,20sleep30-21swapper/300:24:433
924526720,30sleep19258-21modprobe19:14:101
924526720,30sleep19258-21modprobe19:14:101
3159726722,21sleep30-21swapper/320:24:283
3159726722,21sleep30-21swapper/320:24:283
2520926721,30sleep225251-21fschecks_count20:04:312
1781926711,44sleep117820-21cron21:24:111
127712677,13sleep10-21swapper/100:32:571
35426619,31sleep3415-21munin-run23:54:103
2562926618,18sleep20-21swapper/221:49:112
2562926618,18sleep20-21swapper/221:49:112
2315626612,23sleep10-21swapper/121:39:371
154272669,45sleep2362-21snmpd19:33:382
1112026617,36sleep111158-21cpu19:19:311
1909126518,32sleep219161-21df23:09:272
1909126518,32sleep219161-21df23:09:272
1730426523,21sleep10-21swapper/123:04:131
1138726514,30sleep311279-21/usr/sbin/munin22:44:293
1138726514,30sleep311279-21/usr/sbin/munin22:44:293
951526421,20sleep30-21swapper/322:39:123
876826422,23sleep30-21swapper/300:19:343
2976826416,34sleep229862-21sed23:44:122
2826126420,22sleep20-21swapper/220:14:292
2712826422,19sleep30-21swapper/320:09:413
1885126412,36sleep318956-21cat23:09:123
1885126412,36sleep318956-21cat23:09:123
1800126414,21sleep20-21swapper/223:04:392
1800126414,21sleep20-21swapper/223:04:392
602526321,24sleep30-21swapper/319:04:273
43752637,13sleep10-21swapper/122:22:061
3206426317,29sleep232101-21modprobe22:09:102
2882626314,33sleep128857-21nfs_client23:39:351
2819526315,19sleep30-21swapper/321:54:423
1169326316,30sleep111731-21/usr/sbin/munin22:44:351
1169326316,30sleep111731-21/usr/sbin/munin22:44:351
3170426212,33sleep131717-21vmstat22:04:451
2570226212,24sleep10-21swapper/123:29:371
2570226212,24sleep10-21swapper/123:29:371
74592617,10sleep30-21swapper/322:31:493
265492617,14sleep30-21swapper/321:49:573
265492617,14sleep30-21swapper/321:49:573
671026022,21sleep30-21swapper/300:14:133
584426017,26sleep2225-21systemd-journal19:04:142
2104326022,18sleep10-21swapper/123:14:351
1908726019,25sleep319083-21apt-get23:09:243
1908726019,25sleep319083-21apt-get23:09:243
1888526024,24sleep30-21swapper/319:44:253
1292426013,30sleep212933-21/usr/sbin/munin22:49:282
1023926015,26sleep210301-21/usr/sbin/munin22:39:432
702025915,25sleep30-21swapper/320:49:313
577925916,21sleep10-21swapper/120:44:371
420525920,20sleep30-21swapper/300:04:363
39925914,26sleep2433-21munin-run20:29:112
39925914,26sleep2433-21munin-run20:29:112
2405125921,22sleep224093-21sort23:24:382
1944525923,20sleep10-21swapper/123:09:341
1944525923,20sleep10-21swapper/123:09:341
1137725915,18sleep20-21swapper/222:44:282
1137725915,18sleep20-21swapper/222:44:282
676625817,27sleep26811-21ntp_kernel_pll_22:29:382
3024925818,24sleep230278-21irqstats23:44:322
292912586,41sleep30-21swapper/320:16:203
2590425814,27sleep326022-21users20:04:453
2433225818,24sleep124327-21apt-get21:44:261
2433225818,24sleep124327-21apt-get21:44:261
178352587,37sleep20-21swapper/219:39:442
178352587,37sleep20-21swapper/219:39:442
1621825819,23sleep216242-21ntp_states19:34:392
1451625813,32sleep314548-21tune2fs22:54:303
1298825819,20sleep10-21swapper/122:49:321
90442577,36sleep20-21swapper/200:19:412
69942576,12sleep10-21swapper/122:29:451
483225712,20sleep20-21swapper/222:24:292
2678225718,16sleep10-21swapper/121:50:031
2678225718,16sleep10-21swapper/121:50:031
229125716,26sleep22348-21df20:34:282
1676725715,22sleep316895-21/usr/sbin/munin22:59:473
703525620,18sleep10-21swapper/100:14:311
225125615,20sleep20-21swapper/223:59:262
2228425622,20sleep20-21swapper/219:54:342
1886225615,22sleep10-21swapper/119:44:231
1055425619,17sleep20-21swapper/220:59:402
897525517,26sleep38489-21/usr/sbin/munin20:54:403
897525517,26sleep38489-21/usr/sbin/munin20:54:403
405225518,20sleep30-21swapper/322:19:453
2717925515,18sleep10-21swapper/120:09:431
2659025514,26sleep226594-21cron23:34:102
2659025514,26sleep226594-21cron23:34:102
2385325517,22sleep123859-21df_inode23:24:321
237325510,30sleep12496-21timerwakeupswit22:14:431
1032025521,22sleep210351-21latency00:24:342
443725413,24sleep34432-21apt-get18:59:273
2820925418,19sleep20-21swapper/223:39:112
1865425411,29sleep318090-21/usr/sbin/munin21:24:453
1791625414,18sleep30-21swapper/319:39:453
1791625414,18sleep30-21swapper/319:39:453
1240825413,15sleep30-21swapper/300:29:443
545425313,24sleep25460-21diskstats20:44:302
2946925319,15sleep30-21swapper/321:59:343
2002925311,20sleep30-21swapper/321:29:383
1624325316,22sleep316249-21ntp_states19:34:403
137422536,37sleep20-21swapper/219:26:512
137422536,37sleep20-21swapper/219:26:512
2711525214,21sleep127172-21irqstats23:34:301
1345525212,19sleep30-21swapper/321:09:513
732025113,19sleep20-21swapper/200:14:362
3273025112,22sleep2353-21perl22:09:432
3273025112,22sleep2353-21perl22:09:432
1211125114,17sleep20-21swapper/221:04:412
1551125013,23sleep30-21swapper/319:34:113
126625015,16sleep10-21swapper/123:54:411
380124912,19sleep23868-21df00:04:272
2611924913,21sleep226112-21apt-get21:49:242
2611924913,21sleep226112-21apt-get21:49:242
2247824913,21sleep222502-21irqstats23:19:372
2110524917,16sleep30-21swapper/323:14:373
2318824816,14sleep20-21swapper/221:39:382
459124613,17sleep24566-21latency_hist22:24:132
483124511,19sleep14828-21apt-get22:24:281
238472457,15sleep20-21swapper/219:59:352
150102448,19sleep10-21swapper/121:14:251
150102448,19sleep10-21swapper/121:14:251
128652449,15sleep20-21swapper/200:34:112
878424311,15sleep20-21swapper/220:54:352
878424311,15sleep20-21swapper/220:54:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional