You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 06:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Dec 02, 2025 00:44:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20487211059,1086sleep221461-1kworker/2:1H23:58:402
29292284212,812sleep329283-21apt-get22:43:203
7708279816,765sleep17724-21/usr/sbin/munin21:33:361
7708279816,765sleep17724-21/usr/sbin/munin21:33:361
2055240323,362sleep12125-21/usr/sbin/munin22:58:371
20659236913,249sleep020675-21modprobe20:33:050
31356231413,287sleep131365-21ntpq19:23:521
271822285110,153sleep00-21swapper/000:18:370
26978227268,189sleep027122-21date20:53:080
27826226565,86sleep027858-21fschecks_count22:38:250
15290225851,194sleep015369-21cat23:43:070
590225311,222sleep0672-21timerandwakeup19:28:390
14802224019,203sleep214944-21latency_hist21:58:082
70222378,210sleep0845-21grep22:53:390
24583223121,192sleep124635-21/usr/sbin/munin22:28:281
22571223165,147sleep022596-21latency_hist22:23:060
2610122277,209sleep05666-21snmpd00:14:000
23952222017,187sleep323996-21ntpq00:08:303
7219221130,163sleep07221-21/usr/sbin/munin21:33:180
7219221130,163sleep07221-21/usr/sbin/munin21:33:180
2176521989,17sleep00-21swapper/000:03:060
18397219849,88sleep013252-21kworker/0:122:08:280
2506921968,149sleep027199-1kworker/0:2H19:08:070
24591219222,10sleep00-21swapper/000:10:010
5966218916,155sleep06027-21latency_hist23:13:060
1640218812,151sleep00-21swapper/021:13:350
1640218812,151sleep00-21swapper/021:13:350
16862218417,147sleep317003-21grep23:48:063
16862218417,147sleep317003-21grep23:48:063
2975821829,159sleep029798-21processes22:43:340
3436217510,55sleep00-21swapper/019:38:240
26853217425,30sleep00-21swapper/019:11:220
26853217425,30sleep00-21swapper/019:11:220
2248821736,151sleep00-21swapper/000:03:430
2248821736,151sleep00-21swapper/000:03:430
5189217126,34sleep20-21swapper/223:08:362
15571217111,98sleep00-21swapper/021:58:340
8642216528,116sleep00-21swapper/019:53:390
9031216316,133sleep19065-21irqstats21:38:281
1061421559,47sleep00-21swapper/021:43:280
2407021545,11sleep00-21swapper/022:27:540
13298215429,110sleep013401-21grep20:08:370
4119215324,27sleep00-21swapper/021:23:210
898021528,62sleep09030-21basename21:38:270
1788721517,130sleep017963-21latency20:23:320
16943215012,121sleep116966-21/usr/sbin/munin22:03:281
2731521495,29sleep00-21swapper/022:36:230
21915214718,92sleep00-21swapper/020:33:580
21915214718,92sleep00-21swapper/020:33:580
1879521476,30sleep00-21swapper/023:53:330
1879521476,30sleep00-21swapper/023:53:330
2836021465,39sleep00-21swapper/020:56:270
14275214621,109sleep00-21swapper/020:13:220
14275214621,109sleep00-21swapper/020:13:220
3034021457,122sleep00-21swapper/021:03:220
2873621445,29sleep00-21swapper/020:58:090
1654921435,11sleep00-21swapper/022:03:150
1289121426,99sleep00-21swapper/023:33:340
2532121415,29sleep00-21swapper/020:47:430
1050721417,27sleep00-21swapper/019:58:480
30857213515,69sleep00-21swapper/019:23:180
15238213518,96sleep215264-21modprobe23:43:052
13042213414,35sleep113055-21/usr/sbin/munin20:08:301
13593213228,89sleep013659-21grep21:53:260
9810213111,83sleep09904-21cut23:23:360
25316213022,74sleep00-21swapper/022:29:040
12264213014,17sleep30-21swapper/321:48:323
384212915,102sleep00-21swapper/021:09:150
22431212914,19sleep20-21swapper/200:03:402
22431212914,19sleep20-21swapper/200:03:402
294021289,17sleep00-21swapper/021:18:320
32173212629,85sleep032239-21latency_hist22:53:070
32173212629,85sleep032239-21latency_hist22:53:070
23207212410,44sleep00-21swapper/020:38:380
30092212218,91sleep130128-21latency_hist21:03:061
29803212191,15sleep00-21swapper/000:28:060
1662321217,98sleep216693-21sensors_temp20:18:572
26836211920,83sleep126955-21timerwakeupswit22:33:531
6524211419,34sleep00-21swapper/021:28:380
1315221149,89sleep013216-21sendmail-msp21:53:050
764821135,79sleep00-21swapper/019:52:180
11101211315,66sleep00-21swapper/020:03:120
5505211210,57sleep00-21swapper/019:43:370
316321128,89sleep03272-21fschecks_time23:03:250
25645211216,80sleep125712-21df20:48:241
1187621125,73sleep00-21swapper/023:30:060
1092421129,21sleep30-21swapper/323:28:243
30779211120,78sleep130848-21sed00:28:391
30779211120,78sleep130848-21sed00:28:391
9681211020,70sleep29763-21fschecks_time19:58:302
1803421105,77sleep00-21swapper/023:49:130
1803421105,77sleep00-21swapper/023:49:130
10820210813,78sleep210881-21grep21:43:342
1620821069,54sleep00-21swapper/020:18:470
24842210516,69sleep1284-21dbus-daemon00:13:041
19931210529,52sleep319945-21apt-get23:58:043
19931210529,52sleep319945-21apt-get23:58:043
29096210411,34sleep00-21swapper/019:18:210
28240210428,36sleep028375-21unixbench_multi19:13:490
24036210417,65sleep223550-21/usr/sbin/munin00:08:332
22864210218,68sleep121988-21/usr/sbin/munin00:03:481
22864210218,68sleep121988-21/usr/sbin/munin00:03:481
22902210030,45sleep322966-21fschecks_time22:23:253
2466429922,60sleep224740-21wc20:43:372
3100329732,10sleep00-21swapper/000:29:110
3100329732,10sleep00-21swapper/000:29:110
255429715,69sleep32555-21sshd23:00:173
1673529731,11sleep00-21swapper/023:47:230
1673529731,11sleep00-21swapper/023:47:230
825529637,32sleep00-21swapper/023:18:350
751729614,62sleep00-21swapper/023:18:070
1403329617,66sleep214094-21latency_hist20:13:082
2612829311,27sleep226129-21sshd00:14:232
253629331,10sleep00-21swapper/023:00:010
906229212,66sleep39136-21latency_hist23:23:073
2097729110,16sleep20-21swapper/220:33:192
2097729110,16sleep20-21swapper/220:33:192
901229018,56sleep19019-21modprobe23:23:051
621029019,57sleep16177-21latency_hist19:48:071
1374229029,27sleep10-21swapper/123:38:071
2982828917,57sleep129956-21latency_hist00:28:071
2708028914,52sleep327056-21gpgv19:13:073
2708028914,52sleep327056-21gpgv19:13:073
2569128811,61sleep125102-21munin-node19:08:161
2569128811,61sleep125102-21munin-node19:08:161
6752870,6sleep10-21swapper/119:28:391
2748928751,20sleep10-21swapper/120:53:291
2513228729,27sleep30-21swapper/300:13:153
197172876,62sleep10-21swapper/122:13:211
69828614,53sleep2717-21apt21:13:052
174828417,11sleep00-21swapper/019:33:280
174828417,11sleep00-21swapper/019:33:280
778828312,56sleep17732-21latency_hist19:53:071
1437028323,31sleep10-21swapper/120:13:271
1437028323,31sleep10-21swapper/120:13:271
1842428226,31sleep218552-21latency_hist23:53:062
1842428226,31sleep218552-21latency_hist23:53:062
3085228115,50sleep330368-21apt-get19:23:173
2793628115,49sleep227997-21if_eth022:38:272
1165328122,36sleep311799-21latency_hist21:48:073
436928026,27sleep10-21swapper/123:08:061
3060628023,35sleep20-21swapper/221:03:282
72327917,49sleep3775-21unin-run21:13:063
2769527941,18sleep30-21swapper/320:53:353
2063527915,10sleep00-21swapper/022:13:450
1550327925,39sleep315567-21find21:58:333
947627821,40sleep39606-21unixbench_multi21:38:383
370327827,28sleep30-21swapper/323:03:363
2004527813,48sleep220182-21latency_hist23:58:072
2004527813,48sleep220182-21latency_hist23:58:072
2786127727,26sleep10-21swapper/122:38:261
593927617,28sleep20-21swapper/221:28:282
2698427613,46sleep227032-21latency00:18:312
2445927615,47sleep324477-21fschecks_count22:28:263
2231627623,26sleep30-21swapper/318:58:263
1773727625,26sleep10-21swapper/120:23:281
1257827621,35sleep212733-21grep21:48:392
2680127519,26sleep30-21swapper/300:18:283
1987827517,30sleep20-21swapper/220:28:472
1306027519,33sleep313189-21timerwakeupswit23:33:373
478127425,21sleep30-21swapper/321:23:363
3249927418,43sleep332530-21fschecks_count22:53:263
1948727322,29sleep10-21swapper/120:28:251
1900027321,29sleep30-21swapper/320:28:073
3138327212,41sleep131452-21/usr/sbin/munin22:48:351
3138327212,41sleep131452-21/usr/sbin/munin22:48:351
1784827215,39sleep21-21systemd22:08:052
110327220,26sleep10-21swapper/121:13:241
110327220,26sleep10-21swapper/121:13:241
938927114,15sleep20-21swapper/223:23:262
665827113,30sleep10-21swapper/123:13:311
2898227119,38sleep229034-21interrupts20:58:272
1947927121,30sleep30-21swapper/322:13:073
1860727120,33sleep218666-21find22:08:342
542827021,27sleep20-21swapper/221:28:072
2930327016,38sleep329343-21/usr/sbin/munin19:18:283
2412627020,24sleep30-21swapper/319:03:303
1573627016,24sleep20-21swapper/221:58:362
156827031,22sleep31636-21grep21:13:343
156827031,22sleep31636-21grep21:13:343
1448127015,41sleep314558-21cat23:38:333
665926913,43sleep26709-21if_eth019:48:282
2979926917,37sleep329801-21systemd-run00:28:053
2671126916,28sleep10-21swapper/100:18:221
2225726929,25sleep222313-21tune2fs18:58:252
1635726918,35sleep316407-21sed20:18:503
617826813,34sleep26197-21apt-get23:13:162
536726819,35sleep25432-21ps19:43:352
2853326819,34sleep228622-21latency_hist20:58:072
2751526821,28sleep127012-21apt-get19:13:171
2331726821,26sleep30-21swapper/300:08:063
2331726821,26sleep30-21swapper/300:08:063
985926716,24sleep10-21swapper/119:58:331
888126715,24sleep20-21swapper/221:38:252
667126721,24sleep30-21swapper/323:13:343
373526710,42sleep13780-21ntp_kernel_pll_19:38:421
3244126717,33sleep232508-21df19:28:252
180592676,14sleep30-21swapper/323:49:323
180592676,14sleep30-21swapper/323:49:323
439726621,20sleep20-21swapper/223:08:072
2542026617,24sleep20-21swapper/219:08:132
2542026617,24sleep20-21swapper/219:08:132
1731726620,21sleep10-21swapper/123:48:291
1731726620,21sleep10-21swapper/123:48:291
165526616,26sleep30-21swapper/319:33:273
165526616,26sleep30-21swapper/319:33:273
1094026612,24sleep10-21swapper/123:28:261
702626517,25sleep30-21swapper/319:48:373
2515726516,30sleep125152-21apt-get00:13:171
2402726518,30sleep124107-21latency19:03:281
2379626517,29sleep223814-21awk20:43:042
1683226523,21sleep20-21swapper/223:48:052
1683226523,21sleep20-21swapper/223:48:052
3002326415,33sleep230039-21acpi00:28:162
3002326415,33sleep230039-21acpi00:28:162
2501826419,27sleep225046-21munin-run19:08:052
2463626419,25sleep20-21swapper/222:28:292
2244026427,18sleep10-21swapper/118:58:281
2229226425,23sleep30-21swapper/320:38:093
2229226425,23sleep30-21swapper/320:38:093
1779526426,22sleep20-21swapper/220:23:302
156226417,29sleep11557-21apt-get19:33:211
156226417,29sleep11557-21apt-get19:33:211
1408726418,26sleep30-21swapper/321:53:373
613026318,23sleep30-21swapper/321:28:333
280922636,14sleep20-21swapper/219:13:472
257126318,27sleep22567-21apt-get21:18:222
2184426321,24sleep10-21swapper/122:18:391
1645726318,30sleep116494-21sed20:18:521
1629026317,25sleep10-21swapper/123:43:361
1629026317,25sleep10-21swapper/123:43:361
1250026314,22sleep10-21swapper/121:48:381
3164426221,19sleep10-21swapper/121:08:071
1821226219,19sleep10-21swapper/122:08:251
1244626221,25sleep212541-21latency_hist20:08:072
307626113,29sleep10-21swapper/121:18:371
2619226115,18sleep20-21swapper/222:33:222
2428826120,19sleep10-21swapper/120:43:281
2349726129,15sleep023509-21kworker/0:019:03:070
1409726115,20sleep30-21swapper/320:13:093
1409726115,20sleep30-21swapper/320:13:093
875926025,17sleep10-21swapper/119:53:401
700426016,22sleep10-21swapper/121:33:071
470026018,26sleep34724-21fschecks_count23:08:253
3059926012,31sleep130705-21latency_hist22:48:071
2955026011,19sleep10-21swapper/119:18:341
2658826015,22sleep30-21swapper/322:33:473
1132026024,20sleep20-21swapper/223:28:352
81992597,16sleep30-21swapper/321:34:143
81992597,16sleep30-21swapper/321:34:143
2172325917,26sleep321750-21munin-run00:03:053
2098325914,20sleep10-21swapper/120:33:221
2098325914,20sleep10-21swapper/120:33:221
207842599,16sleep20-21swapper/222:15:412
2052325916,26sleep120450-21apt-get23:59:071
976925817,25sleep39808-21/usr/sbin/munin23:23:353
287725818,25sleep22964-21apt-config19:38:062
287725818,25sleep22964-21apt-config19:38:062
2239525813,30sleep322398-21sshd22:21:273
2007825817,19sleep30-21swapper/320:28:523
1993425817,26sleep319983-21irqstats22:13:283
1788625816,26sleep317956-21cut22:08:063
1478925817,26sleep114800-21ls21:58:071
123512586,13sleep30-21swapper/320:07:313
122725815,25sleep11257-21munin-run22:58:051
764225716,25sleep20-21swapper/221:33:322
764225716,25sleep20-21swapper/221:33:322
3209425717,24sleep232112-21idleruntime-cro22:53:052
3209425717,24sleep232112-21idleruntime-cro22:53:052
2831725720,17sleep30-21swapper/322:38:363
2689725718,24sleep226931-21munin-run20:53:062
2252325716,24sleep222530-21modprobe22:23:042
152125715,25sleep21515-21apt-get22:58:222
1427125715,25sleep214267-21apt-get20:13:202
1427125715,25sleep214267-21apt-get20:13:202
802925614,26sleep38025-21apt-get19:53:223
2989325616,23sleep20-21swapper/222:43:362
2889625616,21sleep30-21swapper/319:18:083
2853625617,24sleep228486-21apt-get00:23:482
1907625616,23sleep10-21swapper/123:53:421
1907625616,23sleep10-21swapper/123:53:421
1502725613,26sleep115019-21apt-get21:58:191
1113525614,25sleep111129-21apt-get20:03:211
1034525615,24sleep110335-21apt-get21:43:211
102025614,23sleep21089-21df21:13:212
102025614,23sleep21089-21df21:13:212
451525515,17sleep10-21swapper/121:23:301
334825513,25sleep23343-21apt-get19:38:182
3233925516,23sleep132343-21ntp_states21:08:321
2903925521,18sleep30-21swapper/320:58:283
2323125514,24sleep123268-21ntp_states22:23:311
775725414,24sleep27749-21apt-get23:18:212
1242725415,22sleep212423-21apt-get23:33:192
800025316,22sleep2225-21systemd-journal19:53:172
51325314,19sleep30-21swapper/319:28:383
3031025313,21sleep30-21swapper/321:03:163
2474125317,17sleep30-21swapper/320:43:383
1291325312,25sleep312954-21irqstats20:08:283
3249225214,16sleep20-21swapper/222:53:252
3157925218,14sleep30-21swapper/322:48:373
3157925218,14sleep30-21swapper/322:48:373
268532526,34sleep30-21swapper/320:52:303
1465425216,16sleep10-21swapper/123:38:351
976725116,16sleep30-21swapper/319:58:313
957225113,21sleep10-21swapper/123:23:291
498025113,22sleep34984-21diskstats19:43:243
363525113,22sleep33724-21cat19:38:403
3005025112,21sleep330117-21df00:28:213
3005025112,21sleep330117-21df00:28:213
2381825117,21sleep3284-21dbus-daemon20:43:053
2290025112,25sleep122507-21/usr/sbin/munin20:38:311
2289725116,21sleep222901-21diskstats22:23:242
3048425015,20sleep230374-21latency_hist19:23:082
390124811,16sleep30-21swapper/321:23:073
468624711,20sleep14692-21df_inode23:08:231
3088624614,18sleep20-21swapper/219:23:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional