You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 13:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Mar 11, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
163532729119,7256sleep321740-21kworker/3:123:58:223
2957426068,583sleep229622-21ntpq21:13:482
2963253223,491sleep23012-21/usr/sbin/munin21:33:382
24933249319,457sleep324955-21/usr/sbin/munin00:23:493
13384248958,403sleep00-21swapper/020:23:320
16436240910,343sleep0191-21jbd2/mmcblk2p2-23:58:230
2260523539,77sleep022642-21awk00:18:210
12306227566,112sleep012322-21latency_hist22:03:220
11606225053,89sleep011742-21latency_hist20:18:240
16196222961,99sleep013051-21kworker/0:120:33:210
6080222216,183sleep06227-21unixbench_singl19:58:540
29468222265,92sleep00-21swapper/021:13:460
15360222118,188sleep215401-21systemd-run22:13:222
29338221712,88sleep029443-21tail22:54:010
27567221618,175sleep327568-21cpu00:33:373
2050522098,56sleep00-21swapper/000:08:520
2050522098,56sleep00-21swapper/000:08:520
604922079,128sleep06077-21kernelversion23:23:430
1376220731,122sleep01461-21timerwakeupswit19:44:090
10759220217,164sleep110858-21latency_hist21:58:231
10759220217,164sleep110858-21latency_hist21:58:231
28107219895,90sleep028108-21ntp_states21:08:490
28107219895,90sleep028108-21ntp_states21:08:490
4303219711,131sleep04358-21latency_hist21:38:240
2301321927,171sleep00-21swapper/000:18:410
2557321830,30sleep225577-21fschecks_time19:18:422
953121786,12sleep00-21swapper/021:53:580
953121786,12sleep00-21swapper/021:53:580
2358216824,118sleep30-21swapper/323:13:213
6569216474,66sleep00-21swapper/021:43:490
2417121648,142sleep024196-21awk19:13:450
230921637,14sleep20-21swapper/219:48:392
28954215915,128sleep229099-21latency_hist21:13:232
28954215915,128sleep229099-21latency_hist21:13:232
13865215844,100sleep013962-21latency_hist22:08:230
32624215452,85sleep032580-21apt-get23:03:540
3134215341,98sleep03220-21mailstats23:13:510
28387215310,89sleep028539-21date22:53:240
19198215111,125sleep00-21swapper/022:23:410
22141215062,68sleep00-21swapper/022:33:260
28958214721,107sleep329003-21awk22:53:473
254021465,11sleep00-21swapper/021:31:380
25466214552,80sleep025486-21dpkg22:43:310
8875214419,104sleep10-21swapper/123:33:371
20159214142,27sleep020231-21switchtime20:43:510
20159214142,27sleep020231-21switchtime20:43:510
5178214012,111sleep05297-21timerwakeupswit21:38:520
882321395,11sleep10-21swapper/120:08:521
882321395,11sleep10-21swapper/120:08:521
12476213969,53sleep00-21swapper/023:43:500
12476213969,53sleep00-21swapper/023:43:500
31287213827,88sleep2222-21systemd-journal19:38:212
31287213827,88sleep2222-21systemd-journal19:38:212
356821378,113sleep03725-21cat19:53:250
28596213749,78sleep013441-21snmpd00:35:240
8183213350,66sleep08226-21gpgv20:08:220
11267213316,33sleep111333-21grep21:58:431
5050213019,96sleep05084-21modprobe19:58:210
2935621300,33sleep329360-21unixbench_singl19:28:533
27279212912,18sleep10-21swapper/122:48:451
26091212860,14sleep00-21swapper/000:28:380
2647212632,66sleep22641-21apt-get23:13:362
1806121258,96sleep00-21swapper/020:38:380
1418221219,97sleep00-21swapper/022:08:390
31721211817,88sleep331802-21latency_hist23:03:243
2664521188,22sleep00-21swapper/019:23:220
19543211810,94sleep019651-21sed00:08:240
19543211810,94sleep019651-21sed00:08:240
16348211720,76sleep216352-21cron23:58:212
17224211648,14sleep00-21swapper/020:33:540
7789211448,18sleep07823-21ps23:28:480
26097211411,86sleep326199-21timerandwakeup22:43:503
21241211413,81sleep221251-21/usr/sbin/munin20:48:412
9490211312,19sleep00-21swapper/023:33:530
24510211312,85sleep00-21swapper/020:58:320
852211221,69sleep1869-21grep23:08:231
5780211225,60sleep15774-21apt-get23:23:361
1601211111,75sleep01073-21/usr/sbin/munin23:08:500
30675211045,16sleep030371-21/usr/sbin/munin22:59:090
22337211018,72sleep221856-21apt-get22:33:322
9248210921,68sleep20-21swapper/223:33:462
32180210913,28sleep00-21swapper/019:38:520
32180210913,28sleep00-21swapper/019:38:520
21331210816,72sleep10-21swapper/100:13:371
11968210819,46sleep30-21swapper/323:43:323
11968210819,46sleep30-21swapper/323:43:323
25248210412,17sleep00-21swapper/022:43:230
2343210418,66sleep12448-21fschecks_time19:48:421
15650210417,71sleep215703-21ls22:13:382
12292210412,23sleep00-21swapper/020:18:480
12292210412,23sleep00-21swapper/020:18:480
17926210219,70sleep00-21swapper/000:03:070
22536210010,74sleep022659-21apt-key20:53:230
2071829925,51sleep120232-21/usr/sbin/munin19:03:491
2115629821,58sleep321205-21cpuspeed20:48:383
831429722,54sleep28435-21apt-config20:08:242
2929129721,26sleep10-21swapper/121:13:431
1667929725,47sleep216673-21apt-get23:58:372
3240229522,15sleep00-21swapper/021:23:390
77292949,64sleep00-21swapper/021:48:410
77292949,64sleep00-21swapper/021:48:410
3039529426,13sleep00-21swapper/019:33:470
3039529426,13sleep00-21swapper/019:33:470
2739929411,21sleep00-21swapper/019:23:500
843629311,59sleep30-21swapper/320:08:263
739629320,58sleep37521-21latency_hist21:48:243
2437629311,68sleep224446-21switchtime19:13:512
2444829216,61sleep324575-21timerwakeupswit19:13:523
2425229214,62sleep224402-21cat00:23:232
2834529113,66sleep328383-21latency_hist22:53:233
52572905,39sleep00-21swapper/023:20:200
579728912,61sleep15825-21munin-run21:43:211
2507128920,51sleep125132-21sendmail-msp19:18:221
2268428919,31sleep20-21swapper/200:18:232
319122885,10sleep00-21swapper/021:21:450
2098328810,53sleep10-21swapper/122:28:471
1002028810,19sleep010092-21latency_hist20:13:230
1002028810,19sleep010092-21latency_hist20:13:230
782628724,44sleep17905-21awk23:28:491
50952870,13sleep15089-21munin-run19:58:211
240622865,70sleep00-21swapper/020:56:260
2363028621,45sleep323651-21latency_hist22:38:213
1250328612,60sleep112615-21timerwakeupswit20:18:531
1250328612,60sleep112615-21timerwakeupswit20:18:531
2742328534,25sleep10-21swapper/119:23:511
270302855,68sleep00-21swapper/021:04:210
266182857,12sleep10-21swapper/122:46:391
1034628512,58sleep210359-21fschecks_count20:13:412
938428410,15sleep30-21swapper/321:53:283
938428410,15sleep30-21swapper/321:53:283
2867728426,30sleep20-21swapper/222:53:422
236612848,17sleep00-21swapper/022:38:220
1823228417,38sleep10-21swapper/100:03:361
1823228417,38sleep10-21swapper/100:03:361
179112848,15sleep00-21swapper/022:19:050
179112848,15sleep00-21swapper/022:19:050
1572628457,12sleep00-21swapper/022:13:400
287302836,14sleep00-21swapper/019:28:420
2702328320,41sleep327080-21grep19:23:413
1931628213,54sleep319341-21awk20:43:213
1347128223,34sleep20-21swapper/220:23:382
109112829,56sleep00-21swapper/023:38:490
109112829,56sleep00-21swapper/023:38:490
2801928114,18sleep20-21swapper/200:33:492
2332528113,53sleep123343-21/usr/sbin/munin20:54:011
907028021,28sleep30-21swapper/321:53:243
907028021,28sleep30-21swapper/321:53:243
889228018,45sleep18957-21systemd-run21:53:221
889228018,45sleep18957-21systemd-run21:53:221
2610628025,37sleep226122-21fschecks_count00:28:402
2237328018,44sleep122357-21apt-get22:33:391
2033528020,28sleep20-21swapper/222:28:232
2389427917,46sleep1222-21systemd-journal00:19:391
513027820,35sleep25077-21latency_hist19:58:232
1818027819,36sleep218318-21grep22:19:112
1818027819,36sleep218318-21grep22:19:112
1624727820,16sleep20-21swapper/220:33:232
145112786,12sleep00-21swapper/023:49:390
1108227825,35sleep20-21swapper/221:58:402
2701527718,24sleep20-21swapper/222:48:342
2089927734,25sleep00-21swapper/019:03:520
2023327719,39sleep120354-21grep20:43:521
2023327719,39sleep120354-21grep20:43:521
975527612,51sleep213441-21snmpd20:10:352
975527612,51sleep213441-21snmpd20:10:352
490827612,49sleep15033-21cat23:18:521
2313727624,28sleep20-21swapper/220:53:582
1351727616,27sleep20-21swapper/223:48:322
118427616,45sleep21210-21fschecks_count23:08:402
272627522,30sleep12812-21latency_hist21:33:241
3253627417,23sleep30-21swapper/323:03:533
2616227422,27sleep30-21swapper/321:03:403
1781127418,33sleep117827-21cat20:38:231
3245327317,32sleep21-21systemd23:03:512
3136927317,34sleep131493-21cat19:38:241
3136927317,34sleep131493-21cat19:38:241
257192727,11sleep00-21swapper/000:28:140
2468627221,34sleep124838-21sed22:38:521
2184427218,22sleep20-21swapper/219:08:382
2032627226,25sleep30-21swapper/319:03:393
1100327215,32sleep311128-21sed20:13:553
44927119,28sleep30-21swapper/319:43:233
44927119,28sleep30-21swapper/319:43:233
2984627116,33sleep129906-21modprobe22:58:211
219162708,46sleep00-21swapper/019:08:400
2039127023,23sleep30-21swapper/300:08:513
2039127023,23sleep30-21swapper/300:08:513
1873626924,25sleep10-21swapper/122:23:221
1873626924,25sleep10-21swapper/122:23:221
732426810,43sleep17365-21ntpq20:03:471
446926815,37sleep34582-21sed19:53:523
2732726821,25sleep10-21swapper/100:33:231
2552126811,44sleep30-21swapper/320:59:483
1221326815,27sleep20-21swapper/220:18:462
1221326815,27sleep20-21swapper/220:18:462
668526719,24sleep20-21swapper/220:03:242
2777826715,28sleep10-21swapper/100:33:441
584926610,41sleep35858-21/usr/sbin/munin23:23:383
474426616,36sleep34208-21/usr/sbin/munin23:18:503
341826618,24sleep30-21swapper/323:13:543
244902666,12sleep30-21swapper/322:38:503
1381326621,22sleep20-21swapper/222:08:212
3052326512,27sleep10-21swapper/122:59:061
1005426518,24sleep20-21swapper/221:54:122
1005426518,24sleep20-21swapper/221:54:122
3158026415,33sleep131581-21munin-node19:38:331
3158026415,33sleep131581-21munin-node19:38:331
2578026414,34sleep125882-21sed00:28:231
2310026415,34sleep123102-21cron19:13:211
2174026413,36sleep121819-21ls20:48:521
1648026417,16sleep10-21swapper/123:58:241
1323526426,20sleep10-21swapper/122:03:521
1082826419,22sleep10-21swapper/123:38:461
1082826419,22sleep10-21swapper/123:38:461
790726315,22sleep20-21swapper/223:28:502
2648126320,25sleep30-21swapper/300:28:493
1786826313,26sleep30-21swapper/322:19:043
1786826313,26sleep30-21swapper/322:19:043
563926212,24sleep30-21swapper/319:58:453
2894426215,30sleep228964-21/usr/sbin/munin19:28:492
2510626213,26sleep20-21swapper/200:23:522
2205726223,19sleep30-21swapper/322:33:253
187826220,18sleep20-21swapper/221:28:492
1464526215,22sleep30-21swapper/322:08:513
71002616,14sleep10-21swapper/121:45:111
3208226118,20sleep10-21swapper/121:23:231
1996226117,20sleep10-21swapper/100:08:411
1996226117,20sleep10-21swapper/100:08:411
1469926112,33sleep314843-21cat20:28:243
2575626020,25sleep125783-21awk19:18:461
2503326020,20sleep10-21swapper/120:58:491
2483926015,20sleep20-21swapper/222:38:532
2062626018,19sleep20-21swapper/219:03:462
2032226015,26sleep220390-21/usr/sbin/munin00:08:502
2032226015,26sleep220390-21/usr/sbin/munin00:08:502
1485426020,21sleep10-21swapper/123:53:231
584825915,22sleep30-21swapper/321:43:233
2594325921,23sleep326030-21switchtime19:18:533
1714625912,32sleep317184-21processes23:58:503
1534025920,18sleep30-21swapper/320:28:483
81925816,16sleep30-21swapper/319:43:433
741025818,20sleep30-21swapper/323:28:393
616725816,27sleep26177-21/usr/sbin/munin21:43:392
550225813,30sleep15580-21fw_packets19:58:431
335925815,20sleep30-21swapper/321:33:473
3212025817,18sleep20-21swapper/221:23:242
2979225814,25sleep10-21swapper/119:33:241
2086425824,19sleep30-21swapper/320:48:223
2086425824,19sleep30-21swapper/320:48:223
1928025814,22sleep20-21swapper/200:05:392
1928025814,22sleep20-21swapper/200:05:392
1899725813,25sleep319016-21apt-get22:23:323
895825714,26sleep29069-21dpkg21:53:232
895825714,26sleep29069-21dpkg21:53:232
502525713,25sleep20-21swapper/221:38:502
301382577,11sleep30-21swapper/321:14:213
2196325716,26sleep222056-21apt-config22:33:242
1864025712,26sleep218736-21/usr/sbin/munin20:38:522
1418725721,15sleep10-21swapper/122:08:401
130392577,11sleep10-21swapper/123:45:361
130392577,11sleep10-21swapper/123:45:361
1244825718,21sleep212454-21ntp_states23:43:472
1244825718,21sleep212454-21ntp_states23:43:472
117525714,25sleep11182-21sshd23:08:391
887225613,26sleep38868-21apt-get23:33:353
3197425621,16sleep20-21swapper/219:38:462
3197425621,16sleep20-21swapper/219:38:462
2091825616,15sleep20-21swapper/220:48:252
2091825616,15sleep20-21swapper/220:48:252
1823425619,18sleep30-21swapper/300:03:383
1823425619,18sleep30-21swapper/300:03:383
118025617,17sleep10-21swapper/121:28:241
1175825614,19sleep30-21swapper/323:43:243
1175825614,19sleep30-21swapper/323:43:243
1069125618,23sleep210720-21irqstats23:38:432
1069125618,23sleep210720-21irqstats23:38:432
69225515,24sleep2688-21apt-get19:43:342
296225512,25sleep12956-21apt-get21:33:371
2764525512,25sleep327640-21apt-get21:08:363
2764525512,25sleep327640-21apt-get21:08:363
1315125517,25sleep30-21swapper/320:23:233
1315125517,25sleep30-21swapper/320:23:233
1276425513,19sleep30-21swapper/322:03:423
118425513,27sleep11249-21awk19:43:531
997325414,28sleep310018-21unin-run20:13:223
997325414,28sleep310018-21unin-run20:13:223
624825412,18sleep20-21swapper/223:23:502
3066225418,20sleep30-21swapper/319:33:543
3066225418,20sleep30-21swapper/319:33:543
2659025419,18sleep20-21swapper/221:03:512
2315125413,17sleep30-21swapper/319:13:233
1699025416,21sleep317077-21gpgv22:18:233
1694025415,24sleep316946-21ntp_states20:33:493
1399125416,22sleep114068-21grep20:23:501
1099325413,24sleep310990-21apt-get21:58:333
1359025313,21sleep30-21swapper/323:48:373
3232325211,23sleep232390-21df21:23:372
3077225212,24sleep330769-21apt-get21:18:333
2693425212,24sleep226928-21apt-get19:23:362
2675125213,18sleep20-21swapper/222:48:212
190025217,17sleep30-21swapper/321:28:503
1728225214,18sleep10-21swapper/123:58:521
1025925212,25sleep110254-21apt-get20:13:361
307825113,23sleep13101-21ntp_states23:13:471
3053425114,18sleep20-21swapper/222:59:072
2689325112,21sleep126986-21users21:03:541
2292425112,22sleep322919-21apt-get00:18:363
2189425016,17sleep30-21swapper/300:13:523
1783025018,16sleep20-21swapper/220:38:242
167192506,32sleep10-21swapper/122:15:471
112725012,20sleep31156-21munin-run21:28:223
2478824915,16sleep20-21swapper/220:58:422
2339024911,16sleep30-21swapper/320:54:033
190982498,18sleep10-21swapper/122:23:381
1502524919,16sleep10-21swapper/120:28:421
435624710,15sleep20-21swapper/223:18:412
1812924713,16sleep10-21swapper/120:38:391
277942468,16sleep10-21swapper/121:08:421
277942468,16sleep10-21swapper/121:08:421
224324511,17sleep30-21swapper/319:48:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional