You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-22 - 09:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Mon Dec 22, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291624916,473sleep02954-21ntpq00:09:580
17300245320,417sleep317317-21/usr/sbin/munin19:44:533
23881241526,330sleep023922-21ntpq21:49:540
22862237558,302sleep022874-21modprobe20:04:290
22862237558,302sleep022874-21modprobe20:04:290
29685235423,276sleep00-21swapper/022:09:410
995423377,320sleep0369-21snmpd19:20:550
32511233331,275sleep232533-21munin-plugin-st22:19:282
31652231155,242sleep031687-21ntpq22:14:580
12701229724,256sleep00-21swapper/022:59:280
12701229724,256sleep00-21swapper/022:59:280
1060226670,177sleep01208-21grep22:20:030
2757322656,248sleep0369-21snmpd22:00:320
201002257118,82sleep020103-21cron21:39:280
23836225612,229sleep123950-21sed20:05:061
23836225612,229sleep123950-21sed20:05:061
15772224910,200sleep015807-21grep21:24:460
15772224910,200sleep015807-21grep21:24:460
25703222255,102sleep025820-21cat21:54:590
7777222114,192sleep17821-21ntpq19:15:021
5463222114,188sleep24922-21apt-get19:09:422
11879222118,187sleep112011-21latency_hist19:29:321
2917522187,141sleep029084-21kworker/u8:220:24:320
14193221012,186sleep3369-21snmpd23:03:233
14193221012,186sleep3369-21snmpd23:03:233
2880422075,162sleep00-21swapper/023:49:470
27892220311,134sleep028046-21unixbench_multi23:45:030
11116220352,67sleep010838-1kworker/0:2H22:54:310
3888220027,152sleep33961-21/usr/sbin/munin00:14:423
1533221960,53sleep05085-21cyclictest19:39:320
6341219219,159sleep16392-21latency_hist22:39:301
3038021848,160sleep030442-21grep23:54:480
9548218223,120sleep00-21swapper/000:30:010
7058218215,151sleep27132-21find20:55:152
5657217932,128sleep05760-21timerwakeupswit22:35:010
3493217715,148sleep23536-21ntpq20:44:572
12358217516,116sleep0228-21systemd-journal21:14:280
1474821695,152sleep00-21swapper/019:38:070
15246216722,119sleep215396-21sed23:05:002
29420216443,68sleep01-21systemd22:09:280
5906216314,133sleep05933-21munin-run20:54:290
2175921608,15sleep00-21swapper/023:25:000
2175921608,15sleep00-21swapper/023:25:000
9430215811,132sleep09563-21latency_hist22:49:310
3548215510,132sleep03615-21fschecks_time22:29:480
29339215212,118sleep00-21swapper/020:24:400
9310215012,65sleep09325-21latency_hist21:04:300
9310215012,65sleep09325-21latency_hist21:04:300
1824721508,104sleep018287-21latency_hist19:49:300
29484214922,33sleep10-21swapper/120:24:491
22231214917,115sleep222262-21latency21:44:512
1979321499,78sleep019884-21cat19:54:310
7222214738,57sleep00-21swapper/000:24:320
20118214714,118sleep020195-21wc23:20:280
20118214714,118sleep020195-21wc23:20:280
10020214629,77sleep00-21swapper/021:04:590
10020214629,77sleep00-21swapper/021:04:590
11535214514,55sleep111540-21entropy22:54:491
11535214514,55sleep111540-21entropy22:54:491
496621437,12sleep00-21swapper/000:15:110
496621437,12sleep00-21swapper/000:15:110
3062421418,12sleep10-21swapper/122:10:061
18609214017,108sleep318577-21latency_hist21:34:303
4419213811,69sleep00-21swapper/020:49:320
7104213765,21sleep00-21swapper/019:14:330
2728421345,12sleep20-21swapper/223:44:492
11439213411,62sleep011557-21grep00:35:230
985921300,13sleep30-21swapper/300:30:043
2134821306,10sleep00-21swapper/021:40:170
763021296,28sleep00-21swapper/022:41:470
1675212623,83sleep30-21swapper/322:24:303
21693212521,79sleep221779-21fschecks_time19:59:462
21693212521,79sleep221779-21fschecks_time19:59:462
21049212019,64sleep00-21swapper/019:56:250
850621195,52sleep00-21swapper/000:27:150
14396211812,93sleep314453-21idleruntime21:19:483
1385121186,27sleep00-21swapper/021:18:400
27616211629,32sleep027667-21latency_hist20:19:310
27616211629,32sleep027667-21latency_hist20:19:310
2242921149,65sleep022493-21ps21:44:570
4996211321,73sleep25115-21latency_hist19:09:302
12648211232,55sleep212645-21apt-get21:14:422
32604211141,53sleep00-21swapper/000:00:330
32604211141,53sleep00-21swapper/000:00:330
25443211116,81sleep225478-21sed21:54:532
792621090,5sleep20-21swapper/222:44:312
15405210823,59sleep315398-21apt-get19:39:443
3219821077,65sleep032207-21munin-run20:34:290
6838210618,63sleep37100-21grep19:14:323
25347210617,70sleep225376-21munin-plugin-st23:39:282
1837921065,38sleep00-21swapper/021:32:090
1741221068,85sleep017530-21latency_hist23:14:310
762521057,14sleep20-21swapper/219:14:592
22262210232,42sleep322285-21modprobe23:29:273
22262210232,42sleep322285-21modprobe23:29:273
13479210227,62sleep012995-21/usr/sbin/munin22:59:570
13479210227,62sleep012995-21/usr/sbin/munin22:59:570
6666210112,20sleep16723-21irqstats20:55:061
11110210130,36sleep20-21swapper/221:09:452
10720210119,62sleep110234-21apt-get00:34:391
9733210013,63sleep28952-21/usr/sbin/munin19:20:042
32156210025,47sleep332180-21munin-run20:34:283
31376210019,66sleep331461-21perl22:14:533
1477021005,80sleep00-21swapper/021:19:570
1857829934,12sleep00-21swapper/023:15:150
490129635,41sleep11-21systemd19:09:291
1463729622,14sleep00-21swapper/023:04:450
2495629420,13sleep00-21swapper/020:09:580
2495629420,13sleep00-21swapper/020:09:580
238492946,17sleep00-21swapper/023:34:290
546829319,15sleep00-21swapper/019:09:440
261302939,18sleep00-21swapper/023:39:580
895129214,51sleep30-21swapper/319:19:413
2418629216,59sleep124234-21fschecks_count23:34:501
13482927,20sleep01393-21ntpq00:04:570
13482927,20sleep01393-21ntpq00:04:570
598129114,64sleep36013-21tune2fs00:19:593
1502229017,51sleep215225-21latency_hist19:39:312
139802909,69sleep114000-21latency_hist21:19:301
846228815,56sleep18470-21/usr/sbin/munin20:59:591
846228815,56sleep18470-21/usr/sbin/munin20:59:591
105202888,16sleep00-21swapper/019:24:470
405928717,46sleep33964-21apt-get00:14:463
405928717,46sleep33964-21apt-get00:14:463
2204828723,39sleep30-21swapper/321:44:483
19192878,35sleep00-21swapper/022:24:430
19192878,35sleep00-21swapper/022:24:430
812928614,59sleep28206-21fschecks_time20:59:522
812928614,59sleep28206-21fschecks_time20:59:522
66012868,16sleep00-21swapper/020:55:050
2767828618,41sleep127742-21ps23:45:001
2575728616,49sleep325835-21grep23:39:493
1456828610,58sleep30-21swapper/323:04:443
132528621,41sleep11372-21latency_hist20:39:311
3242528527,40sleep00-21swapper/020:34:420
2422628518,55sleep00-21swapper/020:07:070
2422628518,55sleep00-21swapper/020:07:070
2238328521,49sleep122509-21/usr/sbin/munin20:00:061
2238328521,49sleep122509-21/usr/sbin/munin20:00:061
20272848,40sleep00-21swapper/020:39:590
1898728421,44sleep219074-21apt-config23:19:292
154032849,55sleep10-21swapper/119:39:421
135982848,58sleep30-21swapper/319:34:433
239512830,8sleep20-21swapper/220:05:062
239512830,8sleep20-21swapper/220:05:062
751928127,25sleep20-21swapper/200:24:502
1855728116,47sleep118591-21awk21:34:281
2065528017,46sleep120656-21sshd23:23:361
2065528017,46sleep120656-21sshd23:23:361
560727919,42sleep15627-21fschecks_count19:10:301
2218427926,29sleep30-21swapper/320:00:033
2218427926,29sleep30-21swapper/320:00:033
12462796,19sleep20-21swapper/200:04:552
12462796,19sleep20-21swapper/200:04:552
115252797,17sleep10-21swapper/121:09:561
327727817,30sleep10-21swapper/100:10:111
2974127820,27sleep20-21swapper/220:24:542
2863027817,40sleep228663-21/usr/sbin/munin22:04:582
2707627813,49sleep127113-21ps21:59:561
2624127823,39sleep226316-21grep23:40:002
1091827821,39sleep110941-21dpkg22:54:281
3210327721,25sleep30-21swapper/300:00:203
3210327721,25sleep30-21swapper/300:00:203
290527720,25sleep10-21swapper/120:44:321
2475427712,44sleep224756-21cron21:54:282
865627620,42sleep38700-21munin-run19:19:293
3152727616,43sleep231646-21/usr/sbin/munin20:29:592
2507427617,30sleep20-21swapper/220:10:002
2507427617,30sleep20-21swapper/220:10:002
2077127623,27sleep20-21swapper/223:24:302
2077127623,27sleep20-21swapper/223:24:302
1648027624,26sleep10-21swapper/121:25:031
1648027624,26sleep10-21swapper/121:25:031
1496527615,40sleep215106-21grep21:19:592
971427419,41sleep29827-21unixbench_singl00:30:042
429927413,42sleep14333-21/usr/sbin/munin00:14:561
429927413,42sleep14333-21/usr/sbin/munin00:14:561
2418127420,13sleep00-21swapper/023:34:490
2342127417,28sleep30-21swapper/320:04:563
2342127417,28sleep30-21swapper/320:04:563
1942827418,25sleep30-21swapper/321:35:023
34612737,49sleep30-21swapper/322:29:433
3007927324,31sleep330103-21netstat22:09:563
2789627318,32sleep30-21swapper/320:19:463
2424427318,24sleep30-21swapper/321:50:013
1357127324,29sleep213648-21switchtime22:59:592
1357127324,29sleep213648-21switchtime22:59:592
123292737,49sleep00-21swapper/019:29:510
123292737,49sleep00-21swapper/019:29:510
2460227224,34sleep31-21systemd23:35:013
218927210,46sleep12274-21latency22:24:521
218927210,46sleep12274-21latency22:24:521
2090827226,26sleep10-21swapper/121:40:011
2040227214,26sleep30-21swapper/319:54:553
1493127220,23sleep10-21swapper/119:39:301
1099627225,29sleep311110-21apt-config22:54:303
1049227214,25sleep30-21swapper/319:24:423
658427124,25sleep30-21swapper/322:39:443
654127120,35sleep36598-21fschecks_count20:55:043
3145627123,28sleep10-21swapper/120:29:581
726627015,40sleep27374-21date00:24:342
1213727015,36sleep3372-21sshd21:11:363
987526914,37sleep39925-21memory21:04:543
987526914,37sleep39925-21memory21:04:543
3252026916,36sleep232597-21/usr/sbin/munin00:00:312
3252026916,36sleep232597-21/usr/sbin/munin00:00:312
2030326912,17sleep10-21swapper/119:54:531
2017726912,16sleep20-21swapper/221:39:302
1281626923,22sleep20-21swapper/222:59:302
1281626923,22sleep20-21swapper/222:59:302
1098226924,26sleep20-21swapper/219:25:012
347626822,34sleep13492-21netstat20:44:561
2832626821,25sleep10-21swapper/120:20:011
1888926813,39sleep318925-21ntpq19:49:543
1702426813,23sleep20-21swapper/223:10:042
3076026716,37sleep130824-21ps23:54:571
2172226721,23sleep20-21swapper/221:44:312
1322526718,25sleep10-21swapper/122:59:501
1322526718,25sleep10-21swapper/122:59:501
722826617,27sleep10-21swapper/100:24:331
237982665,15sleep10-21swapper/123:34:221
2231526616,36sleep122396-21date23:29:291
2231526616,36sleep122396-21date23:29:291
2217226618,20sleep10-21swapper/121:44:501
3163226522,17sleep20-21swapper/223:59:302
2781326510,13sleep30-21swapper/323:45:023
2357826520,21sleep10-21swapper/121:49:471
133526522,17sleep30-21swapper/300:04:563
133526522,17sleep30-21swapper/300:04:563
603626421,19sleep20-21swapper/220:54:312
2715626415,33sleep227165-21/usr/sbin/munin20:15:132
2715626415,33sleep227165-21/usr/sbin/munin20:15:132
261552647,43sleep326156-21sshd21:57:363
2011426413,35sleep120176-21unin-run21:39:291
200626420,26sleep20-21swapper/222:24:492
200626420,26sleep20-21swapper/222:24:492
906426319,23sleep10-21swapper/119:19:511
812626310,40sleep28169-21cpu22:44:442
366926317,24sleep30-21swapper/320:45:033
3089826320,25sleep30-21swapper/323:54:593
281426312,36sleep22834-21munin-run20:44:292
1828826320,22sleep10-21swapper/119:49:311
167222639,13sleep20-21swapper/221:25:362
167222639,13sleep20-21swapper/221:25:362
719426222,24sleep10-21swapper/122:40:031
3263426219,28sleep332687-21/usr/sbin/munin20:34:533
322026220,18sleep30-21swapper/322:29:303
322026220,18sleep30-21swapper/322:29:303
2921526221,22sleep30-21swapper/323:49:583
2538626218,20sleep10-21swapper/120:10:111
2538626218,20sleep10-21swapper/120:10:111
1973826213,28sleep319774-21irqstats23:20:193
1973826213,28sleep319774-21irqstats23:20:193
1214626222,26sleep112193-21cut19:29:481
1214626222,26sleep112193-21cut19:29:481
594926114,21sleep10-21swapper/120:54:301
2932726114,23sleep10-21swapper/123:50:011
2668526116,29sleep226694-21df_inode21:59:452
1907626117,23sleep30-21swapper/323:19:303
1085626120,25sleep210866-21munin-run21:09:292
1085626120,25sleep210866-21munin-run21:09:292
60712607,10sleep20-21swapper/222:36:572
3133326019,22sleep10-21swapper/122:14:501
1185726015,29sleep30-21swapper/300:37:363
472525919,19sleep20-21swapper/200:15:092
472525919,19sleep20-21swapper/200:15:092
365125917,25sleep03647-21apt-get19:04:440
3183725911,35sleep131618-40mandb23:59:331
3183725911,35sleep131618-40mandb23:59:331
277852596,41sleep30-21swapper/322:03:183
1765225912,25sleep10-21swapper/123:14:441
1227025918,25sleep312327-21irqstats19:29:503
1227025918,25sleep312327-21irqstats19:29:503
987425815,25sleep29914-21/usr/sbin/munin22:49:492
44192585,11sleep20-21swapper/222:30:392
2606725820,19sleep10-21swapper/123:39:551
875225721,18sleep30-21swapper/321:00:053
875225721,18sleep30-21swapper/321:00:053
820925715,28sleep18216-21/usr/sbin/munin22:44:471
2380525718,25sleep31-21systemd23:34:283
1791525715,26sleep218013-21sed21:30:012
96825615,25sleep1965-21apt-get00:04:421
96825615,25sleep1965-21apt-get00:04:421
812225616,24sleep38119-21apt-get22:44:413
780125618,20sleep30-21swapper/300:24:563
2850625616,17sleep20-21swapper/220:20:062
2661425613,17sleep30-21swapper/320:14:583
2661425613,17sleep30-21swapper/320:14:583
2106525619,18sleep30-21swapper/321:40:033
2011925620,16sleep20-21swapper/219:54:492
1540425615,24sleep214885-21apt-get19:39:432
137425616,18sleep20-21swapper/220:39:322
1849225514,24sleep118486-21apt-get19:49:441
1659825517,23sleep116680-21mailstats23:09:591
2519225415,15sleep30-21swapper/320:10:053
2519225415,15sleep30-21swapper/320:10:053
2143325415,22sleep221464-21ntp_states23:24:542
2143325415,22sleep221464-21ntp_states23:24:542
1928725414,22sleep219390-21users19:50:012
154025413,25sleep31559-21apt-get20:39:413
1422525413,24sleep114219-21apt-get21:19:421
904425312,24sleep19050-21df_inode00:29:461
431525315,22sleep14362-21users22:30:011
37325315,22sleep2364-21apt-get22:19:442
2998425317,21sleep230078-21memory22:09:552
2936325312,24sleep329359-21apt-get20:24:433
1619725314,18sleep30-21swapper/323:09:493
1148725316,22sleep311523-21df22:54:473
1148725316,22sleep311523-21df22:54:473
500825212,24sleep35003-21apt-get22:34:433
379925220,17sleep30-21swapper/319:04:513
2875325212,23sleep228747-21apt-get23:49:452
15342524,37sleep10-21swapper/122:23:151
1464725213,24sleep114651-21diskstats23:04:461
1307825212,23sleep313102-21ntp_states21:14:553
1252625214,23sleep212590-21ps19:29:582
1252625214,23sleep212590-21ps19:29:582
1040725213,23sleep110460-21users21:05:041
1040725213,23sleep110460-21users21:05:041
516125015,16sleep30-21swapper/320:49:593
292192507,26sleep10-21swapper/122:06:511
2669025014,20sleep10-21swapper/120:15:001
2669025014,20sleep10-21swapper/120:15:001
2541325011,19sleep10-21swapper/121:54:521
2816524810,21sleep328161-21apt-get22:04:433
2315924812,23sleep20-21swapper/223:29:592
1573624812,21sleep315737-21sshd21:24:363
1573624812,21sleep315737-21sshd21:24:363
251524612,18sleep20-21swapper/200:09:442
1694924614,19sleep20-21swapper/219:44:442
108542469,17sleep20-21swapper/200:35:002
74292439,17sleep10-21swapper/100:24:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional