You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-26 - 09:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun Oct 26, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30851261285,6069sleep027892-21kworker/0:121:07:380
30851261285,6069sleep027892-21kworker/0:121:07:380
14211224656,2443sleep029319-1kworker/0:0H20:15:280
13610216199,1597sleep38693-21kworker/3:020:14:383
17645279512,767sleep117691-21/usr/sbin/munin23:50:391
14181276923,728sleep314248-21df23:40:313
18537274410,693sleep018612-40gzip23:55:140
2705227336,693sleep027088-21ntpq20:55:420
5504269285,587sleep00-21swapper/019:48:380
26312248583,387sleep026359-21ntpq19:10:420
29800248313,455sleep329793-21apt-get21:05:313
29800248313,455sleep329793-21apt-get21:05:313
15733247076,114sleep00-21swapper/023:45:330
27585244216,410sleep027667-21fschecks_time19:15:340
2480824067,389sleep0362-21snmpd20:50:130
28135240341,309sleep028272-21unixbench_multi22:40:470
28135240341,309sleep028272-21unixbench_multi22:40:470
5152232635,117sleep05146-21munin-run21:30:150
5152232635,117sleep05146-21munin-run21:30:150
17324231019,273sleep217391-21df20:25:352
2041922876,27sleep00-21swapper/020:35:270
253712240112,23sleep00-21swapper/020:50:370
10377223927,51sleep023702-21cyclictest21:45:340
10377223927,51sleep023702-21cyclictest21:45:340
30681223544,165sleep00-21swapper/019:25:360
3564223417,201sleep23711-21awk19:40:482
6990223256,138sleep07142-21date21:35:160
350222729,172sleep0505-21unixbench_multi22:55:450
11843222328,161sleep011906-21cpuspeed21:50:310
11843222328,161sleep011906-21cpuspeed21:50:310
31809222214,151sleep031846-21modprobe22:55:130
31809222214,151sleep031846-21modprobe22:55:130
2928022220,13sleep023702-21cyclictest00:25:500
9319221846,122sleep09408-21switchtime21:40:450
9319221846,122sleep09408-21switchtime21:40:450
21769221348,105sleep021911-21latency_hist00:05:150
14461221391,30sleep014483-21kernelversion23:40:480
13037221231,129sleep00-21swapper/023:36:120
13037221231,129sleep00-21swapper/023:36:120
949220511,112sleep0978-21modprobe19:35:130
949220511,112sleep0978-21modprobe19:35:130
1369522026,181sleep00-21swapper/021:55:400
198522008,180sleep02033-21unin-run21:20:140
22524219922,122sleep00-21swapper/022:25:160
23907219357,76sleep023955-21sshd00:10:410
23907219357,76sleep023955-21sshd00:10:410
17928219262,22sleep00-21swapper/020:25:500
12392218210,105sleep00-21swapper/020:10:310
9294217615,41sleep00-21swapper/020:00:290
773421747,105sleep00-21swapper/019:55:290
22357217326,136sleep022372-21grep20:40:400
29443217011,141sleep329478-21ntp_kernel_pll_19:20:423
8730216929,28sleep00-21swapper/023:25:130
19559216923,19sleep00-21swapper/023:55:510
10659215542,94sleep00-21swapper/023:30:340
10659215542,94sleep00-21swapper/023:30:340
23746215114,99sleep023899-21latency_hist19:05:160
25984214954,81sleep023702-21cyclictest22:35:340
4516214620,100sleep24667-21grep21:25:452
4516214620,100sleep24667-21grep21:25:452
7290214551,83sleep07318-21grep21:35:370
7290214551,83sleep07318-21grep21:35:370
7345214435,71sleep07487-21gpgv19:55:160
9021214011,114sleep09049-21cpu23:25:290
3016213855,69sleep03149-21sed21:20:460
32228213623,23sleep032334-21cat21:11:090
32228213623,23sleep032334-21cat21:11:090
31231213424,83sleep131386-21grep22:50:451
31231213424,83sleep131386-21grep22:50:451
1851621348,110sleep00-21swapper/022:10:420
1652621339,41sleep00-21swapper/022:05:290
26702213120,73sleep226827-21cat00:20:162
26702213120,73sleep226827-21cat00:20:162
2899212910,27sleep00-21swapper/019:40:340
24776212923,17sleep00-21swapper/022:30:410
24776212923,17sleep00-21swapper/022:30:410
16346212910,84sleep016434-21switchtime20:20:490
25071212616,88sleep125216-21/usr/sbin/munin22:30:461
25071212616,88sleep125216-21/usr/sbin/munin22:30:461
1214212216,87sleep11276-21/usr/sbin/munin21:15:471
15856211920,36sleep115895-21cut20:20:381
5627211725,74sleep25643-21idleruntime-cro19:50:132
22065211750,56sleep022086-21fschecks_count00:05:360
990211221,77sleep11021-21sed23:00:141
504721127,93sleep1195-21jbd2/mmcblk2p2-21:29:441
504721127,93sleep1195-21jbd2/mmcblk2p2-21:29:441
2749221109,58sleep00-21swapper/000:20:460
2749221109,58sleep00-21swapper/000:20:460
1966821106,87sleep00-21swapper/022:15:310
16556211012,79sleep216671-21users20:20:512
25286210921,23sleep025293-21irqstats19:10:170
20212210911,82sleep120282-21wc22:15:441
1542210912,34sleep00-21swapper/019:35:370
409821077,87sleep04156-21latency_hist23:10:150
19133210728,64sleep019159-21kernelversion20:30:380
2901921057,55sleep00-21swapper/019:20:270
17882210512,77sleep317961-21wc23:50:453
1600521056,69sleep00-21swapper/022:02:180
1600521056,69sleep00-21swapper/022:02:180
12358210514,73sleep111851-21apt-get23:35:241
12358210514,73sleep111851-21apt-get23:35:241
10930210511,76sleep210990-21/usr/sbin/munin20:05:362
5270210416,34sleep00-21swapper/023:11:040
28606210427,18sleep028642-21ntpq21:00:390
26266210420,61sleep125875-21/usr/sbin/munin22:35:391
250321044,13sleep20-21swapper/221:20:362
4047210320,65sleep2419-21cron23:10:132
2666321017,83sleep026673-21munin-run00:20:140
2666321017,83sleep026673-21munin-run00:20:140
4850210015,64sleep34914-21find19:45:463
253721006,13sleep00-21swapper/023:05:140
11063210010,34sleep00-21swapper/020:05:380
1554429916,70sleep215622-21switchtime22:00:512
1554429916,70sleep215622-21switchtime22:00:512
790829812,68sleep28047-21/usr/sbin/munin21:35:512
790829812,68sleep28047-21/usr/sbin/munin21:35:512
82172966,14sleep20-21swapper/219:55:592
3028429613,64sleep330353-21sort00:30:393
2533329618,58sleep124822-21apt-get00:15:261
2533329618,58sleep124822-21apt-get00:15:261
1365729629,45sleep213674-21cron20:15:132
1221229616,56sleep10-21swapper/121:50:401
1221229616,56sleep10-21swapper/121:50:401
290032928,46sleep00-21swapper/022:45:310
290032928,46sleep00-21swapper/022:45:310
2617929216,57sleep126209-21latency19:10:391
746329113,61sleep37485-21kernelversion21:35:403
746329113,61sleep37485-21kernelversion21:35:403
62329115,61sleep3630-21modprobe22:56:133
3038429116,24sleep10-21swapper/100:30:411
2631529111,18sleep10-21swapper/120:54:381
640929014,17sleep00-21swapper/023:15:440
382329012,63sleep33892-21df21:25:303
382329012,63sleep33892-21df21:25:303
1966529011,65sleep319666-21/usr/sbin/munin22:15:303
1503229010,64sleep115058-21df22:00:371
1503229010,64sleep115058-21df22:00:371
1199229012,60sleep111854-21latency_hist23:35:161
1199229012,60sleep111854-21latency_hist23:35:161
2869028915,56sleep228696-21/usr/sbin/munin21:00:412
2490028918,55sleep225009-21sed00:15:152
2490028918,55sleep225009-21sed00:15:152
1906028813,58sleep119122-21cat23:55:391
3011228421,20sleep00-21swapper/000:30:360
2536128423,39sleep325354-21apt-get00:15:283
2536128423,39sleep325354-21apt-get00:15:283
2972128328,35sleep10-21swapper/100:30:121
1628728339,26sleep316263-21latency_hist22:05:163
1628728339,26sleep316263-21latency_hist22:05:163
900328228,35sleep1284-21dbus-daemon20:00:141
2754028210,57sleep127567-21df19:15:311
2415428215,47sleep124210-21/usr/sbin/munin19:05:381
86042809,51sleep30-21swapper/319:56:043
2968728023,34sleep129841-21unixbench_multi22:45:491
2968728023,34sleep129841-21unixbench_multi22:45:491
2243328024,30sleep10-21swapper/119:00:331
1286828021,33sleep10-21swapper/120:10:441
3031127918,38sleep330431-21latency_hist22:50:153
3031127918,38sleep330431-21latency_hist22:50:153
127227925,29sleep10-21swapper/119:35:301
1243827914,46sleep312523-21awk21:50:463
1243827914,46sleep312523-21awk21:50:463
2977227813,19sleep30-21swapper/300:30:143
2426227816,46sleep324392-21unixbench_singl00:10:483
2426227816,46sleep324392-21unixbench_singl00:10:483
1721027813,49sleep117351-21unixbench_multi22:05:461
2327327713,46sleep323345-21mailstats22:25:473
2086627711,14sleep20-21swapper/200:00:452
1700827721,25sleep20-21swapper/222:05:432
979627613,49sleep2284-21dbus-daemon20:00:442
90192756,52sleep30-21swapper/323:25:283
647527513,50sleep16539-21/usr/sbin/munin23:15:451
275762756,51sleep30-21swapper/319:15:333
2706327521,31sleep20-21swapper/200:20:362
2706327521,31sleep20-21swapper/200:20:362
2394527532,28sleep323938-21apt-get19:05:323
2659027424,26sleep20-21swapper/219:10:482
2100027418,36sleep321068-21switchtime20:35:463
279227316,15sleep30-21swapper/323:05:333
2551427330,24sleep30-21swapper/319:10:203
239232736,11sleep00-21swapper/022:28:470
2023927321,26sleep20-21swapper/220:35:172
193322736,14sleep30-21swapper/323:55:453
1215127323,31sleep112276-21grep20:10:161
407727219,35sleep34093-21modprobe19:45:143
3188927212,44sleep231970-21latency_hist22:55:152
3188927212,44sleep231970-21latency_hist22:55:152
2691727218,26sleep10-21swapper/100:20:311
2691727218,26sleep10-21swapper/100:20:311
2469227219,25sleep20-21swapper/222:30:392
2469227219,25sleep20-21swapper/222:30:392
1801327213,44sleep117295-21/usr/sbin/munin20:25:511
1707827216,27sleep30-21swapper/320:25:163
11602726,14sleep20-21swapper/221:15:432
3086327114,44sleep330927-21wc19:25:393
2714327115,15sleep30-21swapper/320:55:453
2515027111,26sleep20-21swapper/220:50:312
747427023,25sleep20-21swapper/223:20:312
3233727018,27sleep10-21swapper/121:11:101
3233727018,27sleep10-21swapper/121:11:101
3224027011,44sleep232348-21perl19:30:352
3224027011,44sleep232348-21perl19:30:352
2092227026,24sleep20-21swapper/222:20:132
3222726923,20sleep10-21swapper/119:30:341
3222726923,20sleep10-21swapper/119:30:341
2772026917,14sleep30-21swapper/322:40:383
2772026917,14sleep30-21swapper/322:40:383
2721826912,44sleep126655-21/usr/sbin/munin20:55:471
2481626919,31sleep124833-21/usr/sbin/munin00:15:141
2481626919,31sleep124833-21/usr/sbin/munin00:15:141
1907426922,29sleep319131-21/usr/sbin/munin20:30:373
1493626919,25sleep20-21swapper/223:40:582
1430026915,19sleep20-21swapper/220:15:342
1913926820,24sleep20-21swapper/222:12:382
1873426813,26sleep10-21swapper/122:10:471
1351626811,41sleep113611-21fschecks_time21:55:371
694726713,23sleep30-21swapper/321:35:153
592426721,27sleep30-21swapper/323:15:313
2959426714,37sleep129741-21cut21:05:161
2288526714,28sleep10-21swapper/120:40:491
1513626713,24sleep30-21swapper/320:16:113
920426614,33sleep29241-21proc_pri21:40:432
920426614,33sleep29241-21proc_pri21:40:432
2969126611,14sleep10-21swapper/119:20:481
2914326625,24sleep20-21swapper/219:20:372
254826620,22sleep10-21swapper/123:05:151
1473726621,22sleep30-21swapper/322:00:153
2728926515,20sleep20-21swapper/220:55:482
1240826514,35sleep312475-21df23:35:423
1240826514,35sleep312475-21df23:35:423
1235326522,24sleep20-21swapper/221:50:452
1235326522,24sleep20-21swapper/221:50:452
1032926513,39sleep110377-21latency_hist23:30:151
100226519,18sleep30-21swapper/321:15:403
591926419,24sleep30-21swapper/319:50:313
464526418,27sleep14734-21/usr/sbin/munin19:45:401
33272647,19sleep30-21swapper/321:22:393
2354226411,29sleep10-21swapper/120:45:291
1708026412,24sleep30-21swapper/323:50:163
11732646,46sleep30-21swapper/323:00:163
1084126417,21sleep30-21swapper/323:30:373
1084126417,21sleep30-21swapper/323:30:373
2564726311,33sleep30-21swapper/320:50:453
2486726316,19sleep30-21swapper/322:30:443
2486726316,19sleep30-21swapper/322:30:443
2199226315,30sleep20-21swapper/220:40:302
860426221,23sleep10-21swapper/123:23:411
456026215,20sleep10-21swapper/123:10:401
3134126226,21sleep20-21swapper/200:35:142
3030426220,20sleep10-21swapper/121:05:451
3030426220,20sleep10-21swapper/121:05:451
2489326220,18sleep30-21swapper/320:50:163
2351626218,30sleep223534-21/usr/sbin/munin20:45:262
151722625,20sleep10-21swapper/123:41:501
1106326215,23sleep10-21swapper/121:45:471
1106326215,23sleep10-21swapper/121:45:471
902226115,27sleep19107-21sshd21:40:381
902226115,27sleep19107-21sshd21:40:381
412926112,32sleep14105-21latency_hist19:45:161
2070826116,18sleep30-21swapper/300:00:423
184282619,15sleep30-21swapper/320:28:233
1778526118,23sleep30-21swapper/322:10:133
1600326127,17sleep10-21swapper/123:45:421
1585326115,24sleep20-21swapper/223:45:392
146526112,24sleep20-21swapper/219:35:352
756426011,35sleep1362-21snmpd21:35:421
756426011,35sleep1362-21snmpd21:35:421
2315126017,24sleep223256-21users19:00:472
2179826014,30sleep221890-21/usr/sbin/munin22:20:462
1059526016,20sleep10-21swapper/120:05:151
752925910,33sleep37548-21df_inode23:20:323
2869725922,20sleep30-21swapper/321:00:423
2142125917,25sleep10-21swapper/122:20:371
1290625918,20sleep20-21swapper/223:36:102
1290625918,20sleep20-21swapper/223:36:102
637825815,26sleep26384-21ntp_states23:15:422
434125715,24sleep20-21swapper/223:10:332
3066825720,21sleep20-21swapper/219:25:342
2691425716,26sleep326910-21apt-get00:20:293
2691425716,26sleep326910-21apt-get00:20:293
2647425719,20sleep30-21swapper/322:35:453
2305025714,28sleep123134-21memory22:25:411
188452577,38sleep20-21swapper/223:55:242
67042566,10sleep10-21swapper/121:33:071
2756825615,25sleep227575-21diskstats19:15:322
231112565,10sleep20-21swapper/200:08:412
2255425618,15sleep30-21swapper/300:05:473
2134525620,20sleep20-21swapper/220:35:492
208112565,11sleep10-21swapper/120:35:401
1934425614,20sleep30-21swapper/322:15:133
1421525613,25sleep114205-21apt-get20:15:291
1390225617,21sleep20-21swapper/221:55:462
3253825517,24sleep30-21swapper/319:30:403
3253825517,24sleep30-21swapper/319:30:403
3075925514,25sleep330816-21irqstats22:50:353
3075925514,25sleep330816-21irqstats22:50:353
2198225514,24sleep122049-21df00:05:331
68192544,10sleep20-21swapper/221:34:332
358525412,18sleep10-21swapper/121:25:161
2947625419,16sleep20-21swapper/222:45:462
2947625419,16sleep20-21swapper/222:45:462
2356325412,22sleep223671-21users22:25:512
901825311,23sleep29014-21apt-get23:25:272
717625311,28sleep2284-21dbus-daemon19:55:142
433725313,23sleep34334-21apt-get23:10:303
3216325314,21sleep30-21swapper/321:11:083
3216325314,21sleep30-21swapper/321:11:083
2988025313,23sleep229884-21diskstats21:05:342
2988025313,23sleep229884-21diskstats21:05:342
279025312,25sleep12785-21apt-get23:05:311
1083925314,23sleep310838-21apt-get20:05:293
3059225211,25sleep130589-21apt-get19:25:291
2786725212,22sleep227903-21ntp_states22:40:412
2786725212,22sleep227903-21ntp_states22:40:412
2744925211,24sleep127446-21apt-get22:40:271
2744925211,24sleep127446-21apt-get22:40:271
2121325211,23sleep321208-21apt-get22:20:303
1884125211,18sleep10-21swapper/120:30:271
3165425112,24sleep231661-21df_inode21:10:302
3165425112,24sleep231661-21df_inode21:10:302
2272325118,16sleep30-21swapper/319:00:393
2260825115,16sleep30-21swapper/320:40:473
937025013,22sleep39377-21diskstats20:00:323
2402525012,24sleep224029-21diskstats19:05:352
15222508,16sleep30-21swapper/319:35:363
1057125014,21sleep210567-21apt-get23:30:282
1057125014,21sleep210567-21apt-get23:30:282
321262499,23sleep20-21swapper/222:55:272
2871124910,14sleep20-21swapper/200:25:372
24462495,32sleep20-21swapper/223:04:352
1825524911,23sleep218256-21sshd23:51:412
873824810,21sleep38734-21apt-get21:40:283
873824810,21sleep38734-21apt-get21:40:283
70032485,31sleep10-21swapper/119:53:031
2046224815,16sleep10-21swapper/100:00:361
1074024813,21sleep310745-21ntp_states21:45:413
1074024813,21sleep310745-21ntp_states21:45:413
2610624710,20sleep20-21swapper/222:35:362
1028924512,20sleep210286-21apt-get21:45:282
1028924512,20sleep210286-21apt-get21:45:282
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional