You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 11:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun Nov 16, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26282198816,1954sleep012260-21kworker/u8:121:24:120
825627639,721sleep18292-21ntpq19:59:271
2392026977,673sleep26018-21kworker/u8:020:49:222
26697267810,653sleep126736-21ntpq19:14:301
11058250315,472sleep211067-21/usr/sbin/munin20:09:212
11058250315,472sleep211067-21/usr/sbin/munin20:09:212
3285247213,443sleep33291-21diskstats19:44:233
14392236715,311sleep114455-21latency20:19:251
5461235679,265sleep15503-21cut23:14:301
12184233310,311sleep112245-21cat21:54:241
3169227516,238sleep23191-21latency_hist23:09:022
11692222329,123sleep111721-21cron21:54:001
10104222216,192sleep310109-21ntp_states23:29:303
10104222216,192sleep310109-21ntp_states23:29:303
10326222122,180sleep210340-21/usr/sbin/munin20:04:372
22759221716,126sleep122869-21timerandwakeup20:44:321
26914221618,172sleep10-21swapper/100:24:221
6329221014,123sleep15763-21/usr/sbin/munin21:34:341
29956221020,170sleep129398-21/usr/sbin/munin19:24:311
13654220921,170sleep013667-21fschecks_count21:59:200
23811220620,98sleep10-21swapper/100:14:211
12253220218,23sleep10-21swapper/120:14:001
12253220218,23sleep10-21swapper/120:14:001
30582219928,132sleep130159-21/usr/sbin/munin21:09:271
25889219890,91sleep10-21swapper/100:19:331
25889219890,91sleep10-21swapper/100:19:331
23954219712,103sleep10-21swapper/120:49:291
19679219610,88sleep119888-21apt-config22:19:031
19679219610,88sleep119888-21apt-config22:19:031
4693219335,106sleep10-21swapper/121:29:291
28576219121,99sleep10-21swapper/100:29:221
155862187101,72sleep115651-21awk22:04:291
3504218412,82sleep10-21swapper/123:09:211
640521838,39sleep10-21swapper/119:54:181
18638217189,16sleep10-21swapper/120:34:031
2055421688,13sleep10-21swapper/100:04:161
2055421688,13sleep10-21swapper/100:04:161
26429216232,79sleep10-21swapper/122:39:221
4830216053,23sleep10-21swapper/119:49:221
23247215163,18sleep10-21swapper/119:04:331
23247215163,18sleep10-21swapper/119:04:331
25634215041,86sleep10-21swapper/119:09:371
13859214939,27sleep10-21swapper/120:19:011
29183214713,118sleep129286-21latency_hist19:24:021
17653214317,96sleep00-21swapper/023:54:220
1235213848,63sleep10-21swapper/121:19:241
3888213410,66sleep14026-21grep19:44:371
18412213417,66sleep10-21swapper/123:54:591
7421213265,47sleep10-21swapper/121:39:221
21198213222,95sleep021246-21awk18:59:190
31436213163,50sleep10-21swapper/119:29:311
3124521307,107sleep00-21swapper/019:29:260
1570212917,91sleep10-21swapper/123:04:001
17045212812,101sleep017089-21sed22:09:250
16054212835,56sleep10-21swapper/123:49:221
16054212835,56sleep10-21swapper/123:49:221
9798212729,84sleep19810-21needreboot20:04:271
22202212750,16sleep10-21swapper/122:24:321
1959921239,24sleep10-21swapper/123:59:311
15777212214,92sleep115883-21tune2fs20:24:301
15630212112,23sleep10-21swapper/123:49:021
12101212113,92sleep012183-21cut21:54:230
472212033,16sleep10-21swapper/119:34:281
30623211911,57sleep10-21swapper/122:53:471
22103211920,46sleep30-21swapper/300:09:143
3607211817,82sleep23649-21/usr/sbin/munin19:44:302
20598211721,40sleep10-21swapper/120:39:241
1182721178,21sleep10-21swapper/123:34:331
19004211631,69sleep122560-21cyclictest20:34:111
11772211519,38sleep30-21swapper/321:54:023
6253211422,69sleep26264-21modprobe23:18:592
2827210927,32sleep1362-21snmpd23:05:181
28706210820,39sleep028716-21diskstats21:04:170
4673210528,53sleep34689-21modprobe23:13:593
579210415,20sleep10-21swapper/121:19:001
3204210413,56sleep10-21swapper/121:24:331
10150210422,64sleep10-21swapper/123:29:321
10150210422,64sleep10-21swapper/123:29:321
27291210214,72sleep227319-21proc_pri00:24:312
7309210114,74sleep07312-21acpi21:39:130
6808210145,13sleep10-21swapper/123:19:271
31594210021,22sleep10-21swapper/122:54:441
20121210012,73sleep320123-21sshd20:36:473
19452210018,64sleep20-21swapper/220:34:402
200132998,15sleep00-21swapper/022:19:100
200132998,15sleep00-21swapper/022:19:100
2381929815,42sleep10-21swapper/122:29:331
262629716,63sleep32158-21apt-get21:24:103
2128629513,50sleep321341-21fschecks_count18:59:263
83629314,62sleep2911-21sensors_temp22:59:292
2809629338,14sleep10-21swapper/122:44:261
2809629338,14sleep10-21swapper/122:44:261
241782926,65sleep00-21swapper/000:14:280
2112829212,65sleep221129-21sshd22:21:462
2112829212,65sleep221129-21sshd22:21:462
97862915,12sleep10-21swapper/121:44:351
1893829016,54sleep019095-21unixbench_multi22:14:340
1893829016,54sleep019095-21unixbench_multi22:14:340
120322906,73sleep00-21swapper/020:11:130
120322906,73sleep00-21swapper/020:11:130
2212728912,36sleep10-21swapper/100:09:161
1503028918,48sleep014297-21/usr/sbin/munin23:44:330
2637028724,28sleep00-21swapper/022:39:210
609128522,40sleep26214-21latency_hist19:54:032
626528414,49sleep06327-21grep21:34:330
260432847,66sleep126045-21munin-run22:39:011
1855528411,60sleep018576-21pluginstate20:34:020
203372837,37sleep10-21swapper/122:19:401
203372837,37sleep10-21swapper/122:19:401
3119428213,15sleep30-21swapper/319:29:253
263202828,14sleep10-21swapper/120:54:341
21412826,64sleep12204-21awk19:39:331
1768628218,47sleep317198-21/usr/sbin/munin20:29:333
1303528217,45sleep012547-21/usr/sbin/munin20:14:320
1484728120,39sleep214990-21sed20:19:342
497027912,53sleep04989-21apt-get23:14:130
2930627913,12sleep10-21swapper/121:04:311
136862795,42sleep10-21swapper/123:39:371
1056227916,27sleep00-21swapper/021:49:230
2184827817,45sleep21-21systemd20:44:012
148482788,57sleep114857-21munin-run22:04:011
360527719,17sleep20-21swapper/223:09:222
2824527719,25sleep00-21swapper/019:19:290
2196427713,47sleep322008-21ntpq22:24:263
2118327715,48sleep021278-21sed00:04:350
2118327715,48sleep021278-21sed00:04:350
135627716,46sleep21441-21mailstats21:19:292
2847527610,50sleep028575-21grep00:29:210
521827539,19sleep00-21swapper/019:49:330
2678327517,45sleep026823-21modprobe20:59:000
1254927514,44sleep312620-21sensors_temp21:54:343
3071827416,43sleep330812-21gpgv22:54:013
2274927422,28sleep20-21swapper/200:09:332
2230527420,21sleep20-21swapper/222:24:332
1902427421,26sleep00-21swapper/023:59:140
1675927417,27sleep30-21swapper/322:09:203
871727321,28sleep30-21swapper/323:24:333
871727321,28sleep30-21swapper/323:24:333
704327312,15sleep30-21swapper/321:39:003
3153327312,48sleep031538-21latency_hist21:14:020
282152736,28sleep10-21swapper/121:02:291
2519627324,32sleep025200-21cron20:54:000
2352327314,27sleep20-21swapper/222:29:252
1538827319,37sleep315428-21latency22:04:233
632827215,25sleep30-21swapper/319:54:163
2188427210,18sleep30-21swapper/300:09:023
2188427210,18sleep30-21swapper/300:09:023
1408227216,40sleep214215-21sed23:44:022
857927111,43sleep10-21swapper/123:24:311
857927111,43sleep10-21swapper/123:24:311
3273327113,43sleep332744-21sort22:59:103
268652710,7sleep30-21swapper/322:39:313
2548127129,19sleep00-21swapper/019:09:360
2535727119,25sleep00-21swapper/000:19:200
2535727119,25sleep00-21swapper/000:19:200
2302227115,22sleep30-21swapper/319:04:263
2302227115,22sleep30-21swapper/319:04:263
2176827127,24sleep10-21swapper/118:59:361
2146927125,31sleep221542-21sed18:59:292
3122527020,24sleep30-21swapper/322:54:343
2747527013,38sleep027519-21/usr/sbin/munin20:59:290
2740427012,40sleep027496-21sed00:24:330
2498127014,40sleep225010-21irqstats19:09:262
143327019,26sleep00-21swapper/019:39:030
486626919,21sleep30-21swapper/321:29:323
170112697,13sleep00-21swapper/023:52:320
170112697,13sleep00-21swapper/023:52:320
823326817,23sleep30-21swapper/321:39:373
298472687,45sleep20-21swapper/222:49:322
2503526820,26sleep20-21swapper/200:19:022
1694126816,37sleep217030-21load22:09:232
651626715,26sleep00-21swapper/019:54:210
62062678,47sleep30-21swapper/323:18:243
178012675,29sleep10-21swapper/122:12:171
3237326617,20sleep00-21swapper/021:14:310
3190726615,31sleep032047-21grep22:54:510
2539726619,29sleep20-21swapper/200:19:222
2539726619,29sleep20-21swapper/200:19:222
2054726612,24sleep30-21swapper/322:19:463
2054726612,24sleep30-21swapper/322:19:463
1951726614,22sleep30-21swapper/322:19:013
1951726614,22sleep30-21swapper/322:19:013
1459726621,27sleep30-21swapper/320:19:313
1247826612,41sleep2362-21snmpd23:38:572
79662659,14sleep20-21swapper/219:59:202
430226513,20sleep00-21swapper/021:29:200
291252657,11sleep30-21swapper/321:04:293
2863326528,17sleep20-21swapper/221:04:152
2356426522,22sleep00-21swapper/020:49:030
103502657,12sleep00-21swapper/020:04:390
587426419,21sleep20-21swapper/221:34:232
2843126411,39sleep228493-21switchtime19:19:352
2368026417,25sleep20-21swapper/219:04:382
2368026417,25sleep20-21swapper/219:04:382
2181826411,34sleep021863-21irqstats22:24:230
1337526414,23sleep00-21swapper/023:39:320
2513126318,26sleep20-21swapper/222:34:262
2230026321,25sleep30-21swapper/320:44:223
2222326322,19sleep20-21swapper/220:44:212
1713126315,31sleep317183-40ogrotate23:54:003
1713126315,31sleep317183-40ogrotate23:54:003
1458926315,32sleep314297-21/usr/sbin/munin23:44:233
780426217,21sleep20-21swapper/223:24:002
2683426210,26sleep20-21swapper/220:59:032
2482426219,27sleep324834-21/usr/sbin/munin19:09:233
234012626,12sleep00-21swapper/019:04:360
234012626,12sleep00-21swapper/019:04:360
3230126113,33sleep332371-21sensors_temp21:14:303
2913126116,21sleep30-21swapper/319:24:003
152292616,20sleep00-21swapper/020:22:070
140526115,19sleep30-21swapper/319:39:023
943526016,28sleep39430-21apt-get20:04:173
901226017,16sleep00-21swapper/021:44:220
349226014,19sleep00-21swapper/023:09:200
213026021,19sleep00-21swapper/023:04:240
2086526015,24sleep30-21swapper/320:39:303
1900126016,28sleep319020-21apt-get23:59:133
1755126011,34sleep217590-21memory20:29:272
3223125915,28sleep232271-21sendmail19:34:002
3145325911,23sleep20-21swapper/221:14:002
3013125918,23sleep330280-21sed19:24:333
256442595,12sleep30-21swapper/322:34:353
99025815,22sleep00-21swapper/022:59:310
296325814,18sleep30-21swapper/319:44:033
2722625817,25sleep327264-21ntpq00:24:283
2643925814,24sleep30-21swapper/300:22:503
2643925814,24sleep30-21swapper/300:22:503
2441425815,26sleep024477-21sendmail-msp22:34:000
2340425815,29sleep023428-21munin-run00:13:590
915225716,26sleep29190-21wc21:44:242
50425719,21sleep20-21swapper/219:34:292
271622577,12sleep20-21swapper/220:59:232
2473225715,24sleep024726-21apt-get22:34:160
1520625714,18sleep00-21swapper/022:04:200
1009225715,21sleep20-21swapper/223:29:292
1009225715,21sleep20-21swapper/223:29:292
478025617,17sleep20-21swapper/221:29:302
3082125617,23sleep030664-21latency_hist22:54:020
2783625613,26sleep227828-21apt-get22:44:192
2783625613,26sleep227828-21apt-get22:44:192
2753825616,23sleep30-21swapper/320:59:323
226125612,26sleep32158-21apt-get21:24:023
185925612,26sleep31855-21apt-get23:04:153
185682567,38sleep20-21swapper/223:56:542
967525510,18sleep00-21swapper/023:29:190
967525510,18sleep00-21swapper/023:29:190
86925511,27sleep0863-21apt-get21:19:150
815825514,22sleep30-21swapper/319:59:253
809325518,20sleep20-21swapper/221:39:362
3122425512,26sleep231191-21apt-get22:54:332
293025523,19sleep20-21swapper/221:24:262
2914425516,17sleep00-21swapper/022:49:020
2914425516,17sleep00-21swapper/022:49:020
1792925513,27sleep317953-21proc_pri23:54:293
1625425516,22sleep30-21swapper/320:24:403
940725415,18sleep00-21swapper/023:29:020
940725415,18sleep00-21swapper/023:29:020
3073225413,25sleep030775-21latency_hist19:29:020
2942225414,25sleep229416-21apt-get19:24:152
1902625412,25sleep219020-21apt-get23:59:162
164392549,13sleep20-21swapper/223:49:312
164392549,13sleep20-21swapper/223:49:312
1257325413,25sleep312640-21df20:14:193
1209025413,25sleep212097-21diskstats21:54:212
654025314,23sleep36541-21apt-get23:19:163
2846325316,22sleep30-21swapper/300:29:203
2551025313,23sleep325506-21apt-get20:54:153
248822536,35sleep30-21swapper/300:17:463
2057525312,24sleep220573-21apt-get00:04:172
2057525312,24sleep220573-21apt-get00:04:172
1377625313,19sleep20-21swapper/221:59:222
1159125312,21sleep30-21swapper/323:34:263
918525214,23sleep09195-21munin-run20:04:020
896925220,18sleep30-21swapper/321:44:213
864225213,23sleep28715-21grep23:24:322
864225213,23sleep28715-21grep23:24:322
472925212,17sleep30-21swapper/319:49:153
3026825213,16sleep30-21swapper/321:09:213
2451225211,17sleep30-21swapper/320:49:473
1832825212,23sleep218337-21df_inode22:14:202
1832825212,23sleep218337-21df_inode22:14:202
1565225213,18sleep20-21swapper/222:04:302
2765425113,16sleep20-21swapper/219:19:032
272832515,33sleep30-21swapper/319:15:123
2315425112,23sleep323151-21apt-get22:29:143
1635325113,17sleep00-21swapper/020:24:410
114292519,26sleep311440-21ntp_states20:09:303
114292519,26sleep311440-21ntp_states20:09:303
785725012,18sleep00-21swapper/023:24:020
660025010,15sleep20-21swapper/219:54:232
2127625017,17sleep00-21swapper/020:39:390
165425015,18sleep20-21swapper/219:39:172
164725013,23sleep01650-21apt-get19:39:150
1488325014,15sleep30-21swapper/322:04:033
32092499,23sleep00-21swapper/019:44:200
2837724911,23sleep228373-21apt-get00:29:152
273062496,31sleep00-21swapper/019:15:300
1917524913,16sleep00-21swapper/020:34:340
1439424912,23sleep214400-21df_inode23:44:182
1144224911,22sleep011491-21irqstats23:34:230
1070224813,20sleep210739-21grep21:49:252
61024710,21sleep2629-21munin-run21:19:012
1607624511,19sleep20-21swapper/220:24:352
128442457,16sleep30-21swapper/323:39:193
2864324210,15sleep30-21swapper/322:44:373
2864324210,15sleep30-21swapper/322:44:373
263542426,14sleep20-21swapper/222:39:192
130262429,16sleep20-21swapper/220:14:302
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional