You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 15:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Dec 27, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1365721277410,12749sleep110429-21kworker/u8:008:59:451
10514217096,1648sleep0197-21jbd2/mmcblk2p2-10:34:150
916829218,900sleep09169-21sshd12:12:280
12484282918,795sleep212497-21/usr/sbin/munin10:39:342
13548258218,543sleep213541-21apt-get08:59:262
28853243516,405sleep128896-21ntpq08:04:371
28853243516,405sleep128896-21ntpq08:04:371
20822235639,198sleep020977-21grep11:05:040
20822235639,198sleep020977-21grep11:05:040
2289233614,308sleep32300-21ntpq11:49:373
31622229723,218sleep031663-21fschecks_count08:14:290
2689222916,275sleep0369-21snmpd09:39:550
2261822908,204sleep022643-21awk07:44:370
15951228190,38sleep016101-21unixbench_singl10:49:410
15951228190,38sleep016101-21unixbench_singl10:49:410
250792278102,159sleep225088-21/usr/sbin/munin07:54:122
10496226980,120sleep00-21swapper/007:09:110
29497226014,232sleep329498-21sshd08:06:293
29497226014,232sleep329498-21sshd08:06:293
117222410,26sleep31176-21unixbench_singl11:44:563
1005222376,174sleep00-21swapper/012:14:380
1005222376,174sleep00-21swapper/012:14:380
28804221713,152sleep028837-21latency_hist09:49:120
2820822159,194sleep028251-21cut08:04:110
5000221339,108sleep00-21swapper/010:14:440
21306221320,77sleep00-21swapper/009:24:240
1344622088,98sleep013597-21sort10:44:120
32166220742,103sleep00-21swapper/009:59:290
3040222058,183sleep030446-21cat08:09:370
23875220417,163sleep010545-21cyclictest07:49:290
664322020,5sleep00-21swapper/010:19:440
22902219936,123sleep022923-21modprobe11:14:090
22902219936,123sleep022923-21modprobe11:14:090
20467219934,124sleep020516-21latency_hist07:39:120
12598219110,141sleep012801-21touch12:24:110
1872219065,19sleep00-21swapper/010:04:420
1872219065,19sleep00-21swapper/010:04:420
25535218310,157sleep025589-21if_eth007:54:300
7754217617,111sleep07757-21cron12:09:090
1832021738,101sleep00-21swapper/010:59:270
1375021725,129sleep00-21swapper/010:44:280
439721715,28sleep00-21swapper/011:56:000
24765216227,16sleep00-21swapper/009:34:380
12059216234,58sleep012151-21wc07:14:120
2966321567,38sleep00-21swapper/009:49:400
2975721527,134sleep029849-21latency_hist08:09:120
2975721527,134sleep029849-21latency_hist08:09:120
31065215124,95sleep00-21swapper/011:39:290
31065215124,95sleep00-21swapper/011:39:290
20410215021,109sleep20-21swapper/209:19:422
2106221498,104sleep021092-21sed09:24:120
30666214830,16sleep00-21swapper/009:54:290
30666214830,16sleep00-21swapper/009:54:290
12217214818,25sleep00-21swapper/008:54:520
12217214818,25sleep00-21swapper/008:54:520
25278214330,16sleep00-21swapper/011:19:420
29163213736,29sleep10-21swapper/111:34:131
12101213726,89sleep112102-21/usr/sbin/munin10:39:201
8489213650,72sleep08578-21grep08:44:300
24662213616,100sleep224683-21kernelversion09:34:362
158621367,45sleep00-21swapper/008:19:500
14350213666,59sleep014358-21df07:19:290
7079213421,93sleep17154-21switchtime12:04:421
9092213333,15sleep00-21swapper/010:29:300
7569213310,81sleep00-21swapper/008:39:410
1764921315,11sleep00-21swapper/009:10:510
1927213018,98sleep22000-21fschecks_time11:49:282
19665212817,95sleep119727-21sendmail_mailst07:35:131
29473212411,22sleep00-21swapper/011:34:280
1866321249,32sleep018744-21apt-config07:34:110
24257212219,78sleep10-21swapper/107:49:391
11704212220,81sleep111219-21apt-get08:54:201
11704212220,81sleep111219-21apt-get08:54:201
13726212118,80sleep21049-21sendmail-mta12:25:022
48621208,94sleep00-21swapper/011:44:380
1316521197,22sleep00-21swapper/007:14:400
27856211814,16sleep00-21swapper/011:29:230
1509921189,29sleep015164-21ls09:04:250
13378211816,83sleep313513-21unixbench_multi07:14:433
22008211729,16sleep00-21swapper/007:44:120
12311211619,19sleep30-21swapper/310:39:313
404721155,28sleep00-21swapper/010:14:070
14364211517,85sleep214448-21fschecks_time07:19:312
3499211311,89sleep03535-21latency_hist08:29:110
2881211318,19sleep20-21swapper/208:24:412
6051211115,27sleep00-21swapper/012:02:020
1107321115,30sleep00-21swapper/008:52:250
1107321115,30sleep00-21swapper/008:52:250
9918210814,36sleep30-21swapper/308:49:213
9918210814,36sleep30-21swapper/308:49:213
31659210821,64sleep331802-21grep11:39:443
31659210821,64sleep331802-21grep11:39:443
1449721089,86sleep0369-21snmpd12:29:200
1449721089,86sleep0369-21snmpd12:29:200
12925210833,58sleep0369-21snmpd12:24:190
12494210822,61sleep312591-21latency_hist12:24:103
252421068,17sleep02684-21awk11:49:430
16510210616,76sleep116588-21latency_hist10:54:111
16510210616,76sleep116588-21latency_hist10:54:111
23358210510,34sleep023418-21ls11:14:330
1904721057,15sleep00-21swapper/009:14:430
374821049,12sleep00-21swapper/008:29:280
19865210412,76sleep319960-21fschecks_time09:19:303
1672421049,57sleep00-21swapper/010:54:220
1672421049,57sleep00-21swapper/010:54:220
1964210116,70sleep12039-21latency_hist08:24:111
27620210030,57sleep027699-21date11:29:110
1581621007,14sleep00-21swapper/007:24:260
87132997,52sleep00-21swapper/010:29:100
2507529922,61sleep325206-21unixbench_multi09:34:453
136192997,12sleep00-21swapper/008:59:280
60932986,13sleep00-21swapper/008:34:410
297529814,66sleep33104-21awk08:24:423
967329723,55sleep39731-21cut12:14:293
967329723,55sleep39731-21cut12:14:293
956929626,51sleep29609-21grep07:04:372
3223829614,68sleep232246-21df09:59:312
2342129611,58sleep00-21swapper/009:29:410
2342129611,58sleep00-21swapper/009:29:410
1640529433,16sleep00-21swapper/012:34:330
1404329413,54sleep30-21swapper/308:59:563
1645929322,54sleep316472-21apt10:54:083
1645929322,54sleep316472-21apt10:54:083
305892927,66sleep10-21swapper/109:54:261
305892927,66sleep10-21swapper/109:54:261
223352927,13sleep20-21swapper/207:44:312
1732429013,64sleep317410-21switchtime10:54:413
1732429013,64sleep317410-21switchtime10:54:413
650528914,53sleep36478-21/usr/sbin/munin12:04:283
250728919,49sleep22535-21munin-plugin-st10:09:092
250728919,49sleep22535-21munin-plugin-st10:09:092
536628710,52sleep20-21swapper/208:34:262
2677428713,60sleep126782-21ntpq11:24:401
958528627,32sleep20-21swapper/212:14:242
958528627,32sleep20-21swapper/212:14:242
567628618,19sleep30-21swapper/310:19:113
409728615,56sleep34140-21date10:14:103
2038628529,36sleep220418-21modprobe07:39:102
2205028425,20sleep10-21swapper/107:44:131
2816028312,53sleep328206-21systemd-run08:04:103
198028323,30sleep10-21swapper/110:04:431
198028323,30sleep10-21swapper/110:04:431
1314428312,57sleep213156-21ntpq07:14:382
2312328212,54sleep123217-21/usr/sbin/munin09:29:341
2312328212,54sleep123217-21/usr/sbin/munin09:29:341
284522819,53sleep20-21swapper/208:04:262
284522819,53sleep20-21swapper/208:04:262
2756728127,35sleep227587-21modprobe11:29:092
1580228127,29sleep20-21swapper/212:34:112
1580228127,29sleep20-21swapper/212:34:112
262392809,16sleep10-21swapper/109:39:351
2034828014,51sleep220405-21irqstats11:04:542
2034828014,51sleep220405-21irqstats11:04:542
1710228010,56sleep317109-21ntp_states09:09:373
1180628023,38sleep211807-21cron10:39:092
1376327921,23sleep10-21swapper/110:44:291
1205027929,26sleep30-21swapper/307:14:113
1426327823,30sleep314409-21latency_hist12:29:113
689127731,25sleep30-21swapper/310:20:283
1737227724,22sleep10-21swapper/112:39:111
875427622,33sleep18771-21grep10:29:111
2940027618,40sleep229401-21cpu11:34:262
153412768,16sleep20-21swapper/209:04:312
1030627617,24sleep110410-21date10:34:121
820327516,37sleep28348-21head10:24:442
3204727520,28sleep20-21swapper/208:14:392
2879727525,32sleep30-21swapper/309:49:113
1135527525,29sleep10-21swapper/112:19:321
847027416,42sleep10-21swapper/108:44:271
219552747,49sleep20-21swapper/207:44:102
1745227421,28sleep30-21swapper/307:29:293
1581727416,23sleep10-21swapper/107:24:271
2988527315,15sleep30-21swapper/311:34:393
2181227319,39sleep321898-21mailstats09:24:393
1979527326,25sleep30-21swapper/307:35:153
1954227326,19sleep10-21swapper/109:19:121
1880227321,29sleep20-21swapper/209:14:412
1665927319,32sleep316014-21/usr/sbin/munin12:34:403
67427216,27sleep10-21swapper/111:44:431
398327216,25sleep10-21swapper/108:29:331
176062726,14sleep10-21swapper/107:29:311
1496027220,36sleep314968-21awk12:29:363
1496027220,36sleep314968-21awk12:29:363
1101627219,33sleep311054-21grep10:34:383
816127121,25sleep20-21swapper/208:44:122
2022227115,36sleep320223-21/usr/sbin/munin11:04:513
2022227115,36sleep320223-21/usr/sbin/munin11:04:513
1348827120,22sleep30-21swapper/308:59:153
479127016,40sleep31-21systemd08:31:283
258922709,48sleep325894-21sshd11:22:273
1679727019,36sleep116861-21/usr/sbin/munin09:09:311
2541226919,32sleep225424-21fschecks_count07:54:282
2455226912,38sleep224705-21sed11:19:112
2329126921,23sleep30-21swapper/311:14:323
2109326919,32sleep121240-21latency_hist09:24:131
3095726813,28sleep20-21swapper/209:54:352
3095726813,28sleep20-21swapper/209:54:352
1146926818,22sleep20-21swapper/208:54:132
1146926818,22sleep20-21swapper/208:54:132
60632679,44sleep20-21swapper/212:02:122
2291926720,20sleep30-21swapper/309:29:283
2291926720,20sleep30-21swapper/309:29:283
225202679,12sleep20-21swapper/209:28:382
2165426713,23sleep20-21swapper/211:09:262
2165426713,23sleep20-21swapper/211:09:262
191226718,31sleep31941-21apt08:24:093
1818826712,42sleep318189-21cpu09:14:253
3222226623,19sleep10-21swapper/111:44:101
3222226623,19sleep10-21swapper/111:44:101
2136026620,29sleep121379-21basename11:09:091
2136026620,29sleep121379-21basename11:09:091
1171126616,33sleep311713-21acpi08:54:223
723026519,19sleep10-21swapper/108:39:341
38926524,21sleep20-21swapper/211:44:362
236222656,37sleep20-21swapper/209:29:432
236222656,37sleep20-21swapper/209:29:432
3034826417,25sleep30-21swapper/309:54:113
2572226415,32sleep125820-21latency_hist09:39:111
1475826417,19sleep30-21swapper/307:19:403
1168026418,26sleep20-21swapper/212:19:412
1081626420,27sleep10-21swapper/110:34:321
47362638,13sleep20-21swapper/210:14:382
3233826325,22sleep10-21swapper/109:59:351
2535826320,27sleep125450-21threads11:19:431
1957726316,21sleep20-21swapper/211:04:092
163912638,10sleep20-21swapper/210:53:152
163912638,10sleep20-21swapper/210:53:152
502526222,22sleep30-21swapper/311:59:273
3129626221,18sleep20-21swapper/208:14:112
3065226216,31sleep1287-21dbus-daemon11:39:091
2508926216,20sleep30-21swapper/307:54:133
1985126218,25sleep10-21swapper/111:04:121
1711826218,23sleep10-21swapper/110:54:351
1711826218,23sleep10-21swapper/110:54:351
1518126219,21sleep30-21swapper/309:04:283
896226120,17sleep10-21swapper/107:04:121
804426115,29sleep38039-21apt-get12:09:243
780526113,20sleep20-21swapper/212:09:112
746926117,31sleep37563-21timerandwakeup08:39:403
3124626115,30sleep11-21systemd08:14:101
2941026115,31sleep10-21swapper/109:49:331
2292426119,27sleep122952-21munin-run11:14:101
2292426119,27sleep122952-21munin-run11:14:101
1497026117,19sleep30-21swapper/310:49:123
867126015,30sleep38756-21cat08:44:333
315672606,17sleep30-21swapper/309:55:333
315672606,17sleep30-21swapper/309:55:333
306826020,24sleep23136-21latency10:09:332
1473126011,34sleep114737-21ntp_states07:19:381
811425916,25sleep18123-21df_inode12:09:261
30532594,13sleep10-21swapper/111:53:391
2642825911,32sleep226468-21ps09:39:412
2224925919,21sleep30-21swapper/307:44:283
1547725913,23sleep10-21swapper/109:04:341
1457325919,26sleep114579-21df_inode12:29:251
1457325919,26sleep114579-21df_inode12:29:251
1325825913,27sleep10-21swapper/112:24:531
1215225916,19sleep10-21swapper/107:14:131
3135425812,24sleep30-21swapper/308:14:123
2923125813,19sleep20-21swapper/209:49:302
1664625816,25sleep216642-21apt-get09:09:242
511425723,17sleep10-21swapper/108:34:171
357725716,19sleep10-21swapper/111:54:321
2489525717,22sleep20-21swapper/211:19:322
220225716,24sleep12197-21apt-get08:24:241
1135225717,25sleep111468-21cmp08:54:121
1135225717,25sleep111468-21cmp08:54:121
662625617,23sleep30-21swapper/308:39:123
463625616,19sleep20-21swapper/211:59:092
421625617,16sleep10-21swapper/110:14:121
3243425627,16sleep30-21swapper/309:59:373
3153425612,26sleep331530-21apt-get08:14:243
2757825614,24sleep127587-21diskstats09:44:281
2750425615,24sleep227498-21apt-get09:44:252
2689925615,25sleep226893-21apt-get07:59:252
1415125616,26sleep214227-21mailstats10:44:392
1075225614,25sleep210745-21apt-get07:09:252
900225514,25sleep28998-21apt-get10:29:242
2825425517,24sleep328290-21ntpq11:29:343
262702557,11sleep30-21swapper/307:55:063
961025413,23sleep39626-21ntp_states07:04:383
936925414,25sleep29520-21sed12:14:122
610825417,18sleep10-21swapper/110:19:341
492725412,16sleep10-21swapper/111:59:211
1101425417,17sleep10-21swapper/107:09:321
2795325314,24sleep228061-21tune2fs11:29:292
2319425312,23sleep123189-21apt-get11:14:261
2120525314,15sleep30-21swapper/307:39:393
979725216,19sleep10-21swapper/112:14:311
979725216,19sleep10-21swapper/112:14:311
970625213,24sleep20-21swapper/208:49:122
908725213,24sleep39090-21diskstats10:29:293
73625213,17sleep20-21swapper/208:19:282
3096725212,24sleep230963-21apt-get11:39:232
3096725212,24sleep230963-21apt-get11:39:232
253725214,24sleep32557-21munin-run10:09:103
253725214,24sleep32557-21munin-run10:09:103
2215925215,20sleep322252-21grep11:09:393
2215925215,20sleep322252-21grep11:09:393
2198425220,15sleep10-21swapper/109:24:411
1636325217,18sleep30-21swapper/307:24:413
437225119,16sleep20-21swapper/208:29:422
2741025120,14sleep10-21swapper/107:59:401
2708425114,20sleep30-21swapper/309:42:283
2666225116,19sleep10-21swapper/107:59:121
2244825115,16sleep10-21swapper/107:44:341
1799625118,16sleep20-21swapper/207:29:412
1581325113,22sleep215809-21apt-get07:24:242
1695224914,16sleep20-21swapper/210:54:322
1695224914,16sleep20-21swapper/210:54:322
2631424811,19sleep30-21swapper/311:24:253
1336224714,16sleep20-21swapper/208:59:142
1336224714,16sleep20-21swapper/208:59:142
592224610,20sleep25917-21apt-get10:19:262
513524612,15sleep10-21swapper/110:14:451
1831824611,19sleep30-21swapper/310:59:263
36562459,16sleep30-21swapper/311:54:343
1659424512,17sleep20-21swapper/212:34:392
75312448,14sleep30-21swapper/310:24:283
315372419,15sleep10-21swapper/108:14:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional