You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 03:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Dec 27, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
158112738310,7361sleep311750-21kworker/3:223:26:313
8956281511,791sleep39000-21ntpq23:04:343
11317268217,649sleep211365-21ntpq19:44:412
23751245412,426sleep123800-21/usr/sbin/munin23:54:111
23751245412,426sleep123800-21/usr/sbin/munin23:54:111
1007238417,348sleep31147-21latency_hist00:24:133
11304237018,337sleep111316-21/usr/sbin/munin19:44:401
21562235153,258sleep321501-21latency_hist20:19:143
1911723470,11sleep019116-21modprobe23:39:110
6913233719,300sleep17020-21/usr/sbin/munin19:29:421
12371230423,262sleep012514-21sed23:14:450
12927227782,99sleep30-21swapper/321:34:333
12927227782,99sleep30-21swapper/321:34:333
29246226719,231sleep031574-21kworker/u8:420:44:110
29246226719,231sleep031574-21kworker/u8:420:44:110
19096225377,101sleep319099-21cron23:39:113
25223224320,207sleep125236-21/usr/sbin/munin23:55:201
25223224320,207sleep125236-21/usr/sbin/munin23:55:201
1686224321,204sleep11718-21/usr/sbin/munin00:24:401
32672222728,182sleep132740-21/usr/sbin/munin00:19:391
22153221670,84sleep30-21swapper/322:04:133
22153221670,84sleep30-21swapper/322:04:133
22653221233,81sleep30-21swapper/323:49:453
22653221233,81sleep30-21swapper/323:49:453
23935220381,100sleep30-21swapper/322:09:263
32725220029,111sleep3354-21grep20:54:323
1429219321,89sleep31558-21timerwakeupswit22:39:433
25563219189,77sleep30-21swapper/320:29:443
2339121799,158sleep0369-21snmpd22:05:520
2339121799,158sleep0369-21snmpd22:05:520
22542217712,144sleep022691-21grep20:19:440
30606217542,82sleep30-21swapper/322:29:413
2880421750,35sleep128808-21unixbench_multi20:39:431
2880421750,35sleep128808-21unixbench_multi20:39:431
15269216056,89sleep3314-21rs:main3
8502215244,92sleep38649-21grep19:34:443
25006214712,120sleep025092-21fw_packets20:29:330
24942214773,39sleep30-21swapper/323:55:183
24942214773,39sleep30-21swapper/323:55:183
17011214612,118sleep216874-21latency_hist20:04:152
1647214535,23sleep30-21swapper/319:14:333
13725214511,93sleep313754-21idleruntime-cro19:54:113
3384214450,82sleep33512-21timerwakeupswit00:29:433
18467214419,100sleep118611-21latency_hist20:09:151
6105214113,110sleep20-21swapper/219:29:142
2675221407,11sleep20-21swapper/223:59:492
1605421409,117sleep316100-21processes19:59:403
31770213914,22sleep30-21swapper/320:49:473
19391213622,67sleep319541-21unixbench_singl20:09:453
17345213616,102sleep117560-21latency_hist21:49:131
20955213015,97sleep220956-21cpu23:44:272
20955213015,97sleep220956-21cpu23:44:272
19579213065,44sleep30-21swapper/323:39:313
6700212810,94sleep00-21swapper/019:29:370
1824021279,83sleep318274-21ps21:50:163
11644212632,25sleep30-21swapper/323:14:283
10281212616,86sleep30-21swapper/321:24:403
10014212411,90sleep310140-21timerwakeupswit19:39:453
5463212210,101sleep35510-21ntpq21:09:383
26963212227,15sleep30-21swapper/320:34:413
26963212227,15sleep30-21swapper/320:34:413
5200212110,96sleep35285-21cat22:54:133
7512212018,81sleep30-21swapper/322:59:423
23806212043,16sleep30-21swapper/320:24:443
10842212047,15sleep30-21swapper/323:09:433
5560211816,24sleep30-21swapper/300:39:113
4171211719,32sleep30-21swapper/321:04:583
1740521179,79sleep30-21swapper/320:04:333
28759211531,21sleep30-21swapper/322:24:323
22791211510,71sleep322829-21ntpq22:04:383
22791211510,71sleep322829-21ntpq22:04:383
24171211419,72sleep224206-21latency_hist23:54:162
24171211419,72sleep224206-21latency_hist23:54:162
16226211417,25sleep00-21swapper/021:44:360
3194221135,11sleep30-21swapper/319:09:293
3194221135,11sleep30-21swapper/319:09:293
3157211312,85sleep12670-21apt-get19:19:161
29646211318,68sleep30-21swapper/300:09:443
1116921134,17sleep00-21swapper/023:11:460
759521128,92sleep3369-21snmpd21:15:273
25942211019,69sleep225452-21/usr/sbin/munin22:14:422
30943210825,66sleep030974-21ntp_kernel_pll_00:14:370
30943210825,66sleep030974-21ntp_kernel_pll_00:14:370
17202210616,68sleep30-21swapper/321:49:113
26739210510,23sleep30-21swapper/322:19:113
3269210417,70sleep23275-21df_inode19:19:292
8190210318,51sleep00-21swapper/019:34:370
30307210325,62sleep030405-21sed20:44:430
20589210311,75sleep220195-21/usr/sbin/munin20:14:372
8782210229,57sleep30-21swapper/321:19:393
31892210227,16sleep30-21swapper/300:19:133
31892210227,16sleep30-21swapper/300:19:133
20600210216,19sleep30-21swapper/321:59:133
902121010,11sleep29017-21ntp_states23:04:362
14737210125,41sleep10-21swapper/123:24:231
12710210113,19sleep10-21swapper/119:49:321
3212729911,19sleep20-21swapper/200:19:242
315429821,27sleep00-21swapper/021:04:140
1987829711,71sleep219951-21threads21:54:432
993929617,58sleep210013-21wc19:39:442
747229513,64sleep20-21swapper/222:59:412
48929512,68sleep0561-21sh22:39:110
48929512,68sleep0561-21sh22:39:110
3139429518,55sleep031418-21modprobe22:34:110
3039729543,36sleep130460-21fschecks_time19:04:301
1386929522,54sleep30-21swapper/323:19:413
53752949,63sleep30-21swapper/319:24:433
1218529423,52sleep012216-21idleruntime-cro19:49:110
3260229313,65sleep232610-21acpi20:54:222
3228629310,24sleep30-21swapper/322:34:423
3228629310,24sleep30-21swapper/322:34:423
1974329312,66sleep019788-21processes21:54:410
1566029356,24sleep331671-21cyclictest21:44:103
1566029356,24sleep331671-21cyclictest21:44:103
915929212,62sleep09236-21grep23:04:400
2121129217,52sleep120819-21/usr/sbin/munin21:59:351
2121129217,52sleep120819-21/usr/sbin/munin21:59:351
2528429111,66sleep125330-21cat20:29:381
2423429111,65sleep023766-21apt-get23:54:180
2423429111,65sleep023766-21apt-get23:54:180
1311629115,60sleep213202-21latency21:34:402
1311629115,60sleep213202-21latency21:34:402
2585729023,17sleep325852-21apt-get23:59:263
3041428821,48sleep130451-21grep22:29:351
2912328817,45sleep029191-21fschecks_time00:09:330
23562878,15sleep30-21swapper/322:44:273
1264328714,59sleep112708-21cat21:34:151
58372860,7sleep20-21swapper/221:09:442
2859128613,61sleep328673-21awk20:39:423
2859128613,61sleep328673-21awk20:39:423
1409028612,57sleep014144-21sendmail21:39:110
1409028612,57sleep014144-21sendmail21:39:110
2953628510,59sleep30-21swapper/320:44:263
224602855,45sleep30-21swapper/323:49:203
224602855,45sleep30-21swapper/323:49:203
360728426,42sleep23612-21modprobe22:49:112
1630228412,34sleep30-21swapper/323:29:263
3189228311,51sleep10-21swapper/119:09:261
3189228311,51sleep10-21swapper/119:09:261
676328210,35sleep30-21swapper/319:29:403
484028221,36sleep20-21swapper/221:09:122
3163028129,28sleep00-21swapper/019:09:130
3134528113,53sleep231362-21cut20:49:382
2820228118,43sleep228252-21if_eth020:39:322
2820228118,43sleep228252-21if_eth020:39:322
276372819,37sleep30-21swapper/322:19:443
1379428113,56sleep213867-21switchtime23:19:402
1143628118,20sleep10-21swapper/123:14:131
109172817,60sleep310929-21apt-get19:44:263
522428026,29sleep10-21swapper/119:24:411
217732807,36sleep30-21swapper/320:19:223
1022028015,24sleep20-21swapper/221:24:392
593927911,49sleep30-21swapper/322:54:413
31952799,33sleep30-21swapper/319:19:263
316727916,48sleep03186-21apt-get19:19:230
2226127910,54sleep022334-21latency_hist23:49:130
2226127910,54sleep022334-21latency_hist23:49:130
2090227922,37sleep121044-21grep20:14:441
17402798,31sleep30-21swapper/320:59:243
3068327826,26sleep20-21swapper/219:04:352
1798727815,30sleep00-21swapper/021:50:090
2786427713,44sleep30-21swapper/300:04:393
2391827718,17sleep10-21swapper/120:24:461
514927521,31sleep10-21swapper/122:54:111
3226027515,24sleep20-21swapper/219:09:362
3226027515,24sleep20-21swapper/219:09:362
127727520,32sleep00-21swapper/022:39:400
2875927411,48sleep028799-21latency_hist00:09:120
2745927420,29sleep227498-21ntpq22:19:392
12222749,13sleep30-21swapper/300:24:263
467627315,41sleep04703-21awk00:34:360
2002727322,21sleep20-21swapper/223:39:432
2604727214,41sleep126100-21/usr/sbin/munin23:59:361
1489927223,31sleep00-21swapper/021:39:430
1489927223,31sleep00-21swapper/021:39:430
121827221,25sleep10-21swapper/119:14:191
101827218,22sleep20-21swapper/219:14:162
985427119,23sleep00-21swapper/019:39:420
1598827111,14sleep00-21swapper/019:59:370
1294227114,39sleep013003-21/usr/sbin/munin19:49:400
857527017,27sleep10-21swapper/121:19:331
709527022,31sleep10-21swapper/122:59:321
672127019,27sleep20-21swapper/221:14:342
550927019,24sleep00-21swapper/019:24:440
3219927023,25sleep20-21swapper/222:34:412
3219927023,25sleep20-21swapper/222:34:412
306362705,10sleep10-21swapper/120:47:111
3034527014,28sleep10-21swapper/100:14:141
207982700,8sleep10-21swapper/121:59:141
166732708,14sleep10-21swapper/123:29:361
1035727018,35sleep010434-21timerandwakeup21:24:410
789826917,31sleep27904-21diskstats19:34:312
723726913,42sleep17314-21grep21:14:451
325526920,26sleep13021-21apt-get21:04:151
325026916,35sleep13312-21/usr/sbin/munin00:29:411
217632697,20sleep00-21swapper/021:59:500
217632697,20sleep00-21swapper/021:59:500
198192690,10sleep012-21rcu_sched23:39:350
1913426918,34sleep219141-21ntp_states20:09:402
1844226918,29sleep00-21swapper/020:09:140
178412696,13sleep10-21swapper/123:34:241
117902697,28sleep30-21swapper/321:29:413
488126815,37sleep24959-21ls00:34:422
3088426814,40sleep330983-21cat19:04:413
282026812,35sleep10-21swapper/122:44:421
2148126818,30sleep021507-21munin-run20:19:110
972526716,35sleep19745-21apt-get21:24:221
88772679,41sleep08978-21/usr/sbin/munin21:19:400
2996626713,38sleep229990-21ntp_states20:44:372
2957526715,35sleep229645-21/usr/sbin/munin00:09:432
286426715,33sleep02950-21/usr/sbin/munin00:29:310
2556326711,43sleep025571-21cat22:14:320
2352926710,13sleep00-21swapper/020:24:360
146326720,24sleep00-21swapper/019:14:300
676526612,28sleep20-21swapper/222:59:122
190512665,13sleep00-21swapper/020:09:380
870226519,24sleep20-21swapper/221:19:382
1878226511,42sleep1369-21snmpd21:51:311
1460826513,30sleep20-21swapper/219:54:422
74326418,19sleep00-21swapper/019:14:140
74326418,19sleep00-21swapper/019:14:140
2507926412,21sleep00-21swapper/023:55:190
2507926412,21sleep00-21swapper/023:55:190
915526320,25sleep10-21swapper/119:39:151
3181226320,27sleep131867-21modprobe00:19:111
3181226320,27sleep131867-21modprobe00:19:111
2868726318,27sleep20-21swapper/222:24:312
27482635,47sleep20-21swapper/221:00:492
1735526318,30sleep217402-21irqstats20:04:322
1129426317,27sleep011289-21apt-get21:29:250
2703626218,24sleep00-21swapper/020:34:420
2703626218,24sleep00-21swapper/020:34:420
188052625,46sleep20-21swapper/221:51:492
88402610,7sleep10-21swapper/123:04:311
2523726120,20sleep20-21swapper/222:14:142
331326015,27sleep20-21swapper/200:29:422
251152605,44sleep00-21swapper/022:13:150
2137726017,27sleep121409-21ntp_states23:44:381
2137726017,27sleep121409-21ntp_states23:44:381
1571126022,21sleep10-21swapper/121:44:121
1571126022,21sleep10-21swapper/121:44:121
1513226015,27sleep215164-21/usr/sbin/munin23:24:362
2818025910,40sleep00-21swapper/022:22:270
2774625917,21sleep20-21swapper/220:39:122
2774625917,21sleep20-21swapper/220:39:122
252372597,41sleep20-21swapper/223:55:212
252372597,41sleep20-21swapper/223:55:212
2400225910,37sleep224122-21timerwakeupswit20:24:472
3239125818,22sleep10-21swapper/120:54:131
3229125819,21sleep00-21swapper/000:19:300
2787925812,30sleep127917-21processes00:04:411
2716125813,18sleep10-21swapper/122:19:341
1452425818,20sleep10-21swapper/119:54:411
1064125814,24sleep10-21swapper/119:44:111
435225715,17sleep00-21swapper/022:49:370
2908725715,20sleep00-21swapper/022:24:420
2622525712,20sleep00-21swapper/023:59:390
1370725714,19sleep10-21swapper/123:19:391
1326325715,23sleep113299-21ntp_states21:34:441
1326325715,23sleep113299-21ntp_states21:34:441
1247325713,27sleep212467-21apt-get19:49:252
551325613,25sleep05545-21ntp_states21:09:390
530625617,18sleep20-21swapper/219:24:422
3602569,21sleep00-21swapper/020:54:330
189925611,31sleep01994-21timerwakeupswit00:24:430
703425513,24sleep07077-21/usr/sbin/munin21:14:400
350525511,21sleep00-21swapper/021:04:220
3144525515,24sleep231423-21latency_hist22:34:132
252725511,28sleep22501-21latency_hist00:29:132
2368625517,21sleep10-21swapper/122:09:121
2368625517,21sleep10-21swapper/122:09:121
2292325512,30sleep122924-21sshd20:22:311
1521325511,20sleep00-21swapper/023:24:380
552425413,24sleep10-21swapper/122:54:311
543925412,25sleep05433-21apt-get22:54:260
353625412,25sleep13530-21apt-get21:04:271
3108525413,23sleep031154-21df20:49:320
2985225413,24sleep129883-21munin-run22:29:111
220692544,41sleep20-21swapper/222:03:442
220692544,41sleep20-21swapper/222:03:442
1696925412,29sleep117010-21cut20:04:141
1643825413,15sleep20-21swapper/221:44:422
1319525411,20sleep00-21swapper/023:19:210
1138925419,20sleep20-21swapper/221:29:322
509125312,23sleep15085-21apt-get21:09:281
1875325314,22sleep118761-21df_inode20:09:311
1010025312,24sleep210094-21apt-get23:09:262
3250225212,23sleep132594-21users22:34:441
3250225212,23sleep132594-21users22:34:441
3121825221,16sleep20-21swapper/200:14:432
3121825221,16sleep20-21swapper/200:14:432
272702525,33sleep20-21swapper/200:04:142
2684225215,21sleep126992-21sed22:19:131
1681025216,16sleep20-21swapper/223:29:412
365025112,24sleep03660-21munin-run22:49:130
1627825111,17sleep00-21swapper/023:29:230
1482725110,16sleep00-21swapper/019:54:440
123225113,22sleep21268-21ntp_states22:39:382
80525013,21sleep1802-21apt-get22:39:251
745325013,20sleep07464-21ntp_states22:59:390
2238625013,20sleep222388-21apt-get22:04:252
2238625013,20sleep222388-21apt-get22:04:252
1790925013,21sleep217915-21df_inode23:34:252
2767724912,21sleep127711-21munin-run20:39:111
2767724912,21sleep127711-21munin-run20:39:111
2235324916,17sleep20-21swapper/220:19:422
566124811,22sleep15802-21wc00:39:131
1941324812,21sleep119407-21apt-get23:39:251
1171724811,22sleep211724-21df_inode23:14:312
38992469,13sleep20-21swapper/222:49:272
1441424615,17sleep20-21swapper/221:39:292
1441424615,17sleep20-21swapper/221:39:292
2533124411,16sleep20-21swapper/220:29:392
326272439,17sleep10-21swapper/120:54:251
240082408,15sleep10-21swapper/122:09:281
120162389,12sleep10-21swapper/123:14:381
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional