You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 07:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Mar 12, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
283682509918,5068sleep019034-21kworker/0:123:40:360
283682509918,5068sleep019034-21kworker/0:123:40:360
140882311914,3090sleep3960-21kworker/3:221:13:353
10914216459,1616sleep023822-1kworker/0:2H00:28:360
3017527898,737sleep130151-21latency_hist23:48:201
26257251111,483sleep326305-21/usr/sbin/munin20:08:403
26257251111,483sleep326305-21/usr/sbin/munin20:08:403
7441246411,438sleep27451-21ntpq19:08:442
11204241074,214sleep10-21swapper/119:18:481
210872264101,65sleep121228-21head21:33:481
12921225762,72sleep112955-21ps00:33:461
22038225314,221sleep322138-21fschecks_time21:38:383
10489224711,197sleep110544-21tr22:43:481
234752241108,23sleep10-21swapper/119:58:491
16821224072,52sleep10-21swapper/119:38:341
310222233141,74sleep131112-21latency_hist22:08:201
764222247,72sleep1777-21modprobe23:58:171
1698122019,175sleep317080-21latency_hist21:23:203
12302220159,82sleep10-21swapper/121:08:211
358421996,179sleep13690-21tune2fs22:23:381
15879219715,72sleep10-21swapper/121:18:381
2209218852,22sleep10-21swapper/122:18:411
1624921860,73sleep20-21swapper/221:18:462
20124218111,115sleep10-21swapper/121:33:201
27700217432,123sleep227741-21fschecks_count20:13:392
7190216428,80sleep16552-21/usr/sbin/munin22:33:471
7190216428,80sleep16552-21/usr/sbin/munin22:33:471
8187216127,72sleep18203-21kernelversion20:54:361
7703215226,60sleep10-21swapper/100:18:401
5764215211,123sleep11-21systemd20:48:181
6231214943,59sleep10-21swapper/100:13:471
29842214914,121sleep329964-21sed20:18:493
1596214830,106sleep11687-21cut23:58:451
11296214889,17sleep10-21swapper/100:28:531
30714214610,122sleep330733-21apt-get20:23:293
12125214610,78sleep112249-21cat22:48:491
12125214610,78sleep112249-21cat22:48:491
440214519,71sleep10-21swapper/120:28:481
2634321459,116sleep126042-21/usr/sbin/munin20:08:421
2634321459,116sleep126042-21/usr/sbin/munin20:08:421
3085521435,128sleep00-21swapper/022:06:490
15682214328,13sleep10-21swapper/123:00:361
15682214328,13sleep10-21swapper/123:00:361
1004214111,17sleep10-21swapper/122:13:491
8423213712,108sleep18489-21cpuspeed22:38:441
14201213734,62sleep10-21swapper/119:28:491
17799213618,27sleep00-21swapper/023:08:340
2557821340,6sleep00-21swapper/023:33:200
2557821340,6sleep00-21swapper/023:33:200
31969213313,107sleep232108-21unixbench_singl22:08:492
10911213211,105sleep210899-21apt-get00:28:342
23237213019,92sleep10-21swapper/123:23:491
15356213029,66sleep115419-21cat19:33:381
28626212956,44sleep10-21swapper/123:43:191
28626212956,44sleep10-21swapper/123:43:191
19037212846,26sleep10-21swapper/123:13:191
5604212620,21sleep10-21swapper/122:28:461
22520212649,22sleep10-21swapper/121:38:491
17483212611,80sleep117513-21cut23:08:191
10435212616,92sleep310523-21latency_hist00:28:193
13620212513,14sleep10-21swapper/122:53:461
27685212410,101sleep027693-21df20:13:370
2418621248,26sleep10-21swapper/123:28:291
2418621248,26sleep10-21swapper/123:28:291
10043212234,57sleep10-21swapper/120:58:481
5056211817,82sleep30-21swapper/320:43:523
28204211620,52sleep10-21swapper/120:13:511
355921159,18sleep10-21swapper/100:03:501
32567211558,42sleep10-21swapper/122:13:201
17584211518,79sleep317607-21/usr/sbin/munin21:23:413
17584211518,79sleep317607-21/usr/sbin/munin21:23:413
13044211419,77sleep313084-21cpu22:53:313
2640211329,63sleep22675-21sendmail20:38:182
2757021125,95sleep00-21swapper/021:54:500
2757021125,95sleep00-21swapper/021:54:500
11709211215,77sleep311759-21/usr/sbin/munin22:48:393
11709211215,77sleep311759-21/usr/sbin/munin22:48:393
1655211149,16sleep10-21swapper/120:33:411
787210918,75sleep2847-21unin-run23:58:182
23053210914,74sleep323089-21fschecks_count19:58:393
892021080,7sleep20-21swapper/200:23:202
4197210830,65sleep14189-21apt-get00:08:341
4197210830,65sleep14189-21apt-get00:08:341
32051210815,77sleep132118-21latency_hist20:28:201
25166210715,24sleep10-21swapper/120:03:491
2114821068,62sleep121150-21munin-run19:53:201
2114821068,62sleep121150-21munin-run19:53:201
1519921060,13sleep20-21swapper/222:58:462
1519921060,13sleep20-21swapper/222:58:462
9294210410,81sleep19355-21perl00:23:571
19540210416,72sleep019669-21awk21:28:510
1812921049,62sleep117192-21/usr/sbin/munin21:23:511
1812921049,62sleep117192-21/usr/sbin/munin21:23:511
13264210311,23sleep113396-21grep21:08:491
2946521028,59sleep129471-21munin-run22:03:191
2946521028,59sleep129471-21munin-run22:03:191
7840210017,65sleep37346-21apt-get20:53:293
3074321007,80sleep00-21swapper/020:23:340
18268210012,18sleep10-21swapper/119:43:311
1544429812,69sleep315536-21/usr/sbin/munin19:33:403
2579929715,65sleep325826-21munin-run20:08:193
234729715,66sleep02392-21/usr/sbin/munin22:18:440
124132970,14sleep012-21rcu_sched22:48:510
124132970,14sleep012-21rcu_sched22:48:510
343729611,19sleep10-21swapper/120:38:451
343729611,19sleep10-21swapper/120:38:451
2955929627,18sleep00-21swapper/023:43:510
156342966,12sleep00-21swapper/019:33:450
3041229510,16sleep10-21swapper/123:48:321
196732958,49sleep10-21swapper/123:13:441
693629421,13sleep10-21swapper/120:48:511
1266629414,59sleep212433-21/usr/sbin/munin00:33:392
580829211,68sleep25834-21latency_hist20:48:192
466829214,61sleep04706-21/usr/sbin/munin00:08:500
466829214,61sleep04706-21/usr/sbin/munin00:08:500
1838029215,61sleep018384-21diskstats19:43:380
249752916,69sleep00-21swapper/020:03:440
2199629152,26sleep122082-21timerandwakeup19:53:481
679529020,50sleep06932-21grep20:48:500
3265129010,64sleep232662-21needreboot20:28:412
2700029021,24sleep127035-21cat21:53:461
2700029021,24sleep127035-21cat21:53:461
2629228961,14sleep10-21swapper/121:53:181
2968628829,45sleep129768-21mailstats20:18:471
2616328521,40sleep226255-21sort20:08:392
2616328521,40sleep226255-21sort20:08:392
1248728510,52sleep20-21swapper/219:23:462
1248728510,52sleep20-21swapper/219:23:462
831328419,44sleep38402-21latency_hist19:13:213
2861028411,59sleep328618-21ntpq21:58:453
2861028411,59sleep328618-21ntpq21:58:453
2002928418,23sleep00-21swapper/019:48:380
2002928418,23sleep00-21swapper/019:48:380
1438528415,49sleep314523-21/usr/sbin/munin19:28:513
1121128422,40sleep211236-21munin-run22:48:172
850428320,43sleep28579-21switchtime20:54:452
705328326,37sleep30-21swapper/322:33:453
705328326,37sleep30-21swapper/322:33:453
699028322,16sleep10-21swapper/119:08:301
2308528320,43sleep023234-21grep23:23:480
742428232,37sleep27543-21date20:53:202
664728212,55sleep06656-21df_inode22:33:340
664728212,55sleep06656-21df_inode22:33:340
2357128247,20sleep123615-21grep21:43:371
630328124,27sleep00-21swapper/019:03:500
503128121,44sleep35084-21ls22:28:323
3041328116,47sleep230414-21cpu23:48:332
124492819,18sleep112485-21ntpq19:23:451
124492819,18sleep112485-21ntpq19:23:451
2208428021,34sleep20-21swapper/219:53:492
48772790,14sleep30-21swapper/300:08:523
48772790,14sleep30-21swapper/300:08:523
280812799,56sleep328119-21ps20:13:483
2168927910,50sleep00-21swapper/021:38:190
1448527914,48sleep014520-21/usr/sbin/munin21:13:450
135727917,43sleep21063-21/usr/sbin/munin23:58:382
113327822,35sleep01146-21latency_hist20:33:190
1065027819,39sleep210735-21fschecks_time19:18:372
1877527723,32sleep20-21swapper/219:43:482
671727622,36sleep26734-21munin-run19:08:192
2374627614,39sleep00-21swapper/021:43:400
2193627615,31sleep00-21swapper/021:38:310
150642766,12sleep10-21swapper/121:14:381
77522756,15sleep00-21swapper/020:53:220
1038227511,49sleep210434-21apt-config00:28:182
407027420,32sleep00-21swapper/022:23:490
1441627411,14sleep00-21swapper/022:58:210
3017127327,27sleep20-21swapper/222:03:452
218812735,11sleep10-21swapper/123:20:141
1135327314,16sleep00-21swapper/021:03:430
859027245,15sleep18702-21diskstats19:13:231
266372727,30sleep10-21swapper/123:33:511
266372727,30sleep10-21swapper/123:33:511
2372127222,26sleep30-21swapper/323:28:193
2366427219,34sleep223720-21apt-config23:28:182
741827021,26sleep00-21swapper/000:18:350
273827020,26sleep30-21swapper/300:03:363
1216327016,33sleep312048-21/usr/sbin/munin19:23:383
1216327016,33sleep312048-21/usr/sbin/munin19:23:383
909126922,25sleep00-21swapper/022:39:270
840626920,31sleep08589-21modprobe19:13:220
316712696,14sleep30-21swapper/323:53:183
3098826811,37sleep00-21swapper/023:48:470
2953326815,39sleep329425-21cron22:03:213
511426715,31sleep25006-21/usr/sbin/munin22:28:352
2667026620,28sleep30-21swapper/321:53:403
2667026620,28sleep30-21swapper/321:53:403
2149626611,41sleep021626-21sed23:18:480
1655726617,23sleep00-21swapper/023:03:420
1452526613,29sleep00-21swapper/019:28:520
137832668,12sleep20-21swapper/221:13:062
84826516,24sleep30-21swapper/323:58:193
791026526,19sleep30-21swapper/300:18:463
521526510,16sleep00-21swapper/022:28:370
314826521,25sleep00-21swapper/000:03:450
2874926515,33sleep228831-21find21:58:482
2874926515,33sleep228831-21find21:58:482
158926513,36sleep31639-21/usr/sbin/munin20:33:393
754526417,22sleep30-21swapper/320:53:213
394926425,25sleep23989-21ps22:23:472
241432647,45sleep00-21swapper/020:02:110
55126324,21sleep30-21swapper/322:13:393
2857526327,21sleep30-21swapper/323:43:173
2857526327,21sleep30-21swapper/323:43:173
2179426313,28sleep021841-21ntp_offset19:53:430
2940026217,27sleep229470-21/usr/sbin/munin23:43:492
2054226212,16sleep30-21swapper/323:18:183
15332625,11sleep00-21swapper/022:17:110
125762625,11sleep20-21swapper/222:50:562
125762625,11sleep20-21swapper/222:50:562
1243226211,33sleep00-21swapper/000:33:280
1186926218,19sleep30-21swapper/319:23:223
815826111,19sleep30-21swapper/322:38:213
779126122,23sleep20-21swapper/222:37:352
779126122,23sleep20-21swapper/222:37:352
447926114,27sleep04496-21apt20:43:290
426926116,32sleep04349-21date20:43:210
426926116,32sleep04349-21date20:43:210
3134426118,27sleep231424-21awk20:23:492
2141726111,35sleep321491-21cat23:18:473
1844026114,19sleep20-21swapper/223:08:482
1811526114,20sleep00-21swapper/021:23:500
1811526114,20sleep00-21swapper/021:23:500
1738326117,21sleep20-21swapper/219:38:472
10572615,10sleep30-21swapper/323:58:233
949026018,26sleep39501-21df_inode20:58:343
324226015,21sleep30-21swapper/322:23:193
3204426012,24sleep00-21swapper/023:53:360
2947126015,33sleep329546-21timerandwakeup23:43:503
2852326018,23sleep00-21swapper/021:58:420
2852326018,23sleep00-21swapper/021:58:420
2550426021,23sleep30-21swapper/323:33:203
2550426021,23sleep30-21swapper/323:33:203
202526015,29sleep22114-21grep22:18:382
1371826013,28sleep20-21swapper/222:53:472
995425919,25sleep30-21swapper/322:43:343
701725913,19sleep30-21swapper/300:18:173
468725919,25sleep34688-21sshd22:27:353
356925918,21sleep30-21swapper/320:38:473
356925918,21sleep30-21swapper/320:38:473
2759125919,20sleep20-21swapper/223:38:402
2759125919,20sleep20-21swapper/223:38:402
2584425914,26sleep325936-21users21:48:523
2577625917,27sleep225778-21cron20:08:182
2511025913,26sleep00-21swapper/021:48:370
2202525916,27sleep222037-21/usr/sbin/munin21:38:372
1989225913,25sleep30-21swapper/321:31:363
1751425916,26sleep20-21swapper/223:08:202
1615325914,25sleep316172-21apt-get23:03:283
1589025920,21sleep00-21swapper/023:03:170
1589025920,21sleep00-21swapper/023:03:170
113225915,19sleep20-21swapper/222:13:502
2581525814,28sleep225824-21/usr/sbin/munin23:33:352
2581525814,28sleep225824-21/usr/sbin/munin23:33:352
921725718,23sleep39311-21memory19:13:413
2211725718,22sleep222143-21munin-run23:23:182
2066025715,17sleep20-21swapper/223:18:212
2035825715,24sleep320355-21apt-get21:33:303
1417325714,28sleep214180-21forks21:13:382
581525616,25sleep25828-21diskstats00:13:362
549125619,18sleep30-21swapper/319:03:363
38712565,21sleep20-21swapper/200:07:512
2398725615,20sleep30-21swapper/321:43:473
210012566,37sleep30-21swapper/319:52:003
210012566,37sleep30-21swapper/319:52:003
193772567,40sleep30-21swapper/319:46:033
1576925615,17sleep00-21swapper/021:18:370
1529625617,17sleep30-21swapper/322:58:483
1529625617,17sleep30-21swapper/322:58:483
942125511,20sleep00-21swapper/000:24:000
2512225519,20sleep20-21swapper/221:48:392
1379825516,16sleep30-21swapper/300:38:203
1126425513,18sleep00-21swapper/022:48:190
3233425415,23sleep332360-21df20:28:343
2962925414,24sleep229653-21ntp_states20:18:442
978225313,26sleep29796-21/usr/sbin/munin20:58:412
550225318,18sleep20-21swapper/200:13:202
550225318,18sleep20-21swapper/200:13:202
2620425316,23sleep025718-21/usr/sbin/munin23:33:450
2620425316,23sleep025718-21/usr/sbin/munin23:33:450
2448925316,16sleep20-21swapper/223:28:592
2448925316,16sleep20-21swapper/223:28:592
2385325319,17sleep20-21swapper/221:43:422
2007925311,16sleep00-21swapper/023:13:510
1984025313,23sleep219834-21apt-get19:48:322
1984025313,23sleep219834-21apt-get19:48:322
1723325313,19sleep00-21swapper/019:38:450
902125213,22sleep29027-21diskstats19:13:362
746225218,18sleep00-21swapper/019:08:460
2685325214,19sleep20-21swapper/221:53:432
2685325214,19sleep20-21swapper/221:53:432
267632526,21sleep30-21swapper/323:35:143
267632526,21sleep30-21swapper/323:35:143
179492526,34sleep30-21swapper/319:42:363
1748925212,24sleep217583-21/usr/sbin/munin21:23:402
1748925212,24sleep217583-21/usr/sbin/munin21:23:402
1408225213,23sleep214088-21ntp_states19:28:462
1087725212,24sleep310396-21apt-get00:28:273
1078825213,18sleep00-21swapper/019:18:390
2917125111,24sleep029190-21apt-get20:18:300
1524525113,23sleep215253-21diskstats19:33:352
271552506,36sleep00-21swapper/020:11:070
271552506,36sleep00-21swapper/020:11:070
2138425014,20sleep321381-21apt-get19:53:303
1772225012,22sleep317719-21apt-get23:08:293
1106925015,16sleep30-21swapper/321:03:383
108425013,21sleep01081-21apt-get23:58:280
836924911,22sleep27868-21apt-get22:38:242
708324911,22sleep07096-21munin-run00:18:180
39382496,31sleep20-21swapper/220:39:392
39382496,31sleep20-21swapper/220:39:392
1315324914,19sleep00-21swapper/021:08:480
984324811,21sleep09881-21ntp_states20:58:430
666424810,18sleep10-21swapper/119:08:181
126142488,14sleep30-21swapper/321:08:353
145924713,17sleep00-21swapper/020:33:360
1217124710,20sleep012261-21perl19:23:390
1217124710,20sleep012261-21perl19:23:390
1125324715,17sleep20-21swapper/221:03:412
3166724613,15sleep00-21swapper/022:08:430
190252459,19sleep219039-21irqstats21:28:402
3140624413,17sleep30-21swapper/322:08:383
203532446,15sleep20-21swapper/221:33:292
234162437,13sleep20-21swapper/219:58:462
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional