You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-24 - 06:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Thu Sep 24, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2771421615319,16114sleep324062-21kworker/u8:023:13:033
14555210837,1040sleep114599-21ntpq00:12:481
31600253514,456sleep1314-21rs:main1
2889525316,515sleep1367-21snmpd23:16:401
26286247111,442sleep226327-21ntpq23:07:532
437123970,6sleep30-21swapper/320:12:573
10267239613,368sleep310277-21ntpq20:32:493
24235234420,300sleep324268-21fschecks_count21:17:473
18312232925,96sleep10-21swapper/122:42:451
8036232544,134sleep10-21swapper/120:27:261
8036232544,134sleep10-21swapper/120:27:261
2318231617,199sleep12381-21latency20:07:471
16573229210,242sleep116583-21cpu22:37:401
2723227714,205sleep12776-21sendmail-msp23:37:241
2723227714,205sleep12776-21sendmail-msp23:37:241
12660227228,195sleep112676-21modprobe20:42:241
177152268141,109sleep10-21swapper/120:57:421
1221822646,210sleep1367-21snmpd00:06:061
19729225014,213sleep019755-21awk19:22:240
27212224920,64sleep127356-21unixbench_singl19:43:001
27212224920,64sleep127356-21unixbench_singl19:43:001
6855224362,119sleep10-21swapper/123:48:021
15080223960,100sleep10-21swapper/122:32:441
5749223318,199sleep15837-21fschecks_time22:02:441
21899222820,192sleep121989-21fschecks_time19:27:461
21899222820,192sleep121989-21fschecks_time19:27:461
21707222319,124sleep121770-21mailstats22:52:581
5840222015,190sleep25896-21cat22:02:452
31370221526,177sleep131443-21cat23:22:541
18840221045,59sleep10-21swapper/119:17:491
8963220732,123sleep19102-21unixbench_multi20:27:581
18608220661,58sleep10-21swapper/100:27:261
5866220132,132sleep15914-21apt-config23:47:251
5866220132,132sleep15914-21apt-config23:47:251
29351219010,83sleep10-21swapper/123:17:411
16267218143,88sleep10-21swapper/119:12:281
13049218119,132sleep00-21swapper/020:42:430
18688217121,53sleep218733-21switchtime22:42:542
15930216756,24sleep10-21swapper/100:17:451
13867216517,133sleep113886-21latency_hist00:12:241
20977216341,68sleep10-21swapper/122:52:261
16980215945,90sleep10-21swapper/119:12:481
16980215945,90sleep10-21swapper/119:12:481
20155215241,52sleep10-21swapper/100:32:261
25090215121,105sleep225238-21unixbench_singl23:02:572
9942214950,51sleep10-21swapper/120:32:431
21738214719,31sleep00-21swapper/000:37:260
32452214445,63sleep132518-21fschecks_time23:27:431
3270421418,16sleep10-21swapper/121:42:571
181221347,115sleep11922-21cut20:07:271
28544213266,44sleep10-21swapper/121:32:271
26267213210,109sleep126285-21netstat23:07:521
2657521315,11sleep00-21swapper/019:42:470
2657521315,11sleep00-21swapper/019:42:470
21780213132,18sleep10-21swapper/100:37:271
26191212716,26sleep226192-21cron19:42:252
21458212633,77sleep121512-21processes21:07:561
21458212633,77sleep121512-21processes21:07:561
8290212415,95sleep38329-21ntpq23:52:483
7632212120,86sleep37641-21modprobe23:52:243
9898211760,36sleep10-21swapper/123:57:511
1294421178,37sleep10-21swapper/120:42:361
26067211613,50sleep10-21swapper/121:22:501
11643211331,70sleep111669-21sed22:22:261
23195211228,62sleep10-21swapper/122:57:511
23568211149,46sleep10-21swapper/119:32:471
8528211031,19sleep10-21swapper/123:52:541
32368211018,16sleep20-21swapper/223:27:402
3107210912,22sleep10-21swapper/121:52:541
22878210815,77sleep322927-21grep22:57:453
1872121070,6sleep20-21swapper/200:27:262
2800321056,87sleep128024-21dpkg19:47:361
20401210516,64sleep320549-21unixbench_singl22:48:003
19144210414,75sleep219213-21memory00:27:472
14775210415,16sleep10-21swapper/120:47:491
5183210337,17sleep10-21swapper/120:17:401
5183210337,17sleep10-21swapper/120:17:401
20478210117,68sleep320490-21/usr/sbin/munin00:32:453
7685210023,58sleep27809-21latency_hist23:52:272
15431210020,64sleep0231-21systemd-journal00:17:240
2380529813,15sleep10-21swapper/121:17:251
1731029815,65sleep317315-21modprobe20:57:243
52072975,79sleep10-21swapper/122:00:131
138852979,77sleep113897-21ntpq22:27:511
296942955,11sleep00-21swapper/021:33:110
120329520,13sleep10-21swapper/121:47:501
453929414,63sleep24563-21ntp_states21:57:522
42712948,36sleep1374-21sshd23:42:041
420429416,55sleep04295-21fschecks_time21:57:460
2005629436,14sleep10-21swapper/121:02:591
1080429414,63sleep30-21swapper/322:17:523
326612939,60sleep330846-21kworker/u8:320:02:243
2969929134,15sleep10-21swapper/119:52:451
2885929112,65sleep228947-21fschecks_time21:32:442
201422918,59sleep20-21swapper/221:03:292
2167229018,52sleep021827-21grep21:07:590
2167229018,52sleep021827-21grep21:07:590
707128931,38sleep07236-21sed23:48:050
2433628932,13sleep10-21swapper/121:17:491
1397228923,32sleep00-21swapper/019:02:540
1684128820,23sleep00-21swapper/022:37:470
722328713,16sleep10-21swapper/122:07:481
722328713,16sleep10-21swapper/122:07:481
278528723,42sleep22926-21unixbench_singl20:07:562
1994228724,45sleep220006-21cat22:47:512
1551028720,49sleep215452-21latency_hist00:17:262
253832865,33sleep10-21swapper/123:04:291
1990928522,33sleep00-21swapper/021:02:580
161452847,42sleep116234-21perl20:52:451
1334228413,47sleep20-21swapper/219:02:362
2543028314,15sleep20-21swapper/221:22:262
644028217,41sleep06470-21awk20:22:240
644028217,41sleep06470-21awk20:22:240
938028111,20sleep10-21swapper/122:12:571
2767928116,50sleep227170-21apt-get23:12:382
957628023,34sleep0289-21dbus-daemon20:32:250
848828019,30sleep30-21swapper/320:27:483
591528017,31sleep20-21swapper/223:47:262
591528017,31sleep20-21swapper/223:47:262
3193128034,33sleep10-21swapper/119:57:581
3188928020,35sleep031883-21apt-get21:42:400
297072805,12sleep10-21swapper/121:33:201
159628013,30sleep20-21swapper/221:47:592
1335727919,34sleep113350-21apt-get19:02:391
2404127822,36sleep324196-21grep19:32:573
1303027816,42sleep313046-21sort20:42:423
2044427714,45sleep020447-21fschecks_count19:23:290
39812764,10sleep00-21swapper/023:38:200
361627617,38sleep23691-21grep23:37:572
2613427617,46sleep026185-21processes21:22:530
2362527620,39sleep223645-21sed19:32:482
2014927619,34sleep020213-21find22:47:570
262827519,25sleep30-21swapper/321:52:443
3083927422,33sleep030859-21apt19:57:240
60327315,42sleep3712-21perl20:02:483
2500527310,18sleep10-21swapper/119:37:431
22172738,52sleep02218-21sshd21:52:040
2107227314,43sleep321076-21diskstats21:07:463
2107227314,43sleep321076-21diskstats21:07:463
193327318,35sleep31994-21processes23:32:543
193327318,35sleep31994-21processes23:32:543
2306127218,35sleep20-21swapper/222:57:482
2293227212,41sleep00-21swapper/022:57:460
1555827214,45sleep315645-21timerandwakeup19:07:583
3093827117,37sleep231061-21ls19:57:262
2298627121,23sleep30-21swapper/321:12:553
2069327142,14sleep10-21swapper/119:23:341
789527026,26sleep20-21swapper/223:52:352
2181327014,31sleep20-21swapper/219:27:432
2181327014,31sleep20-21swapper/219:27:432
1812227015,31sleep00-21swapper/022:42:400
1766427020,23sleep00-21swapper/000:22:490
74626921,29sleep20-21swapper/221:47:262
1875426916,24sleep30-21swapper/322:42:553
1411926915,22sleep20-21swapper/222:27:562
1188926920,33sleep10-21swapper/122:22:381
584526813,24sleep00-21swapper/023:47:240
584526813,24sleep00-21swapper/023:47:240
3042626814,28sleep00-21swapper/021:37:430
2983626824,20sleep30-21swapper/319:52:473
1167326810,15sleep20-21swapper/220:37:482
1664526719,32sleep316654-21/usr/sbin/munin22:37:423
874726613,22sleep20-21swapper/220:27:552
261726623,25sleep20-21swapper/221:52:432
1633226624,19sleep30-21swapper/322:37:263
1458326616,35sleep214654-21fschecks_time20:47:462
1287126622,19sleep00-21swapper/000:07:430
1050226612,39sleep010655-21unixbench_multi20:32:540
45932655,10sleep10-21swapper/120:14:011
318792658,16sleep00-21swapper/019:57:570
2065526523,24sleep20-21swapper/200:32:482
1494426519,27sleep014937-21apt-get19:07:380
1197526515,31sleep212032-21fschecks_count22:22:432
48162645,48sleep00-21swapper/020:16:470
3060826414,34sleep330639-21/usr/sbin/munin21:37:463
27392649,15sleep00-21swapper/021:52:450
2502026418,21sleep30-21swapper/319:37:453
2157226420,20sleep20-21swapper/219:27:272
1498726416,29sleep315048-21/usr/sbin/munin20:47:553
1260626418,24sleep20-21swapper/200:07:372
1145526419,19sleep00-21swapper/020:37:420
1143226417,23sleep00-21swapper/000:02:500
1114826422,20sleep30-21swapper/300:02:453
1002426415,31sleep0231-21systemd-journal22:17:240
1082226312,33sleep010884-21latency_hist00:02:260
2322026224,21sleep023367-21grep21:13:000
2181126214,34sleep0367-21snmpd19:27:410
2181126214,34sleep0367-21snmpd19:27:410
2131526219,23sleep00-21swapper/022:52:490
2059326217,28sleep320606-21/usr/sbin/munin19:23:323
1423926225,22sleep30-21swapper/300:12:423
1186426212,31sleep30-21swapper/320:37:533
1007626210,37sleep210113-21sensors23:57:562
702726114,18sleep00-21swapper/022:07:270
3215526120,18sleep30-21swapper/321:42:473
2851926115,23sleep00-21swapper/021:32:260
2749526118,21sleep20-21swapper/221:27:522
2568126116,27sleep025831-21latency_hist23:07:270
241726117,31sleep32463-21ntpq20:07:493
1526326116,24sleep015319-21latency22:32:470
852026013,26sleep30-21swapper/322:12:273
852026013,26sleep30-21swapper/322:12:273
71426015,32sleep0764-21if_eth020:02:490
2598626018,22sleep00-21swapper/023:07:470
1987926018,25sleep320095-21apt-config19:22:273
188472604,11sleep30-21swapper/300:27:403
169782608,11sleep20-21swapper/200:22:022
1675426022,19sleep20-21swapper/222:37:452
1623526014,28sleep216289-21/usr/sbin/munin20:52:472
673125921,25sleep06781-21cpuspeed20:22:400
673125921,25sleep06781-21cpuspeed20:22:400
312825918,25sleep30-21swapper/323:37:463
1601725924,19sleep30-21swapper/300:17:473
965525814,26sleep39667-21vmstat22:12:593
568525815,17sleep20-21swapper/220:17:542
568525815,17sleep20-21swapper/220:17:542
335925818,21sleep00-21swapper/020:12:260
2966425820,18sleep30-21swapper/321:32:573
2821025814,24sleep228263-21if_err_eth019:47:452
2794925810,35sleep028001-21basename23:13:530
2524325816,26sleep325252-21vmstat23:02:583
2188625817,24sleep321892-21df_inode19:27:443
2188625817,24sleep321892-21df_inode19:27:443
1893825818,23sleep00-21swapper/021:02:260
1013625813,21sleep20-21swapper/222:17:262
3119525713,23sleep00-21swapper/023:22:490
280325715,19sleep00-21swapper/023:37:270
280325715,19sleep00-21swapper/023:37:270
2692725714,26sleep30-21swapper/321:27:243
2551225711,25sleep00-21swapper/021:22:290
1979825719,21sleep30-21swapper/321:02:573
1133125719,19sleep20-21swapper/200:02:482
589725617,24sleep35930-21irqstats22:02:463
540225615,24sleep35424-21munin-run22:02:253
302522565,40sleep20-21swapper/223:18:522
146182568,19sleep30-21swapper/319:07:253
71932558,16sleep20-21swapper/222:07:412
71932558,16sleep20-21swapper/222:07:412
672825516,23sleep26724-21apt-get20:22:382
672825516,23sleep26724-21apt-get20:22:382
359725514,23sleep20-21swapper/220:12:412
1281225514,18sleep30-21swapper/300:07:423
1084725513,19sleep00-21swapper/022:17:530
77925413,22sleep2873-21memory20:02:512
3185425414,23sleep231862-21ntp_states19:57:552
3161225414,25sleep231650-21munin-run21:42:252
2388325418,21sleep20-21swapper/221:17:262
1770225412,26sleep317711-21diskstats20:57:403
1573025414,24sleep015742-21apt-get00:17:360
157252545,37sleep00-21swapper/020:51:560
1475625415,20sleep00-21swapper/022:32:260
957325314,24sleep39580-21df_inode23:57:413
495025314,24sleep34952-21memory23:42:473
495025314,24sleep34952-21memory23:42:473
2934025316,19sleep00-21swapper/019:52:280
2185925316,14sleep30-21swapper/322:53:013
1812825319,18sleep20-21swapper/219:17:052
1812825319,18sleep20-21swapper/219:17:052
1732325312,23sleep017388-21systemd-run20:57:250
730725216,18sleep00-21swapper/022:07:510
730725216,18sleep00-21swapper/022:07:510
2926625212,24sleep029261-21apt-get23:17:360
2684425214,17sleep30-21swapper/323:08:273
2574425211,16sleep20-21swapper/221:22:432
2493025212,23sleep024926-21apt-get19:37:380
2348125211,17sleep00-21swapper/019:32:460
1896025217,21sleep219034-21tr19:17:542
182112525,36sleep30-21swapper/300:23:003
1774325212,23sleep217835-21perl20:57:432
1709225217,18sleep30-21swapper/319:12:503
1709225217,18sleep30-21swapper/319:12:503
159682526,17sleep20-21swapper/219:09:222
107825215,22sleep31153-21fschecks_time21:47:473
792025111,23sleep07916-21apt-get23:52:370
2719925114,21sleep327218-21apt-get21:27:403
208772519,15sleep20-21swapper/221:07:282
208772519,15sleep20-21swapper/221:07:282
1188825112,22sleep011884-21apt-get22:22:370
309082506,15sleep30-21swapper/323:22:443
2966125012,18sleep30-21swapper/323:17:483
276152507,33sleep30-21swapper/319:45:493
276152507,33sleep30-21swapper/319:45:493
1343525011,22sleep313433-21apt-get22:27:373
52582499,15sleep30-21swapper/320:17:423
52582499,15sleep30-21swapper/320:17:423
3236624911,22sleep032359-21apt-get23:27:380
1884424912,21sleep018840-21apt-get00:27:370
177924910,20sleep21834-21memory23:32:492
177924910,20sleep21834-21memory23:32:492
1584824912,21sleep215894-21users22:32:572
1042724916,19sleep20-21swapper/222:17:432
1715324813,16sleep00-21swapper/019:12:510
1715324813,16sleep00-21swapper/019:12:510
3144424713,18sleep20-21swapper/223:22:552
2810124711,22sleep328109-21sh19:47:423
2696224712,17sleep20-21swapper/219:42:572
2696224712,17sleep20-21swapper/219:42:572
2178224712,14sleep20-21swapper/222:52:592
118912467,14sleep30-21swapper/322:22:403
965024311,13sleep20-21swapper/220:32:272
67292437,18sleep30-21swapper/320:22:393
67292437,18sleep30-21swapper/320:22:393
325762438,16sleep30-21swapper/323:27:453
1338724210,16sleep20-21swapper/220:42:492
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional