You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 13:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Mar 27, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26710229866,2968sleep225187-21kworker/2:222:55:472
30572193624,1896sleep03948-21kworker/u8:421:22:520
18743267315,640sleep018776-21irqstats00:12:470
8967259574,447sleep28996-21idleruntime-cro21:42:252
15805244822,411sleep215841-21ntpq00:02:512
15805244822,411sleep215841-21ntpq00:02:512
591723560,148sleep224562-21cyclictest21:32:262
12263232721,294sleep212299-21ntpq20:07:502
229722324118,190sleep223108-21latency_hist00:27:272
24112231840,211sleep224261-21latency_hist20:47:282
25167230210,276sleep025206-21ntpq19:07:500
494522830,31sleep224562-21cyclictest23:28:022
1859422787,237sleep218677-21fschecks_time00:12:452
16333223810,135sleep216415-21latency_hist20:22:282
16333223810,135sleep216415-21latency_hist20:22:282
27146223512,166sleep227149-21cron20:57:262
27146223512,166sleep227149-21cron20:57:262
26160222610,204sleep126234-21cat20:52:481
26160222610,204sleep126234-21cat20:52:481
2178222216,191sleep12181-21entropy19:37:441
19668222012,183sleep20-21swapper/220:32:422
7193221518,174sleep27332-21dpkg23:37:282
7193221518,174sleep27332-21dpkg23:37:282
29969221116,170sleep014419-21kworker/u8:323:07:240
12683221142,51sleep20-21swapper/223:52:542
12683221142,51sleep20-21swapper/223:52:542
24709220633,23sleep20-21swapper/220:47:482
237821989,177sleep013441-21snmpd21:20:420
23787219729,131sleep223879-21latency_hist22:47:272
13535218618,149sleep113585-40ogrotate23:57:251
13535218618,149sleep113585-40ogrotate23:57:251
9609218510,82sleep29653-21systemd-run21:47:262
24793217923,76sleep20-21swapper/200:32:402
30548217510,149sleep230563-21acpi21:07:352
18081217410,74sleep20-21swapper/222:27:482
218221649,142sleep22231-21fschecks_count19:37:452
143821590,23sleep31437-21fschecks_time21:17:453
1499321589,76sleep20-21swapper/222:17:512
17599215717,65sleep217748-21latency_hist22:27:282
860421556,36sleep20-21swapper/223:38:452
14179215237,38sleep214319-21unixbench_singl20:13:032
25402215111,118sleep225483-21timerandwakeup19:07:562
29586215032,81sleep229602-21modprobe19:22:262
3683214942,22sleep20-21swapper/219:42:292
3683214942,22sleep20-21swapper/219:42:292
2683721499,74sleep226885-21fschecks_count19:12:492
3006214224,94sleep32525-21apt-get21:22:353
18092213912,24sleep20-21swapper/220:27:372
1230213623,15sleep20-21swapper/221:17:372
69621359,110sleep2835-21latency_hist23:17:282
24023213435,29sleep20-21swapper/222:47:402
28895213037,31sleep20-21swapper/223:02:462
10520213028,67sleep210575-21cat20:02:492
32210212510,75sleep232287-21fschecks_time21:12:442
21103212512,25sleep20-21swapper/222:37:462
21000212515,89sleep321106-21latency_hist20:37:273
16570212510,44sleep20-21swapper/220:22:402
16570212510,44sleep20-21swapper/220:22:402
1043621259,80sleep210475-21modprobe23:47:252
729212416,14sleep20-21swapper/219:32:482
27895212310,48sleep228027-21head22:57:572
27895212310,48sleep228027-21head22:57:572
2901421189,68sleep20-21swapper/221:02:392
377021178,39sleep20-21swapper/223:27:262
22899211216,79sleep022943-21systemd00:27:250
674821119,64sleep26855-21wc21:32:562
5787211014,82sleep35788-21sshd21:31:403
1074121106,69sleep20-21swapper/221:55:342
697821099,35sleep20-21swapper/219:52:392
2533321099,84sleep125401-21sensors_temp19:07:551
1120321097,64sleep210702-21/usr/sbin/munin23:47:532
434321089,65sleep24351-21munin-run21:27:272
1765121088,37sleep20-21swapper/200:07:562
1765121088,37sleep20-21swapper/200:07:562
3875210719,70sleep33376-21apt-get19:42:383
3875210719,70sleep33376-21apt-get19:42:383
17032210716,75sleep317038-21open_files20:22:533
17032210716,75sleep317038-21open_files20:22:533
7530210613,76sleep17609-21switchtime19:52:541
10889210620,65sleep010906-21idleruntime-cro21:57:250
5931210416,74sleep36023-21perl23:32:443
5931210416,74sleep36023-21perl23:32:443
2492210311,79sleep02527-21ntpq19:37:510
30937210117,46sleep331075-21unixbench_multi23:07:563
19449210113,63sleep019540-21tune2fs22:32:470
11177210168,15sleep20-21swapper/221:57:362
7075210021,49sleep07134-21modprobe23:37:250
7075210021,49sleep07134-21modprobe23:37:250
23299210022,63sleep023394-21fschecks_time19:02:460
18213210014,72sleep118249-21ntp_kernel_pll_22:27:511
2036629921,33sleep20-21swapper/200:17:502
58492977,78sleep25884-21ntpq19:48:072
244512977,19sleep20-21swapper/200:31:512
2278829724,51sleep022781-21apt-get20:42:410
1550029712,69sleep015512-21/usr/sbin/munin00:02:450
1550029712,69sleep015512-21/usr/sbin/munin00:02:450
2746329618,45sleep127456-21apt-get20:57:411
2783629511,68sleep327881-21/usr/sbin/munin20:57:513
2441829512,65sleep124442-21fschecks_count20:47:431
1037129512,72sleep00-21swapper/023:46:340
697629419,35sleep00-21swapper/019:52:370
524229414,22sleep10-21swapper/119:47:371
524229414,22sleep10-21swapper/119:47:371
2388629428,34sleep30-21swapper/319:02:573
2370529428,32sleep10-21swapper/119:02:551
2289829417,19sleep10-21swapper/120:42:461
319362939,37sleep20-21swapper/219:27:552
221822925,33sleep20-21swapper/220:38:232
1043629213,62sleep110517-21tune2fs20:02:481
201029125,37sleep20-21swapper/223:20:072
902029010,66sleep09098-21latency_hist21:42:270
3130728918,37sleep20-21swapper/223:09:452
2809228717,52sleep128198-21latency_hist19:17:281
2423228612,55sleep024285-21/usr/sbin/munin22:47:460
86372849,60sleep28675-21grep19:57:442
3109528422,29sleep20-21swapper/219:26:542
196652849,16sleep10-21swapper/120:32:411
558828328,29sleep30-21swapper/323:32:283
263872838,59sleep00-21swapper/019:12:300
2093428319,31sleep10-21swapper/122:37:411
137542836,13sleep20-21swapper/222:12:262
131942837,39sleep213340-21cut22:07:292
3027528213,56sleep030279-21cron21:07:250
271542829,54sleep10-21swapper/122:57:401
271542829,54sleep10-21swapper/122:57:401
142252827,35sleep20-21swapper/223:57:512
505228112,17sleep00-21swapper/021:27:530
2742328118,28sleep20-21swapper/220:57:302
2134328114,43sleep021432-21fschecks_time20:37:450
203682815,65sleep20-21swapper/222:34:152
3202828018,36sleep332103-21find19:27:563
2309028022,40sleep323194-21wc22:42:563
102882807,32sleep20-21swapper/221:52:272
713627922,38sleep17170-21munin-run23:37:271
713627922,38sleep17170-21munin-run23:37:271
381827917,43sleep33945-21apt-config23:27:273
1478727921,33sleep30-21swapper/320:17:283
141922795,63sleep20-21swapper/222:14:442
141922795,63sleep20-21swapper/222:14:442
1095527920,44sleep011036-21timerandwakeup20:02:590
444227818,25sleep00-21swapper/019:44:090
444227818,25sleep00-21swapper/019:44:090
3189427723,31sleep10-21swapper/119:27:541
2619327722,23sleep30-21swapper/319:12:293
2172727710,55sleep121732-21cat00:22:451
180972770,7sleep30-21swapper/322:27:483
739527618,43sleep37461-21modprobe21:37:253
309027620,26sleep00-21swapper/023:22:560
2055727617,24sleep30-21swapper/300:17:553
1570827617,38sleep115847-21tail20:17:591
1570827617,38sleep115847-21tail20:17:591
695227512,43sleep30-21swapper/319:52:363
308392758,18sleep00-21swapper/019:23:370
2345327518,41sleep323506-21irqstats00:27:483
2439127419,27sleep30-21swapper/322:47:493
2188127418,34sleep321990-21ls20:37:563
2143527423,24sleep10-21swapper/120:37:461
1209427419,36sleep012234-21latency_hist23:52:280
1171027420,35sleep011804-21mailstats21:57:540
905027325,29sleep00-21swapper/019:57:540
250927315,42sleep12522-21sh21:22:251
2373627324,31sleep023765-21awk22:47:250
19252737,12sleep00-21swapper/023:19:000
2474027222,28sleep30-21swapper/320:47:493
85952715,17sleep00-21swapper/023:38:390
774827113,35sleep17601-21/usr/sbin/munin23:38:071
1993727119,21sleep30-21swapper/322:32:583
1793727117,29sleep00-21swapper/022:27:460
891827017,40sleep38948-21awk19:57:493
43562699,11sleep00-21swapper/023:27:440
1867926926,20sleep30-21swapper/300:12:463
1441126915,21sleep10-21swapper/123:57:581
64722686,51sleep00-21swapper/019:49:380
2120326821,28sleep00-21swapper/022:37:480
1756826823,23sleep10-21swapper/122:27:271
98772677,47sleep00-21swapper/021:47:400
2558326712,42sleep325618-21grep22:52:413
2384426714,36sleep023918-21find00:27:570
1977626716,33sleep119788-21modprobe00:17:241
103112678,44sleep30-21swapper/320:02:343
316402666,49sleep00-21swapper/021:09:560
1719726620,17sleep00-21swapper/000:07:460
1719726620,17sleep00-21swapper/000:07:460
130126616,35sleep31303-21memory23:17:503
810926521,20sleep30-21swapper/321:37:503
61626519,28sleep3644-21munin-run23:17:253
464126526,21sleep10-21swapper/121:27:421
303126519,27sleep13027-21apt-get21:22:371
1543426521,22sleep30-21swapper/322:18:003
1412926519,23sleep00-21swapper/023:57:490
294142649,12sleep10-21swapper/123:02:571
4892638,14sleep00-21swapper/019:32:380
3053826319,20sleep30-21swapper/319:22:553
3027226323,23sleep10-21swapper/119:22:491
1811926318,27sleep118120-21sshd20:27:401
747326216,20sleep10-21swapper/121:37:271
3249326218,22sleep30-21swapper/321:12:493
2454226224,23sleep224675-21sed19:07:282
2140426216,32sleep021474-21latency_hist00:22:280
2065426219,21sleep10-21swapper/100:17:571
1654126218,25sleep016634-21memory22:22:500
1541726212,25sleep115484-21/usr/sbin/munin00:02:421
1541726212,25sleep115484-21/usr/sbin/munin00:02:421
1509326216,25sleep00-21swapper/022:17:530
146522625,16sleep00-21swapper/020:16:400
1029926220,18sleep30-21swapper/321:52:283
771026117,20sleep10-21swapper/121:37:401
71732617,40sleep00-21swapper/021:34:360
2741626113,35sleep027429-21uname22:57:470
2741626113,35sleep027429-21uname22:57:470
2605926113,24sleep30-21swapper/320:52:463
2605926113,24sleep30-21swapper/320:52:463
252826115,28sleep12555-21sshd23:22:401
2323926115,20sleep10-21swapper/100:27:421
78826016,23sleep30-21swapper/319:32:493
2499626016,30sleep124998-21sshd22:48:391
2486326017,18sleep00-21swapper/000:32:420
1198126020,22sleep10-21swapper/120:07:451
927225919,20sleep30-21swapper/323:42:503
454425918,25sleep14560-21/usr/sbin/munin23:27:531
3138925913,23sleep10-21swapper/123:10:491
2061625915,23sleep10-21swapper/122:37:251
1286125919,17sleep30-21swapper/323:52:593
1286125919,17sleep30-21swapper/323:52:593
4122587,42sleep00-21swapper/023:14:520
2841125819,22sleep30-21swapper/319:17:423
2284825816,27sleep122887-21ntpq22:42:501
1408525812,18sleep10-21swapper/120:13:021
3180525717,25sleep131800-21apt-get23:12:391
3140325711,19sleep00-21swapper/019:27:370
2763825717,21sleep30-21swapper/322:57:543
2763825717,21sleep30-21swapper/322:57:543
2688725717,23sleep326960-21fw_packets19:12:503
1341925721,22sleep00-21swapper/023:55:590
1341925721,22sleep00-21swapper/023:55:590
94212566,39sleep00-21swapper/021:45:020
673925615,21sleep30-21swapper/319:52:283
65082567,11sleep10-21swapper/119:50:081
2556325610,16sleep30-21swapper/300:32:573
1941525616,22sleep319424-21munin-run20:32:263
1479725615,24sleep114804-21df_inode22:17:461
1234725618,20sleep00-21swapper/020:07:520
101625615,19sleep10-21swapper/121:17:281
994725516,18sleep10-21swapper/120:02:271
967125514,26sleep19739-21mailstats23:43:001
3115025513,26sleep131219-21switchtime21:07:551
2722325515,19sleep00-21swapper/020:57:280
2722325515,19sleep00-21swapper/020:57:280
2182425511,25sleep30-21swapper/300:22:483
1743625511,29sleep316937-21/usr/sbin/munin00:07:533
1743625511,29sleep316937-21/usr/sbin/munin00:07:533
1656825517,22sleep016564-21apt-get20:22:380
1656825517,22sleep016564-21apt-get20:22:380
1628225511,25sleep016318-21awk20:22:260
1628225511,25sleep016318-21awk20:22:260
1120625514,25sleep311273-21df21:57:413
1104525514,23sleep111096-21memory23:47:481
2901525419,16sleep30-21swapper/321:02:403
2830225414,24sleep028320-21apt-get19:17:350
246225413,21sleep30-21swapper/319:37:503
1966325414,24sleep019658-21apt-get20:32:400
1480525412,19sleep30-21swapper/323:58:243
1316525417,16sleep10-21swapper/122:07:281
765525317,21sleep30-21swapper/323:37:573
6052536,38sleep00-21swapper/021:13:190
2460325310,29sleep024617-21grep20:47:460
1893725314,24sleep019022-21users20:27:590
1510625312,25sleep315110-21diskstats20:17:443
1510625312,25sleep315110-21diskstats20:17:443
1319925315,24sleep313242-21date20:12:273
1252125315,17sleep10-21swapper/122:02:271
2522125215,22sleep125245-21ntp_states00:32:511
1567225213,24sleep315685-21irqstats00:02:483
1567225213,24sleep315685-21irqstats00:02:483
3067425117,14sleep30-21swapper/321:07:443
2557925115,20sleep125574-21apt-get22:52:391
2223025114,22sleep322239-21munin-run22:42:273
1698225115,20sleep117024-21ntp_states20:22:511
1698225115,20sleep117024-21ntp_states20:22:511
1696125111,24sleep116955-21apt-get00:07:391
1696125111,24sleep116955-21apt-get00:07:391
1670925114,22sleep316720-21ntp_states22:22:533
90625014,19sleep10-21swapper/123:17:351
2964025016,18sleep10-21swapper/119:22:291
1380625012,15sleep00-21swapper/022:12:280
1203625019,17sleep30-21swapper/323:52:273
5062498,16sleep10-21swapper/119:32:391
2638524910,14sleep00-21swapper/022:52:590
2796824818,15sleep00-21swapper/020:57:560
3026224711,19sleep030330-21df23:07:410
1024724713,20sleep110276-21munin-run21:52:261
2870924610,21sleep328705-21apt-get23:02:393
2286624611,21sleep322872-21diskstats20:42:443
1448124615,15sleep30-21swapper/322:17:273
1448124615,15sleep30-21swapper/322:17:273
129332465,29sleep30-21swapper/322:05:083
1092124611,21sleep110965-21munin-run21:57:261
3031324512,18sleep10-21swapper/121:07:261
127942456,27sleep00-21swapper/022:03:190
965424314,15sleep30-21swapper/321:47:273
136942439,15sleep10-21swapper/122:11:401
1334124312,20sleep30-21swapper/322:07:303
1058023916,15sleep30-21swapper/321:53:293
139222389,12sleep10-21swapper/122:12:291
139222389,12sleep10-21swapper/122:12:291
141592346,12sleep00-21swapper/022:14:260
141592346,12sleep00-21swapper/022:14:260
91002335,10sleep10-21swapper/121:42:281
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional