You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-01 - 08:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Aug 01, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25827267221,634sleep325873-21/usr/sbin/munin22:03:063
797253173,401sleep1861-21awk20:43:331
797253173,401sleep1861-21awk20:43:331
1805925078,485sleep318103-21ntpq21:38:103
741244829,402sleep1750-21/usr/sbin/munin19:03:011
22842240419,339sleep122875-21cut20:12:401
127024010,40sleep10-21swapper/100:13:001
27434233511,308sleep027482-21unin-run23:52:410
27434233511,308sleep027482-21unin-run23:52:410
23516233544,281sleep1364-21snmpd21:56:501
17440226316,202sleep117580-21latency_hist21:37:441
23219224495,29sleep123279-21fschecks_count20:13:011
9124224347,160sleep10-21swapper/122:52:591
31680223096,24sleep10-21swapper/100:07:441
11300222613,200sleep011342-21fschecks_count21:18:000
25177222388,24sleep10-21swapper/120:18:051
25177222388,24sleep10-21swapper/120:18:051
7346222118,188sleep27373-21latency_hist19:22:442
28818222129,115sleep1364-21snmpd23:55:321
26367221944,157sleep126473-21fschecks_time20:23:021
6281221415,159sleep16257-21latency_hist21:02:431
18327220497,30sleep10-21swapper/121:38:171
32381220239,117sleep10-21swapper/120:42:441
32381220239,117sleep10-21swapper/120:42:441
7796219830,22sleep112-21rcu_sched19:23:031
989921978,176sleep19972-21latency21:13:051
15451219411,169sleep315461-21forks19:48:083
93921890,9sleep00-21swapper/000:12:430
26087218510,162sleep126212-21timerwakeupswit22:03:121
3651218320,142sleep33843-21apt-key19:12:433
2040221788,157sleep120413-21uname20:03:081
21586217315,107sleep121598-21fschecks_count23:33:001
21586217315,107sleep121598-21fschecks_count23:33:001
569921658,144sleep15715-21grep22:42:541
11488216443,102sleep111553-21ls21:18:051
586221628,112sleep10-21swapper/100:27:591
2864121578,134sleep128703-21grep22:13:021
13030215711,100sleep10-21swapper/119:38:131
2770215315,117sleep22807-21fschecks_count00:18:032
5557215213,123sleep15701-21latency_hist00:27:431
32684215112,123sleep1337-21cat00:08:411
19373215014,26sleep10-21swapper/121:43:011
30001214415,112sleep330132-21timerwakeupswit20:33:133
3474213972,56sleep1364-21snmpd19:12:131
2481521396,11sleep20-21swapper/221:58:172
2700021368,46sleep10-21swapper/122:07:531
16535213519,96sleep316568-21modprobe23:17:403
8352213241,19sleep10-21swapper/121:08:041
31691213282,31sleep131816-21latency_hist22:22:431
14979213110,42sleep1225-21systemd-journal23:12:411
10767213025,23sleep10-21swapper/119:33:001
14269212986,18sleep10-21swapper/119:43:091
6711212769,40sleep10-21swapper/121:03:031
31928212313,28sleep10-21swapper/122:22:551
2254921224,107sleep00-21swapper/020:08:560
1533421227,31sleep10-21swapper/119:47:581
2284121219,95sleep022943-21sed23:37:430
2284121219,95sleep022943-21sed23:37:430
8816211411,86sleep18968-21latency_hist22:52:431
2537621137,91sleep00-21swapper/023:43:120
2537621137,91sleep00-21swapper/023:43:120
19798211211,15sleep10-21swapper/120:02:431
25172211014,77sleep225243-21switchtime23:43:102
25172211014,77sleep225243-21switchtime23:43:102
6174210817,76sleep36211-21sed00:28:053
399021089,16sleep10-21swapper/100:22:421
16573210625,62sleep0225-21systemd-journal23:17:410
13795210616,74sleep313804-21diskstats23:07:593
3936210514,74sleep33951-21apt00:22:403
779221048,16sleep10-21swapper/100:33:081
2588521048,16sleep10-21swapper/123:47:411
2588521048,16sleep10-21swapper/123:47:411
24477210413,74sleep324509-21cat20:17:433
22033210311,77sleep022081-21unin-run21:52:410
1816621037,84sleep118221-21sed23:22:421
16640210319,23sleep20-21swapper/223:17:432
29183210213,73sleep2225-21systemd-journal20:32:442
13430210211,17sleep10-21swapper/123:07:411
30912210110,68sleep10-21swapper/100:02:431
30714210118,21sleep00-21swapper/022:19:310
21385210017,70sleep221473-21cat21:48:152
1614021008,12sleep10-21swapper/123:13:181
291272998,14sleep10-21swapper/120:32:431
1586229910,72sleep10-21swapper/121:32:431
42272987,17sleep14233-21df_inode22:37:571
35629840,23sleep20-21swapper/220:43:012
35629840,23sleep20-21swapper/220:43:012
3249229818,64sleep232516-21/usr/sbin/munin00:08:382
2085929810,34sleep10-21swapper/121:48:021
2748329613,69sleep127545-21latency_hist23:52:421
2748329613,69sleep127545-21latency_hist23:52:421
26112967,15sleep10-21swapper/120:48:131
35652956,11sleep10-21swapper/122:34:141
2524429516,61sleep325367-21timerwakeupswit23:43:113
2524429516,61sleep325367-21timerwakeupswit23:43:113
70022947,11sleep10-21swapper/119:18:551
53342938,52sleep10-21swapper/119:14:111
210292927,12sleep10-21swapper/123:30:171
2024929214,65sleep220296-21if_eth020:03:042
2007829213,20sleep30-21swapper/323:28:013
101762925,12sleep10-21swapper/119:29:531
242392916,11sleep10-21swapper/123:41:231
194129134,36sleep21-21systemd19:07:422
181002915,11sleep10-21swapper/119:56:541
139452916,10sleep10-21swapper/121:26:411
116882915,10sleep10-21swapper/123:00:101
249062907,11sleep10-21swapper/121:59:211
193762907,11sleep10-21swapper/123:24:001
159072905,11sleep20-21swapper/219:48:192
44912895,10sleep10-21swapper/120:55:401
302722895,30sleep10-21swapper/120:33:151
19522898,11sleep10-21swapper/122:28:241
2839328812,60sleep228478-21cat20:28:122
1735828817,52sleep217396-21systemctl21:37:412
942028726,31sleep00-21swapper/021:12:430
867328720,44sleep38618-21latency_hist00:37:423
442628722,37sleep04443-21apt-get19:12:530
3038328714,57sleep230367-21apt-get22:17:592
3035528713,56sleep329840-21apt-get22:17:523
2340428720,47sleep00-21swapper/020:13:040
1698828720,42sleep217019-21fschecks_count19:52:592
922228616,52sleep39280-21if_err_eth022:53:013
1278428615,56sleep012903-21timerandwakeup23:03:120
805328524,48sleep38117-21wc22:48:173
699528519,43sleep07044-21munin-run22:47:410
1652528413,58sleep0364-21snmpd19:51:320
1845528317,30sleep20-21swapper/221:38:182
1371828312,56sleep20-21swapper/223:07:542
2632128216,49sleep226246-21/usr/sbin/munin20:22:592
2249328216,50sleep222504-21/usr/sbin/munin20:08:212
739428117,32sleep20-21swapper/200:32:572
2597828111,16sleep00-21swapper/020:22:400
2597828111,16sleep00-21swapper/020:22:400
159128124,25sleep30-21swapper/320:47:423
159128124,25sleep30-21swapper/320:47:423
1493928114,45sleep214520-21/usr/sbin/munin21:28:202
94028022,24sleep30-21swapper/319:03:073
150828013,52sleep21585-21mailstats22:28:092
2461027921,32sleep30-21swapper/321:58:143
2847927825,24sleep30-21swapper/320:28:133
2826427824,32sleep00-21swapper/022:12:400
2678227815,46sleep226884-21/usr/sbin/munin23:48:112
2678227815,46sleep226884-21/usr/sbin/munin23:48:112
2602927812,52sleep226139-21cat20:22:422
2602927812,52sleep226139-21cat20:22:422
2048427821,25sleep00-21swapper/021:47:400
163027812,46sleep21698-21/usr/sbin/munin00:13:102
1443327827,28sleep00-21swapper/019:43:120
1313327718,27sleep00-21swapper/021:23:050
113027721,44sleep01159-21proc_pri19:03:120
971627622,25sleep20-21swapper/221:13:022
695727618,35sleep36984-21munin-run22:47:403
688827613,14sleep00-21swapper/021:03:060
295742768,22sleep30-21swapper/323:58:133
2838927625,24sleep30-21swapper/323:53:123
1342027618,27sleep30-21swapper/321:23:123
1244027612,46sleep312498-21sed23:03:023
933827520,41sleep29382-21systemd-run21:12:412
745927518,37sleep26975-21apt-get22:47:522
433627520,41sleep04372-21idleruntime22:38:000
3227127522,31sleep00-21swapper/020:42:430
3227127522,31sleep00-21swapper/020:42:430
365627413,45sleep33688-21/usr/sbin/munin20:53:043
1840627428,23sleep20-21swapper/223:22:562
1974627317,38sleep3225-21systemd-journal20:02:413
879127213,43sleep08815-21cut22:52:420
472727222,33sleep24812-21latency_hist20:57:432
1667227220,21sleep00-21swapper/021:33:110
145327219,21sleep00-21swapper/000:13:040
1333627212,43sleep213418-21switchtime21:23:112
326327118,34sleep33411-21grep22:33:153
3219727116,16sleep00-21swapper/022:23:020
816827015,38sleep28262-21/usr/sbin/munin19:23:122
2882427016,25sleep00-21swapper/022:13:050
285027021,32sleep22888-21wc22:33:062
2053627013,40sleep220639-21latency_hist21:47:422
1685627021,24sleep30-21swapper/323:17:563
1515327017,23sleep00-21swapper/021:28:260
961226922,23sleep00-21swapper/022:53:110
904026928,20sleep20-21swapper/222:52:562
3145826924,23sleep20-21swapper/200:07:412
3103326916,31sleep30-21swapper/320:38:013
3103326916,31sleep30-21swapper/320:38:013
3048426921,27sleep20-21swapper/223:58:402
1073726915,36sleep210823-21perl22:58:012
1069526914,41sleep010592-21/usr/sbin/munin22:57:590
464726818,32sleep04683-21munin-run20:57:410
2521726816,39sleep025282-21latency_hist22:02:430
2311326814,29sleep023225-21sshd21:53:140
1658426816,27sleep30-21swapper/321:33:103
884326723,24sleep30-21swapper/319:27:413
523426722,24sleep20-21swapper/220:58:032
1510526714,30sleep20-21swapper/219:47:422
53226625,22sleep30-21swapper/300:09:233
442326622,22sleep20-21swapper/222:38:022
237526614,36sleep02523-21latency_hist22:32:430
1843926619,21sleep30-21swapper/323:22:573
618226512,39sleep26253-21tr22:43:072
461326513,23sleep30-21swapper/322:38:073
318926516,18sleep00-21swapper/000:18:130
2828226511,29sleep00-21swapper/020:28:090
2440226519,31sleep224476-21systemd-run20:17:422
73026416,35sleep2755-21basename22:27:402
3150326415,29sleep331590-21systemd-run00:07:423
3029026422,25sleep00-21swapper/023:58:380
2670126411,38sleep326725-21ntp_states20:23:083
2438726426,16sleep30-21swapper/323:42:423
1506926413,22sleep30-21swapper/321:28:253
1116826414,37sleep311244-21tr19:33:103
1077726418,20sleep20-21swapper/219:33:012
649426315,32sleep36496-21/usr/sbin/munin21:02:533
1703926319,25sleep00-21swapper/023:18:010
169722639,37sleep00-21swapper/019:52:570
963226221,21sleep30-21swapper/321:12:583
3220426212,33sleep20-21swapper/220:42:412
3220426212,33sleep20-21swapper/220:42:412
309726222,23sleep00-21swapper/020:52:400
277022628,13sleep00-21swapper/023:52:530
2289826215,24sleep20-21swapper/221:53:122
106042625,14sleep00-21swapper/021:14:050
973926123,20sleep30-21swapper/319:28:103
603126117,20sleep00-21swapper/000:28:020
496426120,25sleep34957-21apt-get20:57:563
426926117,25sleep23524-21apt-get19:12:512
323342617,13sleep30-21swapper/322:23:053
2973626117,29sleep029763-21netstat20:33:060
2012626112,25sleep020158-21tune2fs20:03:020
954026015,21sleep20-21swapper/219:28:052
44892605,11sleep20-21swapper/200:23:032
435226011,36sleep04438-21fschecks_time00:23:010
123342608,10sleep30-21swapper/321:20:543
35592595,10sleep00-21swapper/022:34:090
2601025914,27sleep026086-21sed22:03:110
2529725920,17sleep30-21swapper/320:18:073
2529725920,17sleep30-21swapper/320:18:073
209192599,35sleep320986-21fw_packets21:48:043
1894525918,24sleep00-21swapper/019:58:100
913325818,16sleep00-21swapper/019:27:540
857325811,34sleep38628-21sendmail_mailst21:08:103
709425810,31sleep27186-21sed21:03:122
612625812,29sleep06132-21/usr/sbin/munin19:18:010
2815825816,24sleep228180-21ntp_states23:53:072
240592587,10sleep20-21swapper/223:39:062
1388125824,18sleep30-21swapper/319:42:593
1900625716,19sleep00-21swapper/021:42:440
1840425714,25sleep018399-21apt-get23:22:530
1372125715,25sleep013715-21apt-get23:07:560
1354025716,27sleep213564-21munin-run19:42:422
1225925714,25sleep012253-21apt-get19:37:560
2998625620,19sleep20-21swapper/222:17:432
1992925611,25sleep219948-21apt-get23:27:522
154025614,26sleep31546-21ntp_states00:13:073
1356525616,25sleep313643-21latency_hist19:42:433
1068625616,18sleep30-21swapper/322:57:583
741025518,19sleep00-21swapper/000:32:590
564825516,21sleep00-21swapper/020:58:140
497925518,20sleep30-21swapper/319:14:053
2305225515,24sleep023071-21apt-get23:37:510
1949325517,16sleep30-21swapper/321:43:033
1712525518,21sleep20-21swapper/223:18:032
3159225411,19sleep00-21swapper/000:07:430
276072548,30sleep327681-21find22:08:113
2186425415,26sleep321944-21latency20:08:103
2029725415,23sleep320311-21irqstats20:03:063
1258125412,18sleep20-21swapper/223:03:062
1111225414,24sleep011136-21ntp_states19:33:070
312625314,23sleep23159-21users19:08:182
2751425322,18sleep20-21swapper/222:08:102
2663625316,17sleep30-21swapper/323:48:083
2663625316,17sleep30-21swapper/323:48:083
2627025314,24sleep026264-21apt-get20:22:570
232425313,16sleep20-21swapper/220:48:102
2189725317,19sleep30-21swapper/323:33:063
2189725317,19sleep30-21swapper/323:33:063
1702325316,22sleep317109-21perl19:53:003
770225210,16sleep00-21swapper/019:23:020
737425211,17sleep30-21swapper/319:22:453
3270325215,22sleep032218-21apt-get20:42:510
3270325215,22sleep032218-21apt-get20:42:510
233725215,15sleep30-21swapper/322:32:423
1875325213,18sleep20-21swapper/219:58:042
420425016,22sleep34205-21/usr/sbin/munin00:22:523
2863025013,22sleep328634-21df_inode22:12:593
2299525016,15sleep30-21swapper/321:53:133
2126224913,14sleep00-21swapper/023:32:420
287224716,15sleep00-21swapper/019:08:160
214772479,16sleep20-21swapper/223:32:532
214772479,16sleep20-21swapper/223:32:532
114242478,13sleep20-21swapper/221:18:022
314824512,16sleep30-21swapper/320:52:433
2555124512,17sleep20-21swapper/222:03:012
575924310,16sleep20-21swapper/200:27:522
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional