You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 12:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Sep 15, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15440285110,831sleep09147-1kworker/0:0H21:33:100
3105625877,568sleep131097-21ntpq00:03:211
12963251317,476sleep013108-21/usr/sbin/munin19:43:290
672024058,359sleep16862-21awk00:28:301
18149239715,199sleep118167-21modprobe20:02:561
5072338125,195sleep1573-21idleruntime22:28:181
6862295121,52sleep1752-21processes20:48:241
6862295121,52sleep1752-21processes20:48:241
23027224619,211sleep323138-21/usr/sbin/munin23:38:273
23027224619,211sleep323138-21/usr/sbin/munin23:38:273
27760223835,156sleep127252-21/usr/sbin/munin23:53:241
21825223629,74sleep121883-21basename21:53:171
3103223231,65sleep13190-21gpgv20:57:571
9953221311,77sleep10-21swapper/122:58:151
3012622138,194sleep1367-21snmpd23:59:501
13521221211,27sleep113579-21latency_hist19:47:581
18393220627,143sleep118515-21cat23:23:271
29153220313,125sleep129189-21cron22:17:561
24420218653,118sleep124486-21sendmail_mailqu23:43:261
4824217445,115sleep14865-21ntpq19:18:221
4750217322,134sleep14760-21modprobe22:42:561
5416217021,58sleep10-21swapper/122:43:201
27845216916,105sleep127859-21fschecks_count20:33:181
8516216829,129sleep1367-21snmpd00:34:151
17102216811,136sleep117129-21fschecks_count21:38:151
3263421677,16sleep10-21swapper/100:08:221
3263421677,16sleep10-21swapper/100:08:221
817221657,143sleep18294-21sort19:28:271
817221657,143sleep18294-21sort19:28:271
4128216543,102sleep04151-21munin-run19:17:560
825421579,109sleep18390-21date21:13:001
7249215414,120sleep27264-21idleruntime-cro19:27:562
7249215414,120sleep27264-21idleruntime-cro19:27:562
21421215110,127sleep321509-21latency_hist21:52:593
4955214942,67sleep15103-21sort21:02:581
481821489,126sleep14863-21ntpq00:23:211
1282021476,12sleep10-21swapper/121:24:061
1282021476,12sleep10-21swapper/121:24:061
26541214517,85sleep10-21swapper/120:28:211
20674214330,67sleep10-21swapper/120:08:301
12763214310,116sleep212810-21unin-run23:07:572
8528214214,109sleep10-21swapper/122:53:221
25122214110,118sleep125162-21ps20:23:241
17489213910,114sleep117612-21latency_hist23:22:591
14682213930,51sleep10-21swapper/123:13:131
2288321389,116sleep122956-21mailstats23:38:251
2288321389,116sleep122956-21mailstats23:38:251
2610221379,16sleep10-21swapper/122:07:591
3095621360,11sleep00-21swapper/022:22:590
1017121368,90sleep110263-21sort21:18:171
30761213410,104sleep130954-21apt-config20:42:591
811721330,14sleep0704-21cobalt_printf22:53:080
1411621317,112sleep114139-21/usr/sbin/munin21:28:261
29232213017,91sleep029288-21tune2fs23:58:480
9458212630,57sleep19494-21ntpq19:33:201
2407212612,73sleep12552-21wc19:13:011
15776212611,23sleep00-21swapper/019:53:260
14329212310,75sleep114402-21sed19:48:271
16779212268,38sleep10-21swapper/121:37:591
27676212128,53sleep127825-21latency_hist22:12:591
31593211848,11sleep10-21swapper/120:43:321
31593211848,11sleep10-21swapper/120:43:321
2614821168,65sleep125701-21/usr/sbin/munin23:48:351
15520211544,34sleep10-21swapper/119:53:191
9586211419,17sleep10-21swapper/121:15:421
24228211333,22sleep10-21swapper/120:21:061
1925321135,13sleep10-21swapper/123:28:071
9389211230,64sleep39424-21systemctl22:57:563
2001211219,10sleep10-21swapper/119:10:461
23623211114,79sleep223699-21systemctl20:18:282
27632211021,23sleep00-21swapper/023:53:190
1969921109,65sleep11-21systemd20:07:571
5749210822,63sleep05898-21sed22:43:280
1916210722,17sleep10-21swapper/122:33:111
13654210719,63sleep213736-21grep23:08:262
13578210732,13sleep10-21swapper/123:08:251
7188210513,77sleep27189-21sshd00:32:062
710221056,11sleep10-21swapper/119:26:011
710221056,11sleep10-21swapper/119:26:011
32499210525,26sleep20-21swapper/219:03:292
23214210423,37sleep00-21swapper/021:58:090
20584210418,62sleep320670-21grep23:32:583
6717210021,15sleep10-21swapper/121:07:591
293821008,81sleep0367-21snmpd20:56:430
23572210011,74sleep023579-21sh20:18:260
2402929821,56sleep324092-21fschecks_time23:43:163
2011129720,15sleep10-21swapper/121:48:091
1005129723,20sleep00-21swapper/021:18:070
2287629623,16sleep10-21swapper/120:17:591
2287629623,16sleep10-21swapper/120:17:591
1545329613,67sleep215520-21df21:33:202
335429567,16sleep13360-21ntp_states00:18:241
1159929533,47sleep10-21swapper/123:03:161
30562940,6sleep20-21swapper/219:13:172
131229435,40sleep10-21swapper/100:13:111
97429324,49sleep31069-21timerandwakeup22:28:283
3072929311,20sleep10-21swapper/122:22:571
1782129316,58sleep217910-21fschecks_time23:23:162
1361329325,49sleep313150-21apt-get21:28:083
1361329325,49sleep313150-21apt-get21:28:083
3256029217,15sleep10-21swapper/122:27:581
2370529215,61sleep223818-21latency_hist23:42:592
2370529215,61sleep223818-21latency_hist23:42:592
3059329119,52sleep330650-21systemd-run20:42:573
2853729118,54sleep212872-1kworker/2:2H23:57:572
2150529010,45sleep121665-21unixbench_multi23:33:291
1752829011,64sleep217651-21timerwakeupswit19:58:292
1373829016,52sleep313886-21unixbench_multi23:08:273
2909828929,36sleep229095-21apt-get23:58:092
237302899,45sleep123809-21mailstats21:58:261
2764828810,66sleep027675-21latency_hist22:12:580
2013528819,48sleep220199-21/usr/sbin/munin20:08:192
827828724,41sleep38413-21grep00:33:283
575628714,60sleep25821-21latency_hist00:27:582
70692869,63sleep17137-21mailstats22:48:251
1221028610,59sleep20-21swapper/219:43:112
991828521,43sleep29408-21apt-get22:58:082
3058928511,63sleep230590-21cron20:42:562
13942847,63sleep00-21swapper/000:13:140
763728320,42sleep27725-21fschecks_time19:28:162
763728320,42sleep27725-21fschecks_time19:28:162
2059528215,50sleep020672-21timerandwakeup20:08:290
319862818,59sleep00-21swapper/019:03:190
2861528120,46sleep328644-21apt-config23:57:583
174428122,41sleep31764-21apt-get20:53:073
1486528114,18sleep00-21swapper/023:13:160
1420428114,37sleep30-21swapper/321:28:283
1248328112,30sleep30-21swapper/319:43:203
935828018,25sleep30-21swapper/319:33:183
259628017,25sleep20-21swapper/222:33:282
256212808,60sleep0367-21snmpd20:24:080
1440328021,18sleep20-21swapper/219:48:282
1103228014,51sleep211033-21sshd21:19:062
3266927932,24sleep30-21swapper/319:03:303
726527815,48sleep37299-21munin-run19:27:573
726527815,48sleep37299-21munin-run19:27:573
319127820,28sleep20-21swapper/220:57:582
1505827825,35sleep015077-21latency_hist19:52:580
1066727816,46sleep210708-21grep19:38:142
789327724,21sleep20-21swapper/222:52:592
570027617,41sleep35701-21cron19:22:563
2802227614,26sleep00-21swapper/020:33:210
1496327612,46sleep014931-21latency_hist21:32:570
323232758,56sleep0367-21snmpd22:25:260
2153427521,29sleep00-21swapper/020:13:130
2153427521,29sleep00-21swapper/020:13:130
203872756,14sleep00-21swapper/021:48:200
2573627422,30sleep025720-21apt-get23:48:140
2519927426,31sleep225216-21apt-get23:47:552
2414927416,42sleep223215-21/usr/sbin/munin21:58:312
2279627419,31sleep322818-21munin-run20:17:573
2279627419,31sleep322818-21munin-run20:17:573
1956427429,11sleep10-21swapper/121:44:371
1969527318,31sleep00-21swapper/020:07:560
1626327315,38sleep016161-21/usr/sbin/munin23:18:170
3238427222,36sleep132498-21timerandwakeup19:03:281
2621427220,36sleep226284-21/usr/sbin/munin20:28:142
989027115,27sleep00-21swapper/021:17:590
2986627123,24sleep20-21swapper/222:18:212
2680127118,24sleep20-21swapper/222:08:242
1737127123,21sleep00-21swapper/019:58:270
1098827110,16sleep30-21swapper/321:18:313
99352708,17sleep30-21swapper/322:58:093
885827012,44sleep28881-21netstat21:13:212
636727020,23sleep30-21swapper/322:47:593
610727020,28sleep35355-21/usr/sbin/munin21:03:373
454027027,23sleep00-21swapper/000:23:150
2822227015,40sleep228242-21/usr/sbin/munin20:33:272
1288327010,16sleep00-21swapper/023:07:590
1229727014,38sleep012339-21/usr/sbin/munin21:23:410
1229727014,38sleep012339-21/usr/sbin/munin21:23:410
489626921,27sleep04919-21munin-run21:02:570
2487826916,35sleep324944-21fschecks_time22:03:173
2350926914,29sleep30-21swapper/321:58:203
175626910,41sleep21814-21users19:08:292
1727826915,36sleep017290-21kernelversion21:38:180
1021926910,45sleep010238-21kernelversion22:58:220
884926818,32sleep08897-21wc19:32:590
884926818,32sleep08897-21wc19:32:590
288532685,13sleep00-21swapper/022:14:120
2166626815,23sleep20-21swapper/223:33:302
709526718,19sleep00-21swapper/021:08:180
361926713,37sleep32875-21/usr/sbin/munin00:18:303
3191526715,36sleep01-21systemd00:07:570
2213926719,23sleep20-21swapper/220:13:282
2213926719,23sleep20-21swapper/220:13:282
2142726715,26sleep00-21swapper/023:33:280
1791326714,38sleep317973-21if_eth023:23:173
1734726710,13sleep30-21swapper/319:58:263
376326615,16sleep00-21swapper/022:38:190
338326620,27sleep00-21swapper/000:18:270
3041926621,28sleep30-21swapper/300:02:593
1864026614,33sleep318646-21/usr/sbin/munin21:43:143
149326617,21sleep20-21swapper/200:13:162
117612665,28sleep10-21swapper/119:40:511
983626513,39sleep39885-21wc21:17:583
3265926511,38sleep232665-21ntp_states00:08:232
3265926511,38sleep232665-21ntp_states00:08:232
2052226514,22sleep30-21swapper/320:08:283
1639426518,24sleep30-21swapper/323:18:203
576826412,31sleep00-21swapper/021:03:300
3108926419,27sleep231108-21apt-get20:43:082
3108926419,27sleep231108-21apt-get20:43:082
2285926416,23sleep00-21swapper/023:38:240
2285926416,23sleep00-21swapper/023:38:240
1741526421,24sleep30-21swapper/321:38:223
815726322,16sleep20-21swapper/200:33:272
611626324,17sleep00-21swapper/019:23:180
611626324,17sleep00-21swapper/019:23:180
442326316,26sleep04442-21apt-get19:18:090
399226317,22sleep30-21swapper/322:38:263
2682526313,23sleep30-21swapper/322:08:253
2174626317,20sleep00-21swapper/021:53:160
210562636,15sleep20-21swapper/221:48:542
1476426313,29sleep20-21swapper/223:13:142
118326316,19sleep00-21swapper/019:08:190
634026213,22sleep20-21swapper/222:47:582
323162627,12sleep30-21swapper/322:25:213
2849126221,18sleep20-21swapper/222:13:252
2785726219,28sleep327875-21/usr/sbin/munin22:13:063
2527226217,21sleep00-21swapper/023:47:580
2051226210,40sleep0367-21snmpd23:32:390
773926116,32sleep07799-21cat00:33:170
421726121,19sleep20-21swapper/220:59:032
325526113,27sleep00-21swapper/022:37:580
3230126117,17sleep30-21swapper/300:08:163
3230126117,17sleep30-21swapper/300:08:163
318772615,10sleep20-21swapper/200:07:262
2192826120,17sleep30-21swapper/321:53:193
153426123,19sleep30-21swapper/320:52:593
153426123,19sleep30-21swapper/320:52:593
1192226118,27sleep011952-21munin-run19:42:560
3180726019,23sleep00-21swapper/020:43:350
3180726019,23sleep00-21swapper/020:43:350
2901326021,24sleep329039-21munin-run20:37:563
1656426018,21sleep20-21swapper/219:57:572
1630726011,29sleep20-21swapper/223:18:192
3257025918,20sleep20-21swapper/222:27:592
290125914,27sleep22893-21apt-get00:18:132
2753425919,20sleep30-21swapper/320:32:593
2570825915,27sleep225216-21apt-get23:48:082
2302325912,23sleep20-21swapper/221:57:592
113225915,28sleep31181-21irqstats19:08:183
1114925923,20sleep30-21swapper/319:38:273
1539525820,18sleep30-21swapper/319:53:173
1207425821,19sleep30-21swapper/323:03:263
661225715,20sleep00-21swapper/000:28:290
54582574,10sleep30-21swapper/300:24:483
388325710,32sleep33895-21sshd19:14:573
1874625715,20sleep20-21swapper/221:43:172
1594725720,17sleep30-21swapper/323:18:003
80882565,10sleep20-21swapper/221:11:272
695325616,25sleep06975-21netstat22:48:200
600625616,24sleep26106-21sed21:03:362
45692566,10sleep30-21swapper/320:59:173
35125614,26sleep0342-21apt-get22:28:130
6792558,29sleep00-21swapper/020:48:230
6792558,29sleep00-21swapper/020:48:230
3220325515,24sleep032199-21apt-get00:08:090
3220325515,24sleep032199-21apt-get00:08:090
3065325517,20sleep00-21swapper/020:42:580
2947025515,24sleep329465-21apt-get22:18:103
2135625515,19sleep30-21swapper/323:33:273
173025514,26sleep31762-21ntp_states00:13:213
1422125515,17sleep30-21swapper/319:48:253
1352725513,28sleep213150-21apt-get21:28:032
1352725513,28sleep213150-21apt-get21:28:032
654325414,24sleep30-21swapper/300:28:273
575325416,22sleep25872-21date19:22:592
2602325415,22sleep326115-21memory23:48:333
1892225416,17sleep20-21swapper/220:03:262
1713125416,23sleep217223-21perl21:38:162
1147425413,17sleep00-21swapper/023:03:060
763825320,17sleep30-21swapper/321:08:303
42625312,17sleep20-21swapper/222:28:152
3065825313,23sleep330653-21apt-get00:03:113
2539725317,17sleep20-21swapper/222:03:282
1901725315,21sleep019023-21ntp_states21:43:230
908725213,24sleep29083-21apt-get19:33:092
839825212,26sleep28415-21uname22:53:192
3108325213,22sleep330604-21apt-get20:43:053
3108325213,22sleep330604-21apt-get20:43:053
2465825213,23sleep324654-21apt-get20:23:103
203825116,18sleep30-21swapper/322:33:173
1544925114,22sleep315430-21apt-get21:33:173
1937525010,20sleep20-21swapper/223:28:172
1816025012,23sleep018179-21ntp_states23:23:220
1414024910,17sleep20-21swapper/221:28:272
753924812,19sleep37536-21apt-get19:28:093
753924812,19sleep37536-21apt-get19:28:093
2516524814,17sleep20-21swapper/220:23:252
193562485,34sleep30-21swapper/320:03:353
94192479,15sleep00-21swapper/019:33:190
429524710,20sleep24393-21users22:38:292
2495824712,16sleep00-21swapper/022:03:180
126824512,20sleep21270-21sshd20:50:112
126824512,20sleep21270-21sshd20:50:112
658624411,17sleep30-21swapper/322:48:113
2311124416,14sleep30-21swapper/320:18:093
262942419,16sleep00-21swapper/020:28:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional