You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 07:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat May 23, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
61992101719,981sleep16201-21ntp_states22:54:211
1990029796,899sleep219943-21ntpq20:14:212
25676285815,825sleep125816-21/usr/sbin/munin23:54:251
26124271818,683sleep026148-21/usr/sbin/munin22:14:230
31778242932,382sleep231719-21latency_hist20:53:582
31778242932,382sleep231719-21latency_hist20:53:582
1161223080,6sleep10-21swapper/123:10:111
1882423058,282sleep2363-21snmpd20:09:282
52342282102,58sleep20-21swapper/219:28:582
52342282102,58sleep20-21swapper/219:28:582
643822737,187sleep26590-21awk22:54:262
7603224686,82sleep27734-21latency_hist21:18:582
21687224263,88sleep221817-21unixbench_multi21:59:272
631223739,142sleep2693-21fschecks_count19:14:162
2308221611,193sleep22325-21open_inodes00:24:222
28522220615,176sleep128528-21ntpq00:04:231
9854220012,74sleep29871-21awk19:43:562
16930219712,172sleep017009-21find21:44:280
16930219712,172sleep017009-21find21:44:280
10402219710,173sleep21-21systemd23:08:562
27771218913,157sleep227802-21grep00:03:552
3008121749,87sleep230146-21awk00:09:242
6250216911,66sleep21-21systemd00:38:562
15476216678,23sleep20-21swapper/219:59:272
16837216316,24sleep30-21swapper/321:44:273
16837216316,24sleep30-21swapper/321:44:273
13812216214,121sleep20-21swapper/223:18:572
2491621607,39sleep20-21swapper/223:54:062
18150215729,116sleep218218-21ls21:49:182
1246215122,104sleep20-21swapper/222:39:162
2996921488,129sleep2363-21snmpd20:46:472
26235214810,17sleep20-21swapper/222:14:252
24683214818,107sleep20-21swapper/223:53:582
741821458,37sleep20-21swapper/219:34:192
4404214521,106sleep14450-21idleruntime-cro21:08:551
4404214521,106sleep14450-21idleruntime-cro21:08:551
14875214230,16sleep20-21swapper/221:39:132
14875214230,16sleep20-21swapper/221:39:132
31706214170,30sleep231836-21sed19:09:282
31706214170,30sleep231836-21sed19:09:282
1559821419,112sleep20-21swapper/223:24:102
11067213713,19sleep20-21swapper/223:10:012
2067521369,112sleep220722-21ntpq23:39:202
14605213672,20sleep20-21swapper/223:19:262
845821355,11sleep20-21swapper/223:02:102
17614213275,16sleep20-21swapper/223:29:232
340821308,16sleep20-21swapper/200:29:192
7841212936,17sleep20-21swapper/221:19:102
23129212749,23sleep20-21swapper/223:48:572
22552212422,77sleep122696-21unixbench_singl23:44:281
1905212422,81sleep31966-21apt-key19:18:573
31014212327,15sleep20-21swapper/222:29:272
31139212117,85sleep331282-21grep22:29:283
32423212016,81sleep332490-21cut20:55:163
32286212025,13sleep20-21swapper/220:54:592
28588212026,16sleep20-21swapper/220:43:572
2358021185,11sleep20-21swapper/222:05:492
632021170,5sleep30-21swapper/300:38:563
2366521175,11sleep20-21swapper/220:25:432
2366521175,11sleep20-21swapper/220:25:432
2573211515,26sleep22636-21if_eth019:19:252
2573211515,26sleep22636-21if_eth019:19:252
1119621155,12sleep20-21swapper/219:46:142
31715211421,16sleep20-21swapper/200:14:252
31715211421,16sleep20-21swapper/200:14:252
14528211422,16sleep20-21swapper/221:38:572
4008211316,70sleep34089-21fschecks_time19:24:143
4008211316,70sleep34089-21fschecks_time19:24:143
29438211330,42sleep20-21swapper/219:04:072
27329211322,14sleep20-21swapper/222:19:142
27329211322,14sleep20-21swapper/222:19:142
4197211121,70sleep34171-21/usr/sbin/munin22:49:103
1844021110,10sleep318434-21munin-run23:33:553
2206221105,11sleep20-21swapper/220:20:022
2206221105,11sleep20-21swapper/220:20:022
11391211013,17sleep20-21swapper/221:29:242
29520210918,72sleep329653-21timerwakeupswit20:44:263
30720210828,57sleep330749-21idleruntime-cro19:08:563
1969821089,17sleep20-21swapper/221:54:192
1869221088,17sleep20-21swapper/223:34:072
2541221079,86sleep225422-21munin-run22:13:572
551021067,85sleep25487-21latency_hist22:53:582
24743210214,31sleep30-21swapper/322:09:263
12247210167,17sleep20-21swapper/219:49:252
2448921009,44sleep20-21swapper/220:29:192
2448921009,44sleep20-21swapper/220:29:192
1045121005,62sleep20-21swapper/221:26:252
7012996,82sleep2363-21snmpd22:37:552
741229719,61sleep17448-21/usr/sbin/munin22:59:141
3172829717,62sleep131776-21apt-key20:53:571
3172829717,62sleep131776-21apt-key20:53:571
1091629715,63sleep310902-21apt-get23:09:133
56729612,63sleep2636-21sort00:19:182
56729612,63sleep2636-21sort00:19:182
40642967,12sleep20-21swapper/221:04:362
40642967,12sleep20-21swapper/221:04:362
3236729616,63sleep332412-21/usr/sbin/munin22:34:213
1639829616,29sleep30-21swapper/320:04:083
575429216,44sleep10-21swapper/100:34:271
2660329216,58sleep326684-21tune2fs23:59:163
173372925,10sleep20-21swapper/220:04:312
920729113,61sleep19312-21timerandwakeup19:39:271
299442917,53sleep20-21swapper/222:26:572
173392916,12sleep20-21swapper/221:45:432
173392916,12sleep20-21swapper/221:45:432
134712917,46sleep20-21swapper/219:54:182
91029014,48sleep10-21swapper/122:38:591
65042905,12sleep20-21swapper/219:31:002
291572905,12sleep20-21swapper/200:06:502
17232906,11sleep20-21swapper/220:59:272
17232906,11sleep20-21swapper/220:59:272
58632896,11sleep20-21swapper/221:12:362
58632896,11sleep20-21swapper/221:12:362
36922895,10sleep20-21swapper/222:46:112
1045428922,45sleep310570-21cat23:08:573
3204728832,29sleep132110-21cut22:34:151
2095428816,52sleep021098-21grep23:39:270
1463028816,49sleep314776-21latency_hist19:58:593
1357428816,55sleep113605-21/usr/sbin/munin21:34:191
269752875,10sleep20-21swapper/220:38:272
936028517,46sleep19361-21/usr/sbin/munin21:24:061
1388728516,50sleep014033-21unixbench_multi21:34:260
547528414,55sleep35516-21cpu19:29:113
3014728415,54sleep330202-21tr00:09:253
2016928421,40sleep320321-21sed21:54:283
150328412,53sleep01504-21sshd19:15:030
1372728414,50sleep113217-21/usr/sbin/munin19:54:251
2855328315,50sleep01-21systemd22:23:560
2855328315,50sleep01-21systemd22:23:560
250428321,36sleep12570-21fschecks_time19:19:241
250428321,36sleep12570-21fschecks_time19:19:241
2954728224,24sleep10-21swapper/119:04:141
151428211,48sleep30-21swapper/322:39:213
1441928212,55sleep014435-21netstat23:19:200
1264828210,58sleep312664-21modprobe21:32:483
2864028125,26sleep00-21swapper/020:43:590
2411528114,48sleep024117-21cpu22:09:100
109328120,26sleep00-21swapper/020:58:570
499428018,30sleep00-21swapper/021:09:190
499428018,30sleep00-21swapper/021:09:190
2160928013,44sleep121684-21find21:59:261
2027527917,26sleep10-21swapper/120:14:281
3091227813,47sleep330918-21ntp_states20:49:393
3091227813,47sleep330918-21ntp_states20:49:393
2311827814,40sleep022632-21/usr/sbin/munin20:24:240
2311827814,40sleep022632-21/usr/sbin/munin20:24:240
211527820,28sleep10-21swapper/120:59:431
211527820,28sleep10-21swapper/120:59:431
1868427813,47sleep118823-21unixbench_singl20:09:271
15112787,59sleep13863-1kworker/1:1H19:15:081
1723427713,50sleep117302-21grep23:29:141
1604027713,13sleep10-21swapper/121:43:561
1604027713,13sleep10-21swapper/121:43:561
2292827616,33sleep30-21swapper/320:24:193
2292827616,33sleep30-21swapper/320:24:193
2147727614,47sleep321516-21ntpq20:19:183
2147727614,47sleep321516-21ntpq20:19:183
913427515,26sleep00-21swapper/019:39:260
909727518,43sleep39129-21idleruntime-cro21:23:553
3169327519,36sleep031721-21munin-run20:53:560
3169327519,36sleep031721-21munin-run20:53:560
1454327418,27sleep30-21swapper/321:38:583
305127311,47sleep03173-21cat19:19:350
305127311,47sleep03173-21cat19:19:350
2635827318,38sleep326448-21timerandwakeup20:34:253
230782739,17sleep00-21swapper/023:48:550
1604927314,42sleep016056-21/usr/sbin/munin23:24:200
390527215,43sleep03926-21latency_hist22:48:550
3177527214,40sleep331852-21timerandwakeup00:14:263
3177527214,40sleep331852-21timerandwakeup00:14:263
162972720,7sleep10-21swapper/123:24:251
793027116,35sleep08018-21tune2fs21:19:160
621427120,21sleep30-21swapper/322:54:233
306272719,44sleep10-21swapper/122:29:181
368027016,28sleep30-21swapper/319:23:583
368027016,28sleep30-21swapper/319:23:583
2151327015,33sleep021606-21mailstats21:59:250
914826919,26sleep10-21swapper/121:23:571
38326822,32sleep3463-21tune2fs00:19:143
38326822,32sleep3463-21tune2fs00:19:143
117426814,36sleep11324-21cat20:58:581
86026710,39sleep0908-21grep22:38:580
2990526722,19sleep00-21swapper/019:04:220
276762675,50sleep00-21swapper/000:02:400
2390926712,29sleep00-21swapper/022:08:580
2041026717,26sleep10-21swapper/123:39:151
1795426723,29sleep117963-21/usr/sbin/munin21:49:131
1701226717,30sleep117140-21/usr/sbin/munin21:44:291
1701226717,30sleep117140-21/usr/sbin/munin21:44:291
1245026716,21sleep00-21swapper/019:49:270
1232026723,25sleep30-21swapper/319:49:263
1038226719,23sleep10-21swapper/123:08:551
835926621,19sleep10-21swapper/119:38:581
3185726613,32sleep032019-21sed00:14:270
3185726613,32sleep032019-21sed00:14:270
1776326612,39sleep117913-21latency_hist20:08:591
1741726620,27sleep00-21swapper/023:29:170
652626512,24sleep30-21swapper/321:14:183
364626521,21sleep30-21swapper/321:04:253
364626521,21sleep30-21swapper/321:04:253
2264926520,24sleep10-21swapper/122:04:151
83626416,23sleep10-21swapper/100:19:251
83626416,23sleep10-21swapper/100:19:251
767926418,19sleep00-21swapper/019:34:250
2970426419,31sleep129767-21fschecks_time00:09:151
2937426415,23sleep00-21swapper/000:08:580
2925426417,24sleep10-21swapper/120:44:201
228426412,36sleep32289-21threads20:59:453
228426412,36sleep32289-21threads20:59:453
2119226421,22sleep10-21swapper/120:19:121
2119226421,22sleep10-21swapper/120:19:121
1149726411,22sleep00-21swapper/019:48:590
1087626414,29sleep010396-21apt-get23:09:020
1061226415,32sleep110676-21ps19:44:231
562726319,20sleep00-21swapper/000:34:260
2547426317,26sleep30-21swapper/323:54:233
1685526323,17sleep30-21swapper/323:28:563
153442635,12sleep00-21swapper/019:59:250
315726212,37sleep03230-21cat00:28:570
315412627,17sleep10-21swapper/100:14:201
315412627,17sleep10-21swapper/100:14:201
312626216,25sleep30-21swapper/300:28:563
3014326215,21sleep00-21swapper/022:28:570
1979826212,22sleep00-21swapper/021:54:210
1495626217,31sleep315024-21grep19:59:153
1281626212,26sleep00-21swapper/021:32:570
392526120,19sleep00-21swapper/019:24:110
392526120,19sleep00-21swapper/019:24:110
2846426115,25sleep328506-21/usr/sbin/munin00:04:213
2054226115,30sleep320584-21sort23:39:173
470226022,20sleep30-21swapper/300:33:563
2902326017,22sleep30-21swapper/322:24:153
251872606,26sleep021160-1kworker/0:1H20:30:220
251872606,26sleep021160-1kworker/0:1H20:30:220
2459726018,27sleep10-21swapper/122:09:241
1526426017,31sleep315342-21basename21:39:223
1526426017,31sleep315342-21basename21:39:223
975825918,21sleep00-21swapper/021:24:210
2701225912,32sleep326951-21latency_hist22:18:583
2386925915,23sleep30-21swapper/322:08:573
191792599,34sleep019187-21systemd-run21:53:560
1057425916,23sleep00-21swapper/023:08:580
3110525816,26sleep031129-21df_inode19:09:130
3110525816,26sleep031129-21df_inode19:09:130
281012588,11sleep10-21swapper/120:39:271
2790625817,25sleep30-21swapper/320:39:253
245225816,28sleep12552-21sed00:24:251
2100225814,28sleep320999-21apt-get21:59:083
1289225816,25sleep112915-21ntp_states23:14:201
111582586,12sleep30-21swapper/319:45:473
340525715,25sleep03389-21apt-get00:29:170
285325713,19sleep10-21swapper/122:44:161
25752576,16sleep00-21swapper/021:00:220
25752576,16sleep00-21swapper/021:00:220
2432925715,24sleep324413-21perl20:29:163
2432925715,24sleep324413-21perl20:29:163
2078925714,20sleep10-21swapper/121:58:581
1850825710,30sleep317947-21/usr/sbin/munin20:09:253
1796925713,27sleep318035-21grep21:49:153
1569325718,18sleep00-21swapper/021:39:270
1569325718,18sleep00-21swapper/021:39:270
1235925711,22sleep10-21swapper/121:32:211
470725613,25sleep34702-21apt-get21:09:093
470725613,25sleep34702-21apt-get21:09:093
314025614,26sleep03134-21apt-get21:04:100
314025614,26sleep03134-21apt-get21:04:100
268572566,40sleep00-21swapper/020:36:560
2284025621,18sleep10-21swapper/120:24:171
2284025621,18sleep10-21swapper/120:24:171
2182125614,25sleep321816-21apt-get23:44:103
135752568,11sleep00-21swapper/023:16:540
127822566,38sleep10-21swapper/119:51:421
696725520,17sleep00-21swapper/021:14:270
3238525514,23sleep132524-21cat19:13:591
3238525514,23sleep132524-21cat19:13:591
3017425512,22sleep10-21swapper/122:28:581
2752325514,18sleep30-21swapper/322:19:193
2752325514,18sleep30-21swapper/322:19:193
2087325513,27sleep021022-21latency_hist20:18:590
1876725515,24sleep118776-21diskstats23:34:111
1324525515,22sleep013235-21apt-get19:54:120
1292325518,19sleep30-21swapper/323:14:223
2650625414,24sleep126500-21apt-get23:59:101
2334825411,24sleep023365-21/usr/sbin/munin23:49:070
1869325416,19sleep30-21swapper/323:34:083
1869025414,23sleep018686-21apt-get23:34:050
1858025416,19sleep00-21swapper/020:09:260
1528625413,19sleep10-21swapper/119:59:221
732525314,23sleep07320-21apt-get22:59:090
3151425317,18sleep30-21swapper/319:09:253
3151425317,18sleep30-21swapper/319:09:253
2660225314,22sleep126655-21users20:34:271
2381725310,18sleep30-21swapper/323:49:233
1413725320,18sleep30-21swapper/323:19:153
107325314,23sleep01092-21apt-get22:39:060
874725218,17sleep30-21swapper/319:39:173
863225214,23sleep38717-21apt-key23:03:563
2164425214,24sleep021715-21swap20:19:230
2164425214,24sleep021715-21swap20:19:230
70842518,27sleep06999-21kworker/0:222:58:560
547325111,24sleep15468-21apt-get19:29:091
2965525111,24sleep029778-21/usr/sbin/munin20:44:270
274432519,17sleep10-21swapper/122:19:171
274432519,17sleep10-21swapper/122:19:171
2604225117,15sleep10-21swapper/122:14:201
2537625112,24sleep025382-21ntp_states23:54:200
1453625114,17sleep10-21swapper/123:19:251
96892508,17sleep30-21swapper/321:24:203
471125010,14sleep10-21swapper/121:09:111
471125010,14sleep10-21swapper/121:09:111
419525012,22sleep14190-21apt-get22:49:081
3057025013,17sleep00-21swapper/022:29:170
2516025013,19sleep10-21swapper/120:30:021
2516025013,19sleep10-21swapper/120:30:021
273024916,17sleep30-21swapper/322:44:143
72702484,32sleep10-21swapper/121:15:461
440124816,18sleep00-21swapper/022:49:150
386924816,18sleep10-21swapper/100:29:301
2007724713,16sleep30-21swapper/320:14:263
80192466,30sleep30-21swapper/319:35:333
49142455,31sleep10-21swapper/119:25:261
49142455,31sleep10-21swapper/119:25:261
676924410,16sleep00-21swapper/019:33:560
2374224412,17sleep10-21swapper/123:49:211
1772724314,16sleep00-21swapper/020:08:580
3275924211,14sleep00-21swapper/000:19:110
3275924211,14sleep00-21swapper/000:19:110
2958724010,16sleep00-21swapper/000:09:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional