You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 11:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Thu Aug 27, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3805256813,539sleep13815-21ntpq00:24:311
602125485,530sleep06003-21/usr/sbin/munin21:09:070
7367245215,419sleep26429-21/usr/sbin/munin00:34:382
8212244810,421sleep28249-21ntpq21:14:302
30303235615,241sleep030454-21unixbench_singl00:04:400
6743231544,214sleep06809-21wc21:09:310
257222966,279sleep0367-21snmpd20:58:490
257222966,279sleep0367-21snmpd20:58:490
197892289126,139sleep00-21swapper/021:49:370
28813226929,217sleep228829-21sh20:39:052
24066226472,136sleep024204-21unixbench_singl23:44:460
21697225819,220sleep121720-21netstat20:14:291
21697225819,220sleep121720-21netstat20:14:291
7560225591,24sleep00-21swapper/019:29:340
7560225591,24sleep00-21swapper/019:29:340
2674622496,231sleep00-21swapper/022:11:080
31653224652,176sleep031656-21cron22:29:040
2450622218,202sleep1367-21snmpd23:48:021
282832218106,63sleep028429-21grep20:34:390
1592622090,10sleep10-21swapper/123:19:341
22595220414,152sleep0231-21systemd-journal20:19:040
22595220414,152sleep0231-21systemd-journal20:19:040
1954221949,114sleep00-21swapper/020:09:070
10676219416,162sleep010716-21latency23:04:290
14112218115,41sleep00-21swapper/023:14:300
21223218013,110sleep021276-21sendmail-msp23:39:050
1548217643,42sleep01629-21awk22:34:330
2986021758,151sleep030001-21grep20:39:350
948421747,144sleep00-21swapper/021:19:240
26410217323,111sleep026454-21ntpq20:29:300
31320217210,142sleep031086-21/usr/sbin/munin00:09:240
20814217215,52sleep00-21swapper/021:54:370
83721676,140sleep125747-1kworker/1:2H19:09:281
16997216692,26sleep00-21swapper/020:00:020
1156921638,140sleep011718-21cat19:44:070
14985216210,138sleep015134-21unixbench_singl21:34:350
553216021,102sleep0610-21grep19:09:230
17546216020,124sleep00-21swapper/023:24:350
1136215920,122sleep21143-21/usr/sbin/munin22:34:232
4669215691,22sleep00-21swapper/000:29:060
19739215612,126sleep019894-21latency_hist23:34:080
16122215410,91sleep016147-21munin-run19:59:050
920821509,128sleep08821-21/usr/sbin/munin22:59:280
10046215014,120sleep210048-21sshd23:03:132
22264214917,115sleep222411-21head23:39:352
790021479,50sleep00-21swapper/021:14:240
3820214129,19sleep00-21swapper/022:43:130
12828214037,18sleep00-21swapper/021:29:260
6550213810,90sleep06652-21fschecks_time00:34:230
19224213816,99sleep219239-21/usr/sbin/munin21:49:262
2547621369,88sleep025535-21date22:09:060
1044213640,57sleep01054-21modprobe20:54:050
158322134105,18sleep015925-21mailstats23:19:340
114421347,12sleep30-21swapper/322:34:243
28501213111,106sleep028586-21cut23:59:350
19637212716,86sleep319779-21grep21:49:363
1613321276,105sleep00-21swapper/021:39:270
25074212611,100sleep025135-21if_eth023:49:260
14361212416,97sleep00-21swapper/019:51:180
585212225,74sleep20-21swapper/200:14:272
27349212219,90sleep127356-21forks22:14:241
27349212219,90sleep127356-21forks22:14:241
20508212222,79sleep120649-21head20:09:361
1252121219,73sleep012557-21wc23:10:140
28166212020,42sleep328276-21ls20:34:383
4441211725,68sleep24505-21cat22:44:242
27726211713,86sleep227749-21/usr/sbin/munin22:14:332
27726211713,86sleep227749-21/usr/sbin/munin22:14:332
23030211733,66sleep223169-21/usr/sbin/munin21:59:362
30434211619,37sleep30-21swapper/320:44:063
25923211524,64sleep30-21swapper/322:09:263
23296211519,69sleep222821-21apt-get23:44:152
20739211419,75sleep120231-21apt-get21:54:181
14888211418,18sleep30-21swapper/321:34:343
973421115,17sleep00-21swapper/019:36:530
2039121119,54sleep00-21swapper/020:09:350
17945211151,35sleep317963-21modprobe20:04:053
11743211013,25sleep111762-21apt-get19:44:171
13080210913,81sleep313199-21latency_hist19:49:073
12144210920,74sleep212188-21ntp_kernel_pll_19:44:302
32624210814,76sleep332756-21unixbench_multi22:29:363
3123210813,77sleep32611-21apt-get20:59:173
24920210821,58sleep225051-21unixbench_singl22:05:012
2279921088,88sleep022859-21systemd-run23:44:050
32420210720,74sleep132464-21latency_hist00:14:061
22916210621,46sleep00-21swapper/020:19:200
5723210525,39sleep00-21swapper/022:49:190
4001210413,75sleep34064-21/usr/sbin/munin19:19:193
12657210414,29sleep312706-21ntp_states23:10:173
11189210414,19sleep20-21swapper/221:24:192
3444210317,73sleep23474-21fschecks_count00:24:242
24613210312,47sleep00-21swapper/020:24:250
24613210312,47sleep00-21swapper/020:24:250
22395210354,16sleep00-21swapper/021:59:220
7026210215,72sleep37076-21latency_hist22:54:063
2958210210,78sleep22961-21sshd00:22:432
287221027,17sleep00-21swapper/019:14:320
15413210223,57sleep215450-21fschecks_count23:19:242
11331210223,62sleep111398-21sort21:24:421
7637210015,23sleep10-21swapper/122:54:281
10576210010,73sleep210616-21/usr/sbin/munin23:04:262
72712998,49sleep00-21swapper/022:54:190
563429921,56sleep35676-21fschecks_count19:24:243
68092987,49sleep00-21swapper/019:29:050
41682987,40sleep00-21swapper/022:44:140
1071129810,75sleep210781-21systemctl19:39:372
426929715,62sleep24334-21latency19:19:262
319492978,38sleep00-21swapper/022:29:210
315629714,68sleep23140-21apt-get20:59:242
44572969,41sleep01-21systemd21:04:060
100229617,57sleep20-21swapper/219:09:332
1385329412,22sleep113854-21sshd21:32:131
111662947,36sleep00-21swapper/021:24:170
3187629329,44sleep231950-21switchtime19:04:332
1850429313,26sleep218563-21latency20:04:292
780829216,60sleep37731-21latency_hist00:39:073
1216129210,22sleep20-21swapper/223:09:152
49002907,44sleep00-21swapper/000:29:150
3010529021,20sleep012-21rcu_sched22:24:040
2545329017,56sleep125532-21sort23:49:351
1913129011,61sleep119275-21/usr/sbin/munin23:29:371
90028919,44sleep11034-21sed00:14:341
3246228817,18sleep20-21swapper/220:49:262
300042888,66sleep130033-21users20:39:361
181228851,18sleep00-21swapper/000:19:220
15422889,21sleep20-21swapper/220:54:272
15422889,21sleep20-21swapper/220:54:272
3222428716,36sleep10-21swapper/120:49:171
2377928723,40sleep123989-21apt-config22:04:071
3076528530,37sleep330792-21fschecks_count20:44:233
2731728520,29sleep10-21swapper/120:34:071
1806928514,15sleep30-21swapper/321:44:353
1818028414,50sleep318149-21latency_hist23:29:063
1733328410,58sleep117371-21proc_pri23:24:321
2005328316,41sleep320141-21fschecks_time23:34:233
893128214,49sleep38936-21/usr/sbin/munin22:59:233
1972828221,32sleep30-21swapper/323:34:073
1726928221,23sleep217312-21awk23:24:292
417328117,44sleep34146-21/usr/sbin/munin22:44:173
3265928124,42sleep032728-21ls19:09:070
2496528126,32sleep20-21swapper/223:49:242
249172800,7sleep30-21swapper/320:24:313
249172800,7sleep30-21swapper/320:24:313
242532806,11sleep30-21swapper/323:44:493
191728020,26sleep30-21swapper/320:54:363
191728020,26sleep30-21swapper/320:54:363
176528050,19sleep31774-21systemd-run19:14:063
880827914,45sleep38863-21idleruntime19:34:253
757627924,25sleep30-21swapper/321:14:063
1866727827,35sleep00-21swapper/021:49:040
498927725,36sleep25032-21fschecks_count00:29:212
3138427710,15sleep20-21swapper/200:09:262
73252760,6sleep20-21swapper/219:29:272
73252760,6sleep20-21swapper/219:29:272
3015627621,28sleep30-21swapper/322:24:073
1586427621,26sleep10-21swapper/121:39:201
1350127623,22sleep10-21swapper/119:49:251
634927523,23sleep20-21swapper/222:49:372
616427525,34sleep16187-21munin-run00:34:041
3066427519,28sleep20-21swapper/222:24:262
3180227420,11sleep00-21swapper/020:47:180
3060627416,38sleep130372-21/usr/sbin/munin22:24:251
130627419,10sleep00-21swapper/000:16:430
2657127329,25sleep20-21swapper/223:54:252
1799627316,42sleep118091-21latency_hist20:04:071
1500627317,28sleep30-21swapper/323:19:053
508827218,11sleep00-21swapper/019:21:510
2965527219,28sleep30-21swapper/300:04:273
2564927213,13sleep30-21swapper/323:49:373
1513427212,16sleep30-21swapper/319:54:283
1274927217,42sleep00-21swapper/019:45:290
2817727129,29sleep00-21swapper/022:14:400
2817727129,29sleep00-21swapper/022:14:400
2639227118,10sleep00-21swapper/023:54:100
2020327114,42sleep120237-21/usr/sbin/munin23:34:251
1255827111,15sleep10-21swapper/123:10:151
81127019,25sleep10-21swapper/122:34:061
718227010,47sleep37185-21entropy19:29:253
718227010,47sleep37185-21entropy19:29:253
3094627023,31sleep20-21swapper/220:44:262
277502700,7sleep30-21swapper/322:14:333
277502700,7sleep30-21swapper/322:14:333
1295227017,31sleep212717-21/usr/sbin/munin21:29:282
1169727014,39sleep111787-21apt-key23:09:061
861926917,37sleep18623-21find19:34:151
372626911,40sleep13775-21sensors_temp20:59:511
2472726918,36sleep224753-21kernelversion20:24:272
2472726918,36sleep224753-21kernelversion20:24:272
838726832,17sleep30-21swapper/321:14:353
1629326819,27sleep20-21swapper/219:59:072
1094426821,24sleep30-21swapper/321:24:083
376226714,42sleep33702-21latency_hist19:19:073
262426713,37sleep22664-21if_eth019:14:262
252626718,23sleep10-21swapper/120:58:141
252626718,23sleep10-21swapper/120:58:141
1620226717,23sleep10-21swapper/119:59:061
2804226623,24sleep10-21swapper/120:34:361
2130226617,29sleep221410-21latency_hist23:39:072
1510426620,18sleep10-21swapper/123:19:071
974426514,33sleep19755-21/usr/sbin/munin22:59:371
2291726518,26sleep10-21swapper/120:19:211
307902648,46sleep20-21swapper/200:08:382
241926421,23sleep10-21swapper/119:14:211
2113126422,23sleep30-21swapper/321:54:443
1072626418,33sleep110822-21apt-key21:24:061
2936726316,26sleep20-21swapper/220:39:252
2710626322,17sleep10-21swapper/123:54:361
225562639,43sleep20-21swapper/220:18:342
225562639,43sleep20-21swapper/220:18:342
2025426324,19sleep10-21swapper/121:54:061
1554426324,22sleep215576-21awk21:39:052
2931026214,19sleep10-21swapper/122:19:331
292182629,32sleep20-21swapper/222:19:302
2815826217,26sleep228220-21fw_packets23:59:252
2412826212,36sleep123650-21apt-get22:04:111
873426124,20sleep20-21swapper/219:34:242
2015226111,25sleep20-21swapper/220:09:292
627826017,21sleep10-21swapper/122:49:361
3170726017,23sleep331844-21date22:29:073
3132826019,23sleep331324-21apt-get19:04:183
2633226016,26sleep126458-21/usr/sbin/munin22:09:361
2309826018,26sleep223134-21/usr/sbin/munin20:19:262
1095426020,20sleep10-21swapper/119:39:401
3145125920,21sleep10-21swapper/119:04:241
3088825917,23sleep10-21swapper/120:44:251
953825815,20sleep20-21swapper/221:19:262
596525823,19sleep20-21swapper/221:09:052
761325715,24sleep10-21swapper/100:37:471
688325718,17sleep20-21swapper/219:29:072
490525715,26sleep34972-21df00:29:183
2838325721,21sleep30-21swapper/323:59:303
274682577,11sleep30-21swapper/323:56:193
2637725713,26sleep326408-21/usr/sbin/munin20:29:293
207972577,10sleep30-21swapper/320:10:563
1462325717,26sleep214698-21latency_hist19:54:072
1331725721,19sleep30-21swapper/321:29:373
51222569,30sleep15159-21ntp_states21:04:321
37712565,10sleep20-21swapper/222:42:352
2456525610,32sleep324601-21ntpq22:04:543
229725614,16sleep10-21swapper/100:19:351
2180925616,19sleep10-21swapper/123:39:261
2085025612,28sleep220857-21/usr/sbin/munin21:54:392
181025614,26sleep31805-21apt-get00:19:203
908425522,19sleep10-21swapper/121:19:051
526025518,21sleep15309-21unin-run19:24:061
256962559,16sleep10-21swapper/120:28:461
256962559,16sleep10-21swapper/120:28:461
1193925513,18sleep30-21swapper/323:09:083
860625416,16sleep30-21swapper/322:59:073
840625417,19sleep10-21swapper/119:34:071
840625417,19sleep10-21swapper/119:34:071
766725419,15sleep20-21swapper/222:54:292
465725415,24sleep34667-21munin-run00:29:053
238925417,17sleep10-21swapper/122:39:061
1869525415,19sleep30-21swapper/320:04:343
1845825415,24sleep118503-21irqstats20:04:271
122842547,11sleep30-21swapper/321:26:593
752525312,19sleep10-21swapper/121:14:041
73925316,21sleep30-21swapper/300:14:323
471125318,16sleep10-21swapper/122:44:311
2154125313,17sleep30-21swapper/323:39:203
756525213,24sleep27575-21munin-run21:14:052
3222625213,21sleep332221-21apt-get20:49:193
3111425213,15sleep30-21swapper/300:09:193
262925212,23sleep12699-21df22:39:191
2132925213,24sleep321324-21apt-get20:14:193
2132925213,24sleep321324-21apt-get20:14:193
480125114,22sleep34872-21fschecks_time21:04:253
289725112,18sleep30-21swapper/322:39:253
2250325117,18sleep30-21swapper/321:59:253
1667025113,18sleep20-21swapper/219:59:162
112392518,10sleep30-21swapper/319:41:003
2571725016,15sleep20-21swapper/222:09:212
1532925011,17sleep30-21swapper/323:19:213
1382625014,16sleep20-21swapper/223:14:242
1340225016,16sleep30-21swapper/319:49:233
554724912,21sleep15544-21apt-get19:24:181
2956824911,22sleep129563-21apt-get00:04:211
1593624911,23sleep315945-21diskstats21:39:223
1428724914,22sleep114354-21df21:34:191
1740624812,21sleep117403-21apt-get21:44:161
1740724713,16sleep20-21swapper/221:44:182
320242466,16sleep20-21swapper/222:29:232
2800424611,20sleep128003-21apt-get23:59:161
2609624511,21sleep20-21swapper/220:29:242
1838924513,19sleep2231-21systemd-journal23:29:132
1604524415,15sleep20-21swapper/221:39:252
233312437,14sleep30-21swapper/320:19:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional