You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 13:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Thu Mar 26, 2026 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7567264122,600sleep015664-21kworker/u8:023:12:400
26573238156,227sleep126711-21unixbench_multi20:48:001
27530230017,268sleep127605-21fschecks_time20:52:491
29394224810,225sleep029438-21ntpq20:57:530
31992222425,147sleep10-21swapper/119:22:481
31992222425,147sleep10-21swapper/119:22:481
30115221842,122sleep130173-21cut19:17:311
2432822170,9sleep112-21rcu_sched20:42:381
1806122147,45sleep00-21swapper/023:45:520
1806122147,45sleep00-21swapper/023:45:520
1592221414,185sleep31657-21awk22:52:573
6516220814,169sleep16664-21unixbench_singl23:08:011
10016220513,158sleep110067-21sendmail19:57:291
10016220513,158sleep110067-21sendmail19:57:291
9428220213,147sleep10-21swapper/123:17:501
18515219117,153sleep318548-21ntp_kernel_pll_20:22:553
22209218510,158sleep315664-21kworker/u8:023:57:583
155642183110,54sleep10-21swapper/123:37:491
13871218328,139sleep013912-21gpgv21:52:290
12229218235,132sleep112240-21sort23:27:391
2666821769,153sleep126715-21ntpq00:12:541
25548217519,80sleep10-21swapper/122:27:311
7294217444,117sleep17351-21grep19:47:481
7294217444,117sleep17351-21grep19:47:481
2566721746,158sleep113441-21snmpd00:08:381
19542217438,84sleep10-21swapper/122:07:561
22550217112,143sleep222660-21latency_hist20:37:312
4005216710,101sleep14121-21latency_hist23:02:321
26819216415,81sleep10-21swapper/119:07:461
17294216247,26sleep10-21swapper/122:02:311
5901216171,74sleep15890-21munin-run21:27:301
10802216129,96sleep110832-21proc_pri19:58:011
2958921597,104sleep129660-21switchtime20:57:581
30083215243,93sleep130213-21grep00:23:001
20037215213,124sleep220068-21nfs_client20:27:542
1097721529,129sleep111047-21cat23:22:511
1097721529,129sleep111047-21cat23:22:511
16295215121,100sleep215358-21/usr/sbin/munin23:38:022
28989215014,120sleep329008-21apt-get20:57:393
1738214431,54sleep10-21swapper/121:12:521
524921435,72sleep10-21swapper/123:04:131
23153214215,111sleep223181-21netstat20:37:522
5197214012,111sleep05278-21threads21:23:010
15474214010,95sleep115547-21tr20:12:591
24307213710,112sleep024326-21apt-get20:42:380
13778213720,98sleep013797-21apt-get23:32:390
31975213635,84sleep132164-21latency_hist21:07:311
2742621307,108sleep127458-21tune2fs22:32:481
29290212723,49sleep10-21swapper/122:37:561
1727212736,76sleep11852-21timerwakeupswit22:52:591
8071212614,96sleep17545-21/usr/sbin/munin23:13:181
32417212342,43sleep132345-21/usr/sbin/munin21:08:161
14648212312,96sleep314649-21sshd20:11:523
1696321218,94sleep30-21swapper/320:17:543
7518212019,12sleep10-21swapper/121:30:491
7579211921,77sleep27563-21apt-get23:12:462
24097211810,95sleep124096-21munin-run20:42:311
2013821187,100sleep00-21swapper/022:09:530
19088211811,88sleep10-21swapper/123:47:591
19088211811,88sleep10-21swapper/123:47:591
2102321176,12sleep10-21swapper/123:54:121
13913211712,59sleep114002-21gpgv21:52:301
25904211652,47sleep10-21swapper/122:27:471
363821155,11sleep10-21swapper/119:35:241
363821155,11sleep10-21swapper/119:35:241
17339211438,22sleep10-21swapper/123:42:581
17339211438,22sleep10-21swapper/123:42:581
16656211447,22sleep116688-21fschecks_count20:17:481
511521127,96sleep00-21swapper/019:39:060
6973211129,56sleep10-21swapper/119:47:311
31670211114,80sleep131817-21latency_hist19:22:321
12163211022,67sleep212219-21latency20:02:552
12163211022,67sleep212219-21latency20:02:552
1178211019,73sleep21207-21awk21:12:292
4669210929,67sleep14735-21switchtime19:37:581
26496210911,81sleep026570-21find20:47:590
1383621097,80sleep10-21swapper/120:07:551
996210834,27sleep10-21swapper/119:27:451
996210834,27sleep10-21swapper/119:27:451
9513210827,60sleep10-21swapper/121:37:441
24990210837,44sleep224986-21apt-get19:02:422
21279210824,28sleep221337-21munin-run23:57:282
12839210838,21sleep10-21swapper/120:03:301
12839210838,21sleep10-21swapper/120:03:301
3056521077,84sleep130668-21fschecks_time22:42:491
21615210514,28sleep10-21swapper/120:32:541
14524210534,19sleep10-21swapper/121:52:501
1232221049,56sleep112350-21munin-plugin-st21:47:291
2273221015,16sleep10-21swapper/122:18:141
1386210123,55sleep21507-21latency_hist00:37:302
354029910,50sleep13613-21tr21:17:571
6392985,12sleep10-21swapper/122:50:081
2556429820,60sleep125633-21switchtime19:02:581
2578329725,39sleep00-21swapper/022:27:420
2035329725,52sleep10-21swapper/120:28:021
196142978,12sleep30-21swapper/322:07:583
817129611,45sleep10-21swapper/121:32:511
2273429614,64sleep322679-21apt-get22:18:163
2190629611,17sleep00-21swapper/023:57:510
3209929514,59sleep231991-21/usr/sbin/munin22:47:472
2009329513,68sleep320101-21df23:52:453
293482947,15sleep30-21swapper/322:37:593
220672945,55sleep10-21swapper/122:16:361
139122947,45sleep114009-21perl23:32:481
607529318,56sleep35872-21apt-get21:27:323
305129228,41sleep23047-21apt-get21:17:422
1789629226,46sleep10-21swapper/122:02:511
1690329217,53sleep316904-21/usr/sbin/munin23:42:403
1690329217,53sleep316904-21/usr/sbin/munin23:42:403
327052917,37sleep10-21swapper/100:32:491
319872915,54sleep10-21swapper/100:29:381
2526229034,23sleep20-21swapper/200:07:592
548628910,63sleep25624-21modprobe19:42:322
1246128911,66sleep312502-21if_eth023:27:503
2806228815,60sleep028106-21if_eth000:17:500
208028824,35sleep20-21swapper/221:13:012
3132128721,51sleep331350-21/usr/sbin/munin00:27:543
3094628712,53sleep030547-21/usr/sbin/munin21:02:530
1994028712,61sleep019992-21latency20:27:520
931128615,55sleep09384-21ls19:52:590
931128615,55sleep09384-21ls19:52:590
322922860,7sleep20-21swapper/219:22:522
322922860,7sleep20-21swapper/219:22:522
287702869,26sleep127829-21/usr/sbin/munin00:18:031
1507128617,46sleep315111-21sort20:12:493
1782828513,57sleep217861-21munin-run20:22:292
2945028419,46sleep329607-21tail19:13:013
1144328421,44sleep211594-21sed23:23:002
1144328421,44sleep211594-21sed23:23:002
2869728315,51sleep028733-21latency_hist20:57:280
3172628221,41sleep031756-21awk22:47:280
299628218,49sleep03003-21ntp_states19:32:570
299628218,49sleep03003-21ntp_states19:32:570
2514628220,49sleep025216-21idleruntime19:02:490
956828119,41sleep09135-21/usr/sbin/munin23:17:530
2436228119,11sleep10-21swapper/100:06:441
216082818,33sleep10-21swapper/123:57:441
2036628124,38sleep220416-21munin-run22:12:302
1400328116,49sleep214196-21date21:52:312
635428016,40sleep26355-21munin-node21:27:392
649827920,42sleep06579-21fschecks_time21:28:050
2710627915,49sleep227248-21cut22:32:322
2534027919,32sleep30-21swapper/300:08:003
2479727816,47sleep024890-21grep00:07:480
22912787,14sleep00-21swapper/022:56:290
2234627811,43sleep322163-21latency_hist22:17:313
157502780,7sleep00-21swapper/020:13:010
1082627818,30sleep00-21swapper/021:42:310
2449627724,32sleep00-21swapper/000:07:320
2115027721,29sleep20-21swapper/222:12:582
2777127617,44sleep327785-21vmstat19:08:453
2545227624,21sleep30-21swapper/319:02:563
1932327618,11sleep10-21swapper/120:26:481
1636127615,43sleep216500-21latency_hist20:17:332
2781027514,40sleep227415-21/usr/sbin/munin20:52:542
254427512,17sleep20-21swapper/219:32:462
254427512,17sleep20-21swapper/219:32:462
1197827521,40sleep312036-21cut20:02:523
1197827521,40sleep312036-21cut20:02:523
2919927414,41sleep028902-21/usr/sbin/munin22:37:550
2678827437,17sleep00-21swapper/019:07:410
2549627418,38sleep325522-21munin-run22:27:293
2213427428,28sleep022166-21basename22:17:280
919327311,38sleep30-21swapper/321:37:293
2781327312,47sleep327862-21processes22:32:583
185427319,22sleep20-21swapper/222:53:002
626627217,35sleep26332-21wc23:07:582
2559127228,26sleep225599-21modprobe20:47:292
132262726,19sleep00-21swapper/023:28:480
806327113,47sleep38066-21ntpq23:13:163
115352718,10sleep20-21swapper/220:01:382
2930027013,36sleep229345-21ntp_kernel_pll_22:37:582
1479727021,25sleep20-21swapper/220:12:322
578926920,22sleep00-21swapper/023:07:440
2638326920,21sleep00-21swapper/000:12:490
25402696,48sleep30-21swapper/319:32:433
25402696,48sleep30-21swapper/319:32:433
2247126920,28sleep00-21swapper/020:37:290
1792526923,25sleep20-21swapper/222:02:522
1667426918,26sleep30-21swapper/321:58:013
1448726920,21sleep00-21swapper/021:52:490
1161826922,22sleep30-21swapper/321:42:593
747726823,20sleep00-21swapper/019:47:510
747726823,20sleep00-21swapper/019:47:510
3100526819,23sleep00-21swapper/000:27:470
1134526816,30sleep011429-21memory21:42:510
764526722,30sleep37665-21modprobe21:32:293
155532679,45sleep313441-21snmpd21:56:073
1515526715,35sleep315302-21latency_hist23:37:313
1325826721,29sleep313382-21/usr/sbin/munin21:47:593
924526619,25sleep20-21swapper/223:17:472
329026616,28sleep20-21swapper/222:58:212
3093226615,36sleep330943-21needreboot21:02:523
2985026616,27sleep00-21swapper/000:22:550
1314826618,35sleep013174-21/usr/sbin/munin20:07:290
1314826618,35sleep013174-21/usr/sbin/munin20:07:290
1099726614,33sleep011111-21/usr/sbin/munin19:58:040
926426524,25sleep00-21swapper/021:37:300
87332659,42sleep28752-21apt-get19:52:412
87332659,42sleep28752-21apt-get19:52:412
3071126518,27sleep30-21swapper/300:27:303
3033326516,29sleep20-21swapper/221:02:312
1644826510,42sleep00-21swapper/023:39:580
129652658,12sleep20-21swapper/223:28:012
420026418,25sleep20-21swapper/219:37:472
313826415,23sleep30-21swapper/321:17:473
3053826415,21sleep00-21swapper/019:17:500
2496426419,30sleep325086-21cat20:42:583
337626319,25sleep30-21swapper/322:58:223
2812926317,21sleep20-21swapper/219:12:312
1824026317,19sleep20-21swapper/223:47:312
1824026317,19sleep20-21swapper/223:47:312
846926215,29sleep28501-21awk19:52:292
846926215,29sleep28501-21awk19:52:292
718326215,21sleep30-21swapper/323:12:313
598926214,27sleep26074-21apt-config21:27:312
314682625,11sleep30-21swapper/322:44:083
2136526218,20sleep00-21swapper/023:57:310
485426115,17sleep00-21swapper/023:02:590
2121326115,26sleep321206-21apt-get20:32:443
1788126111,35sleep017976-21cut20:22:310
1165226114,23sleep30-21swapper/320:02:313
9492605,42sleep00-21swapper/021:09:320
75626013,18sleep00-21swapper/019:27:310
75626013,18sleep00-21swapper/019:27:310
194926022,21sleep00-21swapper/021:12:590
191232605,11sleep00-21swapper/020:24:130
1600926020,20sleep30-21swapper/323:37:593
1461026018,23sleep20-21swapper/221:52:512
441825917,19sleep00-21swapper/021:22:320
2877825917,25sleep328903-21grep20:57:313
2860325916,26sleep228645-21basename22:37:292
2795125917,17sleep30-21swapper/300:17:493
2323325912,21sleep20-21swapper/200:02:462
20762596,17sleep20-21swapper/219:30:122
20762596,17sleep20-21swapper/219:30:122
855025812,12sleep20-21swapper/221:33:002
3258325822,15sleep30-21swapper/300:32:473
2728825819,20sleep20-21swapper/200:14:292
1862125817,18sleep30-21swapper/323:47:483
1862125817,18sleep30-21swapper/323:47:483
1344025810,31sleep013507-21cpuspeed20:07:450
3250125715,28sleep032569-21systemctl22:47:570
3233025717,22sleep331850-21apt-get21:07:333
2884525718,23sleep228852-21diskstats19:12:482
2404525718,19sleep30-21swapper/320:42:293
22042576,41sleep00-21swapper/019:31:520
22042576,41sleep00-21swapper/019:31:520
1978325719,18sleep00-21swapper/023:52:300
1978325719,18sleep00-21swapper/023:52:300
1624425714,24sleep216275-21/usr/sbin/munin21:57:522
952625613,23sleep29585-21df21:37:452
728025613,27sleep30-21swapper/319:47:453
728025613,27sleep30-21swapper/319:47:453
3033025615,18sleep30-21swapper/319:17:413
2987025616,26sleep229922-21processes00:22:572
2578125612,26sleep325777-21apt-get22:27:413
102982568,34sleep30-21swapper/323:20:553
878525515,25sleep38830-21df19:52:463
878525515,25sleep38830-21df19:52:463
37952556,38sleep00-21swapper/023:00:250
3095425512,25sleep230946-21apt-get00:27:452
3084925516,25sleep230897-21awk22:42:542
232425517,19sleep30-21swapper/319:32:313
232425517,19sleep30-21swapper/319:32:313
2322225513,17sleep00-21swapper/020:37:540
218925525,15sleep30-21swapper/321:13:023
2098725516,18sleep30-21swapper/320:32:323
616125413,24sleep36240-21awk19:42:573
448125414,21sleep30-21swapper/319:37:523
2522125423,20sleep00-21swapper/022:23:540
2500825413,18sleep20-21swapper/222:22:592
1816925417,21sleep00-21swapper/022:02:590
1811925412,24sleep218186-21df20:22:462
1113425416,23sleep211141-21df_inode21:42:452
1087325418,16sleep30-21swapper/323:22:493
1087325418,16sleep30-21swapper/323:22:493
2785325312,24sleep227847-21apt-get00:17:442
2695425313,24sleep327112-21sed00:13:003
2315925314,23sleep323154-21apt-get00:02:433
1752725313,23sleep317523-21apt-get22:02:413
471825216,19sleep20-21swapper/223:02:572
3189825213,23sleep331894-21apt-get19:22:423
3189825213,23sleep331894-21apt-get19:22:423
2671525214,17sleep20-21swapper/220:48:012
1919425214,19sleep20-21swapper/223:48:002
1919425214,19sleep20-21swapper/223:48:002
1597225214,22sleep015968-21apt-get21:57:420
2836125114,16sleep00-21swapper/019:12:330
2265425112,23sleep222153-21apt-get22:17:342
2167825115,23sleep221684-21df_inode23:57:452
3249225012,22sleep032488-21apt-get00:32:400
2595825012,22sleep325964-21diskstats20:47:463
431324912,20sleep34319-21df_inode23:02:463
310272499,16sleep20-21swapper/221:02:552
2268424915,20sleep00-21swapper/022:17:400
1419824912,16sleep20-21swapper/223:32:532
958624811,17sleep30-21swapper/321:37:463
3237824812,15sleep30-21swapper/322:47:523
2760724814,20sleep00-21swapper/022:32:510
109382479,14sleep00-21swapper/023:22:500
109382479,14sleep00-21swapper/023:22:500
3035524410,17sleep20-21swapper/219:17:442
57382408,16sleep00-21swapper/019:42:460
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional