You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 20:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Fri Mar 27, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3043629666,948sleep230475-21ntpq11:57:492
19746274920,714sleep319747-21sshd09:42:323
19746274920,714sleep319747-21sshd09:42:323
12556270517,671sleep312592-21/usr/sbin/munin11:02:423
14017246110,436sleep314025-21ntpq09:22:533
30880242230,49sleep20-21swapper/208:32:522
14705242024,382sleep214862-21awk11:07:542
2604724050,11sleep30-21swapper/310:02:253
12097235521,315sleep012135-21grep09:17:410
13828234218,258sleep213850-21kernelversion09:22:482
5648233223,287sleep05642-21apt-get12:22:390
2491823297,309sleep225029-21latency_hist11:42:262
12872232812,300sleep112919-21ntpq11:02:481
22308232710,276sleep222343-21latency11:32:492
9311229320,255sleep19329-21/usr/sbin/munin10:52:351
5161228228,237sleep25225-21ps10:37:502
15894226711,239sleep315935-21unin-run07:47:253
19557224912,223sleep019631-21find11:23:060
19634224425,196sleep119771-21unixbench_multi11:23:071
7028222514,82sleep27047-21modprobe09:02:242
7028222514,82sleep27047-21modprobe09:02:242
27081222344,127sleep20-21swapper/208:22:252
32682221120,74sleep20-21swapper/210:22:442
18701221140,124sleep218821-21latency_hist11:22:272
28630221037,121sleep228730-21latency_hist08:27:262
28630221037,121sleep228730-21latency_hist08:27:262
26330220928,140sleep226386-21tune2fs10:02:422
26330220928,140sleep226386-21tune2fs10:02:422
22074220712,183sleep013441-21snmpd08:03:500
4500220447,119sleep24542-21ntpq12:17:482
1966220329,121sleep21544-21/usr/sbin/munin10:27:492
22618220043,142sleep222628-21apt-key09:52:242
315721959,170sleep23274-21timerwakeupswit12:12:552
315721959,170sleep23274-21timerwakeupswit12:12:552
14750219512,135sleep214819-21fschecks_time07:43:122
7430219456,47sleep20-21swapper/212:27:462
7817219021,129sleep27895-21mailstats09:02:522
6948218648,82sleep20-21swapper/212:27:252
21033218639,126sleep221044-21modprobe09:47:232
21033218639,126sleep221044-21modprobe09:47:232
1342218133,112sleep21355-21uname08:42:452
1166218123,136sleep31192-21fschecks_count08:42:423
8739217837,20sleep20-21swapper/212:32:262
11657217715,114sleep011770-21unixbench_singl10:57:550
24369217629,114sleep224425-21if_err_eth008:12:442
24369217629,114sleep224425-21if_err_eth008:12:442
8164216231,117sleep20-21swapper/207:22:402
1379221587,140sleep113441-21snmpd07:38:301
1379221587,140sleep113441-21snmpd07:38:301
1572221558,107sleep215828-21ls09:27:572
1259821527,134sleep212659-21latency_hist07:37:262
16175214818,25sleep20-21swapper/207:47:402
20024214783,18sleep20-21swapper/207:57:542
25766214530,77sleep225852-21ls11:42:552
22643214523,15sleep20-21swapper/208:07:392
11812214327,46sleep20-21swapper/209:17:262
9634214111,114sleep19708-21find12:33:301
9618214121,97sleep39769-21unixbench_singl09:07:543
32385213913,112sleep232428-21ntpq08:37:482
14227213937,34sleep214262-21apt-key07:42:272
14227213937,34sleep214262-21apt-key07:42:272
4389213829,71sleep24524-21latency_hist07:12:262
25058213525,47sleep20-21swapper/209:57:492
7793213352,15sleep20-21swapper/210:47:392
1892821308,50sleep218925-21apt-get11:22:362
31623212712,101sleep029314-1kworker/0:0H12:02:390
30927212621,64sleep20-21swapper/210:17:402
29829212316,93sleep329798-21latency_hist11:57:253
28346212315,66sleep20-21swapper/210:07:552
28346212315,66sleep20-21swapper/210:07:552
12467212340,43sleep22852-21cyclictest11:02:372
7081212219,87sleep014977-1kworker/0:1H09:02:260
7081212219,87sleep014977-1kworker/0:1H09:02:260
31890212031,72sleep231596-21/usr/sbin/munin12:02:452
10188212022,53sleep20-21swapper/209:12:262
5848211916,90sleep15861-21fschecks_count08:57:421
5848211916,90sleep15861-21fschecks_count08:57:421
12519211920,81sleep0222-21systemd-journal07:37:240
7881211816,80sleep18030-21tail12:27:571
24187211828,66sleep024181-21apt-get08:12:380
24187211828,66sleep024181-21apt-get08:12:380
1602021188,75sleep216019-21ntp_states11:12:512
29136211410,66sleep229186-21sed08:27:472
29136211410,66sleep229186-21sed08:27:472
17474211427,24sleep30-21swapper/307:52:253
978211320,72sleep3941-21apt-get12:07:553
23641211317,39sleep20-21swapper/209:53:412
980621129,69sleep29312-21/usr/sbin/munin10:52:522
1799221127,63sleep218131-21latency_hist09:37:262
4781211116,72sleep14819-21fschecks_count10:37:411
30607211114,26sleep230668-21modprobe10:17:232
30415211114,79sleep130416-21/usr/sbin/munin08:32:381
2181021089,57sleep20-21swapper/208:03:272
19060210810,82sleep319170-21latency_hist07:57:263
19060210810,82sleep319170-21latency_hist07:57:263
8278210517,66sleep00-21swapper/010:47:540
10607210526,54sleep20-21swapper/207:27:572
695210312,74sleep0832-21awk10:22:540
479421028,47sleep24880-21ls08:52:552
16466210218,63sleep115537-21/usr/sbin/munin11:12:581
13323210211,22sleep213445-21latency_hist09:22:272
3779210117,67sleep13826-21readlink12:17:231
3779210117,67sleep13826-21readlink12:17:231
13345210118,67sleep313434-21mailstats07:37:563
13345210118,67sleep313434-21mailstats07:37:563
448321008,32sleep24632-21date10:37:262
61329910,75sleep3691-21find10:22:533
2846129914,71sleep127977-21apt-get11:52:351
1097029924,60sleep110976-21modprobe07:32:241
275229830,48sleep02760-21modprobe07:07:230
1346729811,73sleep013441-21snmpd11:04:000
1181029811,74sleep111819-21vmstat10:57:561
3048729712,71sleep330495-21df08:32:403
2796329710,26sleep227993-21munin-run11:52:232
291229628,20sleep20-21swapper/208:47:462
749829527,27sleep27522-21modprobe10:47:232
2976029531,48sleep21-21systemd11:57:242
564629410,15sleep20-21swapper/207:12:552
233929419,57sleep02370-21awk08:47:230
307829318,37sleep10-21swapper/107:07:371
1079329316,62sleep310804-21/usr/sbin/munin09:12:473
2642229221,54sleep326547-21grep08:17:543
2183329217,59sleep021977-21latency_hist11:32:270
1856329116,53sleep118698-21latency_hist11:22:261
182312898,37sleep20-21swapper/209:37:382
2491028816,54sleep125049-21unixbench_singl08:12:551
2491028816,54sleep125049-21unixbench_singl08:12:551
2056428819,51sleep020626-21unin-run08:02:240
1488328814,56sleep014932-21irqstats07:43:140
369628715,58sleep33781-21cat10:32:553
1409628711,61sleep014146-21systemd-run07:42:250
1409628711,61sleep014146-21systemd-run07:42:250
135282876,11sleep10-21swapper/109:22:301
270912869,20sleep30-21swapper/310:02:553
270912869,20sleep30-21swapper/310:02:553
246572869,64sleep10-21swapper/109:57:321
2284028611,62sleep022894-21if_eth008:07:450
1823228626,44sleep218320-21mailstats07:52:532
1823228626,44sleep218320-21mailstats07:52:532
109962869,38sleep20-21swapper/210:57:412
2905028517,52sleep31-21systemd10:12:243
2905028517,52sleep31-21systemd10:12:243
2069628512,56sleep120752-21irqstats11:27:431
2809828418,46sleep028179-21apt-config11:52:250
94828313,33sleep20-21swapper/212:07:382
72962836,41sleep27441-21sed07:17:552
207532837,39sleep220805-21latency11:27:442
1784528315,28sleep10-21swapper/107:52:441
1784528315,28sleep10-21swapper/107:52:441
2139828218,49sleep30-21swapper/309:47:413
2585428118,46sleep326001-21unixbench_multi11:42:563
1276828117,28sleep10-21swapper/109:17:541
1090928113,30sleep00-21swapper/009:12:520
3017728016,45sleep030294-21kworker/u8:308:32:250
3017728016,45sleep030294-21kworker/u8:308:32:250
2816627917,25sleep10-21swapper/110:07:501
2816627917,25sleep10-21swapper/110:07:501
2556527916,31sleep10-21swapper/111:42:491
207062798,13sleep00-21swapper/009:43:090
207062798,13sleep00-21swapper/009:43:090
1755727817,44sleep117568-21vmstat09:32:561
3033727711,31sleep00-21swapper/011:57:470
2825727713,47sleep028277-21/usr/sbin/munin10:07:530
2825727713,47sleep028277-21/usr/sbin/munin10:07:530
157927731,22sleep20-21swapper/207:02:422
119827717,42sleep01251-21/usr/sbin/munin12:08:290
157027617,26sleep10-21swapper/110:27:401
2329327513,48sleep023316-21munin-run11:37:230
2113327514,33sleep121082-21apt-get08:03:061
2108927513,48sleep121225-21latency_hist09:47:261
2108927513,48sleep121225-21latency_hist09:47:261
2409727423,26sleep00-21swapper/011:37:520
943327310,16sleep30-21swapper/312:33:243
936027318,20sleep00-21swapper/009:07:510
334927318,27sleep30-21swapper/307:07:473
1532427319,27sleep10-21swapper/111:12:251
341127214,27sleep00-21swapper/012:12:570
341127214,27sleep00-21swapper/012:12:570
158327222,25sleep10-21swapper/108:42:521
2258327121,32sleep122597-21apt-get09:52:231
1370927114,38sleep013771-21/usr/sbin/munin09:22:460
272627034,18sleep30-21swapper/308:47:433
249592709,45sleep025038-21cat09:57:470
1177127019,36sleep111811-21latency_hist09:17:251
688426922,29sleep00-21swapper/007:17:450
3199126917,30sleep10-21swapper/108:37:391
1934526916,24sleep30-21swapper/307:57:393
161172699,49sleep00-21swapper/009:29:120
728226815,29sleep00-21swapper/012:27:440
65212687,15sleep10-21swapper/110:42:461
522726820,31sleep30-21swapper/310:37:513
500626819,26sleep35025-21apt-get07:12:353
3068626822,25sleep00-21swapper/010:17:250
542826712,37sleep05578-21latency_hist12:22:270
1123226720,26sleep30-21swapper/307:32:343
541026620,25sleep00-21swapper/007:12:480
540926619,20sleep30-21swapper/312:22:263
465026613,19sleep30-21swapper/308:52:523
275262668,12sleep00-21swapper/008:22:440
275262668,12sleep00-21swapper/008:22:440
2646626612,36sleep00-21swapper/011:47:250
2586926615,29sleep00-21swapper/008:17:430
694526514,36sleep16947-21systemd-run12:27:241
552426517,24sleep10-21swapper/108:57:261
3178826524,23sleep00-21swapper/008:37:260
2736226512,19sleep10-21swapper/111:47:541
2062826516,29sleep120608-21gpgv08:02:251
165102655,11sleep00-21swapper/011:13:210
2444926416,31sleep324577-21ls09:57:263
2276226420,24sleep30-21swapper/308:07:443
1645226418,29sleep316601-21latency_hist09:32:263
48626317,31sleep1603-21latency_hist12:07:251
2777926316,20sleep30-21swapper/308:22:513
2777926316,20sleep30-21swapper/308:22:513
276526321,27sleep12820-21systemd-run07:07:241
2428626312,36sleep0758-21sendmail-mta09:56:180
2225026318,21sleep10-21swapper/111:32:481
1667726315,26sleep00-21swapper/007:47:540
503326211,37sleep05092-21users08:52:570
300842628,14sleep30-21swapper/310:12:563
2897426214,26sleep00-21swapper/011:53:060
2691426217,23sleep30-21swapper/311:47:443
2105826210,34sleep30-21swapper/308:02:313
792626117,20sleep10-21swapper/107:22:261
674626124,23sleep30-21swapper/310:42:523
3230526119,25sleep332453-21grep12:02:553
300572616,46sleep10-21swapper/108:31:331
300572616,46sleep10-21swapper/108:31:331
2739626114,20sleep10-21swapper/108:22:411
2739626114,20sleep10-21swapper/108:22:411
1624126118,21sleep30-21swapper/311:12:563
960926018,20sleep00-21swapper/010:52:460
94142604,48sleep00-21swapper/007:27:180
708226026,20sleep30-21swapper/307:17:523
2895326017,29sleep329009-21tune2fs08:27:443
2895326017,29sleep329009-21tune2fs08:27:443
1945926018,21sleep30-21swapper/311:23:053
1869126010,22sleep30-21swapper/309:37:513
1814726010,15sleep00-21swapper/011:19:020
1527626018,21sleep10-21swapper/107:43:231
719425917,25sleep37189-21apt-get12:27:393
609425916,18sleep30-21swapper/312:22:503
231042598,20sleep10-21swapper/109:52:351
1552725916,27sleep115563-21ntp_states09:27:511
971125815,26sleep09704-21apt-get07:27:390
51112586,16sleep10-21swapper/112:19:121
282325813,29sleep02857-21munin-run10:32:230
215325815,22sleep30-21swapper/310:27:543
3223325714,17sleep00-21swapper/010:22:260
321325715,27sleep13282-21switchtime08:47:531
3138425616,18sleep10-21swapper/110:17:511
312662565,39sleep10-21swapper/112:01:311
312625612,19sleep10-21swapper/110:32:401
2597225617,22sleep10-21swapper/108:17:441
2048425615,24sleep320475-21apt-get11:27:373
1401825616,22sleep314085-21df11:07:393
1017525611,29sleep110187-21munin-run09:12:251
816325512,25sleep18156-21apt-get07:22:391
803525511,25sleep10-21swapper/110:47:461
385825513,17sleep30-21swapper/312:17:253
385825513,17sleep30-21swapper/312:17:253
3260625514,23sleep132681-21fw_packets10:22:431
312525513,24sleep03118-21apt-get10:32:390
274882559,37sleep10-21swapper/110:07:131
274882559,37sleep10-21swapper/110:07:131
2442625517,17sleep30-21swapper/308:12:453
2442625517,17sleep30-21swapper/308:12:453
2360725514,24sleep123602-21apt-get11:37:371
978825410,17sleep30-21swapper/307:27:423
864025417,20sleep30-21swapper/309:07:263
38482546,37sleep00-21swapper/008:51:480
2189925414,16sleep00-21swapper/009:47:540
1721725412,27sleep317226-21sshd11:17:403
1665625413,25sleep316648-21apt-get09:32:383
740425311,23sleep00-21swapper/009:02:420
3243025311,27sleep332441-21ntp_states08:37:493
3199025315,23sleep031984-21apt-get08:37:380
2370525312,25sleep30-21swapper/311:37:433
2249825310,25sleep30-21swapper/311:32:543
1840325314,21sleep00-21swapper/007:52:550
1840325314,21sleep00-21swapper/007:52:550
1091325317,18sleep30-21swapper/310:57:373
100425319,17sleep10-21swapper/112:08:141
985325213,15sleep30-21swapper/310:52:533
930025215,19sleep00-21swapper/012:33:210
1952825212,25sleep019537-21munin-run09:42:240
1864125214,22sleep118683-21ntp_states09:37:491
178242527,35sleep00-21swapper/009:36:150
778725112,23sleep37782-21apt-get10:47:363
57922515,37sleep00-21swapper/010:40:190
3222025112,22sleep332230-21munin-run10:22:253
3028625112,23sleep330335-21irqstats11:57:463
149525112,22sleep31562-21df07:02:383
715525013,12sleep10-21swapper/109:02:271
2631525010,25sleep026324-21df_inode10:02:400
2631525010,25sleep026324-21df_inode10:02:400
1401525011,22sleep014011-21apt-get11:07:360
510924911,22sleep15116-21diskstats07:12:411
305624913,21sleep03074-21apt-get07:07:360
82872489,16sleep30-21swapper/307:22:453
705224813,18sleep30-21swapper/309:02:253
705224813,18sleep30-21swapper/309:02:253
650324814,20sleep00-21swapper/010:42:450
1209024611,20sleep312095-21diskstats09:17:403
2916124310,17sleep10-21swapper/110:12:261
2264224311,16sleep10-21swapper/108:07:381
122172438,23sleep10-21swapper/107:33:331
161742408,17sleep10-21swapper/107:47:391
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional