You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 13:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri May 22, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29403210436,999sleep028740-1kworker/0:0H23:14:170
372329658,883sleep03912-21apt-config21:54:020
8781289017,855sleep28817-21/usr/sbin/munin22:09:182
22801276522,725sleep322851-21/usr/sbin/munin00:34:203
20679271813,692sleep320681-21sshd19:23:073
2887624318,411sleep028912-21ntpq21:29:390
23636240113,371sleep023727-21fschecks_time21:14:210
2393223928,368sleep024049-21cat00:39:010
8999235212,324sleep29040-21/usr/sbin/munin20:29:002
17635230711,252sleep017650-21modprobe00:18:580
1866228211,196sleep01364-21apt-get23:29:150
27548226793,57sleep00-21swapper/019:44:160
65422259113,57sleep00-21swapper/023:44:130
65422259113,57sleep00-21swapper/023:44:130
19683224711,216sleep019715-21fschecks_count22:44:170
14227223965,103sleep014261-21wc20:44:200
17217223638,180sleep017368-21unixbench_singl22:34:320
6296222514,197sleep36322-21latency_hist23:44:003
2766222570,86sleep02788-21awk20:08:590
22637221714,142sleep022417-21latency_hist22:54:020
916321909,166sleep09227-21ps22:09:280
247502183100,37sleep00-21swapper/019:34:270
14843218324,143sleep314857-21awk19:04:303
5678218218,143sleep25826-21grep23:39:312
2116421819,66sleep021306-21head21:04:310
2116421819,66sleep021306-21head21:04:310
3250821807,156sleep00-21swapper/023:24:110
31966218018,139sleep032011-21apt-key19:59:020
3587217929,132sleep33612-21apt-key21:54:013
30075217910,158sleep00-21swapper/019:51:060
24275217510,149sleep024423-21date22:59:030
11831217011,61sleep011836-21cpu22:19:180
7691216924,69sleep07769-21switchtime22:04:280
25773216921,131sleep325775-21cron23:03:593
432216810,14sleep20-21swapper/219:59:492
27393216726,119sleep227533-21unixbench_singl21:24:322
22151216726,42sleep00-21swapper/021:09:200
1579321677,39sleep00-21swapper/020:49:410
28272216192,52sleep00-21swapper/023:09:290
21678215810,134sleep021800-21timerwakeupswit00:29:320
9657215613,124sleep09265-21/usr/sbin/munin20:29:220
14541215421,117sleep114056-21apt-get00:09:101
14541215421,117sleep114056-21apt-get00:09:101
1179221477,71sleep00-21swapper/020:35:080
9229214613,116sleep19299-21sensors22:09:291
2048421455,29sleep00-21swapper/000:25:280
20123214410,49sleep00-21swapper/019:19:450
32398214127,92sleep132395-21apt-get19:59:111
8831213920,99sleep18956-21grep23:49:311
8831213920,99sleep18956-21grep23:49:311
31204213116,19sleep00-21swapper/021:38:110
24501212917,13sleep00-21swapper/022:59:110
20747212723,86sleep220765-21idleruntime-cro19:23:592
19861212718,16sleep00-21swapper/021:00:190
2704921265,13sleep00-21swapper/023:05:290
26272212518,95sleep00-21swapper/021:22:320
32733212321,17sleep20-21swapper/223:24:212
28281212216,85sleep328395-21users19:44:313
10893212225,79sleep310925-21basename23:58:583
4319212129,79sleep04368-21systemctl20:14:000
2377621216,12sleep00-21swapper/019:30:030
13375212135,16sleep00-21swapper/022:24:140
17882212043,41sleep30-21swapper/319:14:113
1349021205,13sleep00-21swapper/020:41:380
9005211918,89sleep00-21swapper/023:49:350
9005211918,89sleep00-21swapper/023:49:350
15940211811,92sleep024936-21munin-node19:09:110
14107211811,92sleep014230-21dpkg00:09:010
4677211716,81sleep24695-21sh23:38:592
18674211710,92sleep018803-21cat22:39:290
27284211619,18sleep10-21swapper/121:24:311
10921211624,71sleep310962-21uniq20:34:183
3788211513,86sleep33831-21ntp_kernel_pll_23:34:233
396121137,63sleep00-21swapper/023:34:280
32621211215,12sleep00-21swapper/021:41:100
627211117,78sleep3702-21switchtime23:24:303
1158021110,7sleep30-21swapper/322:18:593
26135211017,76sleep126142-21sh23:04:171
16012211028,11sleep00-21swapper/022:31:410
2634921069,84sleep026393-21ntpq19:39:230
1793021068,17sleep00-21swapper/000:19:100
6231210570,18sleep00-21swapper/022:00:080
30156210522,67sleep230297-21awk23:14:442
14055210512,80sleep314105-21apt-key00:09:003
1562121046,86sleep00-21swapper/000:11:190
1562121046,86sleep00-21swapper/000:11:190
7224210320,11sleep00-21swapper/020:21:030
21556210317,62sleep221635-21mailstats00:29:302
18881210317,70sleep118934-21cpuspeed20:59:151
8809210220,10sleep00-21swapper/020:26:320
1727721029,18sleep00-21swapper/020:54:100
87921019,48sleep032377-21/usr/sbin/munin19:59:550
3161821018,54sleep031746-21timerwakeupswit23:19:300
15349210021,54sleep115442-21latency_hist20:49:021
1087521005,84sleep00-21swapper/023:58:430
440629817,66sleep24556-21latency_hist20:14:022
3107529826,52sleep131165-21fschecks_time23:19:191
2903329710,74sleep129114-21switchtime21:29:441
2322029719,59sleep10-21swapper/100:34:301
277129611,42sleep02840-21switchtime21:49:300
234692965,70sleep10-21swapper/122:54:321
1805229612,67sleep118054-21grep22:39:131
595629517,60sleep16070-21latency_hist20:19:021
2132029523,53sleep321384-21cat00:29:233
165029517,63sleep31775-21apt-key23:29:023
284752949,70sleep20-21swapper/221:29:252
585329316,62sleep15925-21/usr/sbin/munin22:00:001
1563629329,40sleep115671-21munin-plugin-st19:09:001
2707329214,20sleep30-21swapper/321:24:253
2675629211,44sleep30-21swapper/319:39:313
2654629216,57sleep126624-21threads19:39:281
2606429213,40sleep20-21swapper/223:04:142
218629222,49sleep22317-21grep20:04:292
215302927,47sleep00-21swapper/019:24:250
112232927,13sleep00-21swapper/023:59:120
2509529111,65sleep225096-21sshd21:19:072
711928913,56sleep27138-21apt-get22:04:092
2226428920,48sleep222301-21grep21:09:222
1574728922,43sleep215739-21apt-get20:49:112
3229128720,49sleep232328-21latency_hist23:24:002
2446428728,27sleep30-21swapper/319:34:223
3197328615,34sleep30-21swapper/321:39:233
3184828628,38sleep231867-21apt-get19:59:002
3074728625,34sleep30-21swapper/323:19:013
2501028622,50sleep225138-21timerwakeupswit19:34:332
472428520,23sleep20-21swapper/221:54:532
2480628325,39sleep224834-21modprobe21:18:592
1500728217,21sleep30-21swapper/322:29:183
278662805,42sleep00-21swapper/021:28:090
1853328016,30sleep20-21swapper/222:39:272
1643728023,31sleep10-21swapper/119:09:321
1140828014,50sleep211496-21sed20:34:292
2121727913,49sleep221226-21df_inode22:49:162
1985727912,32sleep30-21swapper/322:44:203
190342795,42sleep00-21swapper/019:17:100
1173727924,36sleep111830-21grep23:59:331
84527832,27sleep2886-21idleruntime21:44:192
84527832,27sleep2886-21idleruntime21:44:192
63242787,56sleep06478-21latency_hist23:44:010
2246027832,25sleep10-21swapper/121:09:291
670027717,42sleep26795-21cut20:19:272
2230327712,49sleep322362-21memory21:09:233
1997827724,36sleep10-21swapper/100:24:281
56052765,60sleep00-21swapper/020:15:380
2880527629,21sleep10-21swapper/119:49:011
2354027615,45sleep2363-21snmpd22:54:332
1791127624,29sleep10-21swapper/120:54:311
1771427630,26sleep30-21swapper/300:19:013
127027622,30sleep30-21swapper/320:04:023
111832765,41sleep00-21swapper/022:14:580
16872755,59sleep00-21swapper/021:45:500
16872755,59sleep00-21swapper/021:45:500
1450627516,29sleep00-21swapper/019:04:230
1219827517,35sleep10-21swapper/122:19:271
1945427422,35sleep319528-21cat20:59:303
1571927421,33sleep115739-21apt-get20:49:101
1468427419,28sleep30-21swapper/322:29:023
2104427218,30sleep30-21swapper/321:04:303
2104427218,30sleep30-21swapper/321:04:303
1499627220,36sleep215135-21sort19:04:332
76127115,40sleep1844-21fschecks_time21:44:181
76127115,40sleep1844-21fschecks_time21:44:181
436927130,24sleep10-21swapper/120:14:011
3153927123,24sleep30-21swapper/323:19:293
1925327123,25sleep20-21swapper/219:19:022
1583727114,40sleep215880-21gpgv00:13:592
1583727114,40sleep215880-21gpgv00:13:592
1416727112,24sleep30-21swapper/320:44:193
2494027024,27sleep10-21swapper/119:34:321
2155427017,24sleep10-21swapper/119:24:261
1618027015,42sleep2363-21snmpd22:33:532
1544527014,25sleep20-21swapper/220:49:032
1191627022,19sleep20-21swapper/223:59:382
1859626921,36sleep318669-21grep22:39:283
1705226923,23sleep30-21swapper/320:54:013
1377026912,42sleep213803-21ntp_kernel_pll_22:24:242
1305126919,35sleep113136-21cat00:04:211
1304126922,31sleep113159-21grep20:39:291
1241126921,25sleep20-21swapper/220:39:142
1234426913,42sleep312345-21sshd00:02:053
661226821,25sleep20-21swapper/223:44:162
661226821,25sleep20-21swapper/223:44:162
452426824,26sleep34580-21sendmail_mailqu21:54:503
3066926818,28sleep10-21swapper/121:34:331
3003926814,37sleep130155-21sed23:14:431
2815026816,24sleep20-21swapper/219:44:302
1727226820,22sleep20-21swapper/219:14:012
1709126824,24sleep30-21swapper/322:34:313
937726721,23sleep30-21swapper/322:09:313
2887026719,21sleep30-21swapper/323:14:003
284226722,26sleep12971-21timerwakeupswit21:49:311
2334326717,31sleep223488-21/usr/sbin/munin00:34:312
231126723,25sleep20-21swapper/221:49:192
1734926723,25sleep30-21swapper/319:14:023
1351526719,23sleep20-21swapper/200:04:312
1347126719,25sleep10-21swapper/122:24:191
2732326620,22sleep30-21swapper/323:08:593
2401726622,24sleep10-21swapper/121:14:301
1977126520,16sleep30-21swapper/300:24:223
939826420,23sleep20-21swapper/223:54:012
939826420,23sleep20-21swapper/223:54:012
3223926415,33sleep132360-21ls21:39:291
2961426417,22sleep30-21swapper/319:49:283
2471826422,22sleep10-21swapper/122:59:201
241126418,27sleep22479-21grep23:29:422
2130826420,26sleep10-21swapper/121:04:321
2130826420,26sleep10-21swapper/121:04:321
1536826421,25sleep30-21swapper/300:09:403
1536826421,25sleep30-21swapper/300:09:403
1209826425,21sleep30-21swapper/322:19:253
822726316,24sleep30-21swapper/320:24:293
2282726317,25sleep10-21swapper/119:29:151
2160826315,20sleep10-21swapper/122:49:271
2081126313,26sleep220881-21cut00:29:022
1784926326,20sleep10-21swapper/119:14:091
1098026319,19sleep10-21swapper/123:59:001
1001626311,39sleep210056-21ntp_kernel_pll_23:54:252
780026214,29sleep10-21swapper/123:48:591
780026214,29sleep10-21swapper/123:48:591
731926221,27sleep30-21swapper/322:04:193
631626216,26sleep10-21swapper/122:00:091
532026222,18sleep30-21swapper/320:14:313
2640326219,21sleep20-21swapper/219:39:252
2580226222,18sleep10-21swapper/121:19:301
2413526218,26sleep324203-21latency_hist19:34:013
2095026223,22sleep10-21swapper/119:24:021
127692628,13sleep20-21swapper/222:19:562
2980226121,17sleep30-21swapper/321:34:023
2254726116,27sleep322635-21apt-config22:54:013
1704326123,23sleep10-21swapper/100:15:161
714326010,35sleep37217-21find23:44:293
714326010,35sleep37217-21find23:44:293
3103826016,18sleep10-21swapper/119:54:261
1058526021,25sleep210587-21systemd-run20:34:012
774425916,24sleep10-21swapper/120:24:151
350625922,21sleep30-21swapper/320:09:273
1060725915,25sleep20-21swapper/222:14:232
1032525919,25sleep110353-21cut22:14:181
2202325820,17sleep20-21swapper/219:24:352
1737225816,21sleep10-21swapper/122:34:331
2781925721,16sleep20-21swapper/223:09:192
929025613,25sleep39284-21apt-get20:29:133
844825612,22sleep10-21swapper/122:09:011
3194925613,26sleep231972-21/usr/sbin/munin21:39:222
317525614,23sleep10-21swapper/123:34:011
3049725610,28sleep330583-21mailstats21:34:313
3037825619,19sleep20-21swapper/221:34:252
161282568,36sleep10-21swapper/122:33:121
1091225615,18sleep10-21swapper/120:34:161
2902825522,17sleep10-21swapper/123:14:021
2427125524,18sleep30-21swapper/322:59:023
745325416,23sleep37501-21unin-run20:24:003
595725416,20sleep30-21swapper/322:00:023
368525420,15sleep10-21swapper/123:34:211
316332548,34sleep30-21swapper/319:56:143
2982225411,22sleep10-21swapper/119:49:301
2582525414,19sleep20-21swapper/223:04:022
1999525415,18sleep20-21swapper/222:44:232
189492547,15sleep20-21swapper/200:20:382
1864025420,19sleep30-21swapper/320:59:013
164325411,16sleep30-21swapper/320:04:183
1626125412,24sleep215863-21apt-get00:14:042
1289625415,19sleep30-21swapper/320:39:273
2904025314,22sleep229037-21apt-get19:49:102
2300825319,17sleep20-21swapper/219:29:202
2381325213,23sleep323518-21/usr/sbin/munin21:14:243
2043325211,24sleep220429-21apt-get21:04:132
2043325211,24sleep220429-21apt-get21:04:132
1690625210,15sleep20-21swapper/222:34:262
59982516,34sleep10-21swapper/123:40:431
184212519,29sleep118475-21processes00:19:281
977525015,22sleep19859-21grep20:29:271
809025013,18sleep30-21swapper/323:49:133
809025013,18sleep30-21swapper/323:49:133
690625015,22sleep26970-21latency_hist22:04:012
3264925017,17sleep10-21swapper/123:24:201
2151325013,17sleep10-21swapper/100:29:291
206842506,32sleep10-21swapper/122:46:441
61782499,16sleep10-21swapper/120:19:101
2512424912,21sleep325121-21apt-get21:19:123
1937124916,18sleep219429-21mailstats20:59:292
1800824915,18sleep20-21swapper/220:54:322
150724712,17sleep10-21swapper/120:04:121
521824614,15sleep20-21swapper/220:14:302
1822324514,15sleep30-21swapper/300:19:223
2128024212,15sleep30-21swapper/319:24:163
7472407,15sleep30-21swapper/321:44:163
7472407,15sleep30-21swapper/321:44:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional