You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 14:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Mon Apr 27, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
949240197,3993sleep325926-21kworker/3:123:55:363
2378427566,743sleep02804-1kworker/0:2H21:41:000
27032274315,713sleep227126-21/usr/sbin/munin21:51:002
29729244072,348sleep329863-21unixbench_multi23:41:073
9611237616,204sleep39766-21unixbench_multi20:56:093
9099234678,252sleep39193-21mailstats22:36:073
9099234678,252sleep39193-21mailstats22:36:073
26477233926,229sleep026520-21unin-run21:50:370
1562233843,280sleep31587-21awk20:31:553
23849228215,98sleep323898-21apt-config23:25:363
9677227014,202sleep39788-21sort00:21:083
13521225712,230sleep013563-21ntpq19:26:030
15105224045,95sleep30-21swapper/321:15:403
15105224045,95sleep30-21swapper/321:15:403
4009223445,174sleep34126-21sed20:40:393
4009223445,174sleep34126-21sed20:40:393
3174223021,193sleep13233-21/usr/sbin/munin20:36:051
3174223021,193sleep13233-21/usr/sbin/munin20:36:051
26141222075,71sleep30-21swapper/320:06:073
21532221077,77sleep321587-21find23:16:083
19918220914,133sleep319992-21switchtime19:46:083
2703222087,191sleep313441-21snmpd23:33:363
11394220616,167sleep311418-21munin-plugin-st22:45:353
999622039,181sleep013441-21snmpd00:21:210
1981822037,143sleep319879-21cat21:30:383
1981822037,143sleep319879-21cat21:30:383
4624217712,16sleep20-21swapper/200:05:572
12040217611,87sleep312081-21grep19:21:093
10634217114,142sleep310750-21cat19:16:083
26470216463,85sleep326473-21cron21:50:363
20066215810,132sleep320145-21timerandwakeup23:11:073
7513215627,71sleep30-21swapper/322:31:383
15227215615,128sleep315301-21switchtime19:31:063
3216521558,49sleep30-21swapper/320:26:013
11357215319,112sleep311509-21head00:26:093
23131215221,115sleep00-21swapper/021:40:360
15300214656,28sleep315364-21mailstats22:56:053
15300214656,28sleep315364-21mailstats22:56:053
17875214417,101sleep217915-21fschecks_count19:40:542
4461213941,26sleep30-21swapper/300:05:543
28463213913,110sleep228533-21latency_hist20:15:392
7826213814,39sleep37890-21munin-node19:10:393
26968213818,101sleep00-21swapper/021:50:580
6156213342,68sleep16180-21sh19:05:361
30862213161,54sleep330986-21timerandwakeup20:21:093
13490213023,26sleep36269-21cyclictest19:26:023
4378212520,84sleep04527-21head22:21:100
12631212544,69sleep312683-21processes00:31:053
2091212335,64sleep30-21swapper/322:15:483
1779121209,92sleep30-21swapper/319:40:513
850421199,92sleep128398-21kworker/u8:420:54:141
7055211914,91sleep27140-21switchtime19:06:062
30990211816,75sleep031135-21head20:21:100
304211866,29sleep3465-21latency_hist22:10:393
28808211813,90sleep228791-21latency_hist23:40:372
12127211826,38sleep312017-21/usr/sbin/munin21:05:543
31284211740,63sleep331440-21grep23:46:083
1327621147,89sleep30-21swapper/322:50:523
6558211219,70sleep06702-21latency_hist22:30:380
6558211219,70sleep06702-21latency_hist22:30:380
25597211215,19sleep30-21swapper/323:30:053
17470211217,74sleep00-21swapper/021:21:060
17470211217,74sleep00-21swapper/021:21:060
11663211215,82sleep211709-21fschecks_count19:21:002
7999211146,46sleep08061-21sendmail_mailqu00:16:040
12272211067,22sleep312366-21threads22:46:083
28019210942,56sleep31-21systemd21:55:373
3641210710,42sleep33659-21apt-get22:20:483
19263210759,26sleep30-21swapper/323:10:393
7282210620,68sleep20-21swapper/200:15:382
20486210635,14sleep30-21swapper/321:31:023
16946210535,15sleep30-21swapper/321:20:483
16946210535,15sleep30-21swapper/321:20:483
27384210317,71sleep027430-21sed20:11:000
24092210332,13sleep30-21swapper/320:00:543
21738210314,41sleep021791-21users19:51:100
23990210215,63sleep124112-21latency_hist23:25:381
7495210030,14sleep30-21swapper/300:15:473
31989210016,20sleep00-21swapper/022:06:080
27588210012,18sleep00-21swapper/023:35:550
12873210016,63sleep012889-21latency_hist19:25:380
2086229932,13sleep30-21swapper/319:50:493
287932986,28sleep30-21swapper/321:56:053
83522976,11sleep20-21swapper/219:10:462
38102968,76sleep313441-21snmpd00:02:023
302582968,40sleep30-21swapper/322:01:023
175462966,11sleep30-21swapper/323:04:373
933129516,64sleep19379-21grep20:56:031
45662950,6sleep10-21swapper/100:05:551
36312956,14sleep10-21swapper/100:01:071
2105029522,49sleep121044-21apt-get23:15:531
1486029525,46sleep014891-21apt-get21:15:360
1486029525,46sleep014891-21apt-get21:15:360
1366729523,56sleep30-21swapper/321:10:553
1366729523,56sleep30-21swapper/321:10:553
1037929512,68sleep110497-21latency_hist00:25:381
103629520,57sleep11165-21latency_hist23:55:381
1406029413,68sleep214114-21processes21:11:062
1406029413,68sleep214114-21processes21:11:062
304929310,68sleep00-21swapper/022:16:250
220182939,63sleep30-21swapper/321:36:153
1438029316,60sleep214419-21modprobe19:30:372
120562925,12sleep10-21swapper/122:46:021
611029110,69sleep313441-21snmpd19:05:023
2571529017,55sleep225805-21grep21:46:072
2886728913,61sleep128926-21cat20:15:581
136128960,15sleep30-21swapper/323:55:573
37128812,60sleep0524-21awk23:51:090
2965328813,58sleep029684-21latency_hist22:00:390
2362728720,45sleep223150-21apt-get21:40:462
2133928718,53sleep221346-21/usr/sbin/munin19:51:042
1674328713,58sleep30-21swapper/319:36:073
65122869,62sleep36556-21apt-key22:30:373
65122869,62sleep36556-21apt-key22:30:373
298852865,28sleep30-21swapper/320:19:393
288402869,17sleep30-21swapper/323:40:383
1088628614,55sleep210945-21/usr/sbin/munin00:26:002
73522859,15sleep30-21swapper/320:50:523
437928515,52sleep04423-21/usr/sbin/munin00:05:510
2343928513,56sleep30-21swapper/321:40:393
120328512,14sleep30-21swapper/322:11:073
812628416,55sleep28217-21sed00:16:062
291428420,44sleep02930-21sort00:00:540
2376828411,16sleep30-21swapper/320:00:383
84828326,39sleep1611-21apt-get20:30:401
2620028313,28sleep00-21swapper/023:30:580
81072827,12sleep30-21swapper/319:10:433
1955028222,40sleep019627-21fschecks_time23:10:560
1477628212,60sleep00-21swapper/022:55:420
1477628212,60sleep00-21swapper/022:55:420
2751728151,17sleep327563-21ntpq20:11:033
60632806,59sleep36077-21kernelversion20:46:003
60632806,59sleep36077-21kernelversion20:46:003
2718728025,35sleep127200-21modprobe23:35:361
1243328020,40sleep112452-21sed00:30:591
1216928018,38sleep112256-21find19:21:111
433227936,25sleep30-21swapper/320:40:553
1542027913,15sleep00-21swapper/021:15:530
1542027913,15sleep00-21swapper/021:15:530
910327815,25sleep00-21swapper/020:55:580
61352786,61sleep36190-21cat00:10:593
859227715,45sleep08586-21apt-get22:35:510
859227715,45sleep08586-21apt-get22:35:510
342727719,26sleep10-21swapper/122:20:381
2989027712,50sleep00-21swapper/022:00:510
2033827713,49sleep120340-21sshd19:47:151
1969027714,48sleep219728-21irqstats23:10:582
1022427725,29sleep00-21swapper/022:40:560
296852768,52sleep129835-21latency_hist22:00:401
287502759,14sleep20-21swapper/221:56:042
1607527518,42sleep116079-21modprobe23:00:351
1607527518,42sleep116079-21modprobe23:00:351
780827312,43sleep00-21swapper/020:51:060
2340927317,18sleep10-21swapper/119:56:361
1809927316,36sleep217892-21/usr/sbin/munin23:05:572
135652737,14sleep10-21swapper/119:26:041
109152738,48sleep30-21swapper/321:01:023
1696727212,44sleep117046-21find23:01:081
310727117,38sleep23152-21ntp_states20:36:022
310727117,38sleep23152-21ntp_states20:36:022
3074927113,41sleep030813-21idleruntime23:45:560
3052027121,25sleep20-21swapper/222:01:092
2411427112,29sleep20-21swapper/223:25:392
1243727119,28sleep10-21swapper/121:06:011
1144727110,47sleep111545-21cat22:45:371
602427013,39sleep213441-21snmpd22:26:082
602427013,39sleep213441-21snmpd22:26:082
2821027015,34sleep127482-21/usr/sbin/munin23:36:081
1755427018,25sleep20-21swapper/219:40:382
3152726916,28sleep20-21swapper/222:05:582
3087226924,23sleep10-21swapper/122:02:051
2652126922,20sleep10-21swapper/121:50:381
179926921,30sleep01830-21munin-run22:15:360
1543626919,26sleep10-21swapper/119:31:081
146726912,41sleep21561-21memory20:31:542
1230826924,24sleep20-21swapper/221:05:582
861026822,21sleep20-21swapper/220:55:362
647326819,32sleep06476-21apt-get19:05:480
527726725,22sleep20-21swapper/219:01:102
302282676,49sleep00-21swapper/023:44:310
259026720,19sleep00-21swapper/000:00:380
1084926710,12sleep10-21swapper/121:01:001
2583626615,30sleep025931-21/usr/sbin/munin20:06:000
1012226619,28sleep010229-21grep19:15:570
700726517,25sleep10-21swapper/122:30:521
2411226517,31sleep124217-21fschecks_time20:00:561
2326126521,26sleep20-21swapper/221:40:382
1704926516,33sleep20-21swapper/221:20:562
1704926516,33sleep20-21swapper/221:20:562
1687526515,32sleep016966-21cut23:01:070
71022648,38sleep07115-21latency_hist20:50:370
71022648,38sleep07115-21latency_hist20:50:370
651026415,35sleep06589-21sort00:11:080
2566126419,21sleep10-21swapper/120:05:571
2177926415,20sleep20-21swapper/223:16:112
1386926415,33sleep113997-21sed22:51:071
1208626416,30sleep20-21swapper/222:46:032
3171126316,22sleep10-21swapper/122:06:011
2878826317,23sleep00-21swapper/020:15:570
2267926317,21sleep00-21swapper/023:20:570
219426316,32sleep22200-21diskstats22:15:552
193112635,11sleep00-21swapper/019:45:450
188012635,11sleep10-21swapper/119:42:231
1800826315,21sleep10-21swapper/123:05:561
1236926317,22sleep00-21swapper/022:46:090
100542636,11sleep20-21swapper/200:22:082
95342628,11sleep20-21swapper/220:56:082
686526214,21sleep20-21swapper/220:48:052
686526214,21sleep20-21swapper/220:48:052
669026215,28sleep16748-21/usr/sbin/munin19:05:571
2945126214,19sleep10-21swapper/123:41:011
1216526214,31sleep012250-21fw_packets21:05:550
700626114,27sleep06496-21apt-get22:30:510
518026112,34sleep05184-21/usr/sbin/munin19:01:080
3225526118,24sleep232275-21/usr/sbin/munin20:26:042
238426124,22sleep10-21swapper/120:35:361
2254626113,23sleep00-21swapper/019:55:540
1673626112,21sleep10-21swapper/121:20:381
1673626112,21sleep10-21swapper/121:20:381
1633226125,20sleep10-21swapper/119:35:571
1323126112,20sleep20-21swapper/222:50:512
1109426112,33sleep011160-21/usr/sbin/munin00:26:060
942626013,21sleep20-21swapper/222:36:102
942626013,21sleep20-21swapper/222:36:102
823626011,32sleep18351-21fschecks_time19:10:451
49172606,11sleep10-21swapper/122:25:311
2733526018,25sleep127423-21sed21:51:071
161282605,12sleep00-21swapper/019:35:400
1378626015,30sleep013867-21cat22:51:060
134072605,46sleep20-21swapper/200:35:242
3179425917,25sleep031789-21apt-get20:25:500
2687625917,28sleep226946-21latency_hist20:10:382
253842595,12sleep20-21swapper/223:27:212
2493125917,18sleep00-21swapper/023:27:030
2240125917,25sleep10-21swapper/121:36:371
1912625915,17sleep20-21swapper/221:26:082
1912625915,17sleep20-21swapper/221:26:082
1308125917,22sleep00-21swapper/000:31:140
109525916,23sleep00-21swapper/020:30:470
659025817,23sleep10-21swapper/100:11:091
2071425817,19sleep00-21swapper/021:31:070
1800025816,26sleep00-21swapper/023:05:550
1034825815,19sleep20-21swapper/222:40:582
579825716,24sleep15865-21df20:45:541
579825716,24sleep15865-21df20:45:541
482325715,18sleep20-21swapper/220:41:062
2434625718,24sleep123865-21apt-get23:25:471
844025612,26sleep08434-21apt-get19:10:520
555525616,19sleep10-21swapper/120:45:381
521425618,21sleep10-21swapper/122:25:511
521425618,21sleep10-21swapper/122:25:511
300022566,36sleep20-21swapper/223:41:382
219972566,13sleep10-21swapper/119:53:411
210642565,11sleep20-21swapper/221:32:222
1905025616,18sleep10-21swapper/121:26:071
1905025616,18sleep10-21swapper/121:26:071
149402566,13sleep00-21swapper/019:30:590
1489725620,23sleep114952-21systemd-run21:15:371
1489725620,23sleep114952-21systemd-run21:15:371
52125513,25sleep1516-21apt-get22:10:491
2682625516,21sleep20-21swapper/223:31:092
2406625514,20sleep20-21swapper/220:00:532
168025518,19sleep20-21swapper/223:56:042
1197225515,16sleep10-21swapper/100:30:391
110525517,17sleep20-21swapper/222:11:062
1002325514,24sleep210019-21apt-get19:15:512
751925412,25sleep17514-21apt-get00:15:491
332725417,19sleep20-21swapper/200:01:042
1869625412,26sleep00-21swapper/021:25:580
1869625412,26sleep00-21swapper/021:25:580
759425313,24sleep27664-21sensors_temp22:31:422
388625313,22sleep23935-21irqstats22:20:592
2401625311,24sleep024010-21apt-get20:00:510
1963025318,20sleep10-21swapper/123:10:571
178622538,18sleep00-21swapper/019:40:520
2365725213,22sleep123650-21apt-get21:40:491
2113025210,17sleep00-21swapper/023:15:560
2005925214,22sleep120126-21df21:30:521
127425211,26sleep11271-21apt-get23:55:511
303122518,28sleep130325-21df_inode20:20:551
2372825110,16sleep20-21swapper/220:00:372
1480025113,23sleep214802-21apt-get22:55:482
1480025113,23sleep214802-21apt-get22:55:482
1056725111,21sleep00-21swapper/021:00:550
2870025013,22sleep228696-21apt-get20:15:512
1722825011,14sleep10-21swapper/121:20:591
1722825011,14sleep10-21swapper/121:20:591
1636925011,23sleep216436-21df23:00:532
1592425012,23sleep215930-21ntp_states21:16:442
1592425012,23sleep215930-21ntp_states21:16:442
3236024911,22sleep232417-21irqstats23:50:592
2711624916,19sleep10-21swapper/120:10:531
1515424911,20sleep20-21swapper/219:31:052
1087824917,18sleep20-21swapper/221:01:012
1023124912,20sleep10-21swapper/119:15:581
3190824714,19sleep10-21swapper/123:50:381
89092469,20sleep18959-21memory22:36:011
89092469,20sleep18959-21memory22:36:011
318312455,31sleep20-21swapper/223:50:142
306652436,14sleep20-21swapper/220:21:042
26752439,16sleep00-21swapper/020:35:470
26752439,16sleep00-21swapper/020:35:470
42342418,15sleep00-21swapper/020:40:480
195512387,14sleep20-21swapper/219:45:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional