You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 11:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Fri Jun 26, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
55822660311,6572sleep329544-21kworker/3:223:59:513
18592215676,1545sleep27486-1kworker/2:1H19:35:092
18592215676,1545sleep27486-1kworker/2:1H19:35:092
4734269019,654sleep14779-21/usr/sbin/munin23:55:211
2782225436,467sleep2195-21jbd2/mmcblk2p2-21:45:082
5995239037,338sleep35999-21df_inode00:00:163
10121238340,325sleep110162-21/usr/sbin/munin19:10:121
6435231720,277sleep16439-21cron22:19:511
48812288115,158sleep24897-21idleruntime-cro22:14:522
13416226070,21sleep20-21swapper/219:19:552
2648922579,235sleep126498-21ntpq21:40:181
1264222337,213sleep212705-21mailstats00:20:222
24516223214,196sleep224541-21munin-run19:54:522
19805222564,22sleep20-21swapper/221:20:042
9032220414,175sleep19116-21expr20:45:391
167522049,58sleep21869-21sed22:04:542
167522049,58sleep21869-21sed22:04:542
2048219413,165sleep01559-21apt-get22:05:040
14992219411,165sleep215027-21proc_pri22:45:202
2239721909,165sleep222496-21fschecks_time23:10:122
32365218916,158sleep232399-21wc23:40:202
32365218916,158sleep232399-21wc23:40:202
28910218812,164sleep2360-21ntpd23:30:152
28910218812,164sleep2360-21ntpd23:30:152
16508218518,148sleep216524-21modprobe19:29:522
6989218147,120sleep27032-21ntp_kernel_pll_20:40:142
189021810,33sleep20-21swapper/223:45:242
189021810,33sleep20-21swapper/223:45:242
26984218018,144sleep326979-21apt-get23:25:073
591721769,155sleep27486-1kworker/2:1H00:00:112
1374221756,121sleep213780-21ntpq21:00:342
1374221756,121sleep213780-21ntpq21:00:342
21419217122,128sleep321429-21modprobe19:44:513
27792216727,119sleep30-21swapper/320:04:543
15524216723,51sleep20-21swapper/200:30:152
15385216616,135sleep215461-21sendmail_mailtr21:05:222
15385216616,135sleep215461-21sendmail_mailtr21:05:222
3033521617,67sleep27486-1kworker/2:1H23:35:152
3033521617,67sleep27486-1kworker/2:1H23:35:152
12664216068,18sleep20-21swapper/222:39:512
12664216068,18sleep20-21swapper/222:39:512
10433216014,42sleep20-21swapper/222:30:222
10433216014,42sleep20-21swapper/222:30:222
899621589,134sleep20-21swapper/200:10:042
12113215716,117sleep20-21swapper/220:55:212
12113215716,117sleep20-21swapper/220:55:212
1701921529,68sleep217116-21cat21:10:232
1701921529,68sleep217116-21cat21:10:232
22169215114,114sleep20-21swapper/219:45:192
17167215114,123sleep117203-21ntpq19:30:161
10422214815,75sleep20-21swapper/220:50:202
203221478,120sleep20-21swapper/220:25:122
27291214511,54sleep227319-21idleruntime-cro21:44:512
23902214264,36sleep223825-21/usr/sbin/munin23:15:082
19867214110,116sleep219978-21ls23:00:222
28387213727,98sleep228440-21basename20:05:462
24964213544,17sleep20-21swapper/219:55:122
24964213544,17sleep20-21swapper/219:55:122
15011213520,98sleep115022-21ls19:24:541
9612213412,44sleep20-21swapper/222:29:542
9693213218,97sleep39827-21latency_hist19:09:543
11259213112,20sleep20-21swapper/219:14:542
950721286,53sleep20-21swapper/220:47:542
2730721277,51sleep20-21swapper/220:00:372
9560212687,22sleep00-21swapper/022:29:520
3497212529,84sleep23487-21apt-get20:30:092
10038212218,85sleep010119-21grep19:10:110
636821219,100sleep26383-21latency_hist20:39:542
24222212137,19sleep20-21swapper/221:34:542
21073212025,77sleep10-21swapper/121:24:511
3299211715,91sleep20-21swapper/222:09:262
23308211718,81sleep123345-21sed21:30:171
14157211712,17sleep20-21swapper/200:25:182
14157211712,17sleep20-21swapper/200:25:182
31841211624,71sleep131241-21/usr/sbin/munin20:15:221
31841211624,71sleep131241-21/usr/sbin/munin20:15:221
13287211624,73sleep113414-21cmp19:19:541
9623211525,66sleep19652-21munin-run19:09:521
11911211521,23sleep10-21swapper/100:19:521
16430211429,66sleep20-21swapper/222:50:172
24929211210,89sleep224958-21ps21:35:222
7468211126,71sleep29740-21cyclictest00:05:072
32257211110,83sleep232341-21apt-config21:59:532
25715211125,67sleep325742-21basename21:39:513
21471211112,83sleep221599-21ls19:44:542
517321109,51sleep25218-21grep22:15:082
31103211021,62sleep331224-21grep23:35:333
31103211021,62sleep331224-21grep23:35:333
1916821106,37sleep20-21swapper/221:18:432
15254211012,80sleep315298-21/usr/sbin/munin00:30:083
9275210914,27sleep39328-21users20:45:413
17554210923,11sleep20-21swapper/200:35:242
21634210810,86sleep221658-21kernelversion21:25:172
1333121077,46sleep10-21swapper/121:00:121
1333121077,46sleep10-21swapper/121:00:121
28937210623,54sleep129076-21sed20:05:571
19504210614,76sleep119679-21latency_hist21:19:551
547221058,30sleep20-21swapper/223:58:252
31268210512,15sleep20-21swapper/220:15:072
31268210512,15sleep20-21swapper/220:15:072
3074321046,30sleep20-21swapper/220:11:522
25434210420,14sleep20-21swapper/223:20:052
25310210219,67sleep325401-21mailstats19:55:213
25310210219,67sleep325401-21mailstats19:55:213
19773210214,69sleep119864-21switchtime23:00:211
13286210212,75sleep312784-21apt-get20:59:583
13286210212,75sleep312784-21apt-get20:59:583
7432210110,78sleep16733-21/usr/sbin/munin22:20:241
23494210113,28sleep00-21swapper/019:50:130
22011210164,22sleep20-21swapper/223:09:512
1761210117,64sleep1968-21/usr/sbin/munin23:45:241
1761210117,64sleep1968-21/usr/sbin/munin23:45:241
1502421018,20sleep215171-21date19:24:552
15736210012,72sleep015798-21/usr/sbin/munin00:30:210
1836729963,22sleep218509-21grep22:55:262
168712997,15sleep20-21swapper/219:30:082
2983929616,64sleep029925-21gpgv23:34:530
2983929616,64sleep029925-21gpgv23:34:530
2168029515,65sleep121687-21basename19:45:011
3090729318,56sleep130981-21cat23:35:311
3090729318,56sleep130981-21cat23:35:311
1699129311,57sleep10-21swapper/122:50:281
131922936,13sleep20-21swapper/222:40:112
106552937,14sleep20-21swapper/200:15:132
16552926,74sleep21701-21systemctl20:24:522
2145229124,44sleep321492-21fschecks_count21:25:143
197782888,24sleep00-21swapper/021:19:580
1134328827,34sleep311351-21munin-node19:14:553
563928716,58sleep35692-21processes22:15:213
419128711,64sleep14313-21cat20:30:491
1867228718,46sleep318819-21grep21:15:243
154528621,49sleep01568-21munin-run22:04:520
154528621,49sleep01568-21munin-run22:04:520
1027928619,36sleep10-21swapper/100:14:521
787228520,44sleep37904-21idleruntime-cro20:44:523
600628417,47sleep16072-21fschecks_time00:00:181
2099228416,54sleep121026-21irqstats23:05:141
94828317,50sleep31017-21switchtime20:20:213
877228314,48sleep38810-21/usr/sbin/munin22:25:233
675928322,46sleep20-21swapper/222:20:092
2344328311,59sleep0364-21snmpd23:13:210
2431228211,57sleep124326-21awk23:15:201
1666128211,55sleep116710-21irqstats21:10:141
1666128211,55sleep116710-21irqstats21:10:141
836828115,43sleep08339-21/usr/sbin/munin19:05:070
35652815,43sleep20-21swapper/223:51:552
1882428119,28sleep00-21swapper/021:15:260
1286128117,43sleep312791-21latency_hist20:59:543
1286128117,43sleep312791-21latency_hist20:59:543
736828020,40sleep17508-21/usr/sbin/munin20:40:221
305442806,11sleep20-21swapper/221:53:482
2543528020,36sleep325430-21apt-get23:20:063
1969328022,30sleep00-21swapper/023:00:200
815127920,40sleep38299-21unixbench_multi00:05:253
297192798,56sleep10-21swapper/120:10:051
282092797,41sleep20-21swapper/223:29:262
2540427916,46sleep024780-21/usr/sbin/munin19:55:220
2540427916,46sleep024780-21/usr/sbin/munin19:55:220
370827819,36sleep10-21swapper/122:10:101
3035127814,53sleep00-21swapper/021:51:190
2296627823,32sleep30-21swapper/319:49:523
3203827716,31sleep30-21swapper/320:15:243
3203827716,31sleep30-21swapper/320:15:243
3027927727,28sleep30-21swapper/321:50:343
1585027718,43sleep316000-21latency_hist22:49:543
1369427719,38sleep313160-21apt-get19:20:043
1147927711,51sleep011488-21df_inode22:35:090
1147927711,51sleep011488-21df_inode22:35:090
2300127616,22sleep30-21swapper/321:30:103
2249727614,46sleep322549-21if_eth023:10:133
382727524,29sleep00-21swapper/023:54:520
3062827417,35sleep130636-21modprobe21:54:521
2764727422,37sleep11-21systemd20:04:521
2133627412,47sleep021353-21apt21:25:030
1518827426,25sleep10-21swapper/122:45:231
1278927424,22sleep00-21swapper/000:20:240
2828327314,44sleep028293-21unin-run23:29:520
2275727311,47sleep122763-21ntp_states23:10:191
1398827319,25sleep00-21swapper/021:00:400
1398827319,25sleep00-21swapper/021:00:400
1150427312,46sleep111505-21sshd20:55:031
1150427312,46sleep111505-21sshd20:55:031
328127216,40sleep03216-21latency_hist20:29:550
2814227218,39sleep327665-21apt-get20:05:023
2732227221,30sleep327361-21unin-run21:44:523
2387027224,24sleep10-21swapper/119:50:221
1887927218,32sleep318895-21modprobe22:59:513
1613427218,40sleep316153-21df_inode22:50:103
3189927115,32sleep10-21swapper/123:40:071
3189927115,32sleep10-21swapper/123:40:071
2889727012,17sleep10-21swapper/123:30:141
2889727012,17sleep10-21swapper/123:30:141
1859027012,35sleep118559-21/usr/sbin/munin19:35:081
1859027012,35sleep118559-21/usr/sbin/munin19:35:081
97926926,21sleep30-21swapper/322:01:283
97926926,21sleep30-21swapper/322:01:283
801326919,34sleep08022-21modprobe22:24:510
480326919,36sleep34934-21latency_hist20:34:543
3149326913,42sleep331566-21sed21:55:263
2748526912,43sleep027487-21sshd20:02:490
1818926916,32sleep30-21swapper/319:34:543
1325026916,34sleep313259-21cut22:40:123
2795026818,28sleep10-21swapper/121:45:191
2489226819,24sleep10-21swapper/119:55:111
2489226819,24sleep10-21swapper/119:55:111
1992526823,21sleep30-21swapper/319:39:553
1992526823,21sleep30-21swapper/319:39:553
187026816,28sleep30-21swapper/322:04:553
1690526823,21sleep00-21swapper/021:10:210
1690526823,21sleep00-21swapper/021:10:210
166332680,8sleep10-21swapper/119:29:541
1032726819,23sleep10-21swapper/120:50:151
231426718,26sleep30-21swapper/320:25:173
1805326720,31sleep118073-21idleruntime-cro00:39:511
115312679,42sleep311616-21fw_packets22:35:123
115312679,42sleep311616-21fw_packets22:35:123
358726615,31sleep03623-21/usr/sbin/munin20:30:150
252926623,23sleep10-21swapper/120:25:231
2411326612,36sleep324159-21users19:50:243
1977626612,39sleep319276-21apt-get21:19:573
172826618,25sleep00-21swapper/020:24:540
1487426622,23sleep10-21swapper/121:05:091
1487426622,23sleep10-21swapper/121:05:091
1424026612,38sleep114242-21cron22:44:511
881826520,18sleep20-21swapper/219:05:172
551226524,20sleep10-21swapper/120:35:211
3032726518,23sleep00-21swapper/023:35:090
3032726518,23sleep00-21swapper/023:35:090
2487226519,20sleep30-21swapper/323:16:513
21572656,15sleep00-21swapper/023:48:420
21572656,15sleep00-21swapper/023:48:420
1762626524,23sleep10-21swapper/122:55:081
1737526521,25sleep00-21swapper/000:35:220
1314126516,34sleep313166-21munin-run19:19:523
3249526420,20sleep00-21swapper/021:59:550
2850826418,23sleep30-21swapper/321:45:333
1416426418,31sleep314187-21/usr/sbin/munin00:25:193
1416426418,31sleep314187-21/usr/sbin/munin00:25:193
1198126417,30sleep112056-21grep22:35:221
1198126417,30sleep112056-21grep22:35:221
1116126411,36sleep111194-21latency_hist22:34:531
1116126411,36sleep111194-21latency_hist22:34:531
319126321,19sleep30-21swapper/323:50:233
2512626316,18sleep10-21swapper/121:35:251
2034726311,36sleep020405-21/usr/sbin/munin19:40:120
140842636,46sleep00-21swapper/022:42:520
133762634,11sleep30-21swapper/300:24:233
1103426317,30sleep311040-21/usr/sbin/munin00:15:223
99126214,29sleep3987-21apt-get23:45:063
99126214,29sleep3987-21apt-get23:45:063
338326219,19sleep10-21swapper/122:09:541
3273426212,33sleep132723-21apt-get22:00:091
3273426212,33sleep132723-21apt-get22:00:091
2738126211,27sleep127424-21ntpq23:25:171
2648026221,21sleep00-21swapper/021:40:170
1838726211,35sleep018533-21cmp19:34:550
1838726211,35sleep018533-21cmp19:34:550
1769726219,28sleep317708-21/usr/sbin/munin00:35:253
1552226215,22sleep00-21swapper/019:25:140
1321026220,20sleep00-21swapper/019:19:530
1127126215,32sleep011401-21sort20:54:540
1094326217,28sleep010977-21/usr/sbin/munin00:15:190
2814426123,16sleep00-21swapper/020:05:030
2789826123,20sleep00-21swapper/021:45:100
1689726114,31sleep316903-21ntp_states21:10:203
1689726114,31sleep316903-21ntp_states21:10:203
1417426113,21sleep00-21swapper/019:21:140
86002605,43sleep00-21swapper/000:08:160
504126015,27sleep35036-21apt-get20:35:073
2446326015,17sleep00-21swapper/021:35:080
2016326017,26sleep120156-21apt-get19:40:051
722425918,18sleep00-21swapper/000:04:530
651725915,20sleep00-21swapper/022:19:540
6152596,11sleep00-21swapper/023:43:490
6152596,11sleep00-21swapper/023:43:490
48002595,11sleep00-21swapper/022:13:490
3166425918,22sleep00-21swapper/020:15:170
3166425918,22sleep00-21swapper/020:15:170
2979425917,26sleep329821-21munin-run23:34:523
2979425917,26sleep329821-21munin-run23:34:523
161692596,42sleep10-21swapper/100:30:301
3211425812,25sleep10-21swapper/121:59:041
2178625817,20sleep00-21swapper/019:45:080
1932025816,24sleep30-21swapper/323:00:113
1032225814,26sleep010357-21grep22:30:200
1032225814,26sleep010357-21grep22:30:200
566225718,19sleep00-21swapper/023:59:520
5072578,20sleep10-21swapper/120:20:111
3046425716,18sleep00-21swapper/020:10:240
2157525715,25sleep021658-21users23:05:250
1762525715,26sleep017619-21apt-get22:55:070
1032925716,20sleep30-21swapper/300:14:543
724325618,16sleep00-21swapper/020:40:210
399225618,18sleep30-21swapper/322:10:163
3017625614,24sleep330182-21ntp_states20:10:193
2290825615,23sleep022904-21apt-get21:30:030
124592565,39sleep00-21swapper/020:55:550
124592565,39sleep00-21swapper/020:55:550
971625517,16sleep00-21swapper/020:49:530
768525514,22sleep17738-21irqstats00:05:151
1409325515,18sleep00-21swapper/000:25:160
1409325515,18sleep00-21swapper/000:25:160
942325418,18sleep00-21swapper/000:10:180
3240025414,23sleep332474-21/usr/sbin/munin23:40:213
3240025414,23sleep332474-21/usr/sbin/munin23:40:213
2382425420,19sleep00-21swapper/023:15:020
1656625419,18sleep00-21swapper/022:50:230
1524925412,25sleep315245-21apt-get19:25:053
986125215,21sleep39852-21apt-get22:30:073
986125215,21sleep39852-21apt-get22:30:073
2157625214,24sleep121633-21awk21:25:161
1479925217,17sleep00-21swapper/022:45:140
1460525218,18sleep10-21swapper/100:25:261
1460525218,18sleep10-21swapper/100:25:261
512825114,19sleep00-21swapper/020:35:120
1680025113,19sleep00-21swapper/019:30:060
1555825116,18sleep00-21swapper/021:05:240
1555825116,18sleep00-21swapper/021:05:240
71552505,12sleep30-21swapper/322:20:183
1613725014,19sleep30-21swapper/321:09:513
1613725014,19sleep30-21swapper/321:09:513
2590324811,17sleep00-21swapper/023:20:190
2075824812,18sleep30-21swapper/323:05:083
2596124712,14sleep10-21swapper/123:20:201
121602469,19sleep112227-21df00:20:091
1003924611,20sleep310043-21diskstats20:50:093
890324513,15sleep10-21swapper/122:25:251
26172398,14sleep10-21swapper/123:50:081
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional