You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-31 - 10:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Oct 31, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7667211506,1129sleep221576-1kworker/2:0H21:20:162
7667211506,1129sleep221576-1kworker/2:0H21:20:162
6872297030,928sleep221576-1kworker/2:0H23:00:162
6872297030,928sleep221576-1kworker/2:0H23:00:162
2901526896,644sleep229065-21ntpq20:45:232
1576026547,634sleep215796-21ntpq21:45:232
1414124446,427sleep2362-21snmpd23:23:062
22089229925,255sleep222094-21modprobe20:24:562
323982291103,125sleep20-21swapper/220:55:282
323982291103,125sleep20-21swapper/220:55:282
10780228616,231sleep210871-21fschecks_time21:30:172
535522758,255sleep25400-21awk22:55:202
11224225816,220sleep111350-21latency_hist19:49:591
1516422569,109sleep215082-21kworker/u8:221:44:592
31746224052,171sleep231797-21sort00:20:152
8604223646,142sleep28619-21kernelversion23:05:212
8604223646,142sleep28619-21kernelversion23:05:212
11731222532,96sleep20-21swapper/219:50:202
15903221731,124sleep20-21swapper/220:04:582
25718221016,178sleep225865-21awk22:15:312
11934221013,183sleep111942-21awk23:15:251
22751220957,136sleep222796-21ntp_kernel_err20:25:252
12188220713,178sleep312342-21unixbench_singl19:50:293
5558220336,124sleep25567-21latency_hist21:14:592
1570220018,165sleep31589-21idleruntime-cro22:44:553
1570220018,165sleep31589-21idleruntime-cro22:44:553
30209219559,23sleep20-21swapper/222:30:292
7346217918,30sleep10-21swapper/119:35:291
97121718,14sleep00-21swapper/000:25:170
30559216836,116sleep230576-21munin-node19:09:592
11942216670,76sleep011968-21idleruntime-cro21:34:550
19732216542,36sleep20-21swapper/223:40:262
19732216542,36sleep20-21swapper/223:40:262
23608216211,137sleep223652-21awk23:54:552
18169216013,130sleep218208-21awk21:54:562
13123215912,131sleep213211-21fschecks_time19:55:172
7106215732,105sleep26549-21/usr/sbin/munin19:35:262
2150215610,134sleep22193-21/usr/sbin/munin00:29:562
11000215434,56sleep20-21swapper/223:13:002
4947215124,108sleep35078-21grep21:10:273
2587921508,130sleep225922-21tr20:35:222
4670214714,82sleep24728-21modprobe22:54:552
969121457,85sleep29731-21apt-key19:44:592
13027214519,108sleep00-21swapper/023:20:130
1151821455,12sleep00-21swapper/019:50:140
61621449,83sleep2648-21latency19:15:192
61621449,83sleep2648-21latency19:15:192
3448214339,46sleep20-21swapper/219:25:132
3448214339,46sleep20-21swapper/219:25:132
3085721359,77sleep20-21swapper/200:15:302
17672213512,24sleep20-21swapper/220:10:102
1745321358,81sleep20-21swapper/223:34:572
28929213240,69sleep11-21systemd19:04:571
28990213134,82sleep229068-21latency_hist19:04:582
28881212913,97sleep328916-21latency20:45:203
17699212914,98sleep317767-21df20:10:153
2199221285,50sleep20-21swapper/222:05:212
18319212514,97sleep118443-21timerandwakeup23:35:281
1955521239,64sleep21-21systemd21:57:262
558212213,18sleep10-21swapper/119:15:181
558212213,18sleep10-21swapper/119:15:181
25760212216,46sleep225774-21kernelversion00:00:202
25760212216,46sleep225774-21kernelversion00:00:202
21303212017,81sleep121328-21awk22:04:561
13522211721,81sleep013554-21idleruntime-cro21:39:560
3388211523,62sleep13407-21apt-get22:50:101
14266211510,20sleep014267-21sshd19:59:260
3892211422,71sleep33956-21find22:50:283
23775211442,53sleep20-21swapper/222:10:382
6650211115,78sleep36658-21/usr/sbin/munin19:35:153
174021097,88sleep21846-21cat21:00:542
26789210825,61sleep20-21swapper/222:20:192
31215210751,13sleep20-21swapper/219:10:152
31215210751,13sleep20-21swapper/219:10:152
20552210713,79sleep120557-21systemd-run23:44:571
20552210713,79sleep120557-21systemd-run23:44:571
1277210718,71sleep31341-21/usr/sbin/munin21:00:423
3154210614,76sleep13298-21unixbench_singl00:30:311
14656210634,34sleep214682-21fschecks_count23:25:162
28667210546,19sleep20-21swapper/222:25:282
27524210424,67sleep127564-21ps20:40:261
16632210414,74sleep016716-21mailstats20:05:280
28918210336,45sleep028927-21kthreadd19:04:560
2838921039,36sleep20-21swapper/220:44:592
31213210130,35sleep00-21swapper/019:10:130
31213210130,35sleep00-21swapper/019:10:130
30301210116,70sleep330424-21timerwakeupswit22:30:303
537221009,78sleep25413-21ntpq19:30:212
2108210013,70sleep12166-21irqstats19:20:191
2108210013,70sleep12166-21irqstats19:20:191
1614529935,16sleep20-21swapper/220:05:142
508029811,12sleep00-21swapper/019:30:150
498829818,61sleep15032-21cpu22:55:111
301752977,46sleep20-21swapper/220:50:162
2001929735,18sleep20-21swapper/222:00:092
817329620,57sleep18271-21/usr/sbin/munin21:20:351
817329620,57sleep18271-21/usr/sbin/munin21:20:351
295372965,12sleep20-21swapper/200:10:462
92072958,33sleep20-21swapper/221:25:162
92072958,33sleep20-21swapper/221:25:162
2465329525,50sleep324691-21munin-plugin-st22:14:553
2140929519,51sleep320777-21/usr/sbin/munin23:45:273
2140929519,51sleep320777-21/usr/sbin/munin23:45:273
2059329520,20sleep10-21swapper/120:19:591
24412947,33sleep20-21swapper/200:30:122
187752945,12sleep20-21swapper/223:37:352
124792946,32sleep20-21swapper/221:35:182
579029333,15sleep20-21swapper/221:15:112
3206629320,56sleep20-21swapper/222:35:292
3206629320,56sleep20-21swapper/222:35:292
839229216,57sleep38423-21/usr/sbin/munin19:40:203
315629239,17sleep23204-21systemd-run19:24:562
315629239,17sleep23204-21systemd-run19:24:562
2081129223,35sleep30-21swapper/320:20:133
134062927,11sleep30-21swapper/319:55:223
273429114,22sleep20-21swapper/221:05:142
2369629114,60sleep023781-21wc23:54:570
892829017,58sleep00-21swapper/021:24:580
1869729015,62sleep118744-21if_eth021:55:191
2451428912,63sleep00-21swapper/022:13:070
1639428920,53sleep016462-21latency23:30:180
859228815,59sleep18655-21awk19:40:261
2505928814,54sleep124950-21/usr/sbin/munin22:15:181
883828617,54sleep18912-21/usr/sbin/munin23:05:281
883828617,54sleep18912-21/usr/sbin/munin23:05:281
223028613,30sleep22255-21awk22:45:232
207682865,12sleep20-21swapper/223:45:022
207682865,12sleep20-21swapper/223:45:022
3089428423,40sleep130916-21munin-run22:34:561
2995128416,21sleep30-21swapper/300:14:593
2415128418,20sleep30-21swapper/323:55:213
2415128418,20sleep30-21swapper/323:55:213
2056328417,50sleep320647-21latency_hist23:44:593
2056328417,50sleep320647-21latency_hist23:44:593
202482835,12sleep20-21swapper/220:16:162
820928219,42sleep08273-21fschecks_time19:40:170
2386828212,57sleep123874-21ntpq22:10:411
2821928111,53sleep028282-21fw_packets22:25:180
2519128119,41sleep025193-21cron20:34:550
3184628011,15sleep30-21swapper/320:55:173
3184628011,15sleep30-21swapper/320:55:173
2857928013,52sleep128664-21mailstats22:25:271
2623228031,30sleep30-21swapper/322:19:563
1660128020,31sleep30-21swapper/323:30:253
1627628024,38sleep315338-21/usr/sbin/munin21:45:323
694127912,51sleep36942-21sshd21:18:273
2363927918,41sleep123654-21apt20:29:561
2440427819,42sleep20-21swapper/223:55:282
2440427819,42sleep20-21swapper/223:55:282
225627819,25sleep30-21swapper/322:45:243
2101927816,43sleep021077-21/usr/sbin/munin20:20:220
2595727716,38sleep025997-21ps00:00:260
2595727716,38sleep025997-21ps00:00:260
2420927721,36sleep324298-21/usr/sbin/munin20:30:263
179972776,57sleep10-21swapper/121:52:451
2673227619,39sleep126788-21/usr/sbin/munin22:20:181
1500827611,47sleep10-21swapper/123:25:231
70382758,15sleep00-21swapper/023:00:390
70382758,15sleep00-21swapper/023:00:390
606527520,46sleep00-21swapper/022:57:320
3056127520,31sleep30-21swapper/300:15:233
2951527515,44sleep329558-21cat19:05:193
2604327522,22sleep30-21swapper/320:35:273
252602757,56sleep225269-21munin-run20:34:572
2135527522,27sleep30-21swapper/322:04:583
1673427516,24sleep10-21swapper/123:30:271
162227527,24sleep10-21swapper/122:44:571
162227527,24sleep10-21swapper/122:44:571
889027420,37sleep38923-21latency_hist21:24:573
889027420,37sleep38923-21latency_hist21:24:573
3121427410,48sleep10-21swapper/119:10:141
3121427410,48sleep10-21swapper/119:10:141
2652327414,26sleep30-21swapper/322:20:123
2185427413,45sleep321873-21kernelversion22:05:183
185942740,7sleep30-21swapper/323:35:293
1363827413,45sleep313751-21cat23:20:283
962627317,33sleep39709-21grep21:25:303
962627317,33sleep39709-21grep21:25:303
917027315,10sleep20-21swapper/219:42:162
635627329,21sleep00-21swapper/021:15:290
3252227314,42sleep332656-21sed22:39:583
3252227314,42sleep332656-21sed22:39:583
2217527326,28sleep10-21swapper/120:24:591
163442735,59sleep00-21swapper/021:46:260
699527215,43sleep37035-21basename23:00:383
699527215,43sleep37035-21basename23:00:383
654827214,43sleep06550-21acpi19:35:080
3214127213,23sleep00-21swapper/020:55:220
3214127213,23sleep00-21swapper/020:55:220
1582327217,27sleep10-21swapper/121:45:261
30192719,41sleep00-21swapper/021:05:200
2757627115,41sleep20-21swapper/200:05:252
1441427115,29sleep10-21swapper/121:40:291
1287327116,40sleep312997-21timerandwakeup21:35:273
718727026,21sleep10-21swapper/121:19:581
2520327016,31sleep125235-21munin-run23:59:551
2520327016,31sleep125235-21munin-run23:59:551
2449627020,23sleep00-21swapper/023:55:300
2449627020,23sleep00-21swapper/023:55:300
193632708,17sleep30-21swapper/320:15:163
1825027011,43sleep018339-21latency_hist21:54:580
116627013,30sleep10-21swapper/100:25:231
2886326912,22sleep00-21swapper/000:10:240
2167926920,32sleep021696-21fschecks_count22:05:150
549426816,22sleep30-21swapper/319:30:263
394926816,33sleep04022-21/usr/sbin/munin19:25:270
394926816,33sleep04022-21/usr/sbin/munin19:25:270
3151126819,23sleep00-21swapper/020:54:580
3077026826,21sleep10-21swapper/100:15:291
2299126822,29sleep023074-21wc20:25:310
189826812,26sleep30-21swapper/319:20:133
189826812,26sleep30-21swapper/319:20:133
1896026818,32sleep018967-21modprobe23:39:550
1245226818,10sleep20-21swapper/223:16:232
862926716,33sleep38703-21memory23:05:223
862926716,33sleep38703-21memory23:05:223
846026715,33sleep08546-21/usr/sbin/munin23:05:190
846026715,33sleep08546-21/usr/sbin/munin23:05:190
46682675,11sleep10-21swapper/119:29:271
46682675,11sleep10-21swapper/119:29:271
235726713,40sleep02376-21idleruntime-cro21:04:550
317426618,27sleep10-21swapper/122:49:581
313426624,20sleep10-21swapper/121:05:251
2558126617,26sleep30-21swapper/300:00:173
2558126617,26sleep30-21swapper/300:00:173
2271226620,29sleep122721-21/usr/sbin/munin23:50:221
1903926615,24sleep10-21swapper/120:14:581
1895126612,23sleep00-21swapper/021:55:260
1671926614,23sleep10-21swapper/120:05:291
1200726617,24sleep30-21swapper/323:15:273
675526513,37sleep16270-21apt-get23:00:071
675526513,37sleep16270-21apt-get23:00:071
1700726512,38sleep317116-21fschecks_time21:50:173
12842655,11sleep20-21swapper/222:41:192
12842655,11sleep20-21swapper/222:41:192
1073426517,20sleep30-21swapper/321:30:143
971326413,35sleep09782-21switchtime21:25:310
971326413,35sleep09782-21switchtime21:25:310
5852649,44sleep00-21swapper/000:24:450
39126412,37sleep0467-21cut22:40:160
39126412,37sleep0467-21cut22:40:160
2677126420,26sleep00-21swapper/020:39:560
2213826411,21sleep10-21swapper/123:49:581
2213826411,21sleep10-21swapper/123:49:581
29132636,10sleep00-21swapper/019:21:510
29132636,10sleep00-21swapper/019:21:510
1508926314,30sleep015160-21mailstats20:00:310
986026217,29sleep19877-21apt23:10:081
321122626,14sleep30-21swapper/300:20:243
3009426217,24sleep00-21swapper/022:30:270
287512628,12sleep30-21swapper/322:25:293
1952326220,26sleep00-21swapper/023:40:200
1952326220,26sleep00-21swapper/023:40:200
1894326213,33sleep318949-21ntp_states21:55:253
1435326116,19sleep10-21swapper/123:24:581
1147426122,20sleep00-21swapper/023:15:120
2907926022,19sleep00-21swapper/020:45:250
2898426016,27sleep129014-21/usr/sbin/munin20:45:221
279772604,15sleep30-21swapper/320:40:323
2783026015,18sleep00-21swapper/022:24:570
278226010,40sleep00-21swapper/022:45:370
2764826013,33sleep327743-21sed00:05:263
2619326021,21sleep10-21swapper/120:35:291
2228126022,16sleep10-21swapper/122:05:291
65125913,29sleep1740-21ls00:24:581
3025825920,21sleep10-21swapper/120:50:191
1928125920,23sleep019335-21sort20:15:130
1588525911,26sleep00-21swapper/023:29:580
513525812,32sleep0362-21snmpd21:10:410
3181925814,25sleep00-21swapper/022:35:270
3181925814,25sleep00-21swapper/022:35:270
2556625817,21sleep10-21swapper/100:00:151
2556625817,21sleep10-21swapper/100:00:151
13892589,37sleep00-21swapper/019:17:110
13892589,37sleep00-21swapper/019:17:110
1330725815,28sleep113345-21wc19:55:201
100652585,41sleep10-21swapper/121:25:441
100652585,41sleep10-21swapper/121:25:441
2971425717,21sleep10-21swapper/119:05:251
1624025721,22sleep30-21swapper/320:05:193
311125614,25sleep33117-21ntp_states21:05:233
2774625617,16sleep00-21swapper/000:05:270
2088725616,21sleep00-21swapper/023:45:160
2088725616,21sleep00-21swapper/023:45:160
1197225618,23sleep112019-21systemd-run21:34:561
475625517,16sleep30-21swapper/319:29:583
475625517,16sleep30-21swapper/319:29:583
475125516,18sleep00-21swapper/022:54:570
3054225512,25sleep330582-21ntpq20:50:253
2933225513,24sleep029380-21fschecks_count19:05:160
2497425512,27sleep024969-21apt-get22:15:120
2004325514,25sleep020038-21apt-get22:00:110
1432125516,19sleep00-21swapper/021:40:280
1083325511,31sleep31-21systemd19:45:453
3098425415,25sleep331095-21timerwakeupswit19:10:043
3098425415,25sleep331095-21timerwakeupswit19:10:043
2401125416,22sleep10-21swapper/120:30:201
630125317,20sleep30-21swapper/322:59:573
3247125313,23sleep132489-21munin-run22:39:561
3247125313,23sleep132489-21munin-run22:39:561
244225311,25sleep32438-21apt-get00:30:133
1355725312,24sleep113572-21munin-run21:39:571
125312538,12sleep10-21swapper/119:52:291
605925216,20sleep10-21swapper/119:32:261
444925212,23sleep14477-21irqstats21:10:171
2773325212,23sleep327742-21diskstats19:00:153
2734525213,23sleep027429-21memory20:40:210
21812529,32sleep10-21swapper/121:02:381
1859625215,19sleep00-21swapper/020:10:330
141125112,17sleep30-21swapper/300:25:373
809625013,20sleep00-21swapper/021:20:340
80125017,18sleep30-21swapper/319:15:253
80125017,18sleep30-21swapper/319:15:253
2860325011,23sleep128597-21apt-get00:10:171
1537425012,14sleep30-21swapper/323:25:293
1038525012,21sleep010391-21ntp_states19:45:370
3040424911,21sleep030461-21irqstats00:15:200
2704024913,21sleep127038-21apt-get00:05:081
2276724914,21sleep322804-21ntp_states23:50:243
153622499,18sleep30-21swapper/320:00:343
150332498,15sleep10-21swapper/120:00:281
995924813,21sleep09966-21df_inode23:10:160
2291924815,15sleep30-21swapper/320:25:303
1381624815,18sleep30-21swapper/321:40:153
1111424612,18sleep30-21swapper/323:14:263
289724510,21sleep02903-21ntp_states00:30:260
323242428,15sleep10-21swapper/120:55:271
323242428,15sleep10-21swapper/120:55:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional