You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 06:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Dec 05, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21242836313,8331sleep119538-21kworker/u8:419:58:401
281792484448,22sleep228181-21sshd21:20:392
2056323910,23sleep315970-21cyclictest20:57:543
7120237018,332sleep17114-21apt-get21:58:121
8414232111,296sleep08425-21systemd-run22:02:550
246442277112,142sleep324802-21unixbench_multi21:08:293
3021722719,252sleep35666-21snmpd19:45:063
19927221833,23sleep30-21swapper/322:38:173
10044221719,173sleep310107-21fschecks_time23:48:163
2389022149,193sleep324022-21cat22:48:393
17664221419,163sleep30-21swapper/319:07:573
2619220820,92sleep30-21swapper/321:43:163
1013220219,167sleep31149-21timerwakeupswit23:18:263
20661219320,146sleep30-21swapper/300:22:553
20661219320,146sleep30-21swapper/300:22:553
31497219230,25sleep30-21swapper/323:13:173
15897219020,39sleep316049-21latency_hist19:02:563
15530217913,152sleep015574-21ntpq22:23:330
17177217821,142sleep317179-21cron20:47:543
21309217212,145sleep021323-21awk20:58:250
29218216611,140sleep329187-21latency_hist23:07:563
6893216417,125sleep30-21swapper/321:57:573
6893216417,125sleep30-21swapper/321:57:573
21754216311,140sleep321763-21/usr/sbin/munin19:18:183
21754216311,140sleep321763-21/usr/sbin/munin19:18:183
22688216126,25sleep30-21swapper/322:47:543
22688216126,25sleep30-21swapper/322:47:543
19925215810,136sleep319930-21entropy19:13:133
9730215317,26sleep30-21swapper/323:47:563
9730215317,26sleep30-21swapper/323:47:563
284012153108,23sleep30-21swapper/319:38:283
2614421536,62sleep30-21swapper/319:33:113
192382153112,28sleep319080-21latency_hist00:17:593
192382153112,28sleep319080-21latency_hist00:17:593
23197214918,104sleep10-21swapper/122:48:081
30230214818,23sleep30-21swapper/323:09:173
3750214318,112sleep33823-21switchtime20:03:263
26163214317,106sleep10-21swapper/119:33:131
24963213614,110sleep324971-21uname19:28:273
11359213670,20sleep30-21swapper/320:28:213
21141213522,98sleep121203-21sed22:42:561
3824213316,99sleep23930-21grep23:28:172
23008213114,98sleep30-21swapper/300:28:283
23008213114,98sleep30-21swapper/300:28:283
2646213013,103sleep02657-21ntpq23:23:390
16296213011,19sleep30-21swapper/300:08:143
16296213011,19sleep30-21swapper/300:08:143
752212915,28sleep30-21swapper/319:53:293
32577212915,101sleep132636-21latency19:53:191
23215212912,22sleep30-21swapper/319:23:223
23215212912,22sleep30-21swapper/319:23:223
17962212914,96sleep118032-21systemd-run22:32:551
30598212812,104sleep330645-21ntpq21:28:193
30598212812,104sleep330645-21ntpq21:28:193
21459212513,97sleep321568-21cat20:58:283
10128212414,28sleep310275-21grep20:23:323
21477212323,83sleep021523-21tune2fs22:43:150
21477212323,83sleep021523-21tune2fs22:43:150
3794212278,20sleep30-21swapper/323:28:143
1062921227,11sleep00-21swapper/020:27:010
9925212123,76sleep3225-21systemd-journal22:07:543
18073212014,83sleep30-21swapper/322:32:573
1343721200,9sleep00-21swapper/023:58:170
25544211832,44sleep20-21swapper/221:13:082
18402211714,88sleep318489-21fw_packets22:33:173
1495211610,91sleep01611-21sed21:38:290
1495211610,91sleep01611-21sed21:38:290
15747211533,62sleep115795-21modprobe19:02:541
12008211510,30sleep312042-21ps23:53:253
540821140,6sleep30-21swapper/323:33:143
27583211411,87sleep327648-21awk21:18:263
1252211415,80sleep21296-21ntpq21:38:232
1252211415,80sleep21296-21ntpq21:38:232
4065211316,19sleep30-21swapper/321:48:173
2354521135,39sleep30-21swapper/321:06:193
2354521135,39sleep30-21swapper/321:06:193
1354211311,89sleep31557-21apt-config19:57:573
25166211116,20sleep325207-21ntp_kernel_err22:53:193
3073421099,79sleep030741-21cpu19:48:140
7891210713,15sleep30-21swapper/320:18:123
1612210712,79sleep11737-21unixbench_singl21:38:301
1612210712,79sleep11737-21unixbench_singl21:38:301
11251210718,18sleep10-21swapper/120:28:191
18896210613,77sleep119101-21latency_hist20:52:571
29168210419,68sleep129186-21apt-get23:07:541
18723210411,82sleep318233-21/usr/sbin/munin19:08:253
18723210411,82sleep318233-21/usr/sbin/munin19:08:253
685721029,76sleep30-21swapper/323:38:113
14328210162,18sleep30-21swapper/320:38:053
4752210067,16sleep30-21swapper/320:08:113
4075210017,64sleep04166-21fschecks_time21:48:180
32144210013,70sleep30-21swapper/321:33:193
2733521006,12sleep30-21swapper/322:59:203
21158210024,54sleep221318-21latency_hist19:17:562
20307210014,69sleep019817-21/usr/sbin/munin19:13:220
231829816,66sleep12321-21/usr/sbin/munin23:23:321
2301229811,64sleep223143-21latency_hist22:47:572
2301229811,64sleep223143-21latency_hist22:47:572
136102986,14sleep30-21swapper/322:18:203
1028229810,18sleep30-21swapper/322:08:113
1028229810,18sleep30-21swapper/322:08:113
3073229712,20sleep30-21swapper/319:48:133
157362958,12sleep30-21swapper/322:23:403
118032957,11sleep30-21swapper/322:13:113
27252948,68sleep30-21swapper/323:23:423
131832947,12sleep30-21swapper/323:58:133
287302927,69sleep30-21swapper/321:23:183
287302927,69sleep30-21swapper/321:23:183
1479229217,23sleep00-21swapper/022:22:570
1005029212,66sleep210127-21grep20:23:312
997729012,61sleep110059-21latency_hist22:07:561
866629020,40sleep28663-21apt-get22:03:092
2704429021,46sleep127198-21unixbench_multi22:58:301
1722929025,17sleep10-21swapper/120:47:561
146202906,12sleep30-21swapper/300:03:023
1208429013,54sleep211777-21/usr/sbin/munin22:13:182
82612895,13sleep30-21swapper/322:01:243
72812899,11sleep30-21swapper/320:13:523
184622885,10sleep30-21swapper/320:49:323
1669128813,53sleep116754-21sendmail_mailqu00:08:241
1669128813,53sleep116754-21sendmail_mailqu00:08:241
125392886,11sleep30-21swapper/320:33:053
266952876,11sleep30-21swapper/321:16:283
220912876,11sleep30-21swapper/300:26:593
130982879,17sleep20-21swapper/223:58:082
18302866,10sleep30-21swapper/321:39:003
18302866,10sleep30-21swapper/321:39:003
1755428614,54sleep017643-21fschecks_time20:48:160
863528513,52sleep08788-21/usr/sbin/munin20:18:290
112428421,40sleep01138-21apt-get19:57:540
62728321,43sleep2749-21grep19:53:282
357328318,28sleep20-21swapper/220:03:212
1849128317,32sleep00-21swapper/022:33:180
3189228215,49sleep031261-21/usr/sbin/munin23:13:260
310528210,55sleep13231-21unixbench_multi21:43:261
2934028216,44sleep029187-21latency_hist23:07:570
263422820,6sleep10-21swapper/121:13:261
158872825,11sleep00-21swapper/000:07:310
1579628228,34sleep215871-21munin-run19:02:552
3175028114,47sleep131751-21/usr/sbin/munin21:33:051
1639128114,52sleep016452-21processes20:43:270
970728021,31sleep20-21swapper/223:47:552
970728021,31sleep20-21swapper/223:47:552
2575828026,32sleep225772-21modprobe19:32:542
3206127810,55sleep232087-21latency_hist19:52:562
3145827819,38sleep231476-21modprobe21:32:542
3145827819,38sleep231476-21modprobe21:32:542
2926127815,46sleep029264-21/usr/sbin/munin19:43:130
210642775,61sleep00-21swapper/022:42:340
1487327721,28sleep00-21swapper/020:38:390
35227618,40sleep00-21swapper/023:18:060
2996027613,48sleep130108-21sed21:27:561
2996027613,48sleep130108-21sed21:27:561
328927520,25sleep10-21swapper/120:03:161
674727417,40sleep16792-21ntp_states20:13:211
447727413,31sleep10-21swapper/121:48:271
1286227418,25sleep20-21swapper/223:57:562
850227323,31sleep10-21swapper/123:43:141
850227323,31sleep10-21swapper/123:43:141
232227318,33sleep22152-21/usr/sbin/munin23:23:332
1981327314,45sleep119892-21awk20:53:461
2281927220,25sleep10-21swapper/122:47:561
2281927220,25sleep10-21swapper/122:47:561
2087527217,40sleep220891-21awk00:23:072
1889127217,36sleep218987-21timerandwakeup22:33:282
1502127221,32sleep215153-21cat20:38:412
1156527220,25sleep10-21swapper/122:12:571
1156527220,25sleep10-21swapper/122:12:571
440527123,22sleep10-21swapper/123:28:291
1673827120,24sleep20-21swapper/219:03:442
151212716,12sleep10-21swapper/122:23:101
740527011,45sleep07407-21sshd20:15:260
287027010,43sleep00-21swapper/021:43:210
2318927019,33sleep222703-21apt-get22:48:052
2132427020,26sleep10-21swapper/120:58:261
2060927025,22sleep10-21swapper/100:22:531
2060927025,22sleep10-21swapper/100:22:531
193952708,49sleep00-21swapper/020:53:320
187402705,11sleep00-21swapper/000:13:480
187402705,11sleep00-21swapper/000:13:480
1782027011,42sleep017942-21munin-node19:07:580
1782027011,42sleep017942-21munin-node19:07:580
1148827015,29sleep00-21swapper/020:28:260
2608026923,25sleep20-21swapper/219:33:102
1281226926,26sleep10-21swapper/123:57:551
511926814,37sleep05147-21netstat20:08:200
307532689,13sleep00-21swapper/023:10:110
3046326812,42sleep030480-21uname21:28:160
3046326812,42sleep030480-21uname21:28:160
2768626815,38sleep027844-21cat23:02:570
252126816,34sleep22617-21fschecks_time21:43:152
224312688,16sleep10-21swapper/100:28:101
224312688,16sleep10-21swapper/100:28:101
211102689,15sleep00-21swapper/000:23:170
1960426818,27sleep20-21swapper/219:12:562
1960426818,27sleep20-21swapper/219:12:562
1383026820,24sleep00-21swapper/020:37:540
1045026810,41sleep210535-21/usr/sbin/munin23:48:282
712526713,40sleep27171-21sed23:38:182
2270826716,23sleep20-21swapper/219:22:572
2270826716,23sleep20-21swapper/219:22:572
2252626719,22sleep10-21swapper/121:03:141
2252626719,22sleep10-21swapper/121:03:141
1465626713,37sleep114723-21/usr/sbin/munin00:03:121
843826611,37sleep20-21swapper/220:18:272
239102666,49sleep10-21swapper/119:24:091
239102666,49sleep10-21swapper/119:24:091
1950026617,23sleep20-21swapper/200:18:352
1950026617,23sleep20-21swapper/200:18:352
162326616,32sleep01138-21apt-get19:58:070
93826518,28sleep20-21swapper/223:18:252
3065626514,34sleep130657-21ntp_states21:28:211
3065626514,34sleep130657-21ntp_states21:28:211
2683326517,33sleep226896-21sendmail_mailqu22:58:272
1110826516,27sleep20-21swapper/222:08:262
1110826516,27sleep20-21swapper/222:08:262
1827226418,34sleep10-21swapper/100:13:291
1827226418,34sleep10-21swapper/100:13:291
2378926318,25sleep20-21swapper/200:32:552
2378926318,25sleep20-21swapper/200:32:552
1265226316,32sleep112657-21/usr/sbin/munin20:33:151
941426211,38sleep05666-21snmpd23:44:420
941426211,38sleep05666-21snmpd23:44:420
5682629,12sleep10-21swapper/123:18:151
2966626211,31sleep229753-21/usr/sbin/munin19:43:232
2739026217,20sleep00-21swapper/019:37:560
2611826212,30sleep026247-21latency_hist22:57:560
2265726215,33sleep022694-21awk19:22:550
2265726215,33sleep022694-21awk19:22:550
202426221,19sleep20-21swapper/219:58:212
1485326220,21sleep00-21swapper/018:58:170
58212617,12sleep20-21swapper/221:53:162
58212617,12sleep20-21swapper/221:53:162
2736726120,19sleep10-21swapper/121:18:201
1392026117,28sleep213849-21latency_hist20:37:562
1206726111,19sleep10-21swapper/122:13:171
510426018,26sleep15263-21date23:32:571
2917926022,21sleep10-21swapper/119:43:101
1854826016,24sleep10-21swapper/122:33:191
1757626018,27sleep10-21swapper/119:07:551
2404025912,32sleep224046-21diskstats21:08:162
2002525923,19sleep10-21swapper/119:13:151
1827925918,26sleep20-21swapper/200:13:302
1827925918,26sleep20-21swapper/200:13:302
134692599,12sleep10-21swapper/123:58:191
576325815,23sleep15819-21interrupts21:53:151
576325815,23sleep15819-21interrupts21:53:151
514925817,26sleep15183-21/usr/sbin/munin20:08:211
319222585,12sleep10-21swapper/119:52:021
2969525816,27sleep129688-21apt-get23:08:101
2885525816,25sleep20-21swapper/221:23:212
2885525816,25sleep20-21swapper/221:23:212
2811725820,21sleep00-21swapper/019:38:250
1121325818,22sleep011256-21idleruntime-cro23:52:530
842925712,20sleep20-21swapper/222:02:572
58725715,25sleep2617-21/usr/sbin/munin21:37:542
3004125711,29sleep230135-21memory23:09:122
2501425718,22sleep125104-21grep19:28:291
2436325717,19sleep10-21swapper/121:08:231
1465225715,26sleep214647-21apt-get00:03:092
1069825713,21sleep10-21swapper/120:27:541
531625619,22sleep20-21swapper/221:52:562
416125615,25sleep04184-21ntp_states23:28:240
2920425621,18sleep20-21swapper/223:07:552
2541625614,25sleep00-21swapper/019:28:400
2541125615,23sleep10-21swapper/122:53:251
2372325615,18sleep00-21swapper/021:07:570
2372325615,18sleep00-21swapper/021:07:570
2245625613,25sleep022450-21apt-get00:28:130
2245625613,25sleep022450-21apt-get00:28:130
1953425612,26sleep119602-21systemd-run19:12:551
1953425612,26sleep119602-21systemd-run19:12:551
1677925612,29sleep016785-21diskstats22:28:130
538925515,22sleep20-21swapper/223:33:142
3175525517,18sleep20-21swapper/223:13:242
2865425514,24sleep028648-21apt-get21:23:150
2865425514,24sleep028648-21apt-get21:23:150
2716925513,15sleep00-21swapper/021:18:150
1311625510,29sleep013258-21cat22:17:560
3177525413,23sleep031770-21apt-get21:33:080
2181425412,24sleep121848-21ntp_states19:18:201
2181425412,24sleep121848-21ntp_states19:18:201
1739325411,31sleep05666-21snmpd19:05:320
455625315,17sleep20-21swapper/220:07:572
2680725315,25sleep026839-21idleruntime-cro21:17:540
2295025313,23sleep222939-21apt-get19:23:132
2295025313,23sleep222939-21apt-get19:23:132
1667725314,22sleep216696-21apt-get22:28:072
1436625310,31sleep15245-21kworker/u8:122:19:471
109662537,18sleep20-21swapper/220:28:082
661525213,16sleep00-21swapper/023:37:560
555725212,15sleep00-21swapper/021:53:100
555725212,15sleep00-21swapper/021:53:100
382825212,26sleep03960-21timerwakeupswit20:03:270
3258125213,22sleep132595-21munin-run23:17:551
2762725211,25sleep127622-21apt-get19:38:101
2055425213,22sleep20-21swapper/220:57:542
1461225212,24sleep214631-21/usr/sbin/munin22:22:552
1131625220,17sleep00-21swapper/022:10:460
1131625220,17sleep00-21swapper/022:10:460
109962526,10sleep10-21swapper/123:50:081
2479925111,22sleep224792-21apt-get22:53:102
237432518,15sleep00-21swapper/019:23:360
237432518,15sleep00-21swapper/019:23:360
2322225113,21sleep023216-21apt-get22:48:110
1746625112,23sleep217462-21apt-get20:48:102
826225013,23sleep18305-21tr20:18:221
1647025012,24sleep216502-21mailstats20:43:292
1154425012,21sleep011539-21apt-get23:53:100
994624911,23sleep09976-21munin-run22:07:550
288052495,32sleep20-21swapper/219:41:502
2406024912,20sleep024169-21perl21:08:180
1983224911,23sleep219824-21apt-get22:38:122
156142498,17sleep20-21swapper/222:23:352
707124810,17sleep10-21swapper/123:38:171
700324814,17sleep00-21swapper/023:38:160
1946024811,23sleep019464-21diskstats00:18:330
1946024811,23sleep019464-21diskstats00:18:330
220332476,12sleep10-21swapper/100:26:161
2105624611,20sleep221113-21irqstats20:58:192
1628924511,20sleep216295-21diskstats00:08:132
1628924511,20sleep216295-21diskstats00:08:132
1618824411,16sleep00-21swapper/000:08:070
1618824411,16sleep00-21swapper/000:08:070
144092448,14sleep10-21swapper/120:38:121
689724310,16sleep20-21swapper/220:13:262
2280523913,13sleep20-21swapper/221:03:192
2280523913,13sleep20-21swapper/221:03:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional