You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 08:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Sep 16, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39616522148105,34sleep10-21swapper/119:06:271
3961892214298,15sleep30-21swapper/319:09:303
39619142131108,16sleep00-21swapper/019:09:480
39617092130108,15sleep20-21swapper/219:07:122
3962055992921,6cyclictest192248-21cron00:20:003
3962055992720,4cyclictest1-21systemd21:44:563
3962055992622,2cyclictest0-21swapper/322:50:233
3962055992613,9cyclictest4167728-21tr22:00:193
3962055992420,2cyclictest0-21swapper/322:45:133
3962055992415,6cyclictest110922-21cat23:25:273
3962055992412,8cyclictest217200-21tr00:35:173
3962055992318,3cyclictest4076910-21munin-plugin-st21:00:003
396205599231,7cyclictest1747-21sh22:18:103
3962055992315,5cyclictest4173674-21sort22:05:103
3962055992314,6cyclictest174719-21grep00:05:253
40913462221,14sleep04091471-21ntp_192.168.11421:10:190
396205599229,9cyclictest89675-21seq23:14:283
3962055992218,2cyclictest211893-21sh00:30:383
3962055992218,2cyclictest211893-21sh00:30:383
3962055992214,7cyclictest4099908-21unixbench_singl21:15:313
3962055992214,7cyclictest4099908-21unixbench_singl21:15:303
3962055992214,6cyclictest0-21swapper/321:25:143
396205599221,17cyclictest40342-21sh22:41:313
396205599220,21cyclictest205640-21ssh00:27:353
396205599219,8cyclictest162051-21pluginstate00:00:003
3962055992113,6cyclictest0-21swapper/322:10:153
3962055992112,7cyclictest120069-21runrttasks23:32:073
396205599211,18cyclictest4087549-21/usr/sbin/munin21:10:233
3962055992111,8cyclictest4045134-21ntpq20:25:203
396205599211,18cyclictest151417-21runrttasks23:50:583
3962055992111,8cyclictest0-21swapper/300:20:213
396205599211,17cyclictest62398-21grep22:55:173
3962055992111,7cyclictest4162564-21grep21:55:293
396205599211,17cyclictest4108931-21runrttasks21:22:473
3962055992111,7cyclictest138051-21date23:45:013
396205599211,16cyclictest138958-21grep23:45:153
3962055992110,8cyclictest4010781-21seq19:55:113
3962055992110,7cyclictest4124529-21seq21:32:433
396205599209,7cyclictest177739-21apt-get00:10:153
396205599207,8cyclictest2050013-21systemd-journal21:50:003
396205599202,16cyclictest17016-21sh22:27:393
3962055992016,2cyclictest0-21swapper/323:20:243
3962055992015,3cyclictest4142524-21gltestperf21:45:153
3962055992015,3cyclictest4082824-21kthreadcore21:05:143
3962055992015,3cyclictest4036285-21cpuspeed_turbos20:20:123
3962055992015,3cyclictest27108-21cpuspeed_turbos22:35:143
3962055992014,4cyclictest511ktimers/323:06:153
3962055992014,4cyclictest4062079-21cpuspeed_turbos20:45:143
3962055992014,4cyclictest4061014-21unixbench_singl20:40:273
3962055992013,5cyclictest125695-21cut23:35:183
396205599200,6cyclictest95408-21sh23:15:363
396205599200,6cyclictest26205-21ssh22:34:503
640402191,6sleep24087207-21diskmemload22:56:192
41376022190,2sleep20-21swapper/221:40:172
41323632191,14sleep04132377-21ssh21:37:330
396205599199,3cyclictest164778-21ssh00:00:183
396205599198,8cyclictest4130985-21grep21:35:263
396205599191,6cyclictest4081579-21unixbench_multi21:00:273
396205599191,15cyclictest3995247-21/usr/sbin/munin19:40:213
396205599190,17cyclictest4029882-21/usr/sbin/munin20:10:233
396205599190,16cyclictest8270-21rm22:21:083
396205599190,16cyclictest3988579-21ntpq19:30:193
396205599188,7cyclictest4067311-21perl20:50:183
3962055991814,3cyclictest4046434-21cpuspeed_turbos20:30:103
3962055991813,3cyclictest3969142-21kthreadcore19:15:023
3962055991811,5cyclictest3991037-21kthreadcore19:35:163
396205599181,12cyclictest71572-21taskset23:00:413
396205599180,16cyclictest4020989-21/usr/sbin/munin20:05:153
1918262181,3sleep10-21swapper/100:19:121
41561172171,3sleep00-21swapper/021:52:400
40338772170,5sleep2411rcuc/220:15:162
396205599179,5cyclictest945-21snmpd20:02:473
396205599178,7cyclictest0-21swapper/319:20:153
396205599178,6cyclictest3967275-21basename19:15:013
3962055991713,3cyclictest4051607-21cpuspeed_turbos20:35:133
1981452170,3sleep10-21swapper/100:22:051
977582161,4sleep24087207-21diskmemload23:19:202
41085642161,2sleep20-21swapper/221:22:112
3962055991615,1cyclictest492-21hwrng20:15:573
3962055991611,3cyclictest3983516-21ldconfig19:25:203
2065482163,1sleep10-21swapper/100:29:461
1992772161,10sleep2199289-21ssh00:24:102
181832161,3sleep04087207-21diskmemload22:29:400
1207102161,4sleep24087207-21diskmemload23:33:072
824292150,2sleep00-21swapper/023:10:080
799202151,2sleep10-21swapper/123:06:331
657442151,2sleep00-21swapper/022:58:580
41495042150,1sleep10-21swapper/121:49:361
41110212150,10sleep24111072-21ssh21:25:162
396205599156,6cyclictest4003072-21tr19:45:173
3962055991511,3cyclictest4005753-21cpuspeed_turbos19:50:123
1844182151,9sleep04087207-21diskmemload00:14:170
1769032151,2sleep00-21swapper/000:09:130
105552150,2sleep20-21swapper/222:24:412
41702202140,2sleep00-21swapper/022:00:300
39937012141,8sleep03993748-21ntpq19:35:210
1907472141,6sleep00-21swapper/000:17:150
1664782141,9sleep0166541-21awk00:00:230
1389302141,9sleep0140367-21/usr/sbin/munin23:45:150
876662131,7sleep20-21swapper/223:11:222
40880172131,7sleep14088494-21kthreadcore21:10:161
40862082131,8sleep24086260-21ld21:05:202
40553952131,8sleep04051449-21/usr/sbin/munin20:35:240
403452131,7sleep24087207-21diskmemload22:41:322
40265032131,8sleep04027500-21cut20:10:190
268402131,8sleep226858-21packagekitd22:35:022
1283202131,8sleep0128327-21ssh23:37:410
1279452131,7sleep24087207-21diskmemload23:37:082
68982121,7sleep07036-21ssh22:20:180
572222121,7sleep057232-21ssh22:53:040
41885852121,3sleep10-21swapper/122:14:251
41869742121,7sleep04186984-21ssh22:11:480
41506382121,6sleep14151517-21cut21:50:161
40784562120,7sleep24080217-21cut21:00:182
39835712121,8sleep03983624-21ld19:25:230
342322120,1sleep00-21swapper/022:39:300
331202120,8sleep133128-21ssh22:37:461
248572121,7sleep024865-21ssh22:32:440
2074732121,7sleep2207714-21idleruntime00:30:162
2074732121,7sleep2207714-21idleruntime00:30:152
1598772121,6sleep24087207-21diskmemload23:56:512
15422120,8sleep21554-21ssh22:17:512
1132292121,7sleep04087207-21diskmemload23:28:480
1116082121,6sleep10-21swapper/123:26:171
1065252122,5sleep2106547-21apt-get23:25:142
70722111,5sleep17130-21ntpq22:20:191
480292111,3sleep10-21swapper/122:46:121
41820172110,6sleep24182054-21kthreadcore22:10:182
41718642111,6sleep24171874-21ssh22:02:402
41713292111,3sleep10-21swapper/122:01:471
412002111,4sleep10-21swapper/122:42:521
41037392111,6sleep04105368-21kthreadcore21:20:160
40930552111,3sleep20-21swapper/221:11:482
1830522110,6sleep14087207-21diskmemload00:11:511
1692982111,4sleep24087207-21diskmemload00:03:542
164322111,3sleep20-21swapper/222:26:402
1454962111,3sleep10-21swapper/123:49:131
11982111,3sleep10-21swapper/122:17:171
959432101,3sleep10-21swapper/123:16:271
897752101,3sleep10-21swapper/123:14:371
807582101,4sleep04087207-21diskmemload23:07:520
787492101,3sleep20-21swapper/223:05:252
663532100,2sleep10-21swapper/122:59:551
507872101,3sleep10-21swapper/122:50:051
487342101,4sleep24087207-21diskmemload22:47:182
466892101,3sleep00-21swapper/022:45:190
41785612101,3sleep20-21swapper/222:05:532
41635322101,4sleep24087207-21diskmemload21:56:512
41492562101,4sleep24087207-21diskmemload21:49:132
41471362101,4sleep04087207-21diskmemload21:45:540
41409342101,2sleep00-21swapper/021:43:380
41406982100,1sleep10-21swapper/121:43:141
41256212101,4sleep24087207-21diskmemload21:34:262
41180582100,3sleep00-21swapper/021:29:550
41078812101,4sleep14087207-21diskmemload21:21:011
41024412101,6sleep24102451-21ssh21:19:472
41024412101,6sleep24102451-21ssh21:19:472
41000342101,4sleep04087207-21diskmemload21:15:440
41000342101,4sleep04087207-21diskmemload21:15:430
40949172101,5sleep14094970-21ssh21:15:011
40949172101,5sleep14094970-21ssh21:15:011
40707052101,4sleep04070803-21ntpq20:50:220
405862100,4sleep04087207-21diskmemload22:41:540
40514482101,4sleep24051467-21apt-get20:35:132
40311292101,4sleep14031275-21df20:15:121
2142812101,4sleep1214492-21hddtemp_smartct00:35:161
2120182101,3sleep10-21swapper/100:30:591
2120182101,3sleep10-21swapper/100:30:581
1890522101,4sleep2189099-21ntpq00:15:202
1770612101,2sleep10-21swapper/100:09:301
175562101,2sleep10-21swapper/122:28:361
1618802101,2sleep00-21swapper/023:59:590
1543282101,3sleep10-21swapper/123:55:101
1530202101,2sleep10-21swapper/123:53:311
1366592101,2sleep10-21swapper/123:43:161
1271052101,3sleep10-21swapper/123:35:471
1148092101,5sleep1115479-21cut23:30:141
1056572101,3sleep10-21swapper/123:24:161
1053282101,2sleep20-21swapper/223:23:442
90405291,3sleep090037-21apt-get23:15:010
72745291,3sleep24087207-21diskmemload23:02:322
71585291,2sleep10-21swapper/123:00:421
70769291,3sleep04087207-21diskmemload23:00:210
56086291,4sleep24087207-21diskmemload22:51:182
4179460290,2sleep10-21swapper/122:07:201
4177535291,4sleep04177593-21ntpq22:05:180
4163127291,3sleep04087207-21diskmemload21:56:130
4162349291,2sleep10-21swapper/121:55:291
4149781291,2sleep20-21swapper/221:50:002
4126330290,1sleep20-21swapper/221:35:102
4123358291,2sleep10-21swapper/121:30:511
4114301291,3sleep14114456-21strings21:25:191
4077168291,2sleep10-21swapper/121:00:131
4046350291,3sleep24046369-21apt-get20:30:092
4031107291,3sleep04031126-21apt-get20:15:120
4022490294,2sleep20-21swapper/220:05:202
4020705291,4sleep24020725-21munin-run20:05:002
3995269290,6sleep03995413-21df19:40:130
311291,3sleep04087207-21diskmemload22:16:000
24963290,3sleep10-21swapper/122:32:531
205986291,3sleep04087207-21diskmemload00:28:290
205606291,2sleep20-21swapper/200:27:312
197933291,2sleep00-21swapper/000:21:390
18765291,2sleep20-21swapper/222:30:102
175874290,2sleep20-21swapper/200:07:182
152749290,2sleep00-21swapper/023:53:060
150220290,3sleep20-21swapper/223:50:192
145591291,3sleep24087207-21diskmemload23:49:232
136283291,2sleep20-21swapper/223:42:412
119940291,2sleep00-21swapper/023:31:570
103528291,3sleep04087207-21diskmemload23:20:590
4131700281,2sleep10-21swapper/121:36:291
4123259280,2sleep00-21swapper/021:30:420
4082329281,2sleep10-21swapper/121:05:111
4075998280,1sleep20-21swapper/220:55:222
4068092282,3sleep24069827-21kthreadcore20:50:212
4050498281,4sleep04050725-21sed20:30:230
4046090281,3sleep24046100-21latency_hist20:30:002
4036626281,4sleep24037066-21cat20:20:152
4015553280,1sleep20-21swapper/219:59:312
3988506281,3sleep03988579-21ntpq19:30:190
3983625281,4sleep13983696-21tr19:25:241
3978529283,2sleep10-21swapper/119:20:231
3971031281,3sleep13972826-21kthreadcore19:15:181
3969996281,3sleep03970032-21kthreadcore19:15:170
3967050281,3sleep03967121-21dpkg19:15:010
220034281,3sleep20-21swapper/200:38:362
219654281,2sleep04087207-21diskmemload00:37:310
212563280,1sleep00-21swapper/000:32:180
212563280,1sleep00-21swapper/000:32:180
177610281,3sleep2177616-21ssh00:10:032
167998280,1sleep10-21swapper/100:01:521
135023280,2sleep00-21swapper/023:40:470
4070810270,2sleep10-21swapper/120:50:231
4062111271,3sleep24062324-21sshd20:45:162
4061343270,1sleep10-21swapper/120:44:231
4020971271,2sleep10-21swapper/120:05:141
4005407271,3sleep14005515-21latency_hist19:50:001
4000585270,2sleep20-21swapper/219:45:122
3999932270,1sleep20-21swapper/219:43:272
3990024270,3sleep23990234-21hddtemp_smartct19:35:142
3984791270,2sleep10-21swapper/119:30:121
3979306270,1sleep20-21swapper/219:23:122
4072089261,2sleep00-21swapper/020:55:120
4063841261,2sleep10-21swapper/120:45:191
4061949261,2sleep00-21swapper/020:45:140
4057035261,2sleep00-21swapper/020:40:160
4041467261,2sleep10-21swapper/120:25:141
4027502261,2sleep10-21swapper/120:10:201
4010929261,2sleep10-21swapper/119:55:131
4007403261,2sleep20-21swapper/219:50:162
4005670260,1sleep10-21swapper/119:50:121
3995184260,2sleep10-21swapper/119:40:011
3986179260,3sleep23987975-21kthreadcore19:30:182
3975239260,3sleep03976923-21kthreadcore19:20:170
3963122261,2sleep10-21swapper/119:10:171
4081844250,1sleep00-21swapper/021:02:490
4072125250,1sleep10-21swapper/120:55:131
4056743250,1sleep20-21swapper/220:40:152
4051790250,1sleep10-21swapper/120:35:171
4038738250,1sleep00-21swapper/020:20:170
4030799250,3sleep20-21swapper/220:14:262
4010782250,1sleep00-21swapper/019:55:110
4010438250,1sleep00-21swapper/019:54:160
4005054250,1sleep00-21swapper/019:45:420
3989868250,1sleep10-21swapper/119:35:131
3984412250,1sleep20-21swapper/219:28:332
3967675250,2sleep20-21swapper/219:15:022
4082605240,1sleep00-21swapper/021:05:140
4046566240,1sleep10-21swapper/120:30:121
4041365240,1sleep00-21swapper/020:25:140
4036415240,1sleep10-21swapper/120:20:141
4021303240,1sleep00-21swapper/020:05:170
4015900240,1sleep10-21swapper/120:00:141
4015665240,1sleep00-21swapper/020:00:010
3961997240,1sleep20-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional