You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri May 09, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
362629212463,57sleep30-21swapper/319:06:573
36266628556,9sleep20-21swapper/219:07:242
36267028369,9sleep10-21swapper/119:07:281
36260128162,13sleep00-21swapper/019:06:340
5532192350,4sleep136-21ksoftirqd/123:42:291
362999993222,7cyclictest383661-21cron20:00:002
362999992924,3cyclictest505222-21ntpq22:50:192
362999992825,2cyclictest431042-21grep21:30:152
362999992824,2cyclictest595213-21pmu-power00:30:202
362999992823,3cyclictest559899-21fschecks_time23:50:132
362999992622,2cyclictest573921-21cut00:05:152
362999992622,2cyclictest499528-21latency_hist22:45:002
362999992622,2cyclictest364900-21idleruntime-cro19:15:002
362999992621,3cyclictest426378-21tail21:25:202
362999992522,2cyclictest362994-21cyclictest22:10:162
362999992521,2cyclictest444566-21cpuspeed_turbos21:45:132
362999992521,2cyclictest421078-21latency_hist21:20:012
362999992514,7cyclictest491656-21sed22:35:252
362999992511,9cyclictest532595-21memory23:20:202
5538492241,2sleep20-21swapper/223:43:362
363005992421,1cyclictest0-21swapper/300:00:353
363005992420,2cyclictest0-21swapper/321:26:593
362999992421,2cyclictest573387-21latency_hist00:05:012
362999992420,3cyclictest541586-21gltestperf23:30:162
362999992418,3cyclictest0-21swapper/223:35:202
362999992417,4cyclictest412908-21/usr/sbin/munin21:10:212
36299999241,20cyclictest404638-21ntpq20:50:162
36299999241,20cyclictest404638-21ntpq20:50:152
36299999241,17cyclictest425161-21ssh21:24:172
5908902231,9sleep0590938-21ntpq00:25:190
5661022231,3sleep00-21swapper/023:55:400
4434802231,9sleep0443494-21ssh21:44:040
363005992310,1cyclictest0-21swapper/321:58:223
362999992319,2cyclictest719-21rs:main2
362999992319,2cyclictest0-21swapper/223:05:002
362999992318,3cyclictest458873-21missed_timers22:00:182
36299999231,18cyclictest531145-21sshd23:19:182
362999992311,8cyclictest509485-21awk22:55:162
362999992311,8cyclictest457916-21/usr/sbin/munin22:00:002
3819822220,2sleep00-21swapper/019:55:110
363005992218,2cyclictest0-21swapper/323:04:143
363005992218,2cyclictest0-21swapper/323:00:003
363005992217,2cyclictest0-21swapper/321:54:403
363005992216,3cyclictest0-21swapper/323:20:183
36300599221,17cyclictest0-21swapper/321:33:093
36300599220,18cyclictest416552-21taskset21:15:033
362999992218,2cyclictest522558-21idleruntime-cro23:10:002
362999992218,2cyclictest440093-21fschecks_count21:40:142
362999992218,2cyclictest411616-21ssh21:10:002
362999992218,2cyclictest0-21swapper/219:55:002
362999992218,2cyclictest0-21swapper/219:20:012
36299999221,7cyclictest372299-21ntpq19:30:182
362999992217,3cyclictest537656-21pmu-power23:25:252
362999992213,6cyclictest582085-21ls00:15:182
36299999220,20cyclictest406342-21uname20:55:182
363005992117,2cyclictest0-21swapper/321:49:583
363005992117,2cyclictest0-21swapper/300:25:163
363005992116,2cyclictest0-21swapper/323:59:493
363005992115,3cyclictest0-21swapper/322:08:473
362999992118,2cyclictest362994-21cyclictest00:40:002
36299999211,7cyclictest557964-21ssh23:47:522
36299999211,7cyclictest501518-21ssh22:45:512
362999992117,2cyclictest0-21swapper/223:55:132
362999992116,3cyclictest390543-21ntp_kernel_err20:15:172
362999992111,8cyclictest0-21swapper/200:25:192
36299999211,17cyclictest466356-21sh22:08:512
36299999211,17cyclictest394523-21ld20:25:192
36299999211,17cyclictest378175-21grep19:45:132
36300599206,2cyclictest0-21swapper/300:08:303
36300599201,8cyclictest539575-21taskset23:27:523
363005992017,1cyclictest0-21swapper/319:15:013
363005992017,1cyclictest0-21swapper/300:24:243
363005992016,2cyclictest0-21swapper/322:37:183
363005992016,2cyclictest0-21swapper/322:21:533
363005992016,2cyclictest0-21swapper/321:25:013
363005992016,2cyclictest0-21swapper/319:35:003
363005992015,2cyclictest0-21swapper/322:29:183
363005992015,2cyclictest0-21swapper/321:36:143
363005992015,2cyclictest0-21swapper/300:18:303
36300599201,14cyclictest555092-21chrt23:45:133
362999992016,2cyclictest376582-21ntp_offset19:40:172
362999992015,3cyclictest449222-21cpuspeed_turbos21:50:162
36299999201,16cyclictest527073-21ssh23:15:002
362999992011,6cyclictest473865-21ssh22:16:022
36299999201,13cyclictest941-21runrttasks00:22:182
362999992010,8cyclictest0-21swapper/222:21:272
362999992010,7cyclictest577613-21taskset00:10:022
362999992010,6cyclictest435611-21head21:35:142
36299999200,17cyclictest941-21runrttasks22:29:222
5635452191,9sleep0563558-21ssh23:54:280
36300599199,6cyclictest581303-21turbostat00:15:003
36300599196,9cyclictest411544-21turbostat21:10:013
36300599195,12cyclictest0-21swapper/323:08:533
36300599194,12cyclictest0-21swapper/322:41:413
363005991916,1cyclictest0-21swapper/319:55:203
363005991915,2cyclictest0-21swapper/323:40:173
363005991915,2cyclictest0-21swapper/323:35:213
363005991915,2cyclictest0-21swapper/322:20:013
363005991915,2cyclictest0-21swapper/320:20:003
363005991915,2cyclictest0-21swapper/300:32:393
363005991914,2cyclictest0-21swapper/322:33:023
36300599191,2cyclictest0-21swapper/323:32:513
36300599191,14cyclictest0-21swapper/320:10:133
36300599191,13cyclictest0-21swapper/323:15:103
36299999191,7cyclictest369777-21diskstats19:25:152
362999991915,2cyclictest399901-21apt-key20:40:002
362999991915,2cyclictest0-21swapper/222:30:212
362999991915,2cyclictest0-21swapper/220:30:172
362999991912,5cyclictest373807-21/usr/sbin/munin19:35:252
362999991911,6cyclictest402254-21fschecks_time20:45:162
36299999190,6cyclictest386209-21cut20:05:152
363005991815,1cyclictest0-21swapper/323:12:363
363005991815,1cyclictest0-21swapper/322:50:053
363005991815,1cyclictest0-21swapper/322:10:313
363005991815,1cyclictest0-21swapper/320:34:543
363005991815,1cyclictest0-21swapper/320:22:373
363005991814,2cyclictest0-21swapper/322:45:133
363005991814,2cyclictest0-21swapper/321:13:433
363005991814,2cyclictest0-21swapper/320:45:203
363005991814,2cyclictest0-21swapper/319:24:153
363005991813,3cyclictest0-21swapper/322:00:183
36300599181,14cyclictest0-21swapper/323:51:373
36300599181,12cyclictest377503-21turbostat19:45:003
36300599180,11cyclictest409671-21taskset21:05:003
36299999189,6cyclictest368592-21systemctl19:20:172
36299999187,7cyclictest393166-21vmstat20:20:222
36299999181,14cyclictest941-21runrttasks21:00:212
6009772171,3sleep00-21swapper/000:39:220
5767162170,3sleep00-21swapper/000:08:430
5533362171,4sleep0411546-21diskmemload23:42:460
4807972171,3sleep00-21swapper/022:24:520
4666702171,2sleep10-21swapper/122:09:261
36300599173,12cyclictest0-21swapper/319:15:243
363005991714,1cyclictest0-21swapper/320:55:253
363005991714,1cyclictest0-21swapper/320:29:473
363005991714,1cyclictest0-21swapper/320:09:193
363005991714,1cyclictest0-21swapper/320:00:053
363005991714,1cyclictest0-21swapper/319:45:463
363005991713,2cyclictest405310-21taskset20:51:053
363005991713,2cyclictest405310-21taskset20:51:053
363005991712,3cyclictest0-21swapper/321:40:233
36300599171,13cyclictest0-21swapper/319:35:143
36300599171,13cyclictest0-21swapper/300:36:083
5452052161,9sleep1411546-21diskmemload23:34:331
5086132161,10sleep1508643-21munin-run22:55:001
4368542160,6sleep0436864-21ssh21:35:410
4253392161,4sleep0411546-21diskmemload21:24:360
3937922162,9sleep1944-21snmpd20:25:111
363005991614,1cyclictest0-21swapper/320:40:183
362999991611,3cyclictest387977-21grep20:10:142
5851512151,8sleep0411546-21diskmemload00:19:100
4889772150,2sleep00-21swapper/022:33:140
4720392151,7sleep0472204-21diskstats22:15:140
4701802150,5sleep0411546-21diskmemload22:12:330
4460452151,2sleep00-21swapper/021:45:430
4194222151,10sleep0419436-21ssh21:17:290
36300599151,11cyclictest381727-21sleep19:55:003
36300599150,12cyclictest0-21swapper/320:35:133
36299999151,12cyclictest383947-21/usr/sbin/munin20:00:212
5753242141,8sleep1411546-21diskmemload00:06:081
5696712143,7sleep1569834-21missed_timers00:00:181
5158442141,10sleep00-21swapper/023:01:400
4853132141,8sleep0411546-21diskmemload22:29:530
4836042141,8sleep1411546-21diskmemload22:26:541
4278602141,4sleep0427874-21ssh21:26:070
4182862141,9sleep1411546-21diskmemload21:15:421
4134822140,3sleep0411546-21diskmemload21:10:390
36300599140,10cyclictest369619-21chrt19:25:123
5880112131,8sleep0588025-21ssh00:21:340
5802782131,8sleep0580287-21ssh00:13:000
5702512131,8sleep0570288-21ssh00:00:250
5587322131,6sleep1411546-21diskmemload23:49:081
5462602131,3sleep00-21swapper/023:35:160
5260842131,2sleep00-21swapper/023:13:230
5050582131,5sleep0505142-21ntpq22:50:180
4616382131,8sleep0461640-21ssh22:03:390
4545212131,8sleep0454582-21/usr/sbin/munin21:55:170
4527852131,3sleep0411546-21diskmemload21:54:060
3895762131,8sleep0389609-21basename20:15:010
5958292121,5sleep00-21swapper/000:30:490
5251332121,7sleep1525146-21rm23:11:481
5104822121,5sleep00-21swapper/022:55:320
4699652121,6sleep1411546-21diskmemload22:12:151
4658492121,5sleep00-21swapper/022:08:040
4376142121,5sleep1411546-21diskmemload21:36:581
5773472114,3sleep1341rcuc/100:10:011
5388822110,7sleep0538892-21ssh23:26:420
5289192110,1sleep00-21swapper/023:15:350
5160602111,3sleep1516070-21ssh23:02:011
5109932111,4sleep1411546-21diskmemload22:56:201
4932182110,7sleep1493221-21ssh22:37:401
4220142111,5sleep1422106-21ntpq21:20:191
3924472111,5sleep1392521-21ntpq20:20:141
3631692111,4sleep0363191-21apt-get19:10:110
5961082101,3sleep1411546-21diskmemload00:31:251
5874352101,6sleep10-21swapper/100:20:271
5838862101,3sleep1411546-21diskmemload00:16:541
5663682101,3sleep1411546-21diskmemload23:56:051
5622732101,3sleep1411546-21diskmemload23:52:211
5357572101,3sleep1411546-21diskmemload23:24:061
5293732101,4sleep1411546-21diskmemload23:16:221
5187892101,5sleep0518906-21missed_timers23:05:180
5005142101,4sleep1500567-21ntpq22:45:181
4997072101,2sleep00-21swapper/022:45:050
4970582101,3sleep10-21swapper/122:41:291
4937042101,3sleep0411546-21diskmemload22:38:310
4773912101,5sleep1477437-21ntpq22:20:161
4744142101,2sleep10-21swapper/122:16:511
4582022101,4sleep10-21swapper/122:00:131
4521882101,3sleep1411546-21diskmemload21:53:071
4275412101,2sleep10-21swapper/121:25:361
4133642101,4sleep1411546-21diskmemload21:10:281
4045782101,5sleep0404629-21ntpq20:50:150
4045782101,5sleep0404629-21ntpq20:50:150
4026122101,4sleep1402660-21ntpq20:45:191
4007042101,4sleep0400750-21ntpq20:40:170
3865752101,4sleep0386626-21ntpq20:05:180
3816972101,4sleep0381746-21munin-run19:55:000
3745602101,4sleep0374614-21ntpq19:35:200
3732642101,4sleep0373652-21apt-config19:35:000
3720032101,4sleep1372122-21memory19:30:161
3703152101,5sleep1370361-21ntpq19:25:191
598088291,4sleep1598303-21hddtemp_smartct00:35:141
593109291,3sleep1411546-21diskmemload00:28:411
558080291,2sleep00-21swapper/023:48:030
547803291,3sleep1411546-21diskmemload23:36:181
543140291,2sleep00-21swapper/023:31:110
538210291,3sleep1411546-21diskmemload23:25:361
521974291,3sleep1411546-21diskmemload23:09:041
487245291,3sleep1411546-21diskmemload22:30:301
455481291,3sleep10-21swapper/121:55:591
448651291,3sleep1411546-21diskmemload21:49:571
441495291,3sleep1411546-21diskmemload21:40:431
432172290,1sleep00-21swapper/021:30:330
410531291,4sleep0410578-21ntpq21:05:180
408687291,4sleep0407869-21/usr/sbin/munin21:00:210
396493291,3sleep0396542-21ntpq20:30:200
366090290,1sleep00-21swapper/019:15:160
534114280,1sleep00-21swapper/023:21:330
496032280,2sleep00-21swapper/022:40:240
433124280,3sleep1411546-21diskmemload21:32:091
391823280,2sleep00-21swapper/020:20:100
383478280,1sleep10-21swapper/119:57:421
381421281,4sleep1944-21snmpd19:51:281
368124281,4sleep0368293-21grep19:20:130
364874281,4sleep1365201-21tune2fs19:15:011
407423270,1sleep00-21swapper/020:58:010
395761270,2sleep10-21swapper/120:30:131
395269270,1sleep00-21swapper/020:27:110
391303270,1sleep00-21swapper/020:17:010
385901270,1sleep10-21swapper/120:05:071
375780270,1sleep00-21swapper/019:40:050
373806270,2sleep10-21swapper/119:35:131
370930270,1sleep00-21swapper/019:25:260
407503260,1sleep10-21swapper/120:59:021
404636261,2sleep1404686-21ntpq20:50:171
404636261,2sleep1404686-21ntpq20:50:161
401962261,2sleep00-21swapper/020:45:140
399983260,2sleep10-21swapper/120:40:111
397907260,2sleep0398119-21smartctl20:35:120
397718260,2sleep10-21swapper/120:35:091
385340260,1sleep00-21swapper/020:01:050
379448260,1sleep00-21swapper/019:46:220
369149260,1sleep10-21swapper/119:22:391
363733260,2sleep10-21swapper/119:10:161
410580250,2sleep10-21swapper/121:05:191
390010250,2sleep10-21swapper/120:15:131
388888251,2sleep10-21swapper/120:10:241
384957250,2sleep10-21swapper/120:00:221
378216250,2sleep10-21swapper/119:45:151
375787250,1sleep10-21swapper/119:40:101
407892240,1sleep10-21swapper/121:00:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional