You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 13:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sat Nov 22, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3697840212862,62sleep30-21swapper/319:09:353
369778629262,10sleep20-21swapper/219:08:542
369762328053,19sleep00-21swapper/019:06:480
369770727559,10sleep10-21swapper/119:07:531
40569462530,2sleep14056945-21ssh23:27:031
39847992281,15sleep13984835-21ssh22:45:141
3698165992825,1cyclictest0-21swapper/322:40:003
3698165992824,2cyclictest0-21swapper/322:50:223
3698165992823,2cyclictest0-21swapper/321:10:003
40253002251,2sleep00-21swapper/023:09:190
3698165992522,1cyclictest0-21swapper/300:00:013
3698165992521,2cyclictest0-21swapper/322:10:153
39650792241,2sleep00-21swapper/022:30:540
38841732241,6sleep03827241-21diskmemload21:43:360
38513032240,3sleep00-21swapper/021:24:020
369816599243,6cyclictest3788928-21taskset20:30:193
3698165992421,2cyclictest0-21swapper/323:19:563
369816599239,11cyclictest4083372-21turbostat23:49:563
369816599232,15cyclictest3835225-21taskset21:14:393
369816599232,14cyclictest4006557-21taskset22:55:543
369816599232,14cyclictest3925331-21taskset22:08:113
3698165992319,2cyclictest0-21swapper/323:40:193
3698165992311,8cyclictest4152861-21taskset00:27:223
37960772221,16sleep03797538-21cut20:40:140
369816599222,13cyclictest3890421-21taskset21:45:303
3698165992219,1cyclictest0-21swapper/321:27:213
3698165992218,3cyclictest0-21swapper/300:20:123
369816599212,15cyclictest3769169-21chrt20:15:113
369816599212,14cyclictest4091857-21chrt23:50:103
369816599212,13cyclictest4064390-21taskset23:30:533
3698165992118,1cyclictest0-21swapper/323:21:273
3698165992117,2cyclictest0-21swapper/321:50:173
3698165992116,3cyclictest0-21swapper/323:10:203
3698165992116,2cyclictest0-21swapper/321:43:043
369816599211,16cyclictest4022828-21taskset23:05:323
369816599211,16cyclictest3967851-21turbostat22:35:003
369816599210,18cyclictest66550irq/131-enp1s0-TxRx-300:14:163
369816599210,14cyclictest3916191-21taskset22:01:543
41506492201,13sleep14151132-21head00:25:161
40739222200,9sleep244-21ksoftirqd/223:38:032
39274872201,8sleep23928170-21tr22:10:152
369816599202,12cyclictest3851561-21taskset21:24:253
3698165992015,3cyclictest0-21swapper/323:25:283
369816599201,16cyclictest4075086-21chrt23:39:523
369816599201,14cyclictest0-21swapper/320:25:123
369816599201,14cyclictest0-21swapper/320:25:113
39981362191,7sleep20-21swapper/222:50:562
39342452191,3sleep03827241-21diskmemload22:14:060
38517782191,4sleep23827241-21diskmemload21:24:452
38262122191,1sleep00-21swapper/021:05:190
369816599192,13cyclictest4016016-21taskset23:02:433
369816599191,6cyclictest4139180-21turbostat00:20:003
3698165991916,1cyclictest0-21swapper/322:40:213
369816599191,15cyclictest4167280-21chrt00:36:003
369816599191,14cyclictest3984739-21chrt22:45:073
369816599191,14cyclictest3943108-21chrt22:19:543
369816599191,13cyclictest3860483-21chrt21:30:023
369816599191,13cyclictest3758336-21turbostat20:04:593
369816599190,14cyclictest41550822chrt00:30:153
40426982181,4sleep13827241-21diskmemload23:20:101
39076052181,4sleep03827241-21diskmemload21:56:260
39009132181,8sleep13827241-21diskmemload21:53:511
38903972181,3sleep10-21swapper/121:45:291
38598372181,3sleep20-21swapper/221:29:302
38432332180,8sleep03827241-21diskmemload21:19:190
369816599182,12cyclictest4115843-21taskset00:04:373
369816599182,12cyclictest3909720-21taskset21:59:333
369816599182,11cyclictest4120899-21taskset00:05:253
369816599182,11cyclictest4120899-21taskset00:05:253
369816599182,11cyclictest3943509-21taskset22:20:063
369816599182,11cyclictest3816949-21taskset21:00:133
3698165991815,1cyclictest0-21swapper/321:15:253
3698165991815,1cyclictest0-21swapper/320:45:173
369816599181,12cyclictest3951795-21taskset22:25:113
369816599181,11cyclictest3732032-21taskset19:40:003
369816599181,10cyclictest3742740-21taskset19:50:063
369816599180,15cyclictest0-21swapper/321:35:083
40984862171,3sleep13827241-21diskmemload23:53:131
40762132171,7sleep14076249-21kthreadcore23:40:191
40479362171,4sleep23827241-21diskmemload23:20:592
40149322170,1sleep10-21swapper/123:01:061
39744352177,6sleep03827241-21diskmemload22:37:310
39324982170,1sleep10-21swapper/122:11:311
39236452171,4sleep23827241-21diskmemload22:05:392
39174512170,3sleep20-21swapper/222:03:502
38814082171,11sleep13881458-21ntpq21:40:201
38231452171,12sleep13824853-21cut21:05:161
369816599177,6cyclictest3795027-21taskset20:37:323
3698165991714,1cyclictest0-21swapper/320:40:173
3698165991714,1cyclictest0-21swapper/319:50:003
3698165991714,1cyclictest0-21swapper/319:30:153
3698165991713,2cyclictest0-21swapper/320:55:163
369816599171,12cyclictest0-21swapper/319:25:143
369816599171,11cyclictest3753068-21turbostat20:00:003
3698165991710,3cyclictest0-21swapper/320:50:103
41146942160,3sleep20-21swapper/200:02:532
40911782161,2sleep00-21swapper/023:49:320
40655212161,9sleep23827241-21diskmemload23:32:382
40399852161,2sleep10-21swapper/123:16:291
39838752161,3sleep10-21swapper/122:44:141
38830422161,1sleep23827241-21diskmemload21:41:532
38647632161,6sleep23864819-21ntpq21:30:232
38596362161,4sleep03827241-21diskmemload21:29:120
38327382161,3sleep23827241-21diskmemload21:10:542
38276492161,2sleep10-21swapper/121:10:131
38209312161,10sleep23820977-21ntpq21:00:202
369816599161,9cyclictest3735282-21taskset19:40:203
3698165991611,4cyclictest15950irq/132-ahci[0000:00:17.0]19:15:023
369816599161,10cyclictest3715752-21chrt19:20:323
369816599161,10cyclictest3715752-21chrt19:20:323
369816599161,10cyclictest3703462-21taskset19:14:473
369816599160,13cyclictest0-21swapper/320:05:113
41400422151,5sleep04140116-21kernelversion00:20:140
41308992151,9sleep13827241-21diskmemload00:13:381
41181472151,6sleep24118549-21kthreadcore00:05:172
41181472151,6sleep24118549-21kthreadcore00:05:172
40901862151,10sleep14090196-21ssh23:48:001
40136632151,7sleep04013809-21ntpq23:00:210
40097562151,8sleep24009969-21diskstats23:00:152
40019582151,9sleep04002165-21kthreadcore22:55:150
39722092151,4sleep13972331-21ntpq22:35:211
39429842151,9sleep03942996-21ssh22:19:440
39099792151,7sleep23827241-21diskmemload21:59:582
38762122151,2sleep20-21swapper/221:39:092
369816599151,9cyclictest3768638-21taskset20:11:513
369816599150,13cyclictest0-21swapper/320:20:163
41676712141,8sleep13827241-21diskmemload00:36:431
41378252141,2sleep10-21swapper/100:17:231
41072572141,7sleep13827241-21diskmemload23:59:111
40808772141,6sleep20-21swapper/223:41:122
40092362141,6sleep23827241-21diskmemload23:00:002
40055722141,9sleep14005630-21ld22:55:201
39980952140,2sleep1171rcu_preempt22:50:511
39740112141,9sleep23974022-21ssh22:36:532
38742642141,2sleep10-21swapper/121:36:161
38675632141,7sleep13827241-21diskmemload21:33:381
38427492141,7sleep13827241-21diskmemload21:18:351
37146632141,9sleep13715055-21grep19:20:191
37146632141,9sleep13715055-21grep19:20:191
37067032144,6sleep03708412-21cut19:15:170
41529662131,4sleep04152980-21ssh00:27:370
41401172131,8sleep14141526-21cut00:20:151
41241782131,8sleep04124323-21cut00:10:120
41169472131,6sleep04117030-21irqstats00:05:160
41169472131,6sleep04117030-21irqstats00:05:150
41076222133,4sleep0151ktimers/023:59:460
41060412131,8sleep24106047-21ssh23:57:152
40890932131,3sleep24089102-21ssh23:46:172
39667092130,7sleep13827241-21diskmemload22:33:201
39580692131,6sleep00-21swapper/022:27:370
39475472131,6sleep23947699-21ntpq22:20:182
39366832130,8sleep23938332-21kthreadcore22:15:172
38605902131,6sleep03860617-21apt-get21:30:130
37915682131,7sleep23793307-21grep20:35:192
37849622131,8sleep00-21swapper/020:30:120
37644182131,7sleep23765517-21kthreadcore20:10:142
41686012121,7sleep04168610-21ssh00:38:270
41590522121,3sleep24159293-21sed00:30:272
41241352121,7sleep24124144-21ssh00:10:102
41226382121,7sleep14122639-21sh00:08:011
41226382121,7sleep14122639-21sh00:08:011
40829522121,5sleep03827241-21diskmemload23:44:220
40561272121,5sleep00-21swapper/023:25:500
40422482121,5sleep00-21swapper/023:19:530
40411202121,7sleep23827241-21diskmemload23:18:112
40328742121,3sleep10-21swapper/123:13:121
40249272121,6sleep24024937-21ssh23:08:452
39905072121,3sleep20-21swapper/222:46:482
39891102121,5sleep00-21swapper/022:45:250
39595352121,4sleep13827241-21diskmemload22:29:561
39498962121,7sleep03949908-21ssh22:22:400
39358182121,6sleep13936682-21cut22:15:171
39260542121,2sleep10-21swapper/122:09:181
39009262121,4sleep23827241-21diskmemload21:53:522
38923042121,5sleep20-21swapper/221:48:202
38751622121,5sleep00-21swapper/021:37:340
38582012121,4sleep10-21swapper/121:27:011
37902182121,5sleep13790236-21apt-get20:35:141
37414932121,7sleep03741543-21ntp_kernel_pll_19:45:190
41682622111,3sleep20-21swapper/200:37:482
41599452111,4sleep03827241-21diskmemload00:31:330
41451122111,4sleep23827241-21diskmemload00:21:532
41386432111,3sleep23827241-21diskmemload00:19:002
41158502111,6sleep04115858-21sh00:04:390
40989252111,3sleep20-21swapper/223:53:552
40977402111,3sleep03827241-21diskmemload23:52:040
40751482111,4sleep03827241-21diskmemload23:39:570
40677672111,3sleep10-21swapper/123:35:131
40646202111,3sleep00-21swapper/023:31:160
40562862110,1sleep20-21swapper/223:26:042
40318932111,3sleep20-21swapper/223:11:452
40318732111,4sleep03827241-21diskmemload23:11:430
39929512111,3sleep03827241-21diskmemload22:50:100
39830072111,4sleep03827241-21diskmemload22:42:570
39764302111,4sleep23827241-21diskmemload22:40:062
39652792110,1sleep20-21swapper/222:31:122
39478182111,3sleep10-21swapper/122:20:211
39174402111,3sleep10-21swapper/122:03:491
39154022111,5sleep00-21swapper/022:00:430
39035492111,6sleep13904227-21kthreadcore21:55:191
39001332111,3sleep03827241-21diskmemload21:52:420
38914752110,5sleep03827241-21diskmemload21:47:040
38515372111,4sleep13827241-21diskmemload21:24:241
38420782111,3sleep20-21swapper/221:17:362
37467852111,5sleep13746834-21ntpq19:50:211
37319662111,5sleep21-21systemd19:40:012
41547362101,4sleep14154868-21diskstats00:30:141
41475782101,4sleep24147654-21irqstats00:25:132
41384172101,3sleep00-21swapper/000:18:350
40598662101,4sleep14061243-21grep23:30:161
40430042101,3sleep03827241-21diskmemload23:20:130
40178912101,3sleep14017916-21apt-get23:05:111
39573892101,3sleep23827241-21diskmemload22:26:352
39256902101,2sleep00-21swapper/022:08:450
38276172101,3sleep00-21swapper/021:10:120
38047682101,4sleep13804825-21ntpq20:45:201
38010932101,4sleep03801306-21hddtemp_smartct20:45:150
37957942100,1sleep20-21swapper/220:40:122
37836782101,4sleep03783727-21ntpq20:25:190
37836782101,4sleep03783727-21ntpq20:25:190
37744682101,4sleep13774729-21df_inode20:20:151
37731562101,4sleep23773205-21ntpq20:15:192
37694312101,4sleep03769555-21perl20:15:130
37641372100,1sleep00-21swapper/020:10:120
37626312101,4sleep23762680-21ntpq20:05:212
37480162101,4sleep13748035-21apt-get19:55:101
37427472101,4sleep03742766-21apt-get19:50:120
37377862101,5sleep23737995-21hddtemp_smartct19:45:142
37322762101,4sleep23732534-21df_inode19:40:162
37322542101,4sleep13732273-21apt-get19:40:151
37095602101,5sleep20-21swapper/219:15:192
37028532100,2sleep00-21swapper/019:10:240
4112227291,4sleep14112378-21ssh00:00:191
3810494290,1sleep10-21swapper/120:50:211
3804827291,3sleep23804873-21ntpq20:45:212
3778409291,4sleep03778458-21ntpq20:20:220
3725719291,3sleep23725776-21ntpq19:30:222
3698442291,4sleep13698461-21apt-get19:10:101
3821939280,1sleep20-21swapper/221:05:012
3815751281,3sleep13815836-21ps20:55:221
3810442280,1sleep00-21swapper/020:50:200
3806853281,3sleep23806951-21tail20:50:142
3781376283,2sleep1321rcuc/120:25:161
3781376283,2sleep1321rcuc/120:25:161
3759132281,3sleep13759792-21cut20:05:161
3739405281,3sleep13741226-21kthreadcore19:45:161
3722227281,3sleep13723098-21kthreadcore19:30:171
3821282271,3sleep03821547-21unixbench_singl21:00:260
3813206271,3sleep00-21swapper/020:55:180
3783184271,3sleep23783591-21grep20:25:172
3783184271,3sleep23783591-21grep20:25:172
3773061271,2sleep10-21swapper/120:15:181
3752977270,1sleep00-21swapper/019:59:060
3747685270,3sleep20-21swapper/219:54:072
3726932270,2sleep10-21swapper/119:35:101
3718585271,2sleep00-21swapper/019:25:190
3716368270,2sleep20-21swapper/219:25:132
3704056271,2sleep10-21swapper/119:15:021
3699014271,2sleep20-21swapper/219:10:142
3821887260,1sleep10-21swapper/121:04:241
3799846260,2sleep10-21swapper/120:40:231
3789774260,4sleep10-21swapper/120:32:461
3779303260,1sleep20-21swapper/220:23:412
3763264260,1sleep00-21swapper/020:05:510
3758025260,4sleep20-21swapper/220:01:162
3757355261,1sleep00-21swapper/020:00:220
3748597261,2sleep20-21swapper/219:55:162
3731082261,2sleep00-21swapper/019:35:210
3726348260,1sleep00-21swapper/019:30:560
3716035260,1sleep20-21swapper/219:24:092
3716035260,1sleep20-21swapper/219:24:092
3712855261,2sleep00-21swapper/019:20:180
3712855261,2sleep00-21swapper/019:20:180
3815547250,2sleep20-21swapper/220:55:192
3767795250,1sleep10-21swapper/120:10:171
3753332252,1sleep10-21swapper/120:00:111
3720839250,1sleep10-21swapper/119:25:291
3790002240,1sleep00-21swapper/020:35:010
3785239240,1sleep20-21swapper/220:30:142
3732639240,1sleep00-21swapper/019:40:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional