You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 22:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Tue Jul 15, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36738222138100,13sleep20-21swapper/207:06:402
36740462135101,25sleep00-21swapper/007:09:310
36738182132109,15sleep30-21swapper/307:06:373
36739302130107,15sleep10-21swapper/107:08:031
3674177993832,3cyclictest0-21swapper/310:35:143
40415152350,2sleep10-21swapper/111:52:571
3674177993530,3cyclictest3797971-21idleruntime-cro09:10:003
3674177993326,5cyclictest3768011-21tr08:40:003
3674177993126,3cyclictest703-21dbus-daemon07:37:013
3674177992823,3cyclictest709-21polkitd12:05:013
3674177992816,8cyclictest3842271-21apt-get09:40:123
3674177992722,3cyclictest0-21swapper/311:00:223
3674177992614,8cyclictest4080129-21sshd12:16:303
3674177992521,2cyclictest3929342-21gltestperf10:40:163
3674177992513,8cyclictest262-21systemd-journal11:07:303
3674177992512,9cyclictest3863955-21df_inode09:55:143
367417799250,4cyclictest4039389-21ntpq11:50:273
39212462240,2sleep20-21swapper/210:34:572
37999702240,3sleep30-21swapper/309:10:163
367417799249,2cyclictest0-21swapper/309:35:253
3674177992415,6cyclictest4070552-21sort12:10:223
3674177992320,2cyclictest3696223-21turbostat07:30:003
3674177992319,2cyclictest4011698-21ssh11:35:003
3674177992319,2cyclictest4011698-21ssh11:35:003
3674177992318,3cyclictest720-21tpm2-abrmd08:40:243
3674177992317,4cyclictest4043556-21gltestperf11:55:153
3674177992314,6cyclictest3918888-21sh10:30:503
3674177992311,10cyclictest3955463-21ntpq10:55:223
3674177992214,6cyclictest3860386-21grep09:50:173
3674177992213,6cyclictest4092432-21rm12:25:173
3674177992212,7cyclictest4102816-21sed12:30:193
3674177992211,7cyclictest3978740-21sh11:10:323
3674177992211,7cyclictest3829537-21cut09:30:163
3674177992117,2cyclictest4090636-21ls12:25:003
3674177992116,3cyclictest3889257-21ntp_kernel_pll_10:10:213
3674177992116,3cyclictest0-21swapper/311:40:003
367417799211,17cyclictest4111272-21/usr/sbin/munin12:35:213
3674177992110,8cyclictest3897004-21grep10:15:303
3674177992110,7cyclictest3899721-21runrttasks10:20:013
367417799201,7cyclictest3722641-21grep07:55:163
3674177992016,3cyclictest3674172-21cyclictest08:25:163
3674177992016,2cyclictest3996813-21gltestperf11:25:153
3674177992016,2cyclictest0-21swapper/310:55:003
367417799201,5cyclictest3936379-21/usr/sbin/munin10:45:163
367417799201,5cyclictest3814662-21kthreadcore09:20:163
367417799201,5cyclictest3814662-21kthreadcore09:20:163
3674177992013,4cyclictest3741280-21cut08:10:183
3674177992012,6cyclictest4032701-21runrttasks11:46:233
3674177992012,6cyclictest3912432-21ssh10:26:583
3674177992012,6cyclictest3912432-21ssh10:26:583
367417799201,17cyclictest3987111-21taskset11:17:093
3674177992011,7cyclictest0-21swapper/312:05:113
367417799201,16cyclictest3758514-21hddtemp_smartct08:30:153
367417799201,16cyclictest1-21systemd08:22:313
367417799200,5cyclictest3863201-21kworker/u16:3+rpciod11:20:493
40025792191,6sleep03797944-21diskmemload11:27:550
367417799192,13cyclictest945-21snmpd10:00:203
367417799191,16cyclictest3797944-21diskmemload09:48:043
3674177991910,6cyclictest3776795-21/usr/sbin/munin08:45:193
3674177991910,6cyclictest3696921-21fschecks_count07:30:143
367417799190,14cyclictest3797944-21diskmemload09:25:293
3674177991815,2cyclictest3737362-21apt-key08:10:003
3674177991815,2cyclictest3715955-21needreboot07:45:203
3674177991814,3cyclictest3717425-21cpuspeed_turbos07:50:133
3674177991813,3cyclictest4020099-21gltestperf11:40:153
3674177991813,3cyclictest0-21swapper/309:00:003
3674177991811,5cyclictest3689842-21cut07:20:183
367417799181,13cyclictest3805458-21ssh09:15:123
367417799180,6cyclictest3877995-21ssh10:05:063
367417799180,16cyclictest3679048-21taskset07:13:473
3674177991713,2cyclictest0-21swapper/308:15:133
367417799171,2cyclictest0-21swapper/307:40:103
367417799171,14cyclictest3781948-21ntpq08:50:213
40984972161,4sleep03797944-21diskmemload12:29:410
39779652161,10sleep13978018-21ntpq11:10:211
38971902161,3sleep00-21swapper/010:15:400
38392132161,4sleep23839224-21ssh09:35:332
38123062161,3sleep23797944-21diskmemload09:19:582
37064702162,8sleep03706425-31find07:37:080
367417799168,5cyclictest3727525-21df_inode08:00:143
367417799167,6cyclictest3681821-21grep07:15:183
41127252151,6sleep016-21ksoftirqd/012:36:540
40894402151,3sleep03797944-21diskmemload12:23:190
40592382151,3sleep04060213-21kthreadcore12:05:160
40434112151,7sleep24043631-21hddtemp_smartct11:55:152
39867902151,10sleep23986800-21ssh11:16:352
38785572151,2sleep00-21swapper/010:05:180
38405352151,4sleep13840545-21ssh09:38:011
38121862151,2sleep10-21swapper/109:19:451
367417799151,8cyclictest3788531-21tail09:00:173
40482972141,3sleep14048308-21ssh11:56:191
40114512141,2sleep10-21swapper/111:34:381
40114512141,2sleep10-21swapper/111:34:381
39642132141,7sleep243-21ksoftirqd/211:01:242
39497082141,9sleep03949717-21sh10:52:360
37063562140,9sleep23706425-31find07:37:022
36791472141,9sleep13679382-21latency_hist07:15:011
40923332131,8sleep24093996-21grep12:25:182
40878792131,8sleep14087891-21sh12:21:001
40795802131,8sleep14079582-21ssh12:15:431
40732652131,8sleep24073271-21ssh12:13:152
40359402132,6sleep24037447-21cut11:50:222
39721712137,3sleep03972182-21sh11:07:190
38985132131,6sleep23797944-21diskmemload10:18:102
38900412137,3sleep1331ktimers/110:10:521
37111342130,3sleep016-21ksoftirqd/007:40:200
41126622121,6sleep23797944-21diskmemload12:36:482
39659842121,4sleep10-21swapper/111:04:311
39561722121,3sleep10-21swapper/110:55:421
38938562121,7sleep13895565-21grep10:15:211
38765222120,1sleep10-21swapper/110:02:581
38637162121,7sleep03863726-21ssh09:55:070
38603982120,3sleep016-21ksoftirqd/009:50:170
38562582121,7sleep23856282-21grep09:50:002
38462132121,7sleep13846272-21ntpq09:40:211
38320462120,1sleep20-21swapper/209:31:002
41132462111,2sleep10-21swapper/112:37:401
41037912111,3sleep20-21swapper/212:30:392
40951872110,1sleep10-21swapper/112:25:251
40823862111,3sleep20-21swapper/212:19:492
40639832110,6sleep13797944-21diskmemload12:06:291
40324722111,3sleep20-21swapper/211:46:012
40167422111,3sleep20-21swapper/211:35:392
40120712111,4sleep04012094-21apt-get11:35:130
39889082110,1sleep10-21swapper/111:20:151
39827272111,6sleep03983559-21kthreadcore11:15:170
39714512111,3sleep10-21swapper/111:06:001
39338132111,2sleep00-21swapper/010:40:530
39290202116,2sleep10-21swapper/110:40:141
39048732111,6sleep23797944-21diskmemload10:21:312
38773442111,3sleep00-21swapper/010:04:250
38263482111,3sleep10-21swapper/109:28:511
38192252111,3sleep10-21swapper/109:24:141
38192252111,3sleep10-21swapper/109:24:141
38036092111,3sleep10-21swapper/109:11:551
37478662112,5sleep13747885-21apt-get08:20:131
37463152111,5sleep13746361-21ntpq08:15:201
36967242111,5sleep20-21swapper/207:30:122
36868242111,6sleep03688008-21kthreadcore07:20:170
41038182101,4sleep03797944-21diskmemload12:30:410
40723652100,3sleep10-21swapper/112:11:581
40562742101,4sleep03797944-21diskmemload12:01:390
40510012101,4sleep14051025-21apt-get12:00:111
40470712100,6sleep04047130-21ssh11:55:210
40344502101,4sleep03797944-21diskmemload11:49:080
40250152101,4sleep03797944-21diskmemload11:41:340
40245092100,6sleep14024519-21ssh11:40:471
40235742101,4sleep24023634-21ntpq11:40:202
40094392101,2sleep00-21swapper/011:31:240
40094392101,2sleep00-21swapper/011:31:240
39944502100,3sleep00-21swapper/011:22:100
39934852100,1sleep20-21swapper/211:20:322
39863222101,3sleep10-21swapper/111:15:421
39561942101,2sleep00-21swapper/010:55:450
39496072101,3sleep20-21swapper/210:52:262
39485212100,3sleep10-21swapper/110:50:321
39338352101,3sleep20-21swapper/210:40:552
39256372101,4sleep03797944-21diskmemload10:35:240
39200662101,3sleep00-21swapper/010:32:520
39189662101,3sleep10-21swapper/110:30:591
39108952100,6sleep20-21swapper/210:25:202
39108952100,6sleep20-21swapper/210:25:202
39061162101,6sleep03906121-21ssh10:23:500
39048212101,4sleep13797944-21diskmemload10:21:261
38920212100,4sleep03797944-21diskmemload10:14:440
38771712101,2sleep20-21swapper/210:04:072
38634122101,5sleep13863435-21munin-run09:55:001
38543482100,3sleep10-21swapper/109:46:271
38487832100,2sleep23848793-21ssh09:44:472
38264442101,3sleep20-21swapper/209:29:042
38251992101,4sleep03797944-21diskmemload09:26:410
38179212101,2sleep00-21swapper/009:22:000
38179212101,2sleep00-21swapper/009:22:000
38097832101,3sleep00-21swapper/009:15:270
38031062101,3sleep20-21swapper/209:10:562
37426142101,4sleep03742633-21apt-get08:15:110
37006552100,6sleep03696725-21/usr/sbin/munin07:30:230
36914292101,6sleep13691566-21cpuspeed_turbos07:25:151
4087933291,3sleep20-21swapper/212:21:092
4074852291,3sleep04074876-21apt-get12:15:140
4072144291,2sleep00-21swapper/012:11:370
4064320291,2sleep20-21swapper/212:06:582
4040936291,4sleep04040949-21ssh11:52:050
4017242291,3sleep13797944-21diskmemload11:36:261
4008203291,4sleep24008265-21ntpq11:30:222
4008203291,4sleep24008265-21ntpq11:30:222
3979882291,2sleep20-21swapper/211:12:342
3966244291,2sleep20-21swapper/211:05:012
3957395291,3sleep23797944-21diskmemload10:57:512
3941945291,3sleep03797944-21diskmemload10:47:070
3941915290,6sleep13941928-21rm10:47:041
3941083291,3sleep23797944-21diskmemload10:45:402
3912688291,3sleep10-21swapper/110:27:271
3912688291,3sleep10-21swapper/110:27:271
3909461291,4sleep03910759-21memory10:25:180
3909461291,4sleep03910759-21memory10:25:170
3884785291,4sleep23884790-21ssh10:09:372
3882666291,3sleep10-21swapper/110:05:351
3869978290,2sleep10-21swapper/109:59:211
3868343291,3sleep20-21swapper/209:55:342
3856620291,4sleep23856786-21diskstats09:50:122
3849226290,4sleep03797944-21diskmemload09:45:090
3840316291,2sleep00-21swapper/009:37:360
3817227291,3sleep23797944-21diskmemload09:20:302
3817227291,3sleep23797944-21diskmemload09:20:302
3787878291,3sleep03787950-21munin-run09:00:000
3778376291,5sleep13778548-21cut08:50:151
3773206291,3sleep00-21swapper/008:45:130
3768266291,3sleep23768412-21df08:40:142
3763628291,4sleep03765067-21kthreadcore08:35:150
3752854291,3sleep23752873-21apt-get08:25:152
3746363291,4sleep23746416-21ntp_states08:15:212
3741328291,3sleep13741374-21ntpq08:10:191
3737788291,3sleep03738000-21hddtemp_smartct08:10:140
3732364291,4sleep03732516-21df_inode08:05:160
3722494291,4sleep13722686-21cut07:55:161
3705615291,4sleep13705662-21ntpq07:35:211
3695165291,4sleep03695206-21ntpq07:25:220
4057249281,2sleep20-21swapper/212:03:132
4032990280,2sleep10-21swapper/111:46:531
4001329281,3sleep24001339-21ssh11:25:572
3996854281,3sleep13996979-21ssh11:25:161
3978889280,1sleep00-21swapper/011:10:480
3964809280,2sleep00-21swapper/011:02:260
3926932281,3sleep10-21swapper/110:36:491
3925524281,3sleep20-21swapper/210:35:222
3889192281,3sleep20-21swapper/210:10:212
3842248281,3sleep00-21swapper/009:40:120
3833172281,3sleep03797944-21diskmemload09:33:070
3827721282,3sleep13828237-21kthreadcore09:30:151
3804787280,1sleep00-21swapper/009:14:270
3797244281,3sleep13797305-21sed09:05:261
3793181280,2sleep00-21swapper/009:05:130
3788565281,3sleep23788166-21/usr/sbin/munin09:00:182
3782908281,3sleep23782997-21date08:55:012
3731094281,3sleep13731144-21ntpq08:00:211
3712347281,3sleep13712490-21df07:45:151
3707268280,2sleep20-21swapper/207:40:112
3674384280,2sleep20-21swapper/207:10:132
3783169270,2sleep20-21swapper/208:55:132
3768244270,2sleep10-21swapper/108:40:131
3756355270,2sleep03756535-21strings08:25:210
3717016270,1sleep20-21swapper/207:49:482
4100882261,2sleep10-21swapper/112:30:181
3797935260,2sleep23797944-21diskmemload09:10:002
3788352261,2sleep10-21swapper/109:00:171
3779829261,2sleep00-21swapper/008:50:180
3765068261,2sleep10-21swapper/108:35:161
3758123260,2sleep20-21swapper/208:30:122
3758092260,2sleep00-21swapper/008:30:020
3752848260,1sleep10-21swapper/108:25:101
3737069261,2sleep20-21swapper/208:10:002
3731951261,2sleep20-21swapper/208:03:312
3731870260,1sleep00-21swapper/008:02:290
3721171261,2sleep00-21swapper/007:50:230
3721070260,2sleep20-21swapper/207:50:212
3712749261,2sleep03713158-21kthreadcore07:45:180
3696184261,2sleep20-21swapper/207:30:012
3686542261,2sleep20-21swapper/207:20:152
3685166260,3sleep10-21swapper/107:15:251
3682505261,2sleep00-21swapper/007:15:200
3679384261,2sleep20-21swapper/207:15:022
3786737250,1sleep10-21swapper/108:55:201
3774731250,2sleep10-21swapper/108:45:181
3771842250,1sleep00-21swapper/008:40:200
3741376250,1sleep20-21swapper/208:10:202
3732517250,2sleep10-21swapper/108:05:181
3722338250,1sleep00-21swapper/007:55:150
3722299250,3sleep20-21swapper/207:55:012
3721012250,1sleep10-21swapper/107:50:201
3701542250,1sleep10-21swapper/107:35:011
3686386250,1sleep10-21swapper/107:20:131
3676792250,1sleep00-21swapper/007:10:180
3788250240,1sleep00-21swapper/009:00:160
3773382240,1sleep20-21swapper/208:45:152
3762897240,1sleep20-21swapper/208:35:012
3758111240,1sleep10-21swapper/108:30:031
3748106240,1sleep00-21swapper/008:20:160
3747844240,1sleep20-21swapper/208:20:022
3706752240,1sleep10-21swapper/107:40:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional