You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-10-15 - 01:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Oct 14, 2024 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18877472960,7sleep32197014-21PK-Backend10:30:303
19034392940,4sleep20-21swapper/211:15:212
162399870,12rtkit-daemon2197014-21PK-Backend11:50:400
162399840,13rtkit-daemon0-21swapper/011:35:590
162399820,5rtkit-daemon0-21swapper/112:02:441
19264312810,7sleep12197014-21PK-Backend12:20:121
162399760,66rtkit-daemon0-21swapper/307:09:373
181356427358,9sleep20-21swapper/207:07:202
162399730,6rtkit-daemon2197014-21PK-Backend10:13:512
162399730,6rtkit-daemon2197014-21PK-Backend10:13:502
162399700,5rtkit-daemon0-21swapper/209:48:482
181289426938,9sleep00-21swapper/007:05:210
181365426744,14sleep10-21swapper/107:08:251
162399610,9rtkit-daemon1622-21rtkit-daemon10:40:313
162399580,9rtkit-daemon1622-21rtkit-daemon11:05:563
162399580,8rtkit-daemon1622-21rtkit-daemon09:40:402
181393999578,14cyclictest1753-21gnome-shell08:20:463
162399570,8rtkit-daemon1622-21rtkit-daemon12:35:202
162399560,8rtkit-daemon1622-21rtkit-daemon12:36:453
162399560,8rtkit-daemon1622-21rtkit-daemon07:21:313
181393999554,14cyclictest1851840-21chrt08:51:373
181393999554,14cyclictest1851840-21chrt08:51:363
19118312540,19sleep234-21ksoftirqd/211:40:132
18565352540,5sleep2331rcuc/209:05:202
181393999548,16cyclictest1839653-21cron08:19:593
181393999531,39cyclictest42-21ksoftirqd/307:42:293
162399530,8rtkit-daemon1622-21rtkit-daemon09:13:283
162399530,7rtkit-daemon1622-21rtkit-daemon10:47:113
1813937995229,19cyclictest1753-21gnome-shell10:06:212
1813931995232,7cyclictest13-21ksoftirqd/008:15:130
162399520,8rtkit-daemon1622-21rtkit-daemon12:30:173
162399520,29rtkit-daemon0-21swapper/307:34:463
18472032510,6sleep31753-21gnome-shell08:40:133
1813937995027,19cyclictest1753-21gnome-shell12:25:282
181393799501,16cyclictest18472052grep08:40:132
162399500,42rtkit-daemon1832569-21munin-run08:00:012
162399500,26rtkit-daemon1889949-21StreamT~3
162399500,26rtkit-daemon1889949-21StreamT~3
181393799491,15cyclictest1855382-21systemd-udevd09:01:242
181393199495,15cyclictest1617-10pipewire-media-10:30:120
1813939994810,14cyclictest1753-21gnome-shell07:45:163
1813937994824,20cyclictest1753-21gnome-shell10:33:332
1813937994824,20cyclictest1753-21gnome-shell07:15:232
162399480,35rtkit-daemon2197014-21PK-Backend08:33:170
19143372470,5sleep0161rcuc/011:45:250
18421902470,3sleep31842237-21irqstats08:25:213
1813939994710,13cyclictest42-21ksoftirqd/311:43:083
181393199479,13cyclictest0-21swapper/010:46:400
162399470,6rtkit-daemon1622-21rtkit-daemon11:02:562
162399470,32rtkit-daemon1874438-21cat09:55:143
162399470,32rtkit-daemon1753-21gnome-shell12:03:200
181393999469,19cyclictest1753-21gnome-shell07:10:313
181393999468,14cyclictest1879492-21awk10:10:003
181393199461,6cyclictest141rcu_preempt08:45:170
18817122450,6chrt0-21swapper/310:15:123
181393999458,8cyclictest1753-21gnome-shell11:53:113
181393999453,17cyclictest1753-21gnome-shell09:27:063
1813931994528,13cyclictest13-21ksoftirqd/009:35:030
162399450,32rtkit-daemon1753-21gnome-shell09:53:252
18890552440,8chrt1889054-21fw_packets10:35:191
18890552440,8chrt1889054-21fw_packets10:35:191
181393999447,12cyclictest1913963-21/usr/sbin/munin11:45:183
181393999447,11cyclictest1753-21gnome-shell08:36:263
181393999445,10cyclictest2197014-21PK-Backend07:51:393
1813939994410,30cyclictest1753-21gnome-shell08:32:143
1813937994426,13cyclictest438-20systemd-journal10:20:192
1813937994421,19cyclictest1753-21gnome-shell07:10:282
1813933994418,22cyclictest1851010-21sh08:50:151
1813933994418,22cyclictest1851010-21sh08:50:151
181393199446,5cyclictest141rcu_preempt08:02:080
1813931994434,5cyclictest13-21ksoftirqd/010:40:130
162399440,35rtkit-daemon1813926-21cyclictest09:29:562
181393999439,11cyclictest1753-21gnome-shell09:16:433
181393999438,18cyclictest1753-21gnome-shell09:43:383
181393999438,12cyclictest854-21avahi-daemon10:05:013
181393999438,12cyclictest1753-21gnome-shell11:19:023
181393999437,11cyclictest438-20systemd-journal08:05:013
181393999436,15cyclictest1835838-21taskset08:07:173
1813937994318,20cyclictest1753-21gnome-shell11:50:092
181393399439,21cyclictest1753-21gnome-shell10:05:421
181393399437,25cyclictest141rcu_preempt12:35:261
1813933994331,9cyclictest26-21ksoftirqd/109:01:241
1813933994320,13cyclictest1836729-21sh08:10:151
1813931994331,8cyclictest1753-21gnome-shell07:47:220
1813931994331,7cyclictest1753-21gnome-shell07:13:440
1813931994324,15cyclictest13-21ksoftirqd/011:40:110
181393199431,6cyclictest141rcu_preempt07:30:330
181393999429,11cyclictest1753-21gnome-shell11:58:463
181393999424,15cyclictest1837192-21processes08:10:273
181393999423,21cyclictest1753-21gnome-shell08:59:223
181393999423,18cyclictest1753-21gnome-shell10:30:003
181393999423,12cyclictest1753-21gnome-shell09:49:433
1813939994227,11cyclictest1861838-21irqstats09:20:213
181393399427,5cyclictest141rcu_preempt09:33:161
1813933994227,11cyclictest1827209-21/usr/sbin/munin07:45:001
1813933994225,13cyclictest1841144-21fwupdmgr08:21:201
181393399422,24cyclictest3679623-21Isolated14:31:441
1813933994215,15cyclictest26-21ksoftirqd/109:54:191
1813933994210,15cyclictest26-21ksoftirqd/110:55:021
181393199429,19cyclictest141rcu_preempt09:35:000
1813931994231,7cyclictest1753-21gnome-shell11:30:460
1813931994230,8cyclictest1753-21gnome-shell11:00:570
1813931994224,6cyclictest13-21ksoftirqd/011:25:200
1813931994223,6cyclictest13-21ksoftirqd/012:10:360
1813931994223,14cyclictest13-21ksoftirqd/011:55:250
1813931994222,15cyclictest13-21ksoftirqd/012:20:090
1813931994215,6cyclictest141rcu_preempt10:30:010
1813931994215,5cyclictest141rcu_preempt07:55:210
181393999419,10cyclictest1753-21gnome-shell08:45:113
181393999418,29cyclictest1753-21gnome-shell07:15:113
181393999417,17cyclictest1855393-21colord-sane09:01:253
181393999417,12cyclictest1753-21gnome-shell09:31:443
181393999413,10cyclictest42-21ksoftirqd/309:10:013
1813939994110,10cyclictest1753-21gnome-shell11:30:113
1813933994128,10cyclictest1839680-21awk08:20:001
1813933994126,11cyclictest1924105-21chrt12:13:281
1813933994125,11cyclictest904-21systemd-logind07:23:191
1813933994124,14cyclictest1876549-21grep10:00:221
1813933994118,7cyclictest1848557-21StreamT~1
181393199414,17cyclictest13-21ksoftirqd/011:22:510
181393199412,9cyclictest1753-21gnome-shell10:13:240
181393199412,9cyclictest1753-21gnome-shell10:13:240
1813931994129,8cyclictest1753-21gnome-shell10:00:200
1813931994129,7cyclictest1753-21gnome-shell12:06:410
1813931994128,8cyclictest1753-21gnome-shell11:16:530
1813931994128,8cyclictest1753-21gnome-shell07:36:510
1813931994126,11cyclictest1870119-21sshd09:43:190
1813931994123,7cyclictest13-21ksoftirqd/009:49:030
1813931994123,6cyclictest13-21ksoftirqd/009:50:260
1813931994123,6cyclictest13-21ksoftirqd/008:50:200
1813931994123,6cyclictest13-21ksoftirqd/008:50:190
1813931994122,6cyclictest13-21ksoftirqd/010:50:240
1813931994122,6cyclictest13-21ksoftirqd/008:28:080
1813931994122,14cyclictest13-21ksoftirqd/010:55:030
1813931994122,14cyclictest13-21ksoftirqd/010:55:030
1813931994122,14cyclictest13-21ksoftirqd/008:20:170
162399410,5rtkit-daemon0-21swapper/108:56:211
18206242400,6chrt2197014-21PK-Backend07:25:233
181393999407,10cyclictest1910363-21df11:35:143
181393999406,8cyclictest42-21ksoftirqd/312:00:483
181393999405,11cyclictest42-21ksoftirqd/310:53:453
181393799401,15cyclictest19264362cut12:20:122
181393799401,14cyclictest1069-21runrttasks09:22:572
181393399408,4cyclictest141rcu_preempt10:21:441
1813933994025,11cyclictest26-21ksoftirqd/111:40:301
1813933994015,8cyclictest26-21ksoftirqd/112:25:511
1813933994014,6cyclictest141rcu_preempt12:05:141
1813933994014,6cyclictest141rcu_preempt11:05:151
1813933994013,22cyclictest1828935-21StreamT~1
1813933994013,12cyclictest26-21ksoftirqd/111:36:031
181393399401,10cyclictest141rcu_preempt09:55:291
1813931994028,8cyclictest13-21ksoftirqd/007:23:210
1813931994028,7cyclictest1753-21gnome-shell07:44:430
1813931994028,7cyclictest1753-21gnome-shell07:29:110
181393199402,7cyclictest1753-21gnome-shell10:20:520
1813931994023,5cyclictest13-21ksoftirqd/007:50:170
1813931994022,6cyclictest13-21ksoftirqd/011:10:140
1813931994022,6cyclictest13-21ksoftirqd/009:55:310
1813931994022,13cyclictest13-21ksoftirqd/009:00:010
1813931994022,13cyclictest13-21ksoftirqd/007:17:530
1813931994021,6cyclictest13-21ksoftirqd/011:10:010
181393199401,34cyclictest1861098-21turbostat09:20:010
162399400,26rtkit-daemon26-21ksoftirqd/110:48:551
181393999395,10cyclictest1896132-21df_inode10:55:173
181393999395,10cyclictest1896132-21df_inode10:55:173
181393999390,15cyclictest1904660-21apt11:20:113
1813937993923,11cyclictest438-20systemd-journal08:08:262
1813937993916,19cyclictest1753-21gnome-shell12:15:282
1813937993916,19cyclictest1753-21gnome-shell12:15:282
181393399397,14cyclictest26-21ksoftirqd/109:29:001
1813933993923,5cyclictest26-21ksoftirqd/108:09:071
181393399392,33cyclictest1753-21gnome-shell11:15:011
181393399392,33cyclictest1753-21gnome-shell08:28:211
1813933993923,13cyclictest1930891-21/usr/sbin/munin12:30:241
181393399392,15cyclictest18458112kernelversion08:35:211
1813933993913,14cyclictest26-21ksoftirqd/111:52:291
1813933993913,14cyclictest26-21ksoftirqd/109:20:241
1813933993912,14cyclictest26-21ksoftirqd/109:44:431
1813933993911,16cyclictest26-21ksoftirqd/110:57:001
1813933993911,16cyclictest26-21ksoftirqd/110:57:001
1813931993927,7cyclictest1753-21gnome-shell08:13:510
1813931993921,6cyclictest13-21ksoftirqd/010:20:000
1813931993921,5cyclictest13-21ksoftirqd/009:05:010
181393199392,12cyclictest1848302-21chrt08:41:430
1813931993921,13cyclictest13-21ksoftirqd/009:25:000
181393999385,10cyclictest854-21avahi-daemon10:21:353
181393999385,10cyclictest42-21ksoftirqd/307:36:393
181393999384,11cyclictest42-21ksoftirqd/312:07:013
181393999383,11cyclictest1753-21gnome-shell11:11:113
181393999383,10cyclictest1868183-40apt-check09:36:303
181393999382,13cyclictest1832496-21StreamT~3
181393999381,16cyclictest1873195-21gzip09:50:283
181393999381,14cyclictest1925651-21StreamT~3
181393999381,14cyclictest1925651-21StreamT~3
181393999381,14cyclictest1907764-21StreamT~3
181393999381,13cyclictest1929598-21StreamT~3
181393799383,32cyclictest54-21kcompactd011:46:192
1813937993823,11cyclictest1839369-21cron08:17:002
1813937993821,12cyclictest1897273-21/usr/sbin/munin11:00:032
1813937993821,12cyclictest1897273-21/usr/sbin/munin11:00:022
1813937993820,5cyclictest34-21ksoftirqd/210:48:452
1813937993819,15cyclictest3242253-21nautilus09:35:012
1813937993819,14cyclictest1923957-40apt-check12:12:192
1813937993818,13cyclictest3679623-21Isolated14:31:442
181393799381,33cyclictest1753-21gnome-shell08:33:022
181393399389,17cyclictest141rcu_preempt11:25:151
181393399386,14cyclictest26-21ksoftirqd/112:15:301
181393399386,14cyclictest26-21ksoftirqd/112:15:291
1813933993828,7cyclictest1069-21runrttasks10:44:071
1813933993823,10cyclictest26-21ksoftirqd/107:50:191
1813933993822,12cyclictest1903767-21switchtime11:15:291
1813933993819,14cyclictest1909087-21python311:30:291
181393399381,8cyclictest1850110-21StreamT~s1
1813933993813,13cyclictest26-21ksoftirqd/111:21:351
1813933993812,14cyclictest26-21ksoftirqd/108:34:231
1813933993811,11cyclictest26-21ksoftirqd/107:28:361
1813933993810,13cyclictest26-21ksoftirqd/110:30:111
1813933993810,13cyclictest26-21ksoftirqd/107:34:381
181393199382,7cyclictest1753-21gnome-shell12:28:560
1813931993827,7cyclictest1753-21gnome-shell09:13:240
1813931993821,5cyclictest13-21ksoftirqd/009:05:200
1813931993820,6cyclictest13-21ksoftirqd/012:30:170
1813931993816,6cyclictest141rcu_preempt10:10:020
1813931993810,13cyclictest1069-21runrttasks12:35:390
162399380,27rtkit-daemon2197014-21PK-Backend12:16:440
162399380,27rtkit-daemon2197014-21PK-Backend12:16:440
181393999377,16cyclictest42-21ksoftirqd/312:10:283
181393999376,10cyclictest2197014-21PK-Backend11:01:013
181393999370,13cyclictest1881036-21StreamT~3
181393999370,13cyclictest1881036-21StreamT~3
1813937993728,6cyclictest875-21polkitd08:20:452
1813937993726,7cyclictest1753-21gnome-shell11:22:502
1813937993719,13cyclictest3679623-21Isolated14:31:442
1813937993715,6cyclictest141rcu_preempt12:10:022
1813937993713,20cyclictest1753-21gnome-shell10:00:302
1813937993713,20cyclictest1753-21gnome-shell09:15:432
1813933993720,13cyclictest1885811-21ntpq10:25:241
1813933993720,13cyclictest1824618-21ntp_states07:35:241
1813933993720,12cyclictest438-20systemd-journal10:15:001
1813933993720,12cyclictest438-20systemd-journal10:15:001
1813933993713,8cyclictest26-21ksoftirqd/107:10:301
1813933993712,14cyclictest26-21ksoftirqd/111:00:131
1813933993712,13cyclictest26-21ksoftirqd/111:45:121
1813933993711,14cyclictest26-21ksoftirqd/107:15:341
1813933993711,11cyclictest26-21ksoftirqd/109:47:141
1813933993710,12cyclictest26-21ksoftirqd/107:57:121
1813931993727,6cyclictest1834559-21/usr/sbin/munin08:05:160
1813931993726,7cyclictest1753-21gnome-shell08:39:320
1813931993724,9cyclictest1753-21gnome-shell10:36:490
1813931993724,9cyclictest1753-21gnome-shell10:36:490
162399370,26rtkit-daemon26-21ksoftirqd/108:00:591
181393799365,15cyclictest141rcu_preempt08:05:012
1813937993623,8cyclictest3679588-21Privileged14:31:442
181393799362,30cyclictest1846834-21munin-run08:40:002
1813937993621,11cyclictest1931388-21anacron12:31:502
1813937993621,11cyclictest1866854-21dpkg09:35:032
1813937993618,13cyclictest904-21systemd-logind10:26:502
1813937993618,13cyclictest3679623-21Isolated14:31:442
181393399369,11cyclictest26-21ksoftirqd/109:10:001
1813933993620,12cyclictest1917851-21ntpq11:55:241
1813933993611,11cyclictest26-21ksoftirqd/109:16:111
1813933993610,13cyclictest26-21ksoftirqd/109:12:371
1813931993619,6cyclictest13-21ksoftirqd/009:25:480
18672872350,14sleep1251rcuc/109:35:131
1813939993515,15cyclictest3679623-21Isolated14:31:443
181393799352,30cyclictest34-21ksoftirqd/207:44:332
181393799352,16cyclictest1919637-21ntp_states12:00:262
1813937993519,12cyclictest1894735-21ntpq10:50:252
1813937993518,14cyclictest1753-21gnome-shell10:00:002
1813937993518,12cyclictest3679623-21Isolated14:31:442
1813937993518,12cyclictest3679623-21Isolated14:31:442
1813937993517,13cyclictest438-20systemd-journal09:00:012
1813937993514,16cyclictest3679623-21Isolated14:31:442
1813937993512,19cyclictest1822476-40lsb_release07:30:212
181393799351,12cyclictest2197014-21PK-Backend11:38:332
1813937993423,7cyclictest1907932-21cron11:30:002
1813937993422,8cyclictest1917961-21cut11:55:272
1813937993420,10cyclictest1836766-21sshd08:10:202
1813937993417,12cyclictest3679461-21IPC2
1813937993416,13cyclictest1850007-21StreamT~2
1813937993413,5cyclictest141rcu_preempt10:35:322
1813937993413,5cyclictest141rcu_preempt10:35:312
162399340,27rtkit-daemon1753-21gnome-shell10:42:212
18253522332,6sleep2859-21NetworkManager07:39:162
1813937993320,9cyclictest1753-21gnome-shell11:10:212
1813937993319,10cyclictest1882109-21wc10:15:212
1813937993310,19cyclictest1753-21gnome-shell07:49:382
1813937993215,12cyclictest1830556-21StreamT~2
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional