You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-13 - 02:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Celeron 33455 @1500 MHz, Linux 5.15.2-rt19 (Profile)

Latency plot of shadow in rack #c, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Aug 12, 2022 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
61696922910,11sleep1614430-21kworker/1:207:05:211
66422021080,5sleep30-21swapper/309:20:203
617838210057,14sleep30-21swapper/307:09:573
61752029681,9sleep00-21swapper/007:06:010
162399960,76rtkit-daemon0-21swapper/207:08:512
6305852950,5sleep1630589-21turbostat07:45:001
162399840,9rtkit-daemon1622-21rtkit-daemon10:37:532
7132232810,5sleep3411rcuc/311:40:233
6623442810,4sleep10-21swapper/109:15:241
617986997714,32cyclictest699316-21if_err_enp2s011:00:282
61798699761,53cyclictest628200-21python307:35:282
61798699761,49cyclictest656683-21taskset08:59:482
162399760,9rtkit-daemon1622-21rtkit-daemon08:48:120
7147842740,4sleep30-21swapper/311:45:223
6643512740,5sleep1251rcuc/109:20:271
617986997416,40cyclictest1069-21runrttasks08:12:442
617986997318,38cyclictest1069-21runrttasks11:48:382
61798699731,43cyclictest631970-21taskset07:45:472
6397642710,4sleep30-21swapper/308:10:143
61798699703,43cyclictest961-21snmpd10:55:052
61798699701,48cyclictest675063-21/usr/sbin/munin09:50:332
61798699701,40cyclictest670486-21taskset09:39:042
61798699692,47cyclictest725955-21/usr/sbin/munin12:15:272
61798699692,37cyclictest664297-21diskstats09:20:242
61798699691,41cyclictest657668-21taskset09:00:232
617986996815,38cyclictest1069-21runrttasks12:21:312
617986996813,30cyclictest1069-21runrttasks12:26:072
61798699672,39cyclictest638869-21users08:05:312
61798699672,38cyclictest683211-40apt-check10:15:252
61798699662,38cyclictest1753-21gnome-shell10:03:102
61798699662,35cyclictest1753-21gnome-shell08:02:112
617986996619,27cyclictest1827-21JS14:31:442
61798699661,60cyclictest884-21gdbus11:08:022
61798699661,37cyclictest961-21snmpd09:10:292
6395102650,4sleep00-21swapper/008:10:110
61798699652,37cyclictest1753-21gnome-shell11:19:382
61798699652,36cyclictest724692-21taskset12:13:332
61798699651,39cyclictest652892-21taskset08:47:212
61798699651,38cyclictest646960-21/usr/sbin/munin08:30:202
61798699642,33cyclictest713218-21irqstats11:40:212
61798699641,37cyclictest961-21snmpd07:34:112
61798699632,36cyclictest1753-21gnome-shell10:51:422
61798699632,32cyclictest676787-21ntp_states09:55:282
617986996322,36cyclictest438-20systemd-journal10:45:012
617986996322,36cyclictest438-20systemd-journal10:45:002
617986996310,31cyclictest1069-21runrttasks08:22:442
61798699622,37cyclictest717067-21/usr/sbin/munin11:50:292
61798699622,34cyclictest730407-21/usr/sbin/munin12:30:242
61798699621,39cyclictest1069-21runrttasks09:33:382
61798699621,35cyclictest719416-21systemd12:00:012
61798699621,35cyclictest719416-21systemd12:00:002
61798699621,34cyclictest673445-21/usr/sbin/munin09:45:292
61798699621,32cyclictest859-21NetworkManager09:07:292
61798699612,34cyclictest666731-21users09:25:342
61798699611,40cyclictest961-21snmpd07:19:362
61798699611,38cyclictest682508-21taskset10:10:552
61798699611,37cyclictest709284-21apt11:30:192
61798699611,37cyclictest709284-21apt11:30:192
61798699611,34cyclictest961-21snmpd10:06:482
61798199610,56cyclictest1622-21rtkit-daemon08:40:471
617991996010,46cyclictest1622-21rtkit-daemon12:28:173
61798699602,34cyclictest961-21snmpd09:15:472
61798699602,33cyclictest634698-21df_inode07:55:162
61798699602,32cyclictest686723-21apt10:25:152
61798699601,35cyclictest720812-21taskset12:01:102
61798699601,33cyclictest733172-21sed12:35:332
61798199606,29cyclictest1753-21gnome-shell08:35:201
61798699592,33cyclictest711958-21/usr/sbin/munin11:35:312
617986995919,36cyclictest632869-21expr07:50:142
617986995919,36cyclictest632869-21expr07:50:142
617991995822,8cyclictest42-21ksoftirqd/310:52:483
61798699582,32cyclictest1753-21gnome-shell11:23:502
61798699581,33cyclictest961-21snmpd09:41:142
61798699581,32cyclictest648158-21/usr/sbin/munin08:35:222
61798699573,30cyclictest721259-40unattended-upgr12:05:082
617978995724,28cyclictest681444-21ls10:10:160
61798699562,31cyclictest703428-21/usr/sbin/munin11:10:372
61798199564,28cyclictest672524-21date09:45:011
162399560,8rtkit-daemon1622-21rtkit-daemon10:30:312
61798699556,25cyclictest34-21ksoftirqd/207:24:012
617991995416,19cyclictest42-21ksoftirqd/312:20:313
617986995410,39cyclictest1753-21gnome-shell07:10:082
61798199545,26cyclictest644464-21sh08:25:001
617991995315,20cyclictest42-21ksoftirqd/311:08:013
617986995321,29cyclictest1753-21gnome-shell11:27:292
61797899532,31cyclictest1753-21gnome-shell11:43:590
617978995317,30cyclictest698605-21wc11:00:020
617991995242,7cyclictest411rcuc/312:15:123
61798199525,16cyclictest1753-21gnome-shell09:39:051
61797899522,45cyclictest1753-21gnome-shell09:13:040
617978995216,31cyclictest855-21dbus-daemon08:05:010
617978995212,35cyclictest855-21dbus-daemon08:10:010
61799199515,9cyclictest42-21ksoftirqd/312:01:023
617991995141,7cyclictest411rcuc/312:06:213
617991995117,9cyclictest42-21ksoftirqd/310:00:383
617991995115,5cyclictest141rcu_preempt07:53:273
617991995115,5cyclictest141rcu_preempt07:53:273
617991995115,10cyclictest42-21ksoftirqd/311:10:323
617991995113,16cyclictest42-21ksoftirqd/310:57:343
617986995122,25cyclictest1753-21gnome-shell08:18:332
61798199516,26cyclictest1753-21gnome-shell09:31:051
617978995117,29cyclictest1753-21gnome-shell08:56:270
617978995113,32cyclictest641726-21uname08:15:210
6899172500,6chrt689918-21latency_hist10:35:001
617991995041,6cyclictest411rcuc/308:19:443
617991995015,20cyclictest42-21ksoftirqd/309:10:173
617986995021,25cyclictest1753-21gnome-shell08:52:012
617986995018,28cyclictest625332-21latency_hist07:30:022
61798199508,26cyclictest1753-21gnome-shell09:07:521
61797899509,36cyclictest1753-21gnome-shell09:06:140
61797899509,36cyclictest1753-21gnome-shell07:54:580
61797899509,36cyclictest1753-21gnome-shell07:54:570
617978995018,27cyclictest716684-21cut11:50:200
617978995017,29cyclictest675871-21sed09:55:020
617978995016,29cyclictest1753-21gnome-shell12:21:290
617978995016,28cyclictest438-20systemd-journal09:43:110
61797899501,44cyclictest681096-21turbostat10:10:000
617978995013,33cyclictest657537-21wc09:00:220
61799199498,19cyclictest42-21ksoftirqd/309:57:233
617991994916,8cyclictest42-21ksoftirqd/311:03:183
617991994913,20cyclictest42-21ksoftirqd/310:40:013
617986994918,26cyclictest685143-21tr10:20:182
617986994918,26cyclictest685143-21tr10:20:182
61798199495,17cyclictest1753-21gnome-shell07:35:241
61797899499,36cyclictest1753-21gnome-shell12:12:260
61797899499,36cyclictest1753-21gnome-shell08:39:480
61797899494,40cyclictest667801-21idleruntime09:30:280
617978994919,25cyclictest719425-21ls12:00:010
617978994919,25cyclictest719425-21ls12:00:000
617978994915,29cyclictest676568-21expr09:55:250
6925002480,7sleep01358-1kworker/u9:110:40:230
6925002480,7sleep01358-1kworker/u9:110:40:220
617991994822,8cyclictest42-21ksoftirqd/307:47:433
617991994814,19cyclictest42-21ksoftirqd/308:34:013
617991994813,17cyclictest42-21ksoftirqd/308:35:223
61798199485,19cyclictest1753-21gnome-shell07:25:001
61797899488,16cyclictest1753-21gnome-shell07:31:410
61797899484,36cyclictest1753-21gnome-shell10:15:150
617978994817,27cyclictest731007-21sed12:30:270
61797899481,41cyclictest1753-21gnome-shell12:05:050
617978994812,30cyclictest700888-21expr11:05:160
61797899481,14cyclictest6654142grep09:25:010
61799199478,19cyclictest42-21ksoftirqd/307:43:333
61799199477,11cyclictest42-21ksoftirqd/307:20:343
617991994727,8cyclictest42-21ksoftirqd/309:51:553
617991994716,8cyclictest42-21ksoftirqd/312:35:313
617991994715,9cyclictest42-21ksoftirqd/309:41:113
617991994715,17cyclictest42-21ksoftirqd/308:02:103
617991994714,10cyclictest42-21ksoftirqd/310:23:403
617991994714,10cyclictest42-21ksoftirqd/310:23:393
617991994711,19cyclictest42-21ksoftirqd/309:25:333
617986994713,30cyclictest693856-21cstates10:45:152
61797899479,33cyclictest1753-21gnome-shell12:15:250
61797899479,32cyclictest1753-21gnome-shell08:50:120
61797899478,18cyclictest1753-21gnome-shell11:36:480
61797899472,40cyclictest651373-21cron08:45:010
61797899472,10cyclictest1753-21gnome-shell10:50:220
617978994718,24cyclictest693760-21cpuspeed_turbos10:45:150
61797899471,41cyclictest1753-21gnome-shell07:45:130
61797899471,41cyclictest1753-21gnome-shell07:37:240
617978994712,29cyclictest690876-21netstat10:35:240
617978994710,32cyclictest730285-21date12:30:020
61799199467,18cyclictest42-21ksoftirqd/308:24:303
61799199467,10cyclictest42-21ksoftirqd/310:15:303
61799199465,23cyclictest42-21ksoftirqd/308:51:063
617991994637,6cyclictest411rcuc/307:27:363
617991994614,9cyclictest42-21ksoftirqd/310:05:313
617991994613,10cyclictest42-21ksoftirqd/311:32:083
617991994613,10cyclictest42-21ksoftirqd/311:32:083
61799199461,17cyclictest1753-21gnome-shell11:55:123
61799199461,17cyclictest1753-21gnome-shell11:55:123
617991994611,19cyclictest42-21ksoftirqd/308:09:543
617991994610,14cyclictest42-21ksoftirqd/312:35:013
61798699461,41cyclictest0-21swapper/208:42:112
61798699461,40cyclictest1753-21gnome-shell08:26:112
61798199463,20cyclictest680238-21cat10:05:261
61797899462,39cyclictest721263-21cron12:05:000
617978994618,25cyclictest1753-21gnome-shell11:20:290
617978994616,24cyclictest672557-21acpi09:45:120
617978994614,28cyclictest1753-21gnome-shell08:27:340
617978994610,32cyclictest1753-21gnome-shell07:43:190
6450472450,8sleep3411rcuc/308:25:143
61799199459,20cyclictest42-21ksoftirqd/311:15:253
61799199458,21cyclictest42-21ksoftirqd/311:40:003
61799199458,13cyclictest42-21ksoftirqd/309:03:093
61799199457,10cyclictest42-21ksoftirqd/309:45:203
61799199456,10cyclictest42-21ksoftirqd/311:26:533
61799199456,10cyclictest42-21ksoftirqd/307:35:253
61799199454,9cyclictest0-21swapper/308:55:253
617991994534,8cyclictest411rcuc/309:35:093
617991994513,18cyclictest42-21ksoftirqd/310:33:453
617991994513,17cyclictest42-21ksoftirqd/307:15:113
617991994510,20cyclictest42-21ksoftirqd/308:43:343
61797899458,33cyclictest1753-21gnome-shell10:33:350
61797899458,33cyclictest1753-21gnome-shell08:34:080
61797899458,17cyclictest1753-21gnome-shell09:19:380
61797899458,17cyclictest1753-21gnome-shell07:21:570
61797899454,36cyclictest709494-21cstates11:30:210
61797899454,36cyclictest709494-21cstates11:30:210
617978994515,26cyclictest1753-21gnome-shell07:26:050
617978994510,31cyclictest1753-21gnome-shell10:27:580
617978994510,31cyclictest1753-21gnome-shell07:56:580
61799199449,19cyclictest42-21ksoftirqd/309:15:273
61799199447,15cyclictest42-21ksoftirqd/307:59:253
61799199446,10cyclictest42-21ksoftirqd/311:50:243
61799199445,9cyclictest42-21ksoftirqd/310:10:303
61799199445,16cyclictest42-21ksoftirqd/311:21:263
61799199445,10cyclictest42-21ksoftirqd/309:05:313
61799199441,13cyclictest42-21ksoftirqd/307:35:003
61798199445,17cyclictest699145-21cstates11:00:211
617981994419,10cyclictest26-21ksoftirqd/107:30:441
617981994414,25cyclictest631939-21vmstat07:45:301
617981994411,28cyclictest657899-21switchtime09:00:301
61798199441,10cyclictest1762-21gvfs-gphoto2-vo11:08:011
61797899448,17cyclictest1753-21gnome-shell11:49:590
617978994423,17cyclictest679264-21unin-run10:05:010
617978994418,21cyclictest618393-21cstates07:10:150
617978994417,23cyclictest704740-21grep11:15:250
617978994414,26cyclictest1753-21gnome-shell09:35:240
61797899441,39cyclictest1753-21gnome-shell11:12:360
61797899441,38cyclictest1753-21gnome-shell09:27:420
61797899441,38cyclictest1753-21gnome-shell07:16:370
617978994413,26cyclictest0-21swapper/008:20:190
617978994410,30cyclictest1753-21gnome-shell10:22:140
617978994410,30cyclictest1753-21gnome-shell10:22:140
617991994334,6cyclictest411rcuc/310:26:253
617991994312,9cyclictest42-21ksoftirqd/307:10:143
617981994330,8cyclictest682826-21cron10:15:011
617978994316,23cyclictest732778-21missed_timers12:35:250
617978994310,29cyclictest1753-21gnome-shell11:25:400
61799199424,13cyclictest42-21ksoftirqd/310:46:313
61799199424,13cyclictest42-21ksoftirqd/309:31:473
617991994213,11cyclictest42-21ksoftirqd/308:45:253
61798199421,37cyclictest708980-21cron11:30:001
61798199421,25cyclictest714446-21/usr/sbin/munin11:45:261
617981994212,25cyclictest704734-21grep11:15:251
617978994210,27cyclictest699046-21cstates11:00:210
61798199416,11cyclictest1753-21gnome-shell12:04:181
61798199414,20cyclictest687075-21expr10:25:171
61798199414,12cyclictest1753-21gnome-shell08:11:041
61798199413,22cyclictest0-21swapper/110:22:441
61798199413,22cyclictest0-21swapper/110:22:441
61798199413,16cyclictest0-21swapper/108:51:131
617981994110,16cyclictest621573-21chrt07:16:271
6971962400,5chrt0-21swapper/110:55:121
61798199407,11cyclictest0-21swapper/107:25:221
61798199406,11cyclictest695855-21irqstats10:50:221
61798199402,33cyclictest1753-21gnome-shell12:09:011
61798199401,35cyclictest1622-21rtkit-daemon07:54:521
61798199401,35cyclictest1622-21rtkit-daemon07:54:511
617981994011,25cyclictest702516-21grep11:10:201
617981994011,25cyclictest636592-21cat08:00:221
61799199391,13cyclictest42-21ksoftirqd/310:40:133
61799199391,13cyclictest42-21ksoftirqd/310:40:133
61799199391,11cyclictest724697-21taskset12:13:373
61798199399,25cyclictest665621-21/usr/sbin/munin09:25:171
61798199396,9cyclictest0-21swapper/108:59:371
61798199396,10cyclictest0-21swapper/109:53:171
617981993913,14cyclictest26-21ksoftirqd/109:49:461
617981993912,10cyclictest618552-21df_inode07:10:171
617981993910,25cyclictest646916-21interrupts08:30:201
61798199386,10cyclictest729096-21hddtemp_smartct12:25:231
61798199384,13cyclictest0-21swapper/110:35:171
617981993823,11cyclictest639379-21cut08:10:021
617981993819,14cyclictest676413-21df_inode09:55:201
617981993813,22cyclictest694686-21timerwakeupswit10:45:321
617981993812,22cyclictest679343-21appstreamcli10:05:001
617981993812,21cyclictest683211-40apt-check10:15:191
617981993720,13cyclictest713590-21rt-features11:40:291
617981993720,12cyclictest645933-21sshd08:27:131
617981993715,17cyclictest635409-21users07:55:311
617981993711,21cyclictest661325-21python309:10:291
61798199369,23cyclictest716834-21sed11:50:221
61798199369,23cyclictest693187-21taskset10:42:501
61798199369,23cyclictest693187-21taskset10:42:501
61798199368,11cyclictest26-21ksoftirqd/112:10:181
61798199366,8cyclictest0-21swapper/108:46:501
617981993625,7cyclictest1753-21gnome-shell12:15:171
617981993622,10cyclictest730590-21cpuspeed_turbos12:30:181
617981993620,12cyclictest728188-21seq12:22:091
617981993611,20cyclictest641994-21ntp_states08:15:251
617981993610,11cyclictest1753-21gnome-shell12:39:001
617981993520,11cyclictest712528-21cat11:40:021
617981993516,7cyclictest1069-21runrttasks11:21:161
617981993420,10cyclictest718222-21expr11:55:211
617981993420,10cyclictest718222-21expr11:55:211
617981993415,14cyclictest710033-21ntpq11:30:341
617981993415,14cyclictest710033-21ntpq11:30:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional