You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-24 - 16:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Apr 24, 2024 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
436252210351,43sleep30-21swapper/307:06:343
43665899982,72cyclictest1753-21gnome-shell11:05:130
43665899972,90cyclictest547463-21memory12:35:230
43665899972,90cyclictest547463-21memory12:35:230
436658999685,7cyclictest1753-21gnome-shell12:33:240
43665899962,89cyclictest510750-21diskstats10:50:200
5127422950,5sleep3411rcuc/310:55:273
436658999587,5cyclictest0-21swapper/008:56:010
43665899952,88cyclictest0-21swapper/007:34:340
43665899942,88cyclictest0-21swapper/008:34:460
43665899942,82cyclictest0-21swapper/007:51:010
43665899941,89cyclictest0-21swapper/012:25:050
43665899941,89cyclictest0-21swapper/010:46:360
43665899932,88cyclictest0-21swapper/011:55:410
43665899932,88cyclictest0-21swapper/011:24:460
43665899932,87cyclictest0-21swapper/010:55:230
43665899932,87cyclictest0-21swapper/009:30:090
43665899932,87cyclictest0-21swapper/009:16:440
43665899931,88cyclictest0-21swapper/010:40:450
43651629357,10sleep20-21swapper/207:09:552
43665899922,86cyclictest0-21swapper/008:52:510
43665899922,6cyclictest0-21swapper/012:24:030
43665899921,87cyclictest0-21swapper/011:12:050
43665899912,86cyclictest0-21swapper/012:03:030
43665899912,85cyclictest0-21swapper/009:12:200
43665899912,5cyclictest0-21swapper/010:39:170
43665899911,87cyclictest0-21swapper/007:44:510
43665899911,5cyclictest0-21swapper/012:07:560
436663999017,19cyclictest26-21ksoftirqd/111:58:431
436658999083,4cyclictest0-21swapper/008:48:250
43665899902,20cyclictest1753-21gnome-shell11:53:370
43665899901,85cyclictest0-21swapper/008:38:260
43665899901,85cyclictest0-21swapper/008:38:260
43665899891,78cyclictest0-21swapper/009:43:240
43621328973,9sleep00-21swapper/007:06:030
436667998876,8cyclictest1753-21gnome-shell09:48:252
436667998813,23cyclictest34-21ksoftirqd/209:50:222
436667998813,23cyclictest34-21ksoftirqd/209:50:222
43622928856,24sleep10-21swapper/107:06:151
436667998613,12cyclictest34-21ksoftirqd/210:43:022
436658998617,11cyclictest13-21ksoftirqd/007:38:150
43665899852,20cyclictest1753-21gnome-shell11:26:110
43665899852,20cyclictest1753-21gnome-shell11:26:110
43665899852,20cyclictest1753-21gnome-shell11:00:470
43665899852,20cyclictest1753-21gnome-shell09:05:310
43665899852,19cyclictest1753-21gnome-shell08:41:120
43665899851,22cyclictest1753-21gnome-shell09:20:500
436667998420,8cyclictest141rcu_preempt08:55:342
43665899842,20cyclictest1753-21gnome-shell07:26:140
43665899841,19cyclictest1753-21gnome-shell07:47:590
43666799833,22cyclictest34-21ksoftirqd/207:30:182
43666799832,14cyclictest34-21ksoftirqd/209:05:302
436663998319,7cyclictest26-21ksoftirqd/111:13:161
43665899832,20cyclictest1753-21gnome-shell09:04:520
43665899832,20cyclictest1753-21gnome-shell09:04:520
43665899822,18cyclictest1753-21gnome-shell07:18:290
43665899812,19cyclictest1753-21gnome-shell09:46:180
43667299801,59cyclictest1753-21gnome-shell09:05:013
43667299801,59cyclictest1753-21gnome-shell09:05:013
436667998058,6cyclictest141rcu_preempt12:35:112
436667998058,6cyclictest141rcu_preempt12:35:112
43665899801,19cyclictest1753-21gnome-shell11:30:130
436672997953,6cyclictest141rcu_preempt12:03:243
43666799796,56cyclictest34-21ksoftirqd/208:43:242
43666799795,55cyclictest34-21ksoftirqd/211:30:182
436667997946,7cyclictest141rcu_preempt10:45:172
43666799791,14cyclictest1753-21gnome-shell09:05:022
43666799791,14cyclictest1753-21gnome-shell09:05:012
436663997925,11cyclictest881251-21Softwar~cThread12:09:321
43665899795,6cyclictest141rcu_preempt09:28:340
43665899795,6cyclictest141rcu_preempt09:28:330
43667299782,10cyclictest1753-21gnome-shell08:35:243
43667299782,10cyclictest1753-21gnome-shell08:35:243
43666799786,11cyclictest34-21ksoftirqd/211:14:322
43666799782,20cyclictest34-21ksoftirqd/210:58:512
43665899782,13cyclictest0-21swapper/007:20:270
43666799776,17cyclictest0-21swapper/211:55:472
43666799775,55cyclictest34-21ksoftirqd/208:35:192
43666799775,55cyclictest34-21ksoftirqd/208:35:192
436667997727,10cyclictest34-21ksoftirqd/212:34:512
436667997716,7cyclictest141rcu_preempt11:20:282
436663997722,51cyclictest523614-21grep11:25:301
436663997722,51cyclictest523614-21grep11:25:291
436672997627,45cyclictest1753-21gnome-shell09:50:133
436672997627,45cyclictest1753-21gnome-shell09:50:133
436672997626,9cyclictest1753-21gnome-shell08:50:393
436672997626,46cyclictest1753-21gnome-shell11:32:193
43667299762,10cyclictest1753-21gnome-shell09:28:543
43667299762,10cyclictest1753-21gnome-shell09:28:543
43666799767,53cyclictest436653-21cyclictest11:53:332
43666799762,50cyclictest1753-21gnome-shell12:05:332
436667997613,7cyclictest141rcu_preempt07:12:442
436663997616,17cyclictest881251-21Softwar~cThread08:55:341
436658997610,8cyclictest141rcu_preempt07:10:130
43667299753,10cyclictest1753-21gnome-shell12:36:523
43667299753,10cyclictest1753-21gnome-shell12:36:523
436672997523,10cyclictest1753-21gnome-shell09:40:293
436672997515,7cyclictest141rcu_preempt07:40:073
43666799752,10cyclictest0-21swapper/209:32:332
436658997517,14cyclictest880970-21Timer11:17:130
436672997446,5cyclictest141rcu_preempt07:24:093
43667299742,10cyclictest1753-21gnome-shell11:48:553
43667299742,10cyclictest1753-21gnome-shell11:22:293
43667299741,54cyclictest42-21ksoftirqd/312:25:223
43667299741,10cyclictest1753-21gnome-shell10:37:253
43666799743,10cyclictest34-21ksoftirqd/210:35:132
436667997410,7cyclictest141rcu_preempt10:53:402
436663997427,7cyclictest26-21ksoftirqd/107:39:181
436663997413,18cyclictest881251-21Softwar~cThread12:20:031
43667299733,8cyclictest1753-21gnome-shell09:34:433
43667299732,47cyclictest0-21swapper/310:45:323
43667299732,10cyclictest0-21swapper/311:18:163
43667299732,10cyclictest0-21swapper/308:41:263
43667299731,43cyclictest1753-21gnome-shell08:45:343
436667997347,16cyclictest34-21ksoftirqd/211:26:392
436667997347,16cyclictest34-21ksoftirqd/211:26:392
436663997317,13cyclictest469067-21sed08:50:011
43665899732,10cyclictest1753-21gnome-shell10:33:310
436672997243,5cyclictest141rcu_preempt09:12:463
436672997221,9cyclictest1753-21gnome-shell12:19:133
436672997221,9cyclictest1753-21gnome-shell12:19:133
43667299722,13cyclictest1753-21gnome-shell09:36:263
43667299722,11cyclictest1753-21gnome-shell12:05:193
43666799725,15cyclictest0-21swapper/207:41:492
43666799724,12cyclictest34-21ksoftirqd/209:12:482
436667997213,42cyclictest141rcu_preempt09:36:502
436663997223,45cyclictest465468-21latency_hist08:40:031
436663997223,45cyclictest465468-21latency_hist08:40:031
4968672710,8chrt1753-21gnome-shell10:05:490
43667299712,46cyclictest0-21swapper/307:51:393
436672997124,43cyclictest1753-21gnome-shell07:28:193
43667299712,11cyclictest1753-21gnome-shell07:12:373
43667299711,10cyclictest1753-21gnome-shell09:15:173
436667997157,11cyclictest34-21ksoftirqd/207:45:572
436667997156,6cyclictest34-21ksoftirqd/211:18:362
43666799715,11cyclictest34-21ksoftirqd/209:15:352
436667997144,17cyclictest34-21ksoftirqd/207:36:592
436663997117,49cyclictest881251-21Softwar~cThread11:19:291
43665899716,52cyclictest141rcu_preempt11:50:010
43665899714,53cyclictest141rcu_preempt09:50:120
43665899714,53cyclictest141rcu_preempt09:50:110
43665899712,49cyclictest485085-21grep09:35:140
436672997041,5cyclictest141rcu_preempt11:06:433
436672997038,8cyclictest881251-21Softwar~cThread09:48:233
43667299702,45cyclictest0-21swapper/309:06:533
43667299702,11cyclictest1753-21gnome-shell09:00:013
436672997020,9cyclictest1753-21gnome-shell07:46:563
436667997056,5cyclictest34-21ksoftirqd/211:00:462
436667997056,5cyclictest34-21ksoftirqd/209:20:432
436667997045,8cyclictest34-21ksoftirqd/209:41:042
436667997016,44cyclictest1753-21gnome-shell09:25:282
436667997016,44cyclictest1753-21gnome-shell09:25:272
436667997012,6cyclictest141rcu_preempt08:45:242
43666399702,12cyclictest529293-21kworker/1:011:53:461
43667299693,62cyclictest1753-21gnome-shell11:03:013
43667299692,45cyclictest0-21swapper/312:24:533
436672996917,10cyclictest1753-21gnome-shell07:39:183
43666799696,11cyclictest34-21ksoftirqd/207:26:072
43666799694,7cyclictest141rcu_preempt12:23:252
436667996931,12cyclictest881331-21Privileged14:31:442
436667996911,17cyclictest1753-21gnome-shell07:50:152
436663996911,6cyclictest141rcu_preempt09:46:041
436663996911,16cyclictest525246-21cat11:30:281
436672996861,5cyclictest0-21swapper/311:28:253
436672996861,5cyclictest0-21swapper/311:28:253
436672996838,5cyclictest0-21swapper/312:34:423
43667299682,45cyclictest0-21swapper/310:50:503
43666799682,51cyclictest0-21swapper/207:20:242
436663996858,6cyclictest484947-21/usr/sbin/munin09:35:321
436663996852,11cyclictest881251-21Softwar~cThread12:19:411
436663996852,11cyclictest881251-21Softwar~cThread12:19:401
436663996814,49cyclictest881251-21Softwar~cThread07:47:371
436663996811,15cyclictest881251-21Softwar~cThread12:05:011
436672996750,6cyclictest42-21ksoftirqd/311:10:213
436672996750,13cyclictest42-21ksoftirqd/307:30:183
43667299672,8cyclictest1753-21gnome-shell09:22:353
436667996752,10cyclictest1753-21gnome-shell12:01:222
436667996715,11cyclictest1753-21gnome-shell11:05:112
436663996751,11cyclictest881251-21Softwar~cThread10:44:121
436672996644,8cyclictest141rcu_preempt07:18:303
436667996646,16cyclictest34-21ksoftirqd/207:18:572
43666399668,16cyclictest2265126-21snapd12:28:331
43666399662,14cyclictest437073-21cstates07:10:131
436663996612,12cyclictest881251-21Softwar~cThread10:55:021
43667299652,22cyclictest1753-21gnome-shell11:56:063
436667996552,4cyclictest34-21ksoftirqd/208:34:052
436663996531,7cyclictest141rcu_preempt09:18:371
43666399653,11cyclictest141rcu_preempt07:20:191
43666399652,11cyclictest141rcu_preempt09:20:311
43666399652,10cyclictest141rcu_preempt09:27:141
43666399652,10cyclictest141rcu_preempt09:27:141
43666399651,11cyclictest141rcu_preempt07:26:501
4536502640,5sleep30-21swapper/308:05:133
43667299648,52cyclictest463698-21sshd08:34:493
436672996457,4cyclictest0-21swapper/310:33:253
436672996452,8cyclictest1753-21gnome-shell10:41:073
436667996450,9cyclictest1753-21gnome-shell11:49:522
436667996417,33cyclictest34-21ksoftirqd/208:52:262
43666399643,10cyclictest141rcu_preempt07:43:201
43666399642,20cyclictest1753-21gnome-shell11:01:571
43666399642,10cyclictest141rcu_preempt12:35:171
43666399642,10cyclictest141rcu_preempt12:35:161
43666399641,11cyclictest141rcu_preempt09:30:511
436672996348,6cyclictest42-21ksoftirqd/311:51:393
436667996336,10cyclictest34-21ksoftirqd/212:20:012
436667996336,10cyclictest34-21ksoftirqd/212:20:002
43666799632,19cyclictest1753-21gnome-shell12:25:102
43666399639,11cyclictest881251-21Softwar~cThread10:57:421
436663996332,11cyclictest141rcu_preempt11:05:141
43666399632,44cyclictest141rcu_preempt07:30:111
436663996314,7cyclictest26-21ksoftirqd/110:37:471
43666399631,10cyclictest141rcu_preempt09:43:461
43666399624,5cyclictest141rcu_preempt10:47:001
436663996214,7cyclictest26-21ksoftirqd/112:34:281
436663996214,7cyclictest26-21ksoftirqd/111:22:461
436663996214,7cyclictest26-21ksoftirqd/110:34:221
436663996214,7cyclictest26-21ksoftirqd/108:33:501
43666399617,49cyclictest26-21ksoftirqd/108:50:141
43666399612,56cyclictest231irq_work/108:43:001
43666399612,56cyclictest0-21swapper/109:08:141
43666399607,10cyclictest881331-21Privileged14:31:441
43666399607,10cyclictest881331-21Privileged14:31:441
43666399602,52cyclictest1753-21gnome-shell09:50:081
43666399602,52cyclictest1753-21gnome-shell09:50:081
43666399602,47cyclictest1069-21runrttasks09:10:141
43666399602,15cyclictest449107-21sensors07:50:301
436663996013,7cyclictest26-21ksoftirqd/111:48:251
43665899592,5cyclictest0-21swapper/012:19:100
43665899592,5cyclictest0-21swapper/012:19:100
436663995836,15cyclictest1753-21gnome-shell08:18:251
43666399562,14cyclictest880898-21Xwayland07:18:241
436658995619,10cyclictest459277-21expr08:20:240
436658995221,10cyclictest453171-21sh08:05:010
436658995216,12cyclictest494217-21cat10:00:160
436658995215,12cyclictest453418-21/usr/sbin/munin08:05:240
43667299513,8cyclictest0-21swapper/308:18:243
436658995119,10cyclictest13-21ksoftirqd/008:15:160
162399510,10rtkit-daemon1753-21gnome-shell08:00:102
436658994818,8cyclictest1069-21runrttasks11:36:420
436658994815,13cyclictest881251-21Softwar~cThread08:14:510
436658994815,13cyclictest881251-21Softwar~cThread08:14:510
436658994815,11cyclictest528161-21entropy11:40:180
436658994813,11cyclictest499581-21df_inode10:15:160
43666799472,7cyclictest0-21swapper/208:18:242
43665899479,13cyclictest918-21gdbus10:11:560
436658994718,11cyclictest503736-21sed10:25:380
436672994628,15cyclictest1753-21gnome-shell08:10:143
436672994628,15cyclictest1753-21gnome-shell08:10:143
43665899468,18cyclictest1753-21gnome-shell09:56:530
436658994614,10cyclictest450600-21wc07:55:240
43665899449,16cyclictest1753-21gnome-shell08:28:110
43667299431,23cyclictest1753-21gnome-shell10:05:053
436667994325,13cyclictest495906-21grep10:05:162
43667299421,22cyclictest1753-21gnome-shell10:26:103
436667994217,14cyclictest34-21ksoftirqd/210:13:262
436667994211,6cyclictest141rcu_preempt11:35:162
436672994123,13cyclictest493477-21Indexed~#20838209:57:033
436667994124,7cyclictest34-21ksoftirqd/208:21:232
436667994124,6cyclictest34-21ksoftirqd/208:25:242
43666799411,4cyclictest0-21swapper/210:19:152
436663994124,12cyclictest438-20systemd-journal07:55:481
436667994023,14cyclictest34-21ksoftirqd/211:42:482
436667994022,6cyclictest34-21ksoftirqd/209:55:022
436663994016,19cyclictest881251-21Softwar~cThread10:06:331
436658994017,6cyclictest113-1kworker/0:1H12:13:330
436672993921,13cyclictest438-20systemd-journal10:10:173
43667299391,12cyclictest527761-21find11:40:113
436667993928,7cyclictest1069-21runrttasks07:56:322
43666399392,32cyclictest1753-21gnome-shell08:14:051
43666399392,32cyclictest1753-21gnome-shell08:14:051
436663993913,21cyclictest881251-21Softwar~cThread08:04:581
5385742380,5sleep3411rcuc/312:10:183
43667299389,5cyclictest141rcu_preempt08:25:313
436672993822,12cyclictest42-21ksoftirqd/311:35:343
436672993821,13cyclictest452906-21sshd08:01:493
436672993819,14cyclictest458812-21cat08:20:143
436672993818,15cyclictest881251-21Softwar~cThread10:03:353
436667993822,13cyclictest34-21ksoftirqd/210:23:452
436667993821,6cyclictest34-21ksoftirqd/208:07:022
436667993821,12cyclictest538917-21ntpq12:10:252
436663993820,13cyclictest881251-21Softwar~cThread11:43:101
43666399381,23cyclictest1753-21gnome-shell08:28:141
436663993812,22cyclictest498292-21processes10:10:331
436672993723,10cyclictest42-21ksoftirqd/310:19:043
436672993717,8cyclictest141rcu_preempt10:20:243
436667993726,7cyclictest1753-21gnome-shell10:00:152
436667993721,6cyclictest34-21ksoftirqd/208:10:362
436667993721,6cyclictest34-21ksoftirqd/208:10:362
436663993728,6cyclictest460257-21turbostat.cron08:24:591
436663993726,7cyclictest527518-21latency_hist11:39:591
436663993725,8cyclictest64850irq/126-i91510:05:011
436658993710,11cyclictest1069-21runrttasks10:24:090
436672993618,13cyclictest450696-21ntp_kernel_pll_07:55:263
436667993625,7cyclictest1753-21gnome-shell10:27:432
436663993625,7cyclictest538567-21diskstats12:10:141
436663993423,6cyclictest26-21ksoftirqd/109:55:091
436663993416,13cyclictest453931-21kernelversion08:05:211
436663993412,18cyclictest881251-21Softwar~cThread10:19:011
436663993318,11cyclictest501928-21sed10:20:301
436663993212,6cyclictest503058-21df10:25:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional