You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-17 - 12:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri May 17, 2024 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5758112180123,50sleep30-21swapper/319:06:033
5753532149128,14sleep00-21swapper/019:05:230
5760482136121,9sleep20-21swapper/219:08:312
57633099983,90cyclictest881251-21Softwar~cThread23:06:472
6057012930,6sleep10-21swapper/120:30:241
576333999334,9cyclictest1753-21gnome-shell19:20:353
57605729255,31sleep10-21swapper/119:08:381
57633099893,13cyclictest0-21swapper/220:33:452
57633099883,25cyclictest691273-21df_inode00:35:152
162399880,6rtkit-daemon0-21swapper/321:20:103
162399880,11rtkit-daemon1753-21gnome-shell23:18:313
162399880,11rtkit-daemon1753-21gnome-shell23:18:313
57633099865,23cyclictest141rcu_preempt23:12:242
57633099862,28cyclictest0-21swapper/220:35:272
57633099861,28cyclictest1753-21gnome-shell22:10:352
576330998611,51cyclictest881251-21Softwar~cThread20:42:212
57633099855,25cyclictest684533-21memory00:15:222
576330998515,53cyclictest34-21ksoftirqd/219:38:242
576327998578,5cyclictest0-21swapper/119:19:081
57633099844,25cyclictest880726-21firefox19:45:172
57633099843,26cyclictest643129-21chrt22:19:482
57633099842,14cyclictest880726-21firefox23:04:162
576322998433,9cyclictest1753-21gnome-shell19:10:200
162399840,5rtkit-daemon0-21swapper/323:47:363
57633099833,24cyclictest1753-21gnome-shell19:19:532
576330998315,8cyclictest141rcu_preempt20:50:142
576330998311,19cyclictest141rcu_preempt00:11:402
57633099823,22cyclictest576880-21df_inode19:10:182
576330998223,8cyclictest34-21ksoftirqd/221:05:212
57633099822,22cyclictest1753-21gnome-shell20:49:422
57633099822,20cyclictest880726-21firefox23:17:042
57633099822,20cyclictest880726-21firefox23:17:042
162399820,5rtkit-daemon0-21swapper/323:24:133
57633099816,21cyclictest141rcu_preempt19:41:232
57633099814,23cyclictest141rcu_preempt19:34:282
57633099814,22cyclictest141rcu_preempt20:55:502
576327998142,14cyclictest0-21swapper/121:28:271
57632299812,21cyclictest617972-21memory21:05:240
576333998061,6cyclictest42-21ksoftirqd/321:25:133
57633099806,22cyclictest141rcu_preempt00:29:102
57633099804,59cyclictest141rcu_preempt19:20:212
57633099802,21cyclictest0-21swapper/200:24:012
576322998063,6cyclictest13-21ksoftirqd/019:39:360
57633099795,57cyclictest0-21swapper/221:03:522
57633099795,10cyclictest880726-21firefox00:32:542
57633099792,20cyclictest0-21swapper/222:06:482
576322997928,9cyclictest1753-21gnome-shell00:13:150
162399790,5rtkit-daemon0-21swapper/323:26:423
162399790,5rtkit-daemon0-21swapper/323:02:513
162399790,5rtkit-daemon0-21swapper/322:51:063
162399790,5rtkit-daemon0-21swapper/322:51:063
162399790,5rtkit-daemon0-21swapper/321:45:283
576333997861,5cyclictest42-21ksoftirqd/321:00:363
576333997857,13cyclictest42-21ksoftirqd/300:28:583
57633099789,52cyclictest141rcu_preempt21:26:292
57633099789,11cyclictest141rcu_preempt00:05:132
57633099782,21cyclictest0-21swapper/200:03:462
57633099782,14cyclictest623012-21/usr/sbin/munin21:20:232
57633099782,12cyclictest880726-21firefox19:26:012
57632799782,15cyclictest26-21ksoftirqd/100:09:351
576322997855,14cyclictest13-21ksoftirqd/023:04:350
576333997759,13cyclictest42-21ksoftirqd/322:10:463
576333997727,8cyclictest1753-21gnome-shell20:35:313
576333997721,52cyclictest1753-21gnome-shell20:41:053
576333997719,15cyclictest881251-21Softwar~cThread00:39:523
57633099776,15cyclictest141rcu_preempt22:03:482
57633099776,15cyclictest141rcu_preempt22:03:482
57633099772,19cyclictest0-21swapper/223:20:482
57633099772,18cyclictest0-21swapper/220:19:552
576322997766,7cyclictest1753-21gnome-shell20:41:410
57633399762,8cyclictest141rcu_preempt20:57:303
57633099762,23cyclictest0-21swapper/221:19:102
57632799762,11cyclictest26-21ksoftirqd/122:05:271
162399760,6rtkit-daemon1622-21rtkit-daemon00:16:441
576333997553,13cyclictest42-21ksoftirqd/300:10:023
576333997546,9cyclictest577100-21/usr/sbin/munin19:10:273
576322997545,13cyclictest141rcu_preempt20:35:300
576333997459,6cyclictest42-21ksoftirqd/320:18:453
57633399742,16cyclictest1753-21gnome-shell19:45:173
57632799748,12cyclictest26-21ksoftirqd/120:41:491
576322997456,14cyclictest13-21ksoftirqd/023:13:570
162399740,5rtkit-daemon0-21swapper/300:00:423
576333997362,7cyclictest1753-21gnome-shell19:38:033
576333997356,5cyclictest42-21ksoftirqd/323:12:093
576333997334,20cyclictest860997-21snmpd21:08:193
576333997317,16cyclictest42-21ksoftirqd/322:18:583
57632299734,8cyclictest141rcu_preempt00:05:260
162399730,6rtkit-daemon0-21swapper/321:41:523
162399730,5rtkit-daemon0-21swapper/322:25:003
57633399722,64cyclictest1753-21gnome-shell20:50:253
576333997225,43cyclictest1753-21gnome-shell19:43:443
57632799726,10cyclictest0-21swapper/119:20:231
576322997225,43cyclictest13-21ksoftirqd/020:51:440
162399720,5rtkit-daemon0-21swapper/323:39:593
57633399712,10cyclictest1753-21gnome-shell23:05:173
576333997113,52cyclictest881251-21Softwar~cThread00:30:313
576327997152,5cyclictest141rcu_preempt19:10:361
576327997147,12cyclictest881251-21Softwar~cThread20:53:301
57632299718,7cyclictest141rcu_preempt21:01:330
576322997144,11cyclictest141rcu_preempt23:15:370
576322997144,11cyclictest141rcu_preempt23:15:360
576322997144,11cyclictest141rcu_preempt19:40:130
162399710,5rtkit-daemon0-21swapper/321:34:423
162399710,4rtkit-daemon0-21swapper/323:53:093
576333997025,6cyclictest141rcu_preempt22:10:013
57633099703,47cyclictest1753-21gnome-shell19:50:072
57633099703,47cyclictest1753-21gnome-shell19:50:062
576327997035,11cyclictest0-21swapper/120:49:321
57632799702,61cyclictest1753-21gnome-shell22:18:451
57632299708,44cyclictest141rcu_preempt23:08:060
57632299707,7cyclictest141rcu_preempt00:30:500
57632299706,7cyclictest141rcu_preempt00:25:190
576322997046,18cyclictest13-21ksoftirqd/023:20:160
576322997021,7cyclictest13-21ksoftirqd/021:28:250
576322997010,49cyclictest141rcu_preempt00:38:430
57633399693,46cyclictest1753-21gnome-shell20:45:183
57633399693,45cyclictest1753-21gnome-shell00:20:033
57633099695,8cyclictest141rcu_preempt21:10:072
57633099695,8cyclictest141rcu_preempt21:10:062
576327996954,11cyclictest587503-21cpuspeed_turbos19:40:121
576327996952,12cyclictest881251-21Softwar~cThread20:58:271
57632799693,12cyclictest26-21ksoftirqd/100:10:251
57632299698,8cyclictest141rcu_preempt19:33:460
576322996960,5cyclictest13-21ksoftirqd/019:20:200
576322996944,10cyclictest141rcu_preempt20:45:150
576322996939,19cyclictest13-21ksoftirqd/022:05:410
576322996917,35cyclictest141rcu_preempt20:55:270
576327996814,8cyclictest26-21ksoftirqd/100:21:331
576327996812,13cyclictest689217-21cpuspeed00:30:121
576322996860,5cyclictest13-21ksoftirqd/021:22:350
576322996839,10cyclictest141rcu_preempt19:45:370
576322996820,44cyclictest0-21swapper/022:16:440
57632299680,64cyclictest1622-21rtkit-daemon21:53:340
57633399672,7cyclictest141rcu_preempt00:13:133
57633399672,61cyclictest1753-21gnome-shell21:10:133
57633399672,61cyclictest1753-21gnome-shell21:10:133
576327996753,9cyclictest881251-21Softwar~cThread21:22:501
576322996756,7cyclictest684757-21processes00:15:290
576322996744,9cyclictest13-21ksoftirqd/000:20:310
576322996731,7cyclictest141rcu_preempt22:05:010
576322996731,7cyclictest141rcu_preempt22:05:010
576333996626,15cyclictest141rcu_preempt00:17:503
57632299662,17cyclictest1753-21gnome-shell20:18:450
576322996616,34cyclictest141rcu_preempt19:25:560
162399660,8rtkit-daemon1622-21rtkit-daemon19:52:083
162399660,8rtkit-daemon1622-21rtkit-daemon19:52:083
57633399651,59cyclictest1753-21gnome-shell19:34:203
576333996515,46cyclictest42-21ksoftirqd/321:19:023
576327996557,5cyclictest26-21ksoftirqd/121:05:221
576327996542,17cyclictest1753-21gnome-shell23:11:531
576327996513,48cyclictest438-20systemd-journal00:03:541
162399650,10rtkit-daemon1622-21rtkit-daemon19:58:053
576333996428,20cyclictest1753-21gnome-shell21:55:113
57632799643,6cyclictest141rcu_preempt23:08:141
576322996441,16cyclictest13-21ksoftirqd/022:10:150
576333996315,7cyclictest0-21swapper/322:04:323
576333996315,7cyclictest0-21swapper/322:04:323
576333996254,5cyclictest0-21swapper/319:19:513
57632799627,50cyclictest881251-21Softwar~cThread22:13:261
576327996247,10cyclictest692542-21sed00:40:021
57632799623,44cyclictest141rcu_preempt00:27:141
57632799622,17cyclictest662959-21grep23:15:121
57632799622,17cyclictest662959-21grep23:15:111
57633399619,48cyclictest42-21ksoftirqd/320:34:263
57632799612,56cyclictest0-21swapper/120:19:171
57632799612,54cyclictest881251-21Softwar~cThread19:35:221
576327996047,8cyclictest880726-21firefox21:00:041
576327996037,10cyclictest0-21swapper/120:36:491
57632799602,54cyclictest881143-21gmain22:03:521
57632799602,54cyclictest881143-21gmain22:03:521
57632799602,53cyclictest665542-21/usr/sbin/munin23:20:271
576327996011,8cyclictest0-21swapper/119:45:251
576322996052,5cyclictest13-21ksoftirqd/019:19:580
57632299593,13cyclictest1753-21gnome-shell00:05:010
57632299593,11cyclictest1753-21gnome-shell21:19:550
57632299592,7cyclictest141rcu_preempt20:33:590
57633399582,15cyclictest582694-21grep19:25:263
57633399580,54cyclictest1622-21rtkit-daemon20:09:013
57633099582,51cyclictest643156-21apt-get22:20:052
57632799582,10cyclictest1753-21gnome-shell19:25:241
57632799553,42cyclictest0-21swapper/119:33:431
5962222540,20sleep342-21ksoftirqd/320:05:023
57632799543,46cyclictest0-21swapper/123:03:451
576327995417,32cyclictest26-21ksoftirqd/121:19:021
576327995213,17cyclictest26-21ksoftirqd/123:30:271
576333995127,20cyclictest1753-21gnome-shell23:33:343
57632299512,7cyclictest0-21swapper/019:50:060
57632299512,7cyclictest0-21swapper/019:50:060
576333995026,20cyclictest1753-21gnome-shell21:50:223
576327995021,25cyclictest652156-21unin-run22:45:011
57632299502,7cyclictest0-21swapper/022:20:060
57632299502,44cyclictest0-21swapper/021:10:080
57632299502,44cyclictest0-21swapper/021:10:080
57632799492,44cyclictest6190892sleep121:10:091
57632799492,44cyclictest6190892sleep121:10:081
57632799482,5cyclictest0-21swapper/122:20:081
57632799482,43cyclictest0-21swapper/119:50:081
57632799482,43cyclictest0-21swapper/119:50:081
6318532470,17sleep013-21ksoftirqd/021:45:250
57632299475,13cyclictest13-21ksoftirqd/022:55:010
57632299475,13cyclictest13-21ksoftirqd/022:55:000
57632799463,5cyclictest141rcu_preempt20:15:011
576327994615,8cyclictest26-21ksoftirqd/122:50:161
576327994615,8cyclictest26-21ksoftirqd/122:50:161
576333994522,19cyclictest1753-21gnome-shell22:45:333
576333994521,20cyclictest1753-21gnome-shell22:35:213
6286692440,5sleep10-21swapper/121:35:281
576322994421,13cyclictest13-21ksoftirqd/022:30:110
576322994416,8cyclictest17531gnome-shell00:00:010
576333994319,20cyclictest1753-21gnome-shell23:55:253
576327994312,8cyclictest26-21ksoftirqd/121:46:241
576322994324,6cyclictest13-21ksoftirqd/022:25:140
576322994324,6cyclictest13-21ksoftirqd/021:30:290
576330994223,15cyclictest657508-21unin-run23:00:012
576327994215,10cyclictest646766-21cron22:30:031
576322994224,6cyclictest13-21ksoftirqd/023:30:200
576322994223,7cyclictest13-21ksoftirqd/023:45:170
576322994223,7cyclictest13-21ksoftirqd/023:35:230
576322994223,7cyclictest13-21ksoftirqd/022:35:360
576322994223,7cyclictest13-21ksoftirqd/020:20:110
576322994215,11cyclictest881251-21Softwar~cThread20:05:570
576330994124,12cyclictest438-20systemd-journal23:53:232
57632799417,4cyclictest141rcu_preempt23:46:141
576327994125,12cyclictest670125-21ls23:35:111
576327994119,18cyclictest678489-40mandb00:00:021
576327994115,8cyclictest26-21ksoftirqd/121:40:291
576322994122,6cyclictest13-21ksoftirqd/023:54:020
576322994122,6cyclictest13-21ksoftirqd/023:25:300
576322994122,6cyclictest13-21ksoftirqd/020:00:220
576322994119,13cyclictest13-21ksoftirqd/022:42:130
576322994118,8cyclictest141rcu_preempt21:35:100
57633399403,4cyclictest141rcu_preempt22:30:013
576330994016,20cyclictest668978-40lsb_release23:30:262
576327994015,5cyclictest141rcu_preempt23:25:301
576322994022,5cyclictest13-21ksoftirqd/020:10:320
576322994022,13cyclictest593512-21ntpq19:55:250
576322994017,8cyclictest141rcu_preempt22:45:150
162399400,6rtkit-daemon0-21swapper/221:49:292
576333993914,21cyclictest1753-21gnome-shell22:30:493
576330993928,7cyclictest1753-21gnome-shell21:38:052
576330993922,13cyclictest595251-21ntpq20:00:292
576330993922,12cyclictest655743-21/usr/sbin/munin22:55:022
576330993922,12cyclictest655743-21/usr/sbin/munin22:55:022
576330993921,14cyclictest635336-21memory21:55:242
57632799396,29cyclictest635648-21wc21:55:341
57632799391,34cyclictest671745-21/usr/sbin/munin23:40:171
57632799391,34cyclictest671745-21/usr/sbin/munin23:40:161
576322993921,5cyclictest13-21ksoftirqd/020:25:120
576322993920,6cyclictest13-21ksoftirqd/021:55:170
576333993819,14cyclictest881251-21Softwar~cThread20:10:343
576333993813,21cyclictest1753-21gnome-shell23:44:533
576333993813,21cyclictest1753-21gnome-shell23:44:523
57633099382,11cyclictest859-21NetworkManager22:25:152
576330993821,12cyclictest881251-21Softwar~cThread21:32:442
57633099381,16cyclictest880726-21firefox22:47:412
57632799383,18cyclictest26-21ksoftirqd/120:05:261
576327993815,9cyclictest26-21ksoftirqd/121:50:281
576322993818,8cyclictest141rcu_preempt23:42:400
576322993818,8cyclictest141rcu_preempt23:42:390
5955892370,9sleep1880726-21firefox20:00:371
57633399378,11cyclictest141rcu_preempt22:40:083
57633399376,10cyclictest42-21ksoftirqd/322:55:273
576333993712,21cyclictest1753-21gnome-shell20:29:363
57633099371,11cyclictest859-21NetworkManager22:35:192
576330993710,9cyclictest1753-21gnome-shell20:20:232
57632799377,4cyclictest141rcu_preempt20:29:591
576327993720,12cyclictest438-20systemd-journal22:59:561
576327993716,5cyclictest141rcu_preempt20:20:261
576333993611,11cyclictest141rcu_preempt21:35:113
57633099369,6cyclictest141rcu_preempt20:30:022
57633099367,16cyclictest141rcu_preempt23:50:012
57633099366,12cyclictest141rcu_preempt20:08:342
576330993622,9cyclictest438-20systemd-journal22:45:002
576330993620,12cyclictest647672-21cat22:30:232
57633099361,31cyclictest593110-21expr19:55:142
57633099361,16cyclictest672643-21python323:40:302
57633099361,16cyclictest672643-21python323:40:302
576327993613,18cyclictest885440-21Timer22:25:101
576322993620,12cyclictest630746-21seq21:41:590
576322993618,14cyclictest13-21ksoftirqd/022:55:250
576333993522,9cyclictest1753-21gnome-shell20:22:393
57633099352,28cyclictest881251-21Softwar~cThread23:55:392
57633099352,28cyclictest881251-21Softwar~cThread23:27:132
576330993519,12cyclictest630574-21cat21:40:352
576330993516,14cyclictest881251-21Softwar~cThread20:12:072
57632799352,15cyclictest0-21swapper/119:55:191
576327993519,12cyclictest649069-21expr22:35:171
576327993518,5cyclictest141rcu_preempt22:45:361
576327993515,6cyclictest141rcu_preempt23:55:011
576327993515,6cyclictest141rcu_preempt21:31:081
576330993419,11cyclictest633340-21cat21:50:152
576330993416,13cyclictest881251-21Softwar~cThread23:39:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional