You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 08:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Apr 27, 2024 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991160,103rtkit-daemon0-21swapper/219:07:252
1844630991069,92cyclictest1906243-21taskset22:04:320
18446439910321,15cyclictest438-20systemd-journal22:51:483
184463099992,92cyclictest1753-21gnome-shell23:36:250
184412929868,21sleep00-21swapper/019:05:340
18719672970,4sleep20-21swapper/220:25:212
184463099972,91cyclictest0-21swapper/021:44:260
184463099972,7cyclictest0-21swapper/021:30:400
19069662960,5chrt0-21swapper/022:05:210
184463899962,89cyclictest880726-21firefox22:34:152
184463099962,90cyclictest0-21swapper/022:51:470
19254062950,8sleep11753-21gnome-shell23:00:001
1844638999582,8cyclictest880726-21firefox21:51:322
1844638999582,8cyclictest880726-21firefox21:51:322
1844630999586,6cyclictest0-21swapper/021:59:140
1844630999582,8cyclictest880726-21firefox22:40:080
1844630999582,8cyclictest880726-21firefox22:40:070
184463099953,9cyclictest0-21swapper/021:18:260
184463099952,89cyclictest0-21swapper/023:06:390
184463099952,89cyclictest0-21swapper/022:35:290
184463099952,89cyclictest0-21swapper/022:23:390
184463099952,89cyclictest0-21swapper/022:10:590
184463099952,89cyclictest0-21swapper/021:37:490
184463099942,88cyclictest0-21swapper/023:22:100
184463099942,88cyclictest0-21swapper/023:10:340
184463099942,88cyclictest0-21swapper/023:02:580
184463099942,7cyclictest0-21swapper/021:20:540
184463099942,6cyclictest0-21swapper/021:28:450
184463099942,6cyclictest0-21swapper/021:28:440
1844643999382,8cyclictest1919852-21taskset22:41:003
1844643999382,8cyclictest1919852-21taskset22:41:003
1844638999386,5cyclictest34-21ksoftirqd/223:04:552
1844638999339,13cyclictest34-21ksoftirqd/219:59:372
1844630999385,5cyclictest0-21swapper/022:48:570
184463099932,87cyclictest0-21swapper/022:59:200
184463099932,87cyclictest0-21swapper/022:25:540
184463099932,87cyclictest0-21swapper/021:52:390
184463099932,87cyclictest0-21swapper/021:52:390
184463099932,87cyclictest0-21swapper/019:31:240
184463099932,6cyclictest0-21swapper/022:16:250
184463099932,6cyclictest0-21swapper/022:16:250
184463099931,88cyclictest0-21swapper/023:18:140
184463899922,86cyclictest0-21swapper/221:41:282
184463099922,87cyclictest0-21swapper/019:51:430
184463099922,86cyclictest0-21swapper/023:32:250
184463099922,86cyclictest0-21swapper/023:32:250
184463099922,86cyclictest0-21swapper/020:39:580
184463099922,86cyclictest0-21swapper/020:39:580
184463099922,86cyclictest0-21swapper/020:00:590
184463099922,7cyclictest0-21swapper/021:45:520
184463099922,6cyclictest0-21swapper/000:33:500
184464399911,23cyclictest141rcu_preempt21:27:173
184464399911,23cyclictest141rcu_preempt21:27:173
1844634999183,5cyclictest0-21swapper/120:35:571
1844634999183,5cyclictest0-21swapper/120:35:561
184463499912,87cyclictest0-21swapper/121:41:281
184463499912,5cyclictest0-21swapper/122:34:151
1844630999183,5cyclictest0-21swapper/019:58:180
162399910,8rtkit-daemon1862690-21cat20:00:092
18877112900,5sleep30-21swapper/321:10:263
1844638999082,5cyclictest0-21swapper/222:45:472
1844638999079,7cyclictest1907630-21timerwakeupswit22:05:352
1844638999079,7cyclictest1904176-21taskset21:55:322
1844634998941,7cyclictest0-21swapper/119:18:261
184463099892,20cyclictest1753-21gnome-shell00:35:270
184416428872,9sleep30-21swapper/319:06:013
1844634998781,4cyclictest0-21swapper/122:05:351
1844634998781,4cyclictest0-21swapper/121:55:321
184463099872,19cyclictest1753-21gnome-shell19:21:140
1844638998680,4cyclictest0-21swapper/220:35:562
1844638998680,4cyclictest0-21swapper/220:35:562
184463499867,23cyclictest26-21ksoftirqd/119:54:121
184432028672,9sleep10-21swapper/119:08:001
1844638998575,6cyclictest1929239-21cpuspeed_turbos23:10:152
1844634998578,4cyclictest0-21swapper/121:52:361
1844634998578,4cyclictest0-21swapper/121:52:361
18473542840,16sleep2331rcuc/219:15:142
1844643998410,43cyclictest0-21swapper/321:45:023
184463099842,20cyclictest1753-21gnome-shell20:34:580
184463099842,20cyclictest1753-21gnome-shell19:11:410
19119922830,4sleep20-21swapper/222:20:122
1844634998377,4cyclictest0-21swapper/123:10:151
1844634998377,4cyclictest0-21swapper/123:03:521
1844634998377,4cyclictest0-21swapper/121:27:221
1844634998377,4cyclictest0-21swapper/121:27:221
184463099832,20cyclictest1753-21gnome-shell19:48:280
184463099832,19cyclictest1753-21gnome-shell19:19:160
184464399822,53cyclictest1753-21gnome-shell20:39:213
184464399822,53cyclictest1753-21gnome-shell20:39:213
1844643998214,15cyclictest141rcu_preempt23:27:533
1844638998274,5cyclictest0-21swapper/221:27:222
1844638998274,5cyclictest0-21swapper/221:27:222
1844634998275,4cyclictest0-21swapper/123:15:091
184463099821,20cyclictest1753-21gnome-shell19:29:090
184463099821,20cyclictest1753-21gnome-shell19:29:090
184463099821,18cyclictest1753-21gnome-shell23:25:330
184463099821,18cyclictest1753-21gnome-shell22:30:080
184464399812,16cyclictest1933870-40lsb_release23:21:223
184463499819,6cyclictest141rcu_preempt23:36:271
1844634998175,4cyclictest0-21swapper/122:39:521
184464399802,11cyclictest880726-21firefox19:11:503
1844638998073,5cyclictest0-21swapper/222:39:522
1844634998074,4cyclictest0-21swapper/122:23:421
1844643997911,10cyclictest141rcu_preempt22:56:583
1844638997972,4cyclictest0-21swapper/223:15:092
1844638997971,5cyclictest0-21swapper/223:28:202
1844634997973,4cyclictest0-21swapper/123:28:201
184463899782,11cyclictest1753-21gnome-shell22:50:132
184463099781,18cyclictest1753-21gnome-shell23:48:590
1844643997721,15cyclictest881251-21Softwar~cThread23:08:363
1844643997710,13cyclictest880726-21firefox23:14:303
184463899771,21cyclictest1753-21gnome-shell00:35:142
18498282760,6sleep234-21ksoftirqd/219:20:322
184464399764,9cyclictest0-21swapper/322:49:103
1844638997612,43cyclictest141rcu_preempt22:14:042
1844638997610,7cyclictest141rcu_preempt23:24:472
1844643997522,11cyclictest1753-21gnome-shell19:59:273
184463499752,12cyclictest26-21ksoftirqd/119:25:261
184463499752,12cyclictest26-21ksoftirqd/119:25:251
19583652740,6sleep1231irq_work/100:30:331
184463899744,16cyclictest0-21swapper/223:07:042
184463499743,12cyclictest26-21ksoftirqd/123:49:051
184464399734,14cyclictest141rcu_preempt21:45:293
184464399732,15cyclictest141rcu_preempt21:20:223
1844643997315,16cyclictest881251-21Softwar~cThread22:01:303
184464399731,14cyclictest141rcu_preempt23:15:273
1844643997311,12cyclictest141rcu_preempt19:49:593
184463899735,52cyclictest0-21swapper/219:25:042
184463899735,52cyclictest0-21swapper/219:25:042
184463899732,17cyclictest0-21swapper/223:38:122
1844634997365,5cyclictest0-21swapper/121:35:221
1844643997260,7cyclictest880974-21IPDL14:31:443
184464399724,15cyclictest141rcu_preempt22:05:243
1844643997223,44cyclictest42-21ksoftirqd/319:20:003
1844643997216,15cyclictest881251-21Softwar~cThread21:31:183
184463499724,10cyclictest0-21swapper/123:24:561
1844634997214,15cyclictest881251-21Softwar~cThread22:29:281
1844634997210,6cyclictest141rcu_preempt22:51:141
184464399717,13cyclictest880726-21firefox21:37:223
184464399715,7cyclictest141rcu_preempt22:25:253
1844643997155,11cyclictest881251-21Softwar~cThread22:23:333
184464399714,11cyclictest42-21ksoftirqd/321:54:373
184464399714,11cyclictest42-21ksoftirqd/321:54:373
184464399713,15cyclictest141rcu_preempt23:35:153
184464399712,7cyclictest141rcu_preempt19:25:273
184464399712,7cyclictest141rcu_preempt19:25:273
1844643997124,7cyclictest42-21ksoftirqd/322:36:583
1844643997110,8cyclictest42-21ksoftirqd/323:00:593
1844638997163,5cyclictest0-21swapper/219:53:022
1844638997117,48cyclictest881251-21Softwar~cThread22:15:082
1844638997117,48cyclictest881251-21Softwar~cThread22:15:082
184464399708,12cyclictest141rcu_preempt23:34:403
184464399708,12cyclictest141rcu_preempt23:34:403
184464399706,6cyclictest141rcu_preempt20:00:243
184464399703,7cyclictest141rcu_preempt00:35:243
184464399703,63cyclictest1753-21gnome-shell22:11:143
184464399703,12cyclictest880726-21firefox22:18:133
184464399703,12cyclictest880726-21firefox22:18:133
1844634997063,4cyclictest0-21swapper/122:44:571
1844634997063,4cyclictest0-21swapper/122:44:571
184463499704,11cyclictest0-21swapper/123:05:031
1844634997014,51cyclictest881251-21Softwar~cThread22:16:121
1844634997014,51cyclictest881251-21Softwar~cThread22:16:121
162399700,20rtkit-daemon42-21ksoftirqd/319:30:442
184464399692,7cyclictest141rcu_preempt19:20:093
1844643996920,7cyclictest42-21ksoftirqd/323:48:383
184463899699,42cyclictest141rcu_preempt21:18:432
184464399682,7cyclictest141rcu_preempt19:50:293
184464399682,7cyclictest141rcu_preempt19:30:253
1844638996862,4cyclictest0-21swapper/221:35:232
1844638996861,4cyclictest0-21swapper/222:44:572
1844638996861,4cyclictest0-21swapper/222:44:572
1844638996841,9cyclictest1753-21gnome-shell23:50:192
1844638996812,15cyclictest881251-21Softwar~cThread22:00:552
184463899681,16cyclictest880726-21firefox21:21:312
1844634996862,4cyclictest0-21swapper/121:19:531
184463499685,8cyclictest0-21swapper/119:10:081
1844638996759,5cyclictest0-21swapper/222:28:272
1844638996745,5cyclictest141rcu_preempt21:33:072
184463899673,48cyclictest34-21ksoftirqd/222:55:272
184463499678,6cyclictest141rcu_preempt21:32:121
184463499674,59cyclictest1861559-21/usr/sbin/munin19:55:241
1844634996737,16cyclictest26-21ksoftirqd/122:45:221
19519922660,4sleep01951995-21grep00:15:020
184463899664,16cyclictest141rcu_preempt23:32:142
184463899664,16cyclictest141rcu_preempt23:32:142
184463499664,39cyclictest1880026-21kworker/1:021:20:141
1844643996515,46cyclictest1753-21gnome-shell22:34:283
1844634996555,7cyclictest26-21ksoftirqd/120:33:261
184463499652,59cyclictest1753-21gnome-shell19:25:011
1844634996511,49cyclictest1905704-21/usr/sbin/munin22:00:321
184464399642,9cyclictest141rcu_preempt20:35:003
184463899647,15cyclictest1900074-21if_enp2s021:45:232
1844638996421,39cyclictest34-21ksoftirqd/220:34:522
18879142630,4sleep20-21swapper/221:10:292
1844638996347,11cyclictest881251-21Softwar~cThread19:48:412
184463499632,7cyclictest0-21swapper/121:47:091
184463499632,56cyclictest1862701-21/usr/sbin/munin20:00:101
184463499631,19cyclictest1753-21gnome-shell23:30:291
184463499631,19cyclictest1753-21gnome-shell23:30:291
1844630996317,8cyclictest18879152ntpq21:10:290
184463899622,56cyclictest1753-21gnome-shell19:12:422
184463499626,13cyclictest881251-21Softwar~cThread00:35:361
184463499624,42cyclictest1932452-21kworker/1:223:50:141
1844638996143,14cyclictest34-21ksoftirqd/223:48:392
184463499612,17cyclictest1852616-21df_inode19:30:131
184463499602,54cyclictest1753-21gnome-shell19:49:531
18795952590,6chrt0-21swapper/020:45:360
184463899592,15cyclictest1753-21gnome-shell00:34:292
1844630995922,32cyclictest1865166-21ntp_kernel_pll_20:05:230
184464399582,15cyclictest0-21swapper/323:50:113
184463499581,7cyclictest141rcu_preempt22:15:001
184464399576,10cyclictest881251-21Softwar~cThread21:18:523
184464399563,12cyclictest1753-21gnome-shell00:34:223
184463499562,50cyclictest1622-21rtkit-daemon20:51:271
1844630995617,12cyclictest1884724-21cat21:00:310
1844630995617,12cyclictest1884724-21cat21:00:310
162399550,7rtkit-daemon1622-21rtkit-daemon23:59:310
184463099542,28cyclictest0-21swapper/023:50:170
1844638995316,17cyclictest34-21ksoftirqd/200:22:392
1844630995315,14cyclictest1870936-40apt-check20:21:190
162399530,8rtkit-daemon1622-21rtkit-daemon00:28:181
1844630995111,11cyclictest438-20systemd-journal00:06:200
184463099509,16cyclictest1753-21gnome-shell20:59:220
184463099502,20cyclictest1955266-21cron00:25:000
1844630995017,10cyclictest1881192-21sed20:50:320
1844630995011,13cyclictest881251-21Softwar~cThread19:43:470
19483442490,5chrt0-21swapper/200:05:022
184464399491,14cyclictest18678322date20:15:023
1844638994930,8cyclictest34-21ksoftirqd/220:59:312
1844638994913,19cyclictest34-21ksoftirqd/200:27:232
184463099498,14cyclictest1738-21gdbus00:01:560
1844630994913,11cyclictest1956074-21basename00:25:210
184463099490,44cyclictest1622-21rtkit-daemon00:17:560
184463899488,17cyclictest34-21ksoftirqd/220:05:452
184463099489,12cyclictest1871632-21cat20:25:130
184464399476,16cyclictest17531gnome-shell20:17:003
184463899471,7cyclictest1946390-40updatedb.mlocat00:00:012
1844638994711,9cyclictest34-21ksoftirqd/220:44:562
1844634994721,8cyclictest26-21ksoftirqd/120:42:381
1844630994715,7cyclictest1940536-21processes23:40:300
184463099471,22cyclictest1753-21gnome-shell20:16:020
1844630994711,11cyclictest1877372-21cat20:40:230
1844643994615,11cyclictest141rcu_preempt20:25:013
1844638994616,10cyclictest34-21ksoftirqd/219:35:282
184463499467,22cyclictest26-21ksoftirqd/100:20:271
1844634994612,15cyclictest26-21ksoftirqd/100:04:561
184463099461,23cyclictest1753-21gnome-shell19:35:310
162399460,32rtkit-daemon0-21swapper/219:42:492
1844638994516,6cyclictest141rcu_preempt00:07:032
184463499458,20cyclictest26-21ksoftirqd/120:30:011
1844634994518,15cyclictest26-21ksoftirqd/120:15:151
1844630994510,12cyclictest881251-21Softwar~cThread20:14:200
1844643994425,15cyclictest1753-21gnome-shell00:27:453
1844643994414,11cyclictest141rcu_preempt20:45:173
1844638994421,9cyclictest34-21ksoftirqd/220:20:082
1844638994416,8cyclictest34-21ksoftirqd/221:00:282
1844638994416,8cyclictest34-21ksoftirqd/221:00:282
184463499449,5cyclictest141rcu_preempt23:45:001
1844634994416,6cyclictest1753-21gnome-shell20:05:261
184463899433,20cyclictest34-21ksoftirqd/200:15:222
1844638994318,9cyclictest34-21ksoftirqd/220:47:352
1844638994317,9cyclictest34-21ksoftirqd/220:16:342
1844634994316,9cyclictest26-21ksoftirqd/121:13:471
18869492420,4sleep20-21swapper/221:10:002
184464399428,30cyclictest1753-21gnome-shell20:44:403
1844638994217,6cyclictest141rcu_preempt23:40:172
1844638994211,19cyclictest34-21ksoftirqd/200:15:012
184463499425,22cyclictest26-21ksoftirqd/121:01:421
184463499425,22cyclictest26-21ksoftirqd/121:01:411
1844634994213,6cyclictest141rcu_preempt00:16:391
184463499421,18cyclictest26-21ksoftirqd/119:41:231
162399420,6rtkit-daemon0-21swapper/319:41:093
184463899415,18cyclictest34-21ksoftirqd/220:12:492
1844638994119,5cyclictest141rcu_preempt20:53:302
184463499419,9cyclictest26-21ksoftirqd/100:00:001
184463499417,20cyclictest26-21ksoftirqd/120:55:301
1844634994120,8cyclictest26-21ksoftirqd/100:10:571
1844634994110,9cyclictest26-21ksoftirqd/120:25:001
1844630994112,9cyclictest1886226-21open_files21:05:320
1844643994024,11cyclictest881251-21Softwar~cThread23:58:393
1844643994024,11cyclictest42-21ksoftirqd/321:04:243
1844643994024,11cyclictest42-21ksoftirqd/321:04:243
1844643994014,10cyclictest141rcu_preempt00:09:133
1844643994010,25cyclictest1882220-21expr20:55:143
1844634994018,5cyclictest141rcu_preempt19:35:151
1844634994015,6cyclictest141rcu_preempt20:45:391
184464399392,8cyclictest1753-21gnome-shell21:06:433
1844643993924,10cyclictest1855268-21sshd19:37:493
1844643993923,11cyclictest42-21ksoftirqd/320:25:293
1844643993919,15cyclictest855-21dbus-daemon00:25:013
1844643993916,11cyclictest141rcu_preempt00:17:213
1844634993922,13cyclictest1948441-21awk00:05:111
1844634993919,5cyclictest141rcu_preempt21:05:361
1844634993914,5cyclictest141rcu_preempt20:14:461
1844643993824,10cyclictest1939816-21tr23:40:153
1844643993710,12cyclictest141rcu_preempt20:05:113
1844643993624,8cyclictest1753-21gnome-shell00:03:483
1844643993622,10cyclictest42-21ksoftirqd/300:13:213
1844643993419,11cyclictest1881220-21sed20:50:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional