You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-09 - 06:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Celeron 33455 @1500 MHz, Linux 5.15.2-rt19 (Profile)

Latency plot of shadow in rack #c, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackcslot7s.osadl.org (updated Tue Aug 09, 2022 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991030,9rtkit-daemon1622-21rtkit-daemon22:40:150
1623991030,9rtkit-daemon1622-21rtkit-daemon22:40:150
162399990,47rtkit-daemon0-21swapper/019:08:260
28262242970,4chrt0-21swapper/219:50:172
28152142930,6sleep30-21swapper/319:20:143
28190152900,5sleep30-21swapper/319:30:173
281074028570,9sleep10-21swapper/119:08:551
281067028469,9sleep30-21swapper/319:08:003
2810985997821,35cyclictest2812868-21munin-node20:20:121
2810985997821,35cyclictest2812868-21munin-node20:20:121
281070127645,22sleep20-21swapper/219:08:232
28696982740,5sleep00-21swapper/021:55:200
28696982740,5sleep00-21swapper/021:55:190
2810985997419,27cyclictest2907683-40apt-check23:42:261
281098599741,32cyclictest2851430-21sed21:00:341
29134532720,7sleep126-21ksoftirqd/100:00:161
162399700,10rtkit-daemon1622-21rtkit-daemon00:26:410
29193852680,6sleep3411rcuc/300:15:283
281098599682,47cyclictest2871784-40lsb_release22:00:201
2810985996821,23cyclictest1-21systemd19:30:011
281098599662,39cyclictest1069-21runrttasks23:00:241
281098599662,39cyclictest1069-21runrttasks23:00:231
2810985996613,26cyclictest2886906-21latency_hist22:45:011
2810985996613,26cyclictest2886906-21latency_hist22:45:011
2810985996610,30cyclictest2842468-21sed20:35:261
281098599652,30cyclictest1753-21gnome-shell19:20:291
281098599651,35cyclictest2918430-21cat00:15:021
281098599647,33cyclictest2890522-21latency_hist22:55:021
281098599642,30cyclictest1753-21gnome-shell22:10:331
2810985996418,24cyclictest2841111-21sed20:30:321
2810985996417,26cyclictest1069-21runrttasks21:57:041
2810985996417,26cyclictest1069-21runrttasks21:57:041
2810985996411,29cyclictest2903423-21cat23:30:251
281098599639,28cyclictest2896898-21switchtime23:10:321
281098599632,32cyclictest2859376-21/usr/sbin/munin21:25:161
281098599632,32cyclictest1753-21gnome-shell22:15:241
281098599631,36cyclictest2848316-21taskset20:52:421
28346712620,7sleep342-21ksoftirqd/320:15:003
281098599622,29cyclictest2852828-21memory21:05:251
281098599621,36cyclictest2864082-21latency_hist21:40:011
281098599621,34cyclictest1069-21runrttasks23:20:161
281098599621,33cyclictest2926882-21taskset00:36:251
281098599621,33cyclictest2891127-21/usr/sbin/munin22:55:221
162399620,9rtkit-daemon1622-21rtkit-daemon21:21:211
281098599612,30cyclictest1753-21gnome-shell19:51:271
281098599611,36cyclictest2823658-21taskset19:41:301
162399610,10rtkit-daemon1622-21rtkit-daemon22:20:291
281098599601,33cyclictest2921500-21sed00:20:331
281098599601,33cyclictest1069-21runrttasks20:57:151
281098599601,31cyclictest2887401-21expr22:45:171
281098599601,31cyclictest2835333-21hddtemp_smartct20:15:201
281098599601,30cyclictest2833414-21cstates20:10:181
29043742590,9sleep00-21swapper/023:35:000
281098599592,33cyclictest2829645-21cnf-update-db20:00:111
281098599592,30cyclictest1753-21gnome-shell22:35:341
281098599591,33cyclictest2867750-21cut21:50:021
281098599591,30cyclictest2924853-21python300:30:321
281098599582,31cyclictest1753-21gnome-shell22:27:251
281098599582,29cyclictest2912554-21sensors_temp23:55:291
281098599581,34cyclictest2910964-21sed23:50:311
281098599581,34cyclictest2910964-21sed23:50:311
281098599581,30cyclictest2920064-21taskset00:19:461
281098599581,30cyclictest2829351-21https20:00:001
281098599571,35cyclictest2843549-21runrttasks20:40:021
162399570,9rtkit-daemon1622-21rtkit-daemon23:17:271
162399570,31rtkit-daemon1753-21gnome-shell21:16:102
281098599562,30cyclictest1753-21gnome-shell19:30:331
281098599562,29cyclictest1753-21gnome-shell19:10:211
281098599562,27cyclictest2894881-21sed23:05:261
281098599561,31cyclictest2881959-21/usr/sbin/munin22:30:311
281098599561,29cyclictest2923888-21cat00:30:021
2810982995634,8cyclictest13-21ksoftirqd/022:30:020
162399560,46rtkit-daemon34-21ksoftirqd/200:20:322
2810992995527,23cyclictest42-21ksoftirqd/320:20:283
2810992995527,23cyclictest42-21ksoftirqd/320:20:283
281098299554,10cyclictest13-21ksoftirqd/022:00:280
162399550,9rtkit-daemon1622-21rtkit-daemon21:13:590
162399550,8rtkit-daemon1622-21rtkit-daemon23:26:061
281098599542,48cyclictest1753-21gnome-shell20:25:491
281098599542,29cyclictest2845682-21df20:45:171
281098599542,29cyclictest2845682-21df20:45:171
281098599541,30cyclictest2873188-21apt22:05:191
2810982995426,24cyclictest2834080-21smartctl20:10:330
281099299532,23cyclictest141rcu_preempt19:10:323
281098299538,13cyclictest13-21ksoftirqd/020:55:000
2810982995328,8cyclictest13-21ksoftirqd/023:44:470
2810982995318,17cyclictest13-21ksoftirqd/019:30:290
2810982995317,20cyclictest13-21ksoftirqd/021:00:000
28822832520,5sleep30-21swapper/322:30:193
2810982995233,8cyclictest13-21ksoftirqd/023:25:270
2810982995227,5cyclictest141rcu_preempt19:48:550
2810982995215,20cyclictest13-21ksoftirqd/022:10:000
281098299521,11cyclictest2920128-21http00:20:000
2810982995126,5cyclictest141rcu_preempt21:53:060
2810990995015,11cyclictest2855048-21cat21:10:352
281098599505,40cyclictest2908287-21sort23:45:151
281098299504,24cyclictest13-21ksoftirqd/000:15:020
2810982995026,9cyclictest13-21ksoftirqd/023:00:180
2810982995026,9cyclictest13-21ksoftirqd/023:00:180
2810982995026,8cyclictest13-21ksoftirqd/000:05:370
2810990994910,5cyclictest141rcu_preempt23:00:262
2810990994910,5cyclictest141rcu_preempt23:00:262
281098599491,44cyclictest2916580-21taskset00:09:291
281098299497,13cyclictest13-21ksoftirqd/021:20:230
2810982994927,5cyclictest141rcu_preempt22:30:290
2810982994918,8cyclictest2890361-21latency_hist22:55:020
2810982994917,13cyclictest13-21ksoftirqd/019:20:290
2810982994913,20cyclictest13-21ksoftirqd/020:25:260
2810982994911,20cyclictest13-21ksoftirqd/022:25:010
2810982994911,19cyclictest13-21ksoftirqd/020:00:210
2810982994910,18cyclictest13-21ksoftirqd/020:15:530
2810992994825,18cyclictest42-21ksoftirqd/320:00:263
281098599485,39cyclictest1824-21JS14:31:441
281098299486,20cyclictest13-21ksoftirqd/019:30:020
281098299486,18cyclictest13-21ksoftirqd/020:07:060
2810982994828,8cyclictest13-21ksoftirqd/022:57:550
2810982994824,8cyclictest13-21ksoftirqd/022:19:210
2810982994823,20cyclictest2845080-21unin-run20:45:010
2810982994815,7cyclictest13-21ksoftirqd/023:10:260
2810982994815,14cyclictest13-21ksoftirqd/022:10:330
2810982994810,19cyclictest13-21ksoftirqd/022:35:200
2810992994725,18cyclictest42-21ksoftirqd/319:15:143
2810992994725,17cyclictest42-21ksoftirqd/322:35:183
2810992994722,21cyclictest42-21ksoftirqd/322:18:443
281098599472,27cyclictest2821544-21users19:35:331
2810985994713,29cyclictest2831843-21df20:05:221
2810982994723,5cyclictest141rcu_preempt21:31:110
281098299472,13cyclictest13-21ksoftirqd/023:20:020
281098299471,29cyclictest2864741-21entropy21:40:210
281098299471,12cyclictest2913170-21awk00:00:000
29077142460,8sleep234-21ksoftirqd/223:42:222
281099299466,14cyclictest141rcu_preempt20:25:143
281099299465,36cyclictest2856435-21awk21:15:263
2810992994627,8cyclictest1753-21gnome-shell22:52:553
2810992994618,15cyclictest42-21ksoftirqd/300:00:313
2810992994615,6cyclictest141rcu_preempt19:40:453
2810992994614,6cyclictest141rcu_preempt23:10:323
281098599462,40cyclictest1069-21runrttasks21:18:561
281098299469,20cyclictest13-21ksoftirqd/023:40:010
281098299464,11cyclictest13-21ksoftirqd/021:40:010
2810982994627,8cyclictest13-21ksoftirqd/019:19:210
2810982994625,17cyclictest2837691-21sort20:20:310
2810982994625,17cyclictest2837691-21sort20:20:310
2810982994614,10cyclictest13-21ksoftirqd/023:20:160
2810982994611,19cyclictest13-21ksoftirqd/023:50:260
2810982994611,19cyclictest13-21ksoftirqd/023:50:260
2810982994611,10cyclictest13-21ksoftirqd/023:10:000
281098299461,10cyclictest13-21ksoftirqd/000:39:170
281099299458,18cyclictest1753-21gnome-shell21:53:013
2810992994527,7cyclictest1753-21gnome-shell21:07:253
2810992994523,10cyclictest42-21ksoftirqd/321:30:253
2810992994520,21cyclictest1753-21gnome-shell21:40:173
2810990994512,16cyclictest141rcu_preempt22:50:312
281098299458,20cyclictest13-21ksoftirqd/019:10:170
281098299457,12cyclictest2866118-21apt21:45:140
281098299456,16cyclictest13-21ksoftirqd/019:55:290
281098299456,12cyclictest13-21ksoftirqd/019:40:200
2810982994534,8cyclictest161rcuc/022:45:390
2810982994526,8cyclictest13-21ksoftirqd/020:47:000
2810982994526,8cyclictest13-21ksoftirqd/020:47:000
2810982994526,8cyclictest13-21ksoftirqd/019:50:560
2810982994524,8cyclictest13-21ksoftirqd/000:20:570
281098299451,11cyclictest2914911-21munin-run00:05:000
281098299451,11cyclictest13-21ksoftirqd/023:50:000
2810982994511,19cyclictest13-21ksoftirqd/020:30:180
281099299446,12cyclictest42-21ksoftirqd/322:40:413
281099299446,12cyclictest42-21ksoftirqd/322:40:413
281099299446,12cyclictest42-21ksoftirqd/320:53:073
2810992994423,11cyclictest42-21ksoftirqd/319:58:203
281099299442,24cyclictest1753-21gnome-shell00:05:513
2810992994418,7cyclictest2846302-21timerandwakeup20:45:313
2810992994418,7cyclictest2846302-21timerandwakeup20:45:313
2810992994418,6cyclictest141rcu_preempt23:15:513
2810992994412,28cyclictest2924925-21sed00:30:333
281099299441,11cyclictest2816828-21/usr/sbin/munin19:25:293
2810990994426,14cyclictest2813176-21if_enp2s019:15:052
2810990994410,20cyclictest141rcu_preempt20:45:252
2810990994410,20cyclictest141rcu_preempt20:45:242
281098599442,27cyclictest2865647-21chrt21:42:361
281098299446,16cyclictest13-21ksoftirqd/021:15:140
281098299444,14cyclictest13-21ksoftirqd/021:05:180
281098299441,17cyclictest2822357-21apt-config19:40:010
281099299438,17cyclictest1753-21gnome-shell23:04:143
281099299438,17cyclictest1753-21gnome-shell23:04:143
2810992994325,6cyclictest42-21ksoftirqd/319:35:393
2810992994322,10cyclictest42-21ksoftirqd/320:09:303
2810992994318,7cyclictest1753-21gnome-shell23:05:323
2810990994315,6cyclictest141rcu_preempt19:40:302
2810990994315,12cyclictest1753-21gnome-shell22:39:412
2810990994313,12cyclictest141rcu_preempt20:05:212
2810985994318,21cyclictest2853948-21/usr/sbin/munin21:10:241
2810985994316,22cyclictest2904781-21cpuspeed_turbos23:35:161
281098599431,38cyclictest2813115-21munin-node19:15:051
281098599431,36cyclictest2861811-21sed21:30:331
281098299437,17cyclictest13-21ksoftirqd/000:30:320
2810982994327,12cyclictest2842715-21cut20:35:320
2810982994324,8cyclictest13-21ksoftirqd/021:29:570
2810982994318,22cyclictest13-21ksoftirqd/021:00:200
29076992420,6chrt0-21swapper/323:42:193
281099299427,31cyclictest1753-21gnome-shell21:01:403
2810992994225,6cyclictest42-21ksoftirqd/322:01:563
2810992994220,10cyclictest42-21ksoftirqd/321:29:483
2810992994216,22cyclictest1753-21gnome-shell23:39:093
2810992994214,6cyclictest141rcu_preempt23:20:303
2810990994215,10cyclictest1753-21gnome-shell22:24:592
2810990994212,12cyclictest141rcu_preempt00:05:222
281098599421,36cyclictest2868528-21needreboot21:50:241
281099299417,16cyclictest1753-21gnome-shell21:23:513
281099299416,16cyclictest1753-21gnome-shell00:10:343
2810992994117,20cyclictest2873172-21acpi22:05:173
2810992994115,22cyclictest2891652-21timerwakeupswit22:55:313
2810992994114,22cyclictest2913188-40mandb00:00:023
2810990994130,7cyclictest2815095-21/usr/sbin/munin19:20:322
2810990994123,13cyclictest1787-21gdbus19:39:302
2810990994112,15cyclictest1753-21gnome-shell21:48:072
281099299409,27cyclictest1753-21gnome-shell22:25:003
281099299407,6cyclictest141rcu_preempt21:58:463
281099299407,6cyclictest141rcu_preempt21:58:453
281099299406,11cyclictest1069-21runrttasks23:30:263
2810992994022,6cyclictest42-21ksoftirqd/319:50:243
2810992994012,10cyclictest2881713-21perf22:30:003
281099099409,6cyclictest141rcu_preempt19:25:292
281099099408,13cyclictest1753-21gnome-shell21:05:262
281099099407,6cyclictest141rcu_preempt19:35:012
281099099407,10cyclictest34-21ksoftirqd/220:11:372
2810990994023,13cyclictest2900438-21cat23:20:312
2810990994011,6cyclictest141rcu_preempt20:15:162
2810990994011,14cyclictest141rcu_preempt21:00:362
281099299395,16cyclictest1753-21gnome-shell23:27:423
2810992993922,12cyclictest2908590-21df23:45:173
2810992993921,6cyclictest42-21ksoftirqd/323:50:163
2810992993921,6cyclictest42-21ksoftirqd/323:50:163
2810992993921,6cyclictest42-21ksoftirqd/320:42:023
281099299392,13cyclictest1753-21gnome-shell21:39:053
2810992993919,16cyclictest1753-21gnome-shell00:36:133
2810992993917,11cyclictest2875043-21grep22:10:173
2810992993913,7cyclictest1753-21gnome-shell21:14:333
281099099396,9cyclictest34-21ksoftirqd/220:26:342
281099099393,4cyclictest141rcu_preempt20:35:202
281099099393,31cyclictest1753-21gnome-shell22:35:002
2810990993925,10cyclictest2919785-21sshd00:16:122
2810990993925,10cyclictest2825777-21mktemp19:50:012
2810990993924,11cyclictest1753-21gnome-shell21:42:332
2810990993923,11cyclictest2962348-21rs:main2
2810990993922,12cyclictest438-20systemd-journal22:06:122
2810990993922,12cyclictest2912418-21ntpq23:55:252
2810990993921,14cyclictest2863587-21awk21:35:362
2810990993914,6cyclictest141rcu_preempt22:00:182
2810990993913,5cyclictest141rcu_preempt00:30:252
2810990993913,10cyclictest1753-21gnome-shell23:28:212
2810990993911,13cyclictest141rcu_preempt19:55:302
2810992993823,10cyclictest1753-21gnome-shell22:45:143
2810992993822,5cyclictest42-21ksoftirqd/319:49:413
2810992993821,6cyclictest42-21ksoftirqd/320:15:333
2810992993821,12cyclictest2843304-21cron20:40:003
2810992993820,14cyclictest1753-21gnome-shell21:46:053
281099099389,5cyclictest141rcu_preempt20:50:272
281099099386,13cyclictest1753-21gnome-shell20:32:072
281099099386,12cyclictest1753-21gnome-shell20:20:222
281099099386,12cyclictest1753-21gnome-shell20:20:222
281099099382,10cyclictest34-21ksoftirqd/222:25:142
2810990993819,14cyclictest2908597-21df_inode23:45:182
2810990993813,6cyclictest141rcu_preempt23:30:262
2810990993811,7cyclictest141rcu_preempt23:50:142
2810990993811,7cyclictest141rcu_preempt23:50:142
2810990993810,5cyclictest141rcu_preempt22:47:332
2810990993810,15cyclictest141rcu_preempt21:55:212
2810990993810,15cyclictest141rcu_preempt21:55:212
2810990993810,11cyclictest34-21ksoftirqd/223:07:092
2810992993713,20cyclictest391irq_work/320:55:233
2810992993713,20cyclictest1753-21gnome-shell00:20:153
2810992993711,22cyclictest1753-21gnome-shell00:27:263
281099099375,11cyclictest2812195-21timerwakeupswit19:10:312
281099099372,6cyclictest141rcu_preempt00:10:192
281099099372,5cyclictest141rcu_preempt23:10:192
281099099372,31cyclictest1753-21gnome-shell23:18:142
2810990993721,12cyclictest2858012-21load21:20:232
2810992993613,19cyclictest2840743-21missed_timers20:30:253
281099099369,6cyclictest141rcu_preempt22:40:292
281099099369,6cyclictest141rcu_preempt22:40:292
281099099368,5cyclictest141rcu_preempt22:55:232
281099099368,14cyclictest141rcu_preempt21:30:212
2810990993626,7cyclictest1753-21gnome-shell00:29:512
2810990993618,13cyclictest2906192-21appstreamcli23:40:012
281099099361,31cyclictest2860650-21sleep21:30:002
2810990993610,5cyclictest141rcu_preempt20:00:272
281099099352,5cyclictest141rcu_preempt20:55:252
281099099343,6cyclictest141rcu_preempt00:35:222
281099099342,5cyclictest141rcu_preempt21:50:122
281099099341,5cyclictest141rcu_preempt22:10:302
2810990993411,6cyclictest141rcu_preempt20:45:012
2810990993410,11cyclictest141rcu_preempt22:15:322
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional