You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-02-09 - 08:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Sun Feb 09, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5330992522,2cyclictest0-21swapper/000:02:320
533099250,24cyclictest27811-21apt-get19:57:370
5340992423,0cyclictest0-21swapper/322:20:023
5340992421,2cyclictest313-21systemd-journal23:44:593
5330992419,4cyclictest12285-21perf00:20:000
5330992418,5cyclictest24935-21cron19:50:000
533099240,20cyclictest14668-21perf20:35:010
5340992321,1cyclictest32177-21munin-plugin-st20:05:013
5340992321,1cyclictest0-21swapper/323:00:003
5340992321,1cyclictest0-21swapper/319:47:323
534099230,22cyclictest8845-21apt-get00:10:023
534099230,22cyclictest867-21cron23:50:003
533899230,22cyclictest6905-21apt-get00:07:312
533899230,22cyclictest29463-21latency_hist21:05:002
5333992321,1cyclictest25673-21cron23:35:001
533399230,0cyclictest0-21swapper/122:37:271
5330992319,3cyclictest0-21swapper/022:27:320
533099230,0cyclictest0-21swapper/021:27:310
5340992222,0cyclictest0-21swapper/319:32:313
5340992220,1cyclictest2301-21apt-get21:17:323
534099220,21cyclictest3922-21sh00:00:013
534099220,21cyclictest22871-21apt-get22:07:323
534099220,21cyclictest19868-21apt-get20:47:323
534099220,21cyclictest16962-21grep21:50:003
534099220,1cyclictest28095-21awk23:39:593
5338992220,1cyclictest17081-21apt00:29:592
5338992219,2cyclictest4236-21sh21:19:592
5338992219,2cyclictest313-21systemd-journal23:10:002
533899220,21cyclictest12948-21awk19:25:002
533899220,21cyclictest0-21swapper/200:37:272
5333992221,0cyclictest20475-21apt-get22:02:311
5333992221,0cyclictest0-21swapper/123:27:331
5333992220,1cyclictest28496-21apt-get22:20:021
533399220,21cyclictest8852-21turbostat00:10:001
533399220,21cyclictest20147-21sendmail-msp00:40:001
533399220,21cyclictest0-21swapper/119:12:301
533399220,1cyclictest14821-21munin-run23:05:001
533399220,1cyclictest0-21swapper/123:25:011
5330992221,1cyclictest0-21swapper/023:37:250
5330992221,0cyclictest11957-21apt-get23:00:020
5330992221,0cyclictest0-21swapper/022:02:310
5330992221,0cyclictest0-21swapper/021:22:320
5330992220,2cyclictest0-21swapper/020:57:330
5330992220,1cyclictest9874-21sh00:15:010
5330992220,1cyclictest17618-21munin-run21:54:590
5330992220,1cyclictest16978-21idleruntime-cro21:50:000
5330992219,3cyclictest32308-21apt-get21:12:290
5330992218,3cyclictest0-21swapper/023:42:300
533099220,21cyclictest17474-21apt-get20:42:320
533099220,21cyclictest0-21swapper/019:12:300
533099220,21cyclictest0-21swapper/000:27:310
533099220,1cyclictest8147-21apt-get19:17:290
5340992121,0cyclictest0-21swapper/321:22:303
5340992120,1cyclictest2599-21apt-get20:12:323
5340992119,1cyclictest17838-21perf23:15:003
5340992119,1cyclictest0-21swapper/319:55:003
534099210,21cyclictest0-21swapper/323:27:323
534099210,21cyclictest0-21swapper/319:57:373
534099210,21cyclictest0-21swapper/319:12:303
534099210,20cyclictest0-21swapper/320:07:313
534099210,20cyclictest0-21swapper/319:22:313
534099210,1cyclictest17612-21perf21:54:593
5338992120,1cyclictest15372-21awk19:29:592
5338992120,1cyclictest14617-21perf20:35:002
5338992119,1cyclictest873-21dump-pmu-power23:50:002
533899210,20cyclictest0-21swapper/222:47:342
533899210,20cyclictest0-21swapper/222:22:302
533899210,20cyclictest0-21swapper/222:07:312
533899210,20cyclictest0-21swapper/221:37:312
533899210,1cyclictest0-21swapper/220:00:012
5333992121,0cyclictest31358-21apt-get23:47:321
5333992121,0cyclictest0-21swapper/122:25:021
5333992120,1cyclictest0-21swapper/120:12:311
533399210,21cyclictest0-21swapper/120:22:291
533399210,21cyclictest0-21swapper/119:57:371
533399210,20cyclictest23017-21apt-get19:47:341
533399210,20cyclictest0-21swapper/120:47:321
533399210,0cyclictest0-21swapper/123:40:011
5330992121,0cyclictest0-21swapper/000:07:330
5330992118,3cyclictest6963-21apt-get22:47:330
5330992118,3cyclictest0-21swapper/020:17:280
533099210,20cyclictest0-21swapper/023:07:310
533099210,20cyclictest0-21swapper/020:15:000
533099210,0cyclictest0-21swapper/019:37:320
5340992020,0cyclictest0-21swapper/323:02:313
5340992020,0cyclictest0-21swapper/322:47:353
5340992020,0cyclictest0-21swapper/322:02:313
5340992020,0cyclictest0-21swapper/321:07:323
5340992020,0cyclictest0-21swapper/321:02:333
5340992020,0cyclictest0-21swapper/320:37:333
5340992020,0cyclictest0-21swapper/320:17:273
5340992018,1cyclictest0-21swapper/320:45:003
534099200,19cyclictest0-21swapper/321:42:313
534099200,19cyclictest0-21swapper/321:27:303
534099200,0cyclictest0-21swapper/322:37:263
5338992020,0cyclictest22435-21apt-get20:50:052
5338992020,0cyclictest0-21swapper/223:17:262
5338992020,0cyclictest0-21swapper/222:37:272
5338992020,0cyclictest0-21swapper/221:12:292
5338992020,0cyclictest0-21swapper/220:22:292
5338992019,1cyclictest31508-21apt-get22:27:312
5338992019,1cyclictest23109-21apt-get23:27:332
5338992017,2cyclictest10117-21latency_hist19:20:012
533899200,0cyclictest0-21swapper/221:27:302
533899200,0cyclictest0-21swapper/220:37:332
533899200,0cyclictest0-21swapper/220:02:332
533899200,0cyclictest0-21swapper/219:52:312
533899200,0cyclictest0-21swapper/219:47:332
5333992020,0cyclictest9991-21apt-get22:57:261
5333992020,0cyclictest7117-21apt-get21:27:301
5333992020,0cyclictest0-21swapper/123:57:261
5333992020,0cyclictest0-21swapper/121:57:251
5333992020,0cyclictest0-21swapper/121:47:321
5333992020,0cyclictest0-21swapper/120:32:301
5333992018,1cyclictest30219-21apt-get20:02:321
533399200,20cyclictest0-21swapper/120:42:331
533399200,0cyclictest10158-21apt-get21:37:321
533399200,0cyclictest0-21swapper/120:27:311
533399200,0cyclictest0-21swapper/119:42:311
5330992020,0cyclictest2134-21apt-get22:37:260
5330992020,0cyclictest0-21swapper/022:22:310
5330992020,0cyclictest0-21swapper/020:52:290
5330992020,0cyclictest0-21swapper/000:37:260
5330992019,0cyclictest10960-21apt-get19:22:300
533099200,20cyclictest0-21swapper/021:37:310
5340991917,1cyclictest20012-21perf22:00:003
5340991917,1cyclictest0-21swapper/300:34:593
534099190,19cyclictest0-21swapper/322:25:013
5338991919,0cyclictest0-21swapper/223:22:302
5338991919,0cyclictest0-21swapper/223:02:302
5338991919,0cyclictest0-21swapper/221:22:302
5338991919,0cyclictest0-21swapper/220:47:332
5338991918,1cyclictest10213-21apt-get20:27:312
5338991918,0cyclictest26141-21apt-get23:37:272
5338991917,1cyclictest12283-21awk00:19:592
5338991916,2cyclictest11941-21grep22:59:592
5333991919,0cyclictest10333-21apt-get00:17:271
5333991919,0cyclictest0-21swapper/122:07:311
5333991919,0cyclictest0-21swapper/121:12:291
5333991919,0cyclictest0-21swapper/121:02:321
5333991919,0cyclictest0-21swapper/120:57:331
5333991918,1cyclictest25407-21apt-get19:52:311
5333991918,0cyclictest0-21swapper/100:27:321
5333991917,1cyclictest15348-21basename19:29:591
5330991919,0cyclictest0-21swapper/021:02:310
5330991919,0cyclictest0-21swapper/020:02:320
5330991919,0cyclictest0-21swapper/000:22:330
533099190,0cyclictest18290-21apt-get23:17:270
5340991818,0cyclictest0-21swapper/319:42:303
5340991818,0cyclictest0-21swapper/319:37:313
5340991817,0cyclictest0-21swapper/300:07:323
5340991816,1cyclictest27859-21perf22:20:013
5338991818,0cyclictest5747-21apt-get19:12:312
5338991816,1cyclictest0-21swapper/222:54:592
5333991818,0cyclictest21818-21apt-get20:50:051
5333991818,0cyclictest0-21swapper/121:07:301
5333991818,0cyclictest0-21swapper/120:17:271
533399180,17cyclictest4715-21apt-get21:22:311
5330991818,0cyclictest0-21swapper/023:27:320
5330991817,1cyclictest24820-21perf22:09:590
5330991816,1cyclictest0-21swapper/020:25:000
5340991717,0cyclictest0-21swapper/322:42:313
5340991717,0cyclictest0-21swapper/320:57:333
5340991717,0cyclictest0-21swapper/320:52:293
5340991717,0cyclictest0-21swapper/319:27:313
5340991717,0cyclictest0-21swapper/300:27:313
5338991717,0cyclictest0-21swapper/222:17:282
5338991717,0cyclictest0-21swapper/221:47:322
5338991717,0cyclictest0-21swapper/219:32:312
5338991716,0cyclictest4992-21apt-get20:17:272
5338991716,0cyclictest0-21swapper/220:12:312
5333991717,0cyclictest0-21swapper/121:42:321
5330991717,0cyclictest0-21swapper/023:47:320
5330991717,0cyclictest0-21swapper/020:47:320
533099170,16cyclictest6191-21unixbench_multi22:42:390
5340991616,0cyclictest0-21swapper/323:07:303
5340991616,0cyclictest0-21swapper/320:27:313
5340991614,1cyclictest7813-21apt-get20:22:293
5340991614,1cyclictest612-21gdbus22:15:013
5340991614,1cyclictest18715-21diskstats23:17:283
5340991613,2cyclictest10754-21df_inode00:17:283
534099160,15cyclictest578-21irqbalance22:53:183
5338991616,0cyclictest28959-21apt-get23:42:312
5338991616,0cyclictest0-21swapper/220:57:342
5338991616,0cyclictest0-21swapper/200:22:322
5338991614,1cyclictest0-21swapper/222:02:372
533899160,9cyclictest0-21swapper/222:40:122
533899160,15cyclictest20177-21munin-run19:39:592
5333991616,0cyclictest0-21swapper/119:37:331
5333991616,0cyclictest0-21swapper/119:20:021
5333991616,0cyclictest0-21swapper/119:17:291
533399160,15cyclictest93122sleep121:30:181
5330991616,0cyclictest0-21swapper/021:57:250
5330991616,0cyclictest0-21swapper/021:07:310
533099160,0cyclictest0-21swapper/019:27:300
5340991515,0cyclictest4503-21apt-get00:02:323
5340991515,0cyclictest0-21swapper/300:37:263
5340991514,1cyclictest0-21swapper/300:22:323
534099150,1cyclictest21134-21diskstats23:22:313
5338991513,2cyclictest17728-21sh00:35:002
5338991513,1cyclictest14575-21awk21:45:002
5338991512,2cyclictest18240-21cpuspeed_turbos21:57:272
533899150,14cyclictest0-21swapper/220:07:342
5333991515,0cyclictest0-21swapper/121:17:311
5333991514,0cyclictest32613-21apt-get20:07:301
5333991513,1cyclictest26143-21runrttasks23:37:251
5330991515,0cyclictest0-21swapper/022:57:260
5330991515,0cyclictest0-21swapper/020:27:320
5330991512,2cyclictest25456-21latency_hist22:15:000
533099150,1cyclictest0-21swapper/023:56:460
533099150,14cyclictest17477-21/usr/sbin/munin19:32:410
5340991414,0cyclictest0-21swapper/321:35:133
534099140,14cyclictest0-21swapper/321:30:563
534099140,14cyclictest0-21swapper/320:30:303
534099140,0cyclictest0-21swapper/323:34:553
534099140,0cyclictest0-21swapper/322:34:123
534099140,0cyclictest0-21swapper/321:11:313
5338991414,0cyclictest0-21swapper/223:54:142
5338991414,0cyclictest0-21swapper/222:31:322
533899140,14cyclictest0-21swapper/223:55:002
533899140,14cyclictest0-21swapper/223:31:492
533899140,14cyclictest0-21swapper/223:10:332
533899140,14cyclictest0-21swapper/222:12:502
533899140,14cyclictest0-21swapper/221:50:562
533899140,14cyclictest0-21swapper/221:31:092
533899140,14cyclictest0-21swapper/221:07:202
533899140,14cyclictest0-21swapper/200:10:552
533899140,14cyclictest0-21swapper/200:01:262
533899140,0cyclictest0-21swapper/220:40:572
533899140,0cyclictest0-21swapper/219:41:172
5333991414,0cyclictest0-21swapper/123:50:271
5333991414,0cyclictest0-21swapper/123:12:261
5333991414,0cyclictest0-21swapper/122:42:241
5333991414,0cyclictest0-21swapper/122:15:001
5333991414,0cyclictest0-21swapper/122:14:011
5333991414,0cyclictest0-21swapper/120:37:451
533399140,14cyclictest0-21swapper/122:30:411
533399140,14cyclictest0-21swapper/100:10:221
533399140,0cyclictest0-21swapper/122:52:011
533399140,0cyclictest0-21swapper/122:46:081
533399140,0cyclictest0-21swapper/119:30:251
533399140,0cyclictest0-21swapper/100:30:011
5330991414,0cyclictest0-21swapper/023:50:450
5330991414,0cyclictest0-21swapper/023:10:030
5330991414,0cyclictest0-21swapper/021:40:240
5330991414,0cyclictest0-21swapper/020:07:070
5330991414,0cyclictest0-21swapper/000:30:100
5330991412,1cyclictest0-21swapper/021:20:000
533099140,14cyclictest0-21swapper/023:22:350
533099140,14cyclictest0-21swapper/022:30:290
533099140,14cyclictest0-21swapper/021:30:500
533099140,0cyclictest0-21swapper/023:33:080
533099140,0cyclictest0-21swapper/019:50:160
533099140,0cyclictest0-21swapper/019:40:020
5340991313,0cyclictest0-21swapper/323:52:493
5340991313,0cyclictest0-21swapper/319:16:533
5330991212,0cyclictest0-21swapper/022:53:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional