You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-11 - 23:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Jul 11, 2025 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3974575997672,2cyclictest0-21swapper/009:05:010
3974584996935,28cyclictest235-21systemd-journal08:00:012
3974584996935,28cyclictest235-21systemd-journal08:00:012
3974580996927,10cyclictest3983617-21cron10:05:001
3974575996932,8cyclictest3983618-21cron10:05:010
397458499682,18cyclictest3976109-21munin-run07:40:012
397458499681,14cyclictest3976330-21sshd07:44:562
397458499681,14cyclictest3976330-21sshd07:44:552
397458499671,12cyclictest3988809-21sshd11:43:222
397458499642,17cyclictest3985192-21munin-run10:35:022
397458499642,17cyclictest3985192-21munin-run10:35:022
397458499641,15cyclictest3980778-21munin-run09:10:002
3974584996329,30cyclictest3984382-21cron10:20:012
3974584996328,32cyclictest3975212-21latency_hist07:20:032
3974580996343,15cyclictest1-21systemd11:10:001
397458499622,14cyclictest3983393-21munin-run10:00:002
397458499622,13cyclictest3979226-21munin-run08:40:002
3974584996221,37cyclictest3991467-21latency_hist12:35:022
3974584996221,37cyclictest3991467-21latency_hist12:35:022
3974584996114,12cyclictest3984967-21cat10:30:022
397458499611,11cyclictest3983099-21munin-run09:55:002
3974589996037,16cyclictest1-21systemd11:40:003
397458499602,15cyclictest3989096-21munin-run11:50:022
397458499602,11cyclictest3976852-21munin-run07:55:002
397458499601,14cyclictest48-21kcompactd012:20:572
3974589995917,8cyclictest37-21ksoftirqd/312:35:023
3974589995917,8cyclictest37-21ksoftirqd/312:35:013
397458499598,3cyclictest0-21swapper/207:29:372
397458499592,13cyclictest3991661-21modprobe12:39:592
397458499592,13cyclictest3975817-21munin-run07:34:592
397458499591,9cyclictest3987554-21unin-run11:20:012
397458499591,9cyclictest3987554-21unin-run11:20:002
397458499591,17cyclictest3985420-21cron10:40:002
3974580995942,12cyclictest235-21systemd-journal07:25:011
3974580995941,14cyclictest1-21systemd10:10:001
397458999588,15cyclictest3976398-21sed07:45:023
397458999588,15cyclictest3976398-21sed07:45:013
3974584995825,28cyclictest3989441-21latency_hist11:55:012
3974584995814,40cyclictest3978390-21sshd08:20:032
397458499581,17cyclictest3982481-21cat09:40:042
397458499581,15cyclictest3984104-21sshd10:11:222
397458499581,13cyclictest3989874-21systemd-run12:05:002
397458499581,13cyclictest363-21snmpd08:09:292
397458499581,10cyclictest3986198-21sshd10:54:222
3974589995716,7cyclictest37-21ksoftirqd/312:30:013
397458499572,15cyclictest3988583-21modprobe11:39:592
397458499572,15cyclictest3987167-21cut11:10:022
397458499572,13cyclictest272-21dbus-daemon09:05:012
3974584995717,35cyclictest3985823-21latency_hist10:45:012
397458499571,12cyclictest363-21snmpd08:57:422
397458499571,10cyclictest3982042-21sshd09:33:212
3974589995639,12cyclictest1-21systemd07:40:013
3974589995637,14cyclictest1-21systemd08:20:013
3974584995623,26cyclictest306-21in:imuxsock09:15:012
397458499562,13cyclictest3990107-21cron12:10:002
397458499562,12cyclictest363-21snmpd07:12:042
397458499561,13cyclictest3982834-21cron09:50:012
397458499552,16cyclictest3981287-21munin-plugin-st09:20:002
397458499551,13cyclictest363-21snmpd08:13:352
397458499551,12cyclictest363-21snmpd11:25:262
397458499551,11cyclictest363-21snmpd08:51:312
397458499542,8cyclictest3977399-21unin-run08:05:002
397458499542,6cyclictest3986459-21modprobe10:59:592
397458499541,17cyclictest3981786-21sshd09:27:222
397458499541,13cyclictest3987985-21sed11:25:022
397458499541,11cyclictest3990678-21latency_hist12:20:022
397458499541,11cyclictest3990678-21latency_hist12:20:022
397458499532,15cyclictest3989587-21sh12:00:002
397458499532,12cyclictest3987212-21modprobe11:10:022
397458499531,13cyclictest1-21systemd09:45:012
397458499531,12cyclictest3991219-21date12:30:012
397458499531,12cyclictest363-21snmpd07:21:212
3974584995311,10cyclictest306-21in:imuxsock08:40:032
3974584995311,10cyclictest306-21in:imuxsock08:40:022
3974580995336,13cyclictest235-21systemd-journal09:10:011
397458499529,12cyclictest235-21systemd-journal10:10:002
397458499523,12cyclictest3990360-21modprobe12:15:002
397458499522,14cyclictest3983631-21/usr/sbin/munin10:05:012
397458499521,12cyclictest363-21snmpd08:16:502
397458499521,11cyclictest3981617-21date09:25:022
3974575995210,6cyclictest3977069-21kworker/u8:008:40:010
3974589995133,14cyclictest3991633-21sshd12:35:223
397458499512,14cyclictest3978680-21munin-run08:30:012
397458499512,14cyclictest3978680-21munin-run08:30:002
3974580995122,6cyclictest23-21ksoftirqd/111:55:011
3974575995133,14cyclictest1-21systemd08:55:000
3974589995011,7cyclictest37-21ksoftirqd/311:50:003
397458499501,14cyclictest3984681-21true10:25:002
397458499501,12cyclictest363-21snmpd07:49:442
397458499501,10cyclictest1-21systemd11:35:012
397458499501,10cyclictest1-21systemd11:35:012
3974580995033,11cyclictest1-21systemd09:45:011
3974575995030,16cyclictest1-21systemd07:55:000
3974589994932,13cyclictest3981243-21cron09:16:593
3974589994931,14cyclictest3989869-21munin-run12:05:013
3974589994931,14cyclictest3975529-21sshd07:26:213
397458499493,14cyclictest1-21systemd08:50:012
3974575994942,4cyclictest16950irq/224-800000008:05:020
3974589994829,15cyclictest3980983-21sshd09:10:223
3974589994828,16cyclictest3977399-21munin-run08:05:013
397458499486,7cyclictest30-21ksoftirqd/211:05:012
397458999474,15cyclictest3986983-21sshd11:09:223
3974589994733,10cyclictest3976567-21sshd07:46:233
3974589994732,11cyclictest3990874-21sshd12:20:223
3974589994723,20cyclictest3987217-21modprobe11:10:353
397458499471,10cyclictest3978925-21modprobe08:35:002
397457599478,5cyclictest131rcu_preempt09:05:020
3974575994733,10cyclictest272-21dbus-daemon08:30:010
3974575994733,10cyclictest272-21dbus-daemon08:30:010
3974575994730,13cyclictest272-21dbus-daemon11:55:000
3974575994730,13cyclictest272-21dbus-daemon10:40:000
3974589994630,12cyclictest3986760-21munin-run11:05:013
3974589994630,12cyclictest3985945-21sshd10:49:233
397458499461,11cyclictest3985923-21modprobe10:45:022
397457599462,3cyclictest131rcu_preempt11:25:000
397458999459,8cyclictest37-21ksoftirqd/308:10:013
3974589994532,9cyclictest3982574-21sshd09:43:213
3974589994532,9cyclictest3978403-21sshd08:22:223
3974589994531,10cyclictest3983393-21munin-run10:00:013
3974589994531,10cyclictest3977146-21munin-run08:00:023
3974589994531,10cyclictest3977146-21munin-run08:00:013
3974589994529,12cyclictest3981561-21munin-run09:25:013
397458099454,3cyclictest131rcu_preempt08:19:591
3974575994533,8cyclictest306-21in:imuxsock10:50:010
3974575994533,8cyclictest235-21systemd-journal08:45:010
3974575994533,8cyclictest235-21systemd-journal08:45:000
3974575994529,12cyclictest12-21ksoftirqd/012:20:020
3974575994529,12cyclictest12-21ksoftirqd/012:20:020
3974575994521,20cyclictest3985834-21latency_hist10:45:020
3974589994436,4cyclictest3980473-21sshd09:02:213
3974589994431,10cyclictest3986176-21sshd10:50:223
3974589994431,10cyclictest3983855-21sshd10:06:213
3974589994431,10cyclictest3981793-21sshd09:28:223
3974589994429,11cyclictest3990615-21sshd12:19:223
3974589994429,11cyclictest3990615-21sshd12:19:223
3974589994429,11cyclictest3984466-21sed10:20:013
3974589994426,14cyclictest363-21snmpd10:28:433
397458499441,13cyclictest3974700-21latency_hist07:10:012
3974580994435,6cyclictest3981547-21cron09:25:011
397458099441,6cyclictest131rcu_preempt08:55:001
3974575994435,4cyclictest16850irq/134-800000008:20:000
397457599443,14cyclictest3988990-21latency_hist11:45:020
3974575994429,11cyclictest272-21dbus-daemon11:40:000
3974575994428,11cyclictest272-21dbus-daemon07:40:010
3974575994424,12cyclictest272-21dbus-daemon11:20:010
3974575994424,12cyclictest272-21dbus-daemon11:20:000
3974589994334,4cyclictest3975792-21sshd07:33:233
3974589994328,12cyclictest3980188-21sshd08:55:233
3974589994328,11cyclictest3978895-21sshd08:30:033
3974589994328,11cyclictest3978685-21systemd-run08:30:013
3974589994328,11cyclictest3978685-21systemd-run08:30:013
3974589994328,11cyclictest3975040-21munin-run07:20:003
3974589994327,12cyclictest235-21systemd-journal10:20:263
3974589994323,13cyclictest235-21systemd-journal11:55:003
3974589994314,9cyclictest3980735-21sshd09:09:223
3974580994332,8cyclictest3983065-21sshd09:50:231
3974580994332,8cyclictest3977604-21sshd08:06:221
3974580994331,9cyclictest23-21ksoftirqd/109:20:011
397457599432,7cyclictest3984401-21munin-run10:20:010
3974589994229,10cyclictest48-21kcompactd010:13:223
3974589994229,10cyclictest3989552-21sshd11:55:213
3974589994229,10cyclictest3987492-21sshd11:18:223
3974589994229,10cyclictest3987492-21sshd11:18:213
3974589994229,10cyclictest272-21dbus-daemon11:30:013
3974589994227,10cyclictest3979968-21munin-run08:54:593
3974589994222,16cyclictest3977872-21sendmail-mta08:14:233
3974580994232,6cyclictest3987498-21cron11:20:011
3974580994232,6cyclictest3987498-21cron11:20:011
3974580994231,8cyclictest3974977-21sed07:15:021
3974575994231,8cyclictest3977650-21sshd08:10:020
3974575994227,9cyclictest272-21dbus-daemon08:15:000
3974575994224,14cyclictest3978398-21sshd08:21:220
3974575994224,13cyclictest235-21systemd-journal10:30:010
3974575994224,13cyclictest235-21systemd-journal10:15:010
3974589994129,9cyclictest37-21ksoftirqd/310:45:013
3974589994128,9cyclictest48-21kcompactd009:45:043
3974589994128,9cyclictest48-21kcompactd008:38:283
3974589994127,10cyclictest48-21kcompactd010:39:263
3974589994127,10cyclictest412-21sshd07:24:233
3974589994126,11cyclictest3982298-21sshd09:39:233
3974580994129,9cyclictest3979171-21cron08:40:001
3974580994129,9cyclictest23-21ksoftirqd/109:50:011
3974580994126,11cyclictest235-21systemd-journal10:25:001
397458099412,3cyclictest131rcu_preempt11:15:001
3974580994123,12cyclictest3984406-21systemctl10:20:011
3974575994130,7cyclictest272-21dbus-daemon12:15:000
3974575994127,11cyclictest272-21dbus-daemon11:50:000
3974575994123,14cyclictest3986433-21sshd10:55:330
3974575994118,19cyclictest3976597-21latency_hist07:50:020
3974589994032,4cyclictest3988312-21munin-run11:35:013
3974589994032,4cyclictest3988312-21munin-run11:35:003
3974589994031,5cyclictest3976815-21sshd07:50:223
3974589994028,9cyclictest3985152-21sshd10:32:223
3974589994028,9cyclictest3985152-21sshd10:32:213
3974589994028,8cyclictest3979446-21sshd08:44:233
3974589994028,8cyclictest3979446-21sshd08:44:223
3974589994027,7cyclictest3983644-21munin-run10:05:013
3974589994027,10cyclictest3979730-21munin-run08:50:013
397458099407,7cyclictest23-21ksoftirqd/109:28:231
397458099403,4cyclictest131rcu_preempt08:50:011
3974580994029,8cyclictest3975938-21sed07:35:021
3974580994027,9cyclictest23-21ksoftirqd/108:45:021
3974580994027,9cyclictest23-21ksoftirqd/108:45:021
3974580994026,10cyclictest272-21dbus-daemon12:40:001
3974580994024,12cyclictest23-21ksoftirqd/109:35:011
397457599403,7cyclictest1-21systemd12:40:000
3974575994031,6cyclictest19050irq/34-mmc111:08:270
3974575994029,8cyclictest12-21ksoftirqd/010:00:000
3974575994028,9cyclictest235-21systemd-journal12:07:220
3974575994028,8cyclictest235-21systemd-journal11:59:590
3974575994024,11cyclictest272-21dbus-daemon08:50:010
3974575994021,14cyclictest272-21dbus-daemon09:35:010
397457599401,5cyclictest131rcu_preempt12:35:020
397457599401,5cyclictest131rcu_preempt12:35:010
397457599401,34cyclictest0-21swapper/007:25:010
3974589993928,8cyclictest48-21kcompactd007:12:303
3974589993928,8cyclictest3990333-21sshd12:10:233
3974589993925,11cyclictest363-21snmpd11:44:043
3974589993924,11cyclictest48-21kcompactd009:33:593
397458099396,9cyclictest23-21ksoftirqd/112:06:211
3974580993930,5cyclictest3991403-21cron12:35:001
3974580993930,5cyclictest3991403-21cron12:34:591
3974580993929,7cyclictest3984900-21sshd10:27:221
3974580993927,9cyclictest1-21systemd10:00:011
397458099392,4cyclictest131rcu_preempt07:45:011
397458099392,4cyclictest131rcu_preempt07:45:011
3974580993924,11cyclictest1-21systemd11:05:001
397458099392,10cyclictest0-21swapper/107:18:221
397457599399,25cyclictest0-21swapper/008:34:140
397457599393,31cyclictest3977275-21sort08:00:030
397457599393,31cyclictest3977275-21sort08:00:020
3974575993920,13cyclictest272-21dbus-daemon09:20:000
3974589993824,11cyclictest37-21ksoftirqd/310:56:223
397458099384,6cyclictest131rcu_preempt08:21:231
397458099383,4cyclictest131rcu_preempt12:00:001
397458099383,3cyclictest131rcu_preempt10:35:021
397458099383,3cyclictest131rcu_preempt10:35:021
3974580993827,7cyclictest3990592-21sshd12:15:501
3974580993827,7cyclictest3990592-21sshd12:15:501
3974580993827,7cyclictest306-21in:imuxsock09:40:001
3974580993826,9cyclictest3974625-21cat07:10:021
3974580993826,8cyclictest3987992-21sed11:25:011
397458099382,5cyclictest131rcu_preempt10:40:001
397458099382,4cyclictest131rcu_preempt10:58:221
3974580993824,10cyclictest1-21systemd07:40:001
397458099381,3cyclictest131rcu_preempt11:45:031
397457599383,30cyclictest0-21swapper/011:35:020
397457599383,30cyclictest0-21swapper/011:35:020
3974575993828,8cyclictest12-21ksoftirqd/007:15:020
3974575993828,7cyclictest12-21ksoftirqd/009:30:010
3974575993827,8cyclictest3980993-21sshd09:12:220
397457599382,32cyclictest3983097-21munin-plugin-st09:54:590
3974575993822,12cyclictest1-21systemd09:25:010
3974575993818,13cyclictest12-21ksoftirqd/011:05:030
3974589993730,4cyclictest3987773-21sshd11:23:233
3974589993726,9cyclictest48-21kcompactd009:50:033
397458099374,3cyclictest131rcu_preempt12:30:011
3974580993726,8cyclictest23-21ksoftirqd/108:35:021
3974580993725,9cyclictest272-21dbus-daemon08:05:011
3974580993724,10cyclictest1-21systemd12:25:001
3974575993726,8cyclictest3983858-21sshd10:07:210
397457599372,30cyclictest3987247-21latency_hist11:15:010
3974575993722,9cyclictest306-21in:imuxsock09:50:010
397457599371,3cyclictest131rcu_preempt07:10:020
397457599371,32cyclictest3982558-21sshd09:40:190
397457599371,31cyclictest3988059-21munin-run11:30:000
397458099364,4cyclictest131rcu_preempt08:28:191
397458099364,4cyclictest131rcu_preempt08:28:181
397458099364,3cyclictest131rcu_preempt08:56:231
397458099364,3cyclictest131rcu_preempt07:29:521
3974580993627,5cyclictest306-21in:imuxsock08:00:011
3974580993627,5cyclictest306-21in:imuxsock08:00:001
3974580993624,9cyclictest23-21ksoftirqd/108:15:011
3974580993622,9cyclictest235-21systemd-journal10:10:251
3974575993629,4cyclictest0-21swapper/010:32:100
3974575993629,4cyclictest0-21swapper/010:32:090
3974575993628,5cyclictest3975523-21modprobe07:25:010
3974575993628,4cyclictest0-21swapper/007:42:460
3974575993628,4cyclictest0-21swapper/007:42:450
397457599362,5cyclictest3982324-21munin-run09:40:010
397457599362,30cyclictest0-21swapper/010:55:010
3974575993620,11cyclictest12-21ksoftirqd/007:35:010
397457599361,33cyclictest0-21swapper/007:20:010
3974589993529,3cyclictest3990103-21sshd12:09:223
397458099356,3cyclictest131rcu_preempt07:55:011
397458099355,3cyclictest131rcu_preempt11:34:021
397458099355,3cyclictest131rcu_preempt11:34:021
397458099353,3cyclictest131rcu_preempt12:15:011
3974580993525,6cyclictest3986214-21munin-plugin-st10:55:001
397458099352,4cyclictest131rcu_preempt11:30:011
3974580993524,8cyclictest3985941-21sshd10:48:221
3974575993523,8cyclictest235-21systemd-journal09:00:010
397457599352,29cyclictest0-21swapper/012:25:220
397457599351,31cyclictest0-21swapper/012:04:170
397458099343,4cyclictest1-21systemd12:03:221
3974580993427,4cyclictest1-21systemd11:40:011
3974580993425,7cyclictest23-21ksoftirqd/109:00:111
397458099342,3cyclictest131rcu_preempt10:40:301
397458099341,3cyclictest131rcu_preempt11:45:011
3974580993322,8cyclictest23-21ksoftirqd/109:13:271
397457599331,2cyclictest0-21swapper/012:25:010
3974580993210,19cyclictest3976614-21unin-run07:50:001
397457599321,4cyclictest0-21swapper/010:20:260
397458999289,15cyclictest3974595-21latency_hist07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional