You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-23 - 17:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 23, 2025 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2543038997614,9cyclictest2558394-21sshd12:06:122
254303899687,11cyclictest2551225-21cron09:50:002
2543038996623,37cyclictest235-21systemd-journal07:40:012
254303899652,16cyclictest2546107-21munin-run08:10:012
254303899642,16cyclictest2545810-21munin-run08:05:012
2543038996421,39cyclictest412-21sshd09:59:132
254303899632,16cyclictest2557905-21munin-run12:00:002
254303899632,15cyclictest2556386-21munin-run11:30:002
254303899632,14cyclictest2546890-21munin-run08:25:012
2543038996316,41cyclictest272-21dbus-daemon08:55:012
2543038996224,33cyclictest2552895-21latency_hist10:20:022
254303899622,15cyclictest2553293-21munin-run10:30:002
254303199623,8cyclictest12-21ksoftirqd/012:30:010
254303199623,8cyclictest12-21ksoftirqd/012:30:010
2543038996122,32cyclictest272-21dbus-daemon07:35:012
254303899612,14cyclictest2545270-21munin-run07:55:012
254303899612,13cyclictest2543999-21munin-run07:30:002
254303899611,15cyclictest2549729-21munin-run09:20:012
254303899611,14cyclictest363-21snmpd11:45:052
254303899602,15cyclictest2559718-21munin-run12:35:022
254303899602,14cyclictest2558173-21munin-run12:05:012
254303899592,19cyclictest2549179-21munin-run09:10:002
254303899592,19cyclictest2549179-21munin-run09:10:002
254303899592,12cyclictest2555573-21sshd11:14:132
254303899592,12cyclictest2543230-21munin-run07:15:012
254303899592,11cyclictest2550464-21munin-run09:35:002
2543038995917,36cyclictest2552438-21cat10:10:022
2543038995917,36cyclictest2552438-21cat10:10:022
2543031995928,4cyclictest131rcu_preempt09:15:000
2543038995819,35cyclictest2554498-21latency_hist10:50:022
254303899581,51cyclictest2553880-21cat10:40:032
254303899581,51cyclictest2553880-21cat10:40:022
254304199573,5cyclictest131rcu_preempt11:20:003
254303899572,13cyclictest2554618-21sed10:55:022
254303899572,13cyclictest2545035-21/usr/sbin/munin07:50:012
2543035995737,15cyclictest1-21systemd08:30:011
2543031995741,12cyclictest1-21systemd09:10:010
2543031995741,12cyclictest1-21systemd09:10:010
2543031995717,6cyclictest12-21ksoftirqd/008:55:020
254304199567,6cyclictest131rcu_preempt07:50:023
2543041995632,20cyclictest1-21systemd09:55:013
2543041995632,20cyclictest1-21systemd09:55:013
254303899562,19cyclictest2543751-21munin-plugin-st07:24:592
254303899561,14cyclictest2550754-21unin-run09:40:012
254303899561,14cyclictest2550754-21unin-run09:40:012
254303899561,12cyclictest363-21snmpd08:55:502
254303899561,12cyclictest363-21snmpd08:55:502
254303899561,11cyclictest2552024-21unin-run10:05:012
254304199553,6cyclictest131rcu_preempt10:45:003
2543041995515,5cyclictest131rcu_preempt07:25:003
254303899553,10cyclictest2556877-21cat11:40:012
2543038995526,24cyclictest306-21in:imuxsock08:00:002
2543038995520,31cyclictest2557710-21latency_hist11:55:012
254303899551,12cyclictest2553526-21sshd10:34:132
254303899543,20cyclictest2554090-21cat10:45:012
254303899543,15cyclictest2555342-21munin-plugin-st11:10:002
254303899542,15cyclictest2547963-21latency_hist08:45:022
254303199545,9cyclictest131rcu_preempt10:55:010
254303899539,11cyclictest2549462-21cat09:15:012
254303899536,8cyclictest0-21swapper/211:20:002
2543038995326,23cyclictest2544936-21latency_hist07:45:022
254303899532,12cyclictest2548162-21munin-run08:50:012
2543038995320,28cyclictest272-21dbus-daemon12:25:002
2543031995316,5cyclictest2557078-21latency_hist11:40:020
2543031995314,6cyclictest131rcu_preempt07:20:000
2543031995314,6cyclictest131rcu_preempt07:20:000
254304199529,5cyclictest131rcu_preempt07:30:003
2543041995243,6cyclictest37-21ksoftirqd/311:05:023
254304199522,5cyclictest131rcu_preempt08:25:003
2543041995212,6cyclictest131rcu_preempt11:25:013
2543041995212,6cyclictest131rcu_preempt11:25:013
2543041995211,6cyclictest131rcu_preempt12:35:003
254303899522,9cyclictest235-21systemd-journal08:35:012
254303899522,13cyclictest2559995-21/usr/sbin/munin12:40:012
254303899522,13cyclictest2551004-21munin-run09:45:012
2543038995213,3cyclictest208-21jbd2/mmcblk1p2-12:15:202
254303199523,5cyclictest131rcu_preempt07:30:000
2543031995233,14cyclictest235-21systemd-journal10:05:000
2543031995212,5cyclictest131rcu_preempt10:10:010
2543031995212,5cyclictest131rcu_preempt10:10:010
254304199518,6cyclictest131rcu_preempt10:10:003
254304199518,6cyclictest131rcu_preempt10:10:003
2543038995128,20cyclictest2546338-21latency_hist08:15:012
254303899512,13cyclictest2552510-21munin-run10:15:002
254303899512,12cyclictest2543484-21munin-run07:20:012
254303899512,12cyclictest2543484-21munin-run07:20:012
254303899511,13cyclictest2555293-21latency_hist11:05:032
254303899511,12cyclictest2559503-21date12:30:022
254303899511,12cyclictest2559503-21date12:30:022
2543038995110,9cyclictest30-21ksoftirqd/208:39:272
2543038995110,3cyclictest2551457-21kworker/u8:109:53:142
2543038995110,3cyclictest2551457-21kworker/u8:109:53:132
2543035995132,13cyclictest1-21systemd08:25:001
254303199516,5cyclictest131rcu_preempt07:25:000
2543031995116,5cyclictest131rcu_preempt11:20:000
2543031995110,5cyclictest131rcu_preempt08:25:000
254304199507,9cyclictest37-21ksoftirqd/309:25:013
254304199506,5cyclictest131rcu_preempt07:55:023
254304199506,10cyclictest37-21ksoftirqd/310:59:593
2543041995014,6cyclictest131rcu_preempt09:35:003
2543041995013,6cyclictest131rcu_preempt10:15:003
2543038995021,26cyclictest30-21ksoftirqd/210:22:142
2543038995021,26cyclictest30-21ksoftirqd/210:22:132
254303899502,11cyclictest2556126-21munin-run11:25:002
254303899502,11cyclictest2556126-21munin-run11:25:002
2543038995012,33cyclictest0-21swapper/208:20:012
2543038995012,33cyclictest0-21swapper/208:20:012
254303899501,16cyclictest2548923-21munin-run09:05:002
2543035995030,15cyclictest1-21systemd09:20:001
254303199509,10cyclictest12-21ksoftirqd/011:45:010
2543031995017,5cyclictest12-21ksoftirqd/007:57:140
2543031995015,5cyclictest131rcu_preempt09:50:010
254303199501,47cyclictest0-21swapper/009:25:020
2543031995010,6cyclictest131rcu_preempt08:30:010
2543031995010,10cyclictest12-21ksoftirqd/008:40:010
254304199497,8cyclictest37-21ksoftirqd/312:40:013
254304199495,7cyclictest37-21ksoftirqd/312:20:013
254304199493,5cyclictest131rcu_preempt11:40:023
2543041994913,7cyclictest131rcu_preempt09:00:143
2543041994913,6cyclictest131rcu_preempt11:45:013
2543041994910,6cyclictest131rcu_preempt12:25:003
2543041994910,5cyclictest131rcu_preempt08:35:013
2543038994928,18cyclictest2547124-21munin-plugin-st08:30:002
2543035994934,11cyclictest2543479-21cat07:20:001
2543035994934,11cyclictest2543479-21cat07:20:001
254303199494,9cyclictest12-21ksoftirqd/007:15:000
2543031994937,8cyclictest12-21ksoftirqd/009:45:000
2543031994932,13cyclictest12-21ksoftirqd/010:40:000
2543031994932,13cyclictest12-21ksoftirqd/010:40:000
2543031994930,14cyclictest2557902-21run-parts12:00:000
254304199489,6cyclictest131rcu_preempt09:50:013
254304199487,10cyclictest37-21ksoftirqd/308:05:013
254304199482,11cyclictest37-21ksoftirqd/309:30:003
2543041994813,5cyclictest131rcu_preempt09:20:013
2543041994812,6cyclictest131rcu_preempt07:39:143
2543038994831,12cyclictest272-21dbus-daemon11:00:002
254303899482,14cyclictest2550184-21sshd09:25:142
254303199489,6cyclictest131rcu_preempt12:40:020
254303199485,5cyclictest131rcu_preempt12:17:000
2543031994812,9cyclictest131rcu_preempt08:05:010
2543031994811,5cyclictest131rcu_preempt08:10:010
254304199479,6cyclictest131rcu_preempt11:15:023
254304199477,6cyclictest131rcu_preempt11:50:013
254304199476,6cyclictest131rcu_preempt10:05:003
254304199476,12cyclictest37-21ksoftirqd/307:32:143
254304199475,6cyclictest131rcu_preempt08:30:013
2543041994735,8cyclictest361rcuc/307:20:003
2543041994735,8cyclictest361rcuc/307:20:003
2543041994711,6cyclictest131rcu_preempt08:15:013
254303199478,6cyclictest131rcu_preempt09:20:000
2543031994717,26cyclictest12-21ksoftirqd/007:50:020
2543031994711,5cyclictest131rcu_preempt10:50:020
254304199468,5cyclictest131rcu_preempt09:40:003
254304199468,5cyclictest131rcu_preempt09:40:003
254304199463,9cyclictest37-21ksoftirqd/308:40:003
254304199462,4cyclictest131rcu_preempt11:35:003
254304199461,7cyclictest0-21swapper/312:30:013
254304199461,7cyclictest0-21swapper/312:30:013
2543041994611,7cyclictest131rcu_preempt10:40:023
2543041994611,7cyclictest131rcu_preempt10:40:023
2543041994610,4cyclictest131rcu_preempt08:50:013
2543041994610,23cyclictest37-21ksoftirqd/307:10:023
254303899466,5cyclictest0-21swapper/209:25:012
2543035994634,9cyclictest23-21ksoftirqd/111:45:011
2543035994634,7cyclictest2551237-21munin-plugin-st09:50:001
254303199461,9cyclictest12-21ksoftirqd/007:10:020
2543031994614,15cyclictest12-21ksoftirqd/007:35:000
2543031994610,5cyclictest131rcu_preempt08:45:000
254304199457,11cyclictest37-21ksoftirqd/308:55:013
2543041994518,7cyclictest361rcuc/309:45:013
254304199451,5cyclictest131rcu_preempt10:20:023
2543041994510,4cyclictest131rcu_preempt08:10:003
254303899452,14cyclictest2558686-21/usr/sbin/munin12:15:012
2543035994534,6cyclictest307-21in:imklog09:05:011
2543035994531,8cyclictest306-21in:imuxsock10:55:011
2543035994528,12cyclictest272-21dbus-daemon10:40:011
2543035994528,12cyclictest272-21dbus-daemon10:40:011
254303199453,8cyclictest12-21ksoftirqd/010:20:030
2543031994529,12cyclictest272-21dbus-daemon09:30:010
254303199451,32cyclictest19050irq/34-mmc108:15:020
254303199451,32cyclictest19050irq/34-mmc108:15:010
2543031994512,6cyclictest131rcu_preempt10:00:020
2543031994512,6cyclictest131rcu_preempt09:05:010
254304199446,10cyclictest37-21ksoftirqd/310:00:013
254304199443,5cyclictest131rcu_preempt08:43:143
254304199442,34cyclictest2558675-21grep12:15:003
2543041994410,5cyclictest131rcu_preempt11:55:013
2543041994410,5cyclictest131rcu_preempt10:32:133
254303899441,14cyclictest2556655-21latency_hist11:35:012
254303899441,12cyclictest2543092-21latency_hist07:10:022
2543035994431,9cyclictest1-21systemd12:05:011
2543035994430,9cyclictest23-21ksoftirqd/112:20:011
254303599441,8cyclictest2557895-21cron12:00:001
2543031994426,15cyclictest2550696-21sshd09:39:140
2543031994426,15cyclictest2550696-21sshd09:39:140
2543031994422,15cyclictest235-21systemd-journal10:45:000
254304199437,6cyclictest131rcu_preempt10:50:003
254304199436,6cyclictest131rcu_preempt08:20:013
254304199436,6cyclictest131rcu_preempt08:20:013
254304199431,8cyclictest37-21ksoftirqd/309:10:003
254304199431,8cyclictest37-21ksoftirqd/309:10:003
2543041994310,6cyclictest131rcu_preempt10:27:133
2543038994314,23cyclictest0-21swapper/211:45:002
2543035994330,9cyclictest306-21in:imuxsock12:27:131
2543035994330,9cyclictest306-21in:imuxsock12:27:131
2543035994330,8cyclictest235-21systemd-journal11:25:131
2543035994329,11cyclictest23-21ksoftirqd/111:47:131
2543035994325,14cyclictest235-21systemd-journal10:30:001
254303599431,3cyclictest131rcu_preempt07:15:001
254303199439,5cyclictest131rcu_preempt07:40:020
254303199437,5cyclictest131rcu_preempt11:25:010
254303199437,5cyclictest131rcu_preempt11:25:010
254303199436,6cyclictest131rcu_preempt10:35:010
254303199435,10cyclictest12-21ksoftirqd/009:35:010
254303199433,8cyclictest131rcu_preempt11:15:000
2543031994334,4cyclictest306-21in:imuxsock10:15:000
2543031994331,9cyclictest12-21ksoftirqd/011:35:000
2543031994330,9cyclictest235-21systemd-journal08:15:000
2543031994330,8cyclictest12-21ksoftirqd/009:55:010
2543031994330,8cyclictest12-21ksoftirqd/009:55:010
2543031994329,10cyclictest2544747-21sshd07:42:140
254303199432,7cyclictest12-21ksoftirqd/008:35:010
254303199431,37cyclictest2555315-21sshd11:06:130
2543035994233,5cyclictest306-21in:imuxsock10:20:001
2543035994230,8cyclictest306-21in:imuxsock12:20:031
254303599421,3cyclictest131rcu_preempt07:35:001
254303199426,6cyclictest131rcu_preempt11:50:020
254303199426,5cyclictest131rcu_preempt09:00:000
254303199426,5cyclictest131rcu_preempt09:00:000
254303199423,8cyclictest12-21ksoftirqd/010:30:000
2543031994233,6cyclictest12-21ksoftirqd/011:05:010
254304199419,5cyclictest131rcu_preempt08:00:023
254304199416,6cyclictest37-21ksoftirqd/310:54:133
254304199416,4cyclictest131rcu_preempt11:10:003
254304199415,9cyclictest37-21ksoftirqd/307:15:013
254304199415,5cyclictest131rcu_preempt10:25:003
254304199415,5cyclictest131rcu_preempt10:25:003
254304199413,7cyclictest37-21ksoftirqd/311:26:133
2543041994124,13cyclictest272-21dbus-daemon12:10:003
2543035994127,10cyclictest235-21systemd-journal12:34:591
2543035994122,13cyclictest23-21ksoftirqd/109:35:001
254303199416,5cyclictest131rcu_preempt12:25:000
254303199412,6cyclictest2553059-21munin-run10:25:010
254303199412,6cyclictest2553059-21munin-run10:25:010
2543041994032,5cyclictest37-21ksoftirqd/307:45:013
2543041994011,4cyclictest131rcu_preempt12:02:133
2543035994029,8cyclictest2556098-21sshd11:21:131
2543035994029,8cyclictest2556098-21sshd11:21:131
254303599401,3cyclictest131rcu_preempt11:15:001
254303199409,5cyclictest131rcu_preempt12:35:000
254303199405,7cyclictest12-21ksoftirqd/012:08:130
2543031994030,7cyclictest12-21ksoftirqd/007:55:010
2543031994027,9cyclictest12-21ksoftirqd/012:15:020
2543035993930,5cyclictest2548039-21latency_hist08:45:021
2543035993924,10cyclictest306-21in:imuxsock09:55:011
2543035993924,10cyclictest306-21in:imuxsock09:55:001
2543035993920,13cyclictest235-21systemd-journal10:35:001
2543031993926,10cyclictest2554832-21tr11:00:000
254304199385,6cyclictest131rcu_preempt12:00:003
254304199383,8cyclictest37-21ksoftirqd/309:14:143
254303599387,9cyclictest23-21ksoftirqd/112:37:461
254303599385,6cyclictest131rcu_preempt09:40:001
254303599385,6cyclictest131rcu_preempt09:40:001
2543035993828,7cyclictest23-21ksoftirqd/109:45:021
2543035993827,6cyclictest2549156-21cron09:10:001
2543035993827,6cyclictest2549156-21cron09:10:001
2543035993823,10cyclictest235-21systemd-journal11:40:001
2543035993821,11cyclictest306-21in:imuxsock07:55:031
254303199381,12cyclictest12-21ksoftirqd/008:50:010
2543035993725,10cyclictest23-21ksoftirqd/110:00:001
254303599372,3cyclictest131rcu_preempt07:10:021
2543035993723,11cyclictest23-21ksoftirqd/109:15:011
2543035993723,10cyclictest272-21dbus-daemon11:55:001
2543035993721,11cyclictest306-21in:imuxsock11:17:001
254303199372,3cyclictest131rcu_preempt11:55:010
254304199366,5cyclictest131rcu_preempt08:57:143
254304199366,5cyclictest131rcu_preempt08:57:143
2543035993623,10cyclictest23-21ksoftirqd/110:45:001
254303599362,29cyclictest48-21kcompactd008:35:011
2543035993622,10cyclictest23-21ksoftirqd/107:30:011
2543035993620,11cyclictest235-21systemd-journal09:00:001
2543035993620,11cyclictest235-21systemd-journal09:00:001
2543035993617,6cyclictest131rcu_preempt10:10:011
2543035993617,6cyclictest131rcu_preempt10:10:011
2543035993617,13cyclictest23-21ksoftirqd/110:15:021
254303599361,12cyclictest2552015-21modprobe10:05:001
2543035993525,7cyclictest23-21ksoftirqd/108:10:031
2543035993524,8cyclictest23-21ksoftirqd/110:50:001
2543035993521,11cyclictest23-21ksoftirqd/108:55:011
254303199351,10cyclictest12-21ksoftirqd/012:03:130
254303599346,4cyclictest131rcu_preempt08:20:001
254303599346,4cyclictest131rcu_preempt08:20:001
254303599344,3cyclictest131rcu_preempt09:25:021
2543035993425,4cyclictest23-21ksoftirqd/107:50:011
2543035993423,8cyclictest2553153-21sed10:25:021
2543035993423,8cyclictest2553153-21sed10:25:021
2543035993419,11cyclictest235-21systemd-journal07:40:011
2543035993418,4cyclictest131rcu_preempt08:50:001
2543031993424,7cyclictest12-21ksoftirqd/011:30:010
254303599334,3cyclictest131rcu_preempt08:05:001
2543035993327,4cyclictest235-21systemd-journal11:01:131
2543035993326,4cyclictest2550213-21munin-plugin-st09:30:001
2543035993325,4cyclictest2543939-21latency_hist07:25:021
2543035993321,9cyclictest23-21ksoftirqd/111:10:011
2543035993318,11cyclictest1-21systemd11:00:001
2543035993224,6cyclictest23-21ksoftirqd/108:40:011
2543035993221,7cyclictest2556637-21unin-run11:35:001
254303599314,6cyclictest131rcu_preempt12:10:011
2543035993124,4cyclictest2544939-21latency_hist07:45:021
2543035993123,4cyclictest23-21ksoftirqd/108:13:141
2543035993118,9cyclictest306-21in:imuxsock07:55:011
2543035992816,9cyclictest2558684-21systemctl12:15:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional