You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 23:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Nov 26, 2025 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12702299745,3cyclictest131rcu_preempt10:15:012
12702299672,41cyclictest130089-21munin-run08:10:012
12702299672,13cyclictest133672-21sshd09:15:492
12702299653,57cyclictest140463-21sshd11:29:482
12702299653,57cyclictest140463-21sshd11:29:482
127027996431,9cyclictest272-21dbus-daemon08:50:003
12702299641,9cyclictest135257-21sshd09:46:482
127022996414,46cyclictest127189-21sshd07:13:482
12702299622,14cyclictest131887-21sshd08:44:482
12702299622,14cyclictest129999-21cat08:05:032
12702299622,14cyclictest129999-21cat08:05:032
12702299612,5cyclictest136847-21munin-run10:20:012
12702299612,12cyclictest128788-21munin-run07:45:012
12702299611,7cyclictest363-21snmpd09:24:482
12702299602,8cyclictest135038-21munin-run09:45:002
12702299602,5cyclictest141026-21munin-run11:40:012
12702299602,54cyclictest127515-21munin-run07:20:012
12702299602,53cyclictest138889-21sshd10:59:482
12702299601,55cyclictest137337-21sshd10:26:482
12702299593,51cyclictest363-21snmpd11:12:102
12702299593,50cyclictest142010-21cron12:00:002
12702299593,50cyclictest142010-21cron12:00:002
12702299592,7cyclictest363-21snmpd11:45:162
12702299592,52cyclictest138403-21munin-run10:50:002
12702299591,6cyclictest134467-21sshd09:30:482
12702299591,5cyclictest133160-21sshd09:06:482
12702299591,15cyclictest128549-21munin-run07:40:002
12702299591,13cyclictest136299-21sshd10:08:482
127012995950,5cyclictest138909-21sh11:00:000
12702299583,50cyclictest129278-21cron07:55:002
12702299582,51cyclictest135540-21munin-run09:55:012
12702299582,51cyclictest127732-21sshd07:24:182
12702299582,51cyclictest127732-21sshd07:24:172
12702299582,4cyclictest137841-21sshd10:36:482
12702299582,4cyclictest130630-21munin-run08:20:002
12702299581,9cyclictest130307-21sshd08:14:282
12702299581,7cyclictest140213-21sshd11:24:482
12702299581,7cyclictest136048-21sshd10:03:482
12702299581,6cyclictest134215-21sshd09:25:492
12702299581,51cyclictest363-21snmpd08:36:552
12702299581,4cyclictest142265-21sshd12:00:062
12702299581,4cyclictest128420-21cat07:35:032
12702299581,4cyclictest128420-21cat07:35:032
12702299572,6cyclictest363-21snmpd11:15:102
127022995725,28cyclictest272-21dbus-daemon12:40:012
12702299572,4cyclictest129039-21munin-run07:50:012
12702299572,4cyclictest129039-21munin-run07:50:012
12702299572,12cyclictest128004-21munin-run07:29:592
127022995718,29cyclictest30-21ksoftirqd/212:20:022
12702299571,7cyclictest131335-21sshd08:30:482
12702299571,5cyclictest137097-21sshd10:24:482
12702299571,5cyclictest137097-21sshd10:24:472
12702299571,51cyclictest133428-21sshd09:12:472
12702299563,48cyclictest363-21snmpd10:32:092
12702299563,48cyclictest130882-21munin-run08:25:012
127022995623,29cyclictest132974-21cat09:05:012
12702299561,12cyclictest363-21snmpd12:23:152
127012995639,13cyclictest272-21dbus-daemon10:00:000
12702299552,5cyclictest363-21snmpd11:40:152
12702299552,5cyclictest363-21snmpd11:40:142
127022995525,26cyclictest1-21systemd09:59:592
12702299552,4cyclictest132409-21munin-run08:54:592
12702299551,49cyclictest363-21snmpd10:41:362
12702299551,10cyclictest363-21snmpd08:55:532
12702299551,10cyclictest363-21snmpd08:55:532
12702299543,46cyclictest412-21sshd11:03:482
12702299542,6cyclictest134799-21munin-run09:40:012
12702299542,6cyclictest134799-21munin-run09:40:002
12702299541,5cyclictest142779-21sshd12:12:482
12702299541,5cyclictest142779-21sshd12:12:482
12702299541,48cyclictest363-21snmpd07:56:242
12702299533,46cyclictest363-21snmpd12:30:182
12702299532,5cyclictest363-21snmpd11:50:472
12702299532,13cyclictest140694-21sshd11:30:032
12702299531,12cyclictest363-21snmpd08:47:472
12702299522,45cyclictest0-21swapper/210:52:302
12702299521,7cyclictest363-21snmpd12:07:472
12702299512,5cyclictest363-21snmpd11:09:012
12702299511,5cyclictest363-21snmpd12:26:512
12701799505,3cyclictest131rcu_preempt09:59:471
127012995032,14cyclictest1-21systemd12:25:010
127012995032,12cyclictest272-21dbus-daemon08:20:010
127012995031,16cyclictest272-21dbus-daemon12:00:000
127012995031,16cyclictest272-21dbus-daemon12:00:000
12702299493,41cyclictest131089-21modprobe08:26:492
12702299493,41cyclictest131089-21modprobe08:26:492
127027994830,12cyclictest1-21systemd08:15:003
12702299482,10cyclictest127035-21latency_hist07:10:022
12702299482,10cyclictest127035-21latency_hist07:10:012
12701799482,4cyclictest131rcu_preempt08:05:011
12701799482,4cyclictest131rcu_preempt08:05:001
12701799481,3cyclictest131rcu_preempt12:10:001
127012994820,5cyclictest12-21ksoftirqd/010:25:010
127012994820,5cyclictest12-21ksoftirqd/010:25:010
12701799471,4cyclictest131rcu_preempt10:14:571
127027994632,10cyclictest235-21systemd-journal08:34:593
12702799463,11cyclictest37-21ksoftirqd/308:26:483
12702799463,11cyclictest37-21ksoftirqd/308:26:483
127027994627,15cyclictest0-21swapper/307:20:013
12701799463,3cyclictest131rcu_preempt12:17:001
12701799463,3cyclictest131rcu_preempt09:40:021
12701799463,3cyclictest131rcu_preempt09:40:011
12701799462,3cyclictest131rcu_preempt08:36:481
12701299462,9cyclictest139441-21modprobe11:10:010
127012994627,13cyclictest1-21systemd10:29:590
127012994610,31cyclictest16850irq/134-800000011:40:030
127012994610,31cyclictest16850irq/134-800000011:40:030
12702799452,6cyclictest131rcu_preempt11:00:003
12702799451,3cyclictest131rcu_preempt11:50:003
12701799453,4cyclictest131rcu_preempt08:15:491
12701799453,3cyclictest131rcu_preempt10:26:481
12701799452,3cyclictest131rcu_preempt07:52:491
12701799451,3cyclictest131rcu_preempt11:48:481
12701799451,3cyclictest131rcu_preempt07:14:151
12701299455,35cyclictest135717-21grep09:55:010
127012994529,12cyclictest272-21dbus-daemon07:45:010
127027994432,8cyclictest235-21systemd-journal09:40:033
12701799443,4cyclictest131rcu_preempt07:30:001
12701799441,4cyclictest131rcu_preempt10:20:011
127012994432,9cyclictest12-21ksoftirqd/009:20:020
127012994429,11cyclictest272-21dbus-daemon08:55:010
127012994427,13cyclictest1-21systemd08:10:010
12701299442,6cyclictest131rcu_preempt11:30:000
12701299442,6cyclictest131rcu_preempt11:30:000
127012994426,14cyclictest140196-21sshd11:20:490
12702799433,4cyclictest131rcu_preempt08:40:003
127027994323,15cyclictest272-21dbus-daemon07:50:013
127027994323,15cyclictest272-21dbus-daemon07:50:013
12701799433,33cyclictest306-21in:imuxsock11:00:011
127017994331,8cyclictest1-21systemd09:30:001
127012994336,4cyclictest16950irq/224-800000008:00:010
127012994331,9cyclictest143027-21sshd12:16:480
127012994331,7cyclictest306-21in:imuxsock09:25:000
12701299431,6cyclictest131rcu_preempt09:15:010
12701299431,38cyclictest135261-21sshd09:47:490
127027994229,9cyclictest272-21dbus-daemon11:05:013
127027994229,10cyclictest272-21dbus-daemon08:10:013
127027994222,14cyclictest1-21systemd11:15:013
12702799421,4cyclictest131rcu_preempt10:20:003
12702799421,3cyclictest131rcu_preempt10:40:013
127017994230,8cyclictest23-21ksoftirqd/109:10:021
127017994228,10cyclictest235-21systemd-journal10:55:011
127017994223,14cyclictest1-21systemd10:05:001
12701299425,4cyclictest131rcu_preempt12:05:000
127012994231,8cyclictest235-21systemd-journal11:12:480
127012994229,9cyclictest0-21swapper/007:50:010
127012994229,9cyclictest0-21swapper/007:50:010
12701299422,6cyclictest281-21systemd-logind12:35:000
127012994225,13cyclictest137598-21sshd10:33:490
12701299421,6cyclictest144126-21lockfile-create12:40:000
127027994127,10cyclictest1-21systemd12:10:013
12702799412,4cyclictest131rcu_preempt10:05:023
127027994124,13cyclictest272-21dbus-daemon10:15:003
12702799411,3cyclictest131rcu_preempt08:00:013
12702799411,36cyclictest235-21systemd-journal10:55:003
12701799418,22cyclictest142018-21modprobe12:00:001
12701799418,22cyclictest142018-21modprobe12:00:001
127017994131,7cyclictest137849-21sshd10:38:481
127017994119,16cyclictest272-21dbus-daemon09:00:011
127017994119,16cyclictest272-21dbus-daemon09:00:011
127017994118,6cyclictest221rcuc/107:45:011
12701299413,7cyclictest272-21dbus-daemon12:30:010
127012994132,5cyclictest1-21systemd11:05:010
12701299412,8cyclictest308-21rs:main0
127012994128,9cyclictest134732-21sshd09:38:490
127012994128,9cyclictest134732-21sshd09:38:490
12701299411,9cyclictest131427-21latency_hist08:35:000
12701299411,36cyclictest132008-21ls08:45:020
12702799404,3cyclictest131rcu_preempt10:30:003
12702799403,3cyclictest131rcu_preempt10:35:003
12702799403,3cyclictest131rcu_preempt07:54:593
127027994029,8cyclictest1-21systemd12:05:013
127027994028,9cyclictest37-21ksoftirqd/311:07:483
127027994026,10cyclictest272-21dbus-daemon11:19:593
127027994024,10cyclictest272-21dbus-daemon11:45:013
127027994024,10cyclictest272-21dbus-daemon11:45:013
12702799402,3cyclictest131rcu_preempt10:50:003
127027994021,10cyclictest140699-21sshd11:30:483
12702799401,3cyclictest131rcu_preempt10:45:013
127012994027,10cyclictest272-21dbus-daemon10:05:010
127012994024,11cyclictest272-21dbus-daemon10:45:010
12701299402,3cyclictest131rcu_preempt10:10:020
127012994016,20cyclictest16850irq/134-800000011:45:480
12701299401,3cyclictest131rcu_preempt07:40:000
12701299401,35cyclictest138637-21sshd10:54:470
12702799394,3cyclictest131rcu_preempt12:15:013
12702799394,3cyclictest131rcu_preempt12:15:013
12702799394,15cyclictest142020-21modprobe12:00:003
12702799394,15cyclictest142020-21modprobe12:00:003
12702799392,3cyclictest131rcu_preempt12:20:003
127027993923,12cyclictest37-21ksoftirqd/309:40:013
127027993923,12cyclictest37-21ksoftirqd/309:40:013
12702799391,3cyclictest131rcu_preempt09:09:593
127027993913,9cyclictest127008-21kworker/3:107:22:483
127027993913,9cyclictest127008-21kworker/3:107:22:483
12701799396,24cyclictest221rcuc/111:10:011
12701799395,4cyclictest131rcu_preempt11:25:021
127017993928,9cyclictest23-21ksoftirqd/108:45:021
12701799392,6cyclictest131rcu_preempt09:19:591
127017993923,10cyclictest272-21dbus-daemon11:45:011
127017993923,10cyclictest272-21dbus-daemon11:45:011
127017993917,17cyclictest272-21dbus-daemon07:25:011
127017993917,17cyclictest272-21dbus-daemon07:25:001
12701299394,4cyclictest131rcu_preempt07:20:000
127012993933,3cyclictest16950irq/224-800000007:15:020
127012993932,5cyclictest0-21swapper/010:15:020
127012993930,7cyclictest0-21swapper/012:14:480
127012993930,7cyclictest0-21swapper/012:14:480
12701299392,8cyclictest235-21systemd-journal10:39:240
12701299392,7cyclictest139958-21munin-run11:19:590
127012993925,10cyclictest408-21lldpd11:31:450
12702799384,4cyclictest131rcu_preempt12:35:003
127027993828,6cyclictest306-21in:imuxsock12:40:013
12702799382,3cyclictest131rcu_preempt09:25:013
127027993821,12cyclictest1-21systemd08:25:013
127027993820,14cyclictest272-21dbus-daemon11:40:003
12702799381,4cyclictest131rcu_preempt10:00:003
12701799385,3cyclictest131rcu_preempt11:30:011
12701799385,3cyclictest131rcu_preempt11:30:011
127017993831,4cyclictest130310-21sshd08:14:491
127017993827,8cyclictest135275-21cron09:50:001
127017993810,21cyclictest129029-21modprobe07:50:011
127017993810,21cyclictest129029-21modprobe07:50:001
127012993825,9cyclictest133178-21cron09:09:590
127012993822,13cyclictest235-21systemd-journal08:05:020
127012993822,13cyclictest235-21systemd-journal08:05:020
127012993822,11cyclictest1-21systemd07:25:010
127012993822,11cyclictest1-21systemd07:25:010
12701299381,30cyclictest0-21swapper/009:32:460
12701299381,12cyclictest1-21systemd08:11:470
127027993727,7cyclictest37-21ksoftirqd/309:50:023
127027993726,8cyclictest37-21ksoftirqd/308:05:023
127027993726,8cyclictest37-21ksoftirqd/308:05:023
12702799372,3cyclictest131rcu_preempt08:20:023
127027993722,10cyclictest272-21dbus-daemon09:20:013
127027993717,8cyclictest37-21ksoftirqd/307:30:003
12702799371,3cyclictest131rcu_preempt08:55:003
12701799376,4cyclictest131rcu_preempt12:20:231
12701799374,4cyclictest131rcu_preempt09:15:011
12701799373,3cyclictest131rcu_preempt09:25:021
127017993727,6cyclictest140984-21sh11:40:001
127017993727,5cyclictest139689-21munin-plugin-st11:15:001
127017993722,11cyclictest272-21dbus-daemon11:05:011
127017993720,12cyclictest1-21systemd08:55:011
12701299379,23cyclictest0-21swapper/010:48:200
127012993732,3cyclictest16950irq/224-800000011:37:500
127012993724,9cyclictest12-21ksoftirqd/009:05:010
127012993723,10cyclictest408-21lldpd08:57:420
127012993723,10cyclictest408-21lldpd08:57:420
127012993711,22cyclictest16850irq/134-800000007:34:480
127012993711,22cyclictest16850irq/134-800000007:34:470
12702799368,5cyclictest131rcu_preempt09:30:173
127027993627,7cyclictest37-21ksoftirqd/309:05:003
12702799362,5cyclictest131rcu_preempt09:27:493
127027993618,13cyclictest272-21dbus-daemon09:00:013
127027993618,13cyclictest272-21dbus-daemon09:00:013
12702799361,3cyclictest131rcu_preempt09:15:013
127017993628,5cyclictest143593-21latency_hist12:30:021
127017993625,7cyclictest1-21systemd10:10:011
127017993623,10cyclictest23-21ksoftirqd/110:40:031
127017993622,10cyclictest142473-21wc12:05:021
127017993619,5cyclictest23-21ksoftirqd/108:10:011
127017993616,15cyclictest272-21dbus-daemon07:20:001
12701299364,3cyclictest131rcu_preempt08:40:010
127012993620,11cyclictest235-21systemd-journal11:50:020
127012993618,13cyclictest131325-21latency_hist08:30:030
12702799353,4cyclictest131rcu_preempt11:30:013
12702799353,4cyclictest131rcu_preempt11:30:013
127027993525,8cyclictest37-21ksoftirqd/307:36:483
127027993523,9cyclictest37-21ksoftirqd/312:25:243
127027993518,13cyclictest37-21ksoftirqd/308:45:023
127027993518,11cyclictest272-21dbus-daemon10:10:013
12702799351,3cyclictest131rcu_preempt12:25:013
127017993527,5cyclictest139951-21latency_hist11:20:021
127017993525,7cyclictest23-21ksoftirqd/110:20:471
127017993525,7cyclictest23-21ksoftirqd/110:20:461
12701799352,4cyclictest131rcu_preempt12:10:211
12701799352,4cyclictest131rcu_preempt12:10:211
127017993523,9cyclictest23-21ksoftirqd/108:24:591
12701299353,3cyclictest131rcu_preempt07:52:480
12701299351,31cyclictest0-21swapper/008:29:240
12701299351,30cyclictest0-21swapper/009:44:010
12701299351,30cyclictest0-21swapper/009:27:170
12701299351,30cyclictest0-21swapper/008:20:580
127027993425,6cyclictest128316-21latency_hist07:35:023
127027993425,6cyclictest128316-21latency_hist07:35:013
12702799341,3cyclictest131rcu_preempt11:25:013
12702799341,3cyclictest131rcu_preempt07:15:003
12701799349,14cyclictest235-21systemd-journal07:35:001
12701799349,14cyclictest235-21systemd-journal07:34:591
12701799347,20cyclictest128508-21sh07:40:001
12701799344,25cyclictest1-21systemd08:35:011
127017993427,4cyclictest140727-21latency_hist11:35:021
127017993423,8cyclictest143995-21sed12:35:031
12701299342,4cyclictest131rcu_preempt07:30:010
12701299342,3cyclictest131rcu_preempt08:47:470
12701299341,30cyclictest0-21swapper/010:10:360
127027993326,4cyclictest141746-21sshd11:52:483
12701799335,21cyclictest134490-21cron09:35:001
127017993325,5cyclictest137640-21latency_hist10:35:011
127017993320,10cyclictest132126-21sshd08:46:481
127017993318,11cyclictest135024-21cron09:45:001
12701799331,5cyclictest131rcu_preempt11:55:001
127027993222,7cyclictest127037-21sed07:10:023
127027993222,7cyclictest127037-21sed07:10:013
12702799321,4cyclictest131rcu_preempt09:55:003
12702799321,3cyclictest131rcu_preempt10:25:003
12702799321,3cyclictest131rcu_preempt10:25:003
12701799323,5cyclictest131rcu_preempt10:49:591
127017993225,4cyclictest129546-21sendmail08:00:001
127017993220,9cyclictest132998-21wc09:05:001
127017993211,5cyclictest23-21ksoftirqd/108:28:491
127017993211,5cyclictest23-21ksoftirqd/108:28:491
12701299322,3cyclictest131rcu_preempt07:10:020
12701299322,3cyclictest131rcu_preempt07:10:010
127027993121,7cyclictest37-21ksoftirqd/307:41:403
12701799312,4cyclictest131rcu_preempt12:36:481
127017993019,7cyclictest135523-21sshd09:54:481
127017992819,4cyclictest0-21swapper/107:10:021
127017992819,4cyclictest0-21swapper/107:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional