You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 19:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Nov 18, 2025 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
332918399128117,7cyclictest37-21ksoftirqd/307:50:023
332916999853,4cyclictest131rcu_preempt09:15:020
332916999853,4cyclictest131rcu_preempt09:15:010
3329169997828,10cyclictest306-21in:imuxsock09:50:330
3329169997635,8cyclictest272-21dbus-daemon07:40:010
3329183996415,10cyclictest37-21ksoftirqd/309:00:003
3329183996317,10cyclictest37-21ksoftirqd/308:00:013
3329183996115,9cyclictest37-21ksoftirqd/309:30:033
3329183996115,9cyclictest37-21ksoftirqd/309:30:023
3329183996113,10cyclictest37-21ksoftirqd/309:25:003
332917499616,7cyclictest23-21ksoftirqd/109:05:011
3329183996018,7cyclictest37-21ksoftirqd/310:35:023
3329183996014,10cyclictest37-21ksoftirqd/309:15:013
3329183996014,10cyclictest37-21ksoftirqd/309:15:003
3329183995817,7cyclictest37-21ksoftirqd/310:09:593
332917499581,16cyclictest23-21ksoftirqd/108:10:021
332918399567,16cyclictest3331508-21systemd-run07:55:003
3329183995616,10cyclictest37-21ksoftirqd/312:30:003
3329183995615,8cyclictest37-21ksoftirqd/311:50:013
332917499567,9cyclictest23-21ksoftirqd/107:40:031
332918399542,47cyclictest1-21systemd11:10:013
332917499548,12cyclictest23-21ksoftirqd/112:05:021
332917499548,12cyclictest23-21ksoftirqd/112:05:011
332917499546,9cyclictest23-21ksoftirqd/112:25:021
332917499546,9cyclictest23-21ksoftirqd/112:25:021
3329183995337,13cyclictest1-21systemd09:45:013
3329183995337,13cyclictest1-21systemd09:45:013
3329183995334,14cyclictest1-21systemd07:35:013
3329183995312,8cyclictest37-21ksoftirqd/312:35:023
3329178995312,11cyclictest131rcu_preempt09:00:002
332917499531,7cyclictest0-21swapper/110:15:021
332918399528,10cyclictest37-21ksoftirqd/311:30:003
332917499525,9cyclictest23-21ksoftirqd/111:50:011
3329174995237,11cyclictest272-21dbus-daemon11:19:591
332917499522,10cyclictest23-21ksoftirqd/110:04:591
3329174995217,10cyclictest23-21ksoftirqd/110:35:021
3329174995214,9cyclictest23-21ksoftirqd/112:10:011
332916999522,43cyclictest3340859-21munin-plugin-st10:55:010
332916999522,43cyclictest3340859-21munin-plugin-st10:55:010
3329183995134,12cyclictest235-21systemd-journal12:19:593
332917499519,11cyclictest23-21ksoftirqd/109:40:021
332917499514,11cyclictest23-21ksoftirqd/109:17:001
332917499512,12cyclictest23-21ksoftirqd/110:25:021
332917499512,12cyclictest23-21ksoftirqd/110:25:011
332916999514,4cyclictest131rcu_preempt10:15:010
332916999512,4cyclictest131rcu_preempt09:20:000
3329183995034,12cyclictest1-21systemd08:30:013
332917499507,8cyclictest23-21ksoftirqd/111:15:001
332917499506,9cyclictest23-21ksoftirqd/107:55:011
332917499505,9cyclictest23-21ksoftirqd/108:00:021
332917499505,6cyclictest23-21ksoftirqd/111:30:021
332917499504,10cyclictest23-21ksoftirqd/107:20:001
332917499504,10cyclictest23-21ksoftirqd/107:20:001
332917499503,9cyclictest23-21ksoftirqd/112:00:011
332916999502,3cyclictest131rcu_preempt08:00:030
3329169995023,20cyclictest3333823-21modprobe08:40:000
332917499496,8cyclictest23-21ksoftirqd/109:35:021
3329178994826,14cyclictest3329673-21modprobe07:20:002
3329178994826,14cyclictest3329673-21modprobe07:19:592
332917499486,13cyclictest23-21ksoftirqd/108:33:331
332917499485,9cyclictest23-21ksoftirqd/110:50:011
332917499485,6cyclictest23-21ksoftirqd/110:40:021
332917499485,6cyclictest23-21ksoftirqd/110:40:011
332917499485,6cyclictest23-21ksoftirqd/108:25:011
332917499483,10cyclictest23-21ksoftirqd/109:09:591
332917499481,9cyclictest3345447-21cut12:20:021
332917499481,10cyclictest3342432-21latency_hist11:25:011
332917499481,10cyclictest3342432-21latency_hist11:25:011
3329178994731,12cyclictest235-21systemd-journal07:40:012
3329178994729,14cyclictest1-21systemd10:50:012
3329178994728,13cyclictest235-21systemd-journal12:10:012
3329178994728,13cyclictest235-21systemd-journal09:34:592
332917499478,7cyclictest23-21ksoftirqd/110:20:021
332917499477,9cyclictest23-21ksoftirqd/111:05:011
332917499476,8cyclictest23-21ksoftirqd/107:50:001
332917499474,12cyclictest23-21ksoftirqd/111:40:021
332917499474,12cyclictest23-21ksoftirqd/111:40:021
3329174994734,9cyclictest235-21systemd-journal11:10:011
3329174994711,9cyclictest0-21swapper/109:55:011
332916999471,3cyclictest131rcu_preempt08:59:590
3329183994631,11cyclictest235-21systemd-journal07:27:333
3329178994630,12cyclictest1-21systemd07:55:012
332917499467,8cyclictest23-21ksoftirqd/108:20:031
332917499467,6cyclictest23-21ksoftirqd/108:50:031
332917499466,6cyclictest23-21ksoftirqd/108:15:001
332917499465,7cyclictest23-21ksoftirqd/109:45:031
332917499465,7cyclictest23-21ksoftirqd/109:45:021
332917499464,10cyclictest23-21ksoftirqd/108:55:011
3329174994634,9cyclictest23-21ksoftirqd/108:40:011
3329169994633,9cyclictest1-21systemd07:45:010
3329169994631,10cyclictest272-21dbus-daemon12:40:000
3329169994630,11cyclictest235-21systemd-journal10:45:000
332916999461,4cyclictest131rcu_preempt11:45:000
3329183994532,9cyclictest1-21systemd07:15:003
3329183994532,9cyclictest1-21systemd07:15:003
332917499456,8cyclictest23-21ksoftirqd/110:30:021
332917499456,10cyclictest23-21ksoftirqd/107:35:021
332917499454,9cyclictest23-21ksoftirqd/110:45:001
332917499454,7cyclictest23-21ksoftirqd/108:30:001
332917499453,7cyclictest23-21ksoftirqd/107:30:021
332917499451,10cyclictest3335658-21munin-run09:15:011
332917499451,10cyclictest3335658-21munin-run09:15:011
332916999453,3cyclictest131rcu_preempt09:40:020
3329169994528,13cyclictest1-21systemd10:20:010
332916999451,4cyclictest131rcu_preempt10:59:590
332916999451,3cyclictest131rcu_preempt10:25:020
332916999451,3cyclictest131rcu_preempt10:25:020
3329183994429,12cyclictest1-21systemd08:55:013
3329183994428,13cyclictest272-21dbus-daemon09:20:003
3329183994426,14cyclictest3343472-21munin-run11:45:003
3329183994423,17cyclictest3338243-21sshd10:01:343
3329183994410,7cyclictest0-21swapper/309:49:003
332917899449,5cyclictest131rcu_preempt09:42:012
332917899449,5cyclictest131rcu_preempt09:42:012
3329178994429,11cyclictest235-21systemd-journal12:20:002
332917499447,7cyclictest23-21ksoftirqd/110:55:001
332917499447,7cyclictest23-21ksoftirqd/110:54:591
332917499444,8cyclictest23-21ksoftirqd/109:25:011
332917499444,8cyclictest23-21ksoftirqd/107:11:261
332917499444,8cyclictest23-21ksoftirqd/107:11:251
3329174994433,6cyclictest3329976-21cat07:25:001
332917499442,11cyclictest23-21ksoftirqd/111:00:001
332917499441,13cyclictest3334950-21latency_hist09:00:021
332916999444,3cyclictest131rcu_preempt12:00:010
332916999441,4cyclictest131rcu_preempt11:50:010
332916999441,4cyclictest131rcu_preempt11:40:020
332916999441,4cyclictest131rcu_preempt11:40:020
3329183994330,10cyclictest3341370-21sshd11:04:333
3329183994329,10cyclictest3330967-21sshd07:42:343
3329183994329,10cyclictest235-21systemd-journal10:25:023
3329183994327,12cyclictest3338758-21sshd10:14:333
3329183994326,14cyclictest3336674-21sshd09:31:333
3329178994327,12cyclictest1-21systemd08:55:012
332917499436,8cyclictest23-21ksoftirqd/111:55:011
332917499436,14cyclictest23-21ksoftirqd/112:35:001
332917499435,7cyclictest23-21ksoftirqd/110:10:011
332917499434,9cyclictest23-21ksoftirqd/108:45:001
332917499434,6cyclictest23-21ksoftirqd/109:30:021
332917499434,6cyclictest23-21ksoftirqd/109:30:011
3329174994332,8cyclictest3343447-21sshd11:42:341
332916999434,3cyclictest131rcu_preempt07:50:020
3329169994333,7cyclictest3333576-21munin-run08:35:000
3329183994230,9cyclictest363-21snmpd07:24:083
3329183994228,10cyclictest3345515-21sshd12:20:333
3329183994228,10cyclictest3345515-21sshd12:20:333
3329183994228,10cyclictest3343987-21munin-run11:55:003
3329183994227,11cyclictest37-21ksoftirqd/309:10:013
3329183994222,16cyclictest1-21systemd11:40:013
3329183994222,16cyclictest1-21systemd11:40:003
332917899429,5cyclictest131rcu_preempt07:44:202
3329178994233,5cyclictest3332280-21munin-plugin-st08:09:592
3329178994225,12cyclictest235-21systemd-journal10:15:002
3329178994213,17cyclictest3340066-21modprobe10:40:002
3329178994213,17cyclictest3340066-21modprobe10:40:002
332917499427,6cyclictest23-21ksoftirqd/112:40:021
332917499424,8cyclictest23-21ksoftirqd/112:10:321
332916999424,4cyclictest131rcu_preempt08:55:010
3329169994229,10cyclictest235-21systemd-journal08:40:020
3329169994228,11cyclictest3345245-21sshd12:17:330
332916999421,4cyclictest131rcu_preempt09:30:010
332916999421,4cyclictest131rcu_preempt09:30:000
332916999421,4cyclictest131rcu_preempt08:05:010
3329183994133,5cyclictest3332515-21sshd08:14:333
3329183994129,9cyclictest3344980-21sshd12:11:333
3329183994124,14cyclictest3339314-21sh10:25:013
3329183994124,14cyclictest3339314-21sh10:25:013
3329183994122,14cyclictest363-21snmpd08:08:413
3329183994114,21cyclictest3344241-21readlink12:00:013
3329178994131,7cyclictest30-21ksoftirqd/208:50:022
3329178994123,13cyclictest0-21swapper/207:25:002
332917499412,9cyclictest23-21ksoftirqd/110:00:011
332917499412,9cyclictest23-21ksoftirqd/110:00:011
332916999414,4cyclictest131rcu_preempt11:30:020
332916999414,3cyclictest131rcu_preempt11:20:000
332916999413,6cyclictest131rcu_preempt09:45:010
332916999413,6cyclictest131rcu_preempt09:45:010
332916999413,4cyclictest131rcu_preempt09:09:330
3329169994127,11cyclictest3342399-21sshd11:20:340
3329169994127,11cyclictest3342399-21sshd11:20:330
332916999411,3cyclictest131rcu_preempt11:11:330
332916999411,3cyclictest131rcu_preempt07:15:030
332916999411,3cyclictest131rcu_preempt07:15:030
3329183994031,5cyclictest3340037-21sshd10:35:343
3329183994031,5cyclictest3340037-21sshd10:35:343
3329183994028,9cyclictest3344732-21sshd12:07:333
3329183994027,10cyclictest363-21snmpd11:34:223
3329183994025,11cyclictest3339011-21sshd10:17:333
3329183994024,12cyclictest3342164-21munin-run11:20:003
3329183994024,12cyclictest3336938-21sshd09:39:333
3329178994028,9cyclictest1-21systemd12:40:012
3329178994026,11cyclictest3337579-21sed09:50:022
332917899402,3cyclictest131rcu_preempt11:10:022
332917499401,4cyclictest0-21swapper/111:30:021
332916999405,6cyclictest131rcu_preempt10:25:320
3329169994028,9cyclictest1-21systemd12:25:500
3329169994026,11cyclictest1-21systemd08:25:010
332916999402,3cyclictest131rcu_preempt10:35:010
3329183993927,8cyclictest272-21dbus-daemon12:05:013
3329183993927,8cyclictest272-21dbus-daemon12:05:013
3329183993926,10cyclictest37-21ksoftirqd/310:50:003
3329183993926,10cyclictest3330695-21sshd07:39:323
3329183993924,12cyclictest3329633-21sshd07:15:333
3329183993924,12cyclictest3329633-21sshd07:15:323
3329183993919,16cyclictest363-21snmpd09:59:033
3329183993919,16cyclictest363-21snmpd09:59:023
332917899397,4cyclictest131rcu_preempt10:05:002
332917899397,3cyclictest131rcu_preempt07:28:332
3329178993930,5cyclictest3338205-21latency_hist10:00:032
3329178993930,5cyclictest3338205-21latency_hist10:00:022
3329178993928,3cyclictest0-21swapper/209:30:032
3329178993928,3cyclictest0-21swapper/209:30:032
3329178993925,10cyclictest30-21ksoftirqd/210:20:002
332917899391,3cyclictest131rcu_preempt11:05:022
332917499397,7cyclictest23-21ksoftirqd/107:40:341
3329169993929,7cyclictest12-21ksoftirqd/007:35:030
332916999391,4cyclictest131rcu_preempt08:50:010
3329169993914,17cyclictest0-21swapper/010:40:020
3329169993914,17cyclictest0-21swapper/010:40:010
332916999391,3cyclictest131rcu_preempt08:09:590
332916999391,35cyclictest3344753-21latency_hist12:10:020
332918399387,27cyclictest3334095-21cron08:45:003
3329183993827,8cyclictest3335107-21sshd09:00:213
3329183993825,10cyclictest3346272-21sshd12:35:323
3329183993818,15cyclictest3341101-21cron11:00:003
3329178993829,6cyclictest3335143-21grep09:05:002
3329178993827,8cyclictest3344476-21sshd12:01:332
3329178993827,8cyclictest3344476-21sshd12:01:332
3329178993824,10cyclictest272-21dbus-daemon12:15:012
332917899382,3cyclictest131rcu_preempt08:30:022
3329178993823,12cyclictest3342142-21cron11:20:002
332917499385,6cyclictest23-21ksoftirqd/112:25:031
332917499384,7cyclictest23-21ksoftirqd/108:00:101
332916999384,25cyclictest3329173-21latency_hist07:10:010
332916999382,7cyclictest3337466-21sshd09:48:330
3329169993825,10cyclictest3338633-21sed10:10:010
3329169993825,10cyclictest1-21systemd09:04:340
3329169993824,11cyclictest1-21systemd10:00:010
3329169993824,11cyclictest1-21systemd10:00:010
332916999382,3cyclictest131rcu_preempt09:33:580
332916999381,5cyclictest3341402-21unin-run11:05:010
332916999381,3cyclictest131rcu_preempt12:25:010
332916999381,3cyclictest131rcu_preempt12:25:010
3329183993729,4cyclictest3340316-21sshd10:40:333
3329183993724,10cyclictest3332752-21sshd08:15:333
3329183993720,13cyclictest363-21snmpd08:20:313
332917899372,8cyclictest0-21swapper/212:25:032
332917499374,6cyclictest23-21ksoftirqd/109:45:331
332916999375,11cyclictest131rcu_preempt11:55:000
3329169993731,4cyclictest0-21swapper/012:15:010
3329169993730,4cyclictest16950irq/224-800000007:54:590
3329169993726,7cyclictest12-21ksoftirqd/008:30:030
3329169993724,9cyclictest3338241-21sshd10:00:340
3329169993723,11cyclictest3330293-21cat07:30:010
3329169993723,11cyclictest12-21ksoftirqd/009:25:010
3329169993719,12cyclictest235-21systemd-journal11:10:000
332916999371,4cyclictest131rcu_preempt12:35:000
332916999371,4cyclictest131rcu_preempt07:20:020
332916999371,4cyclictest131rcu_preempt07:20:010
332916999371,3cyclictest131rcu_preempt12:00:330
332916999371,3cyclictest131rcu_preempt12:00:330
3329183993629,4cyclictest3341875-21sshd11:14:333
3329183993629,4cyclictest3333576-21munin-run08:35:003
3329183993622,9cyclictest272-21dbus-daemon11:25:013
3329183993622,9cyclictest272-21dbus-daemon11:25:003
3329178993624,7cyclictest3335395-21modprobe09:10:002
332917899361,3cyclictest131rcu_preempt08:35:022
3329169993625,8cyclictest3330155-21sed07:25:020
3329169993620,12cyclictest235-21systemd-journal11:30:010
332916999361,3cyclictest131rcu_preempt10:50:020
3329183993526,6cyclictest3334444-21latency_hist08:50:023
3329183993521,10cyclictest412-21sshd09:50:333
3329183993521,10cyclictest363-21snmpd08:03:403
3329183993520,12cyclictest363-21snmpd10:53:573
3329183993520,12cyclictest363-21snmpd10:53:573
332917899355,3cyclictest131rcu_preempt08:39:592
3329178993528,4cyclictest0-21swapper/207:10:022
3329178993525,6cyclictest412-21sshd12:29:342
3329178993524,8cyclictest235-21systemd-journal10:51:562
3329178993524,8cyclictest235-21systemd-journal10:51:552
3329178993523,8cyclictest3337115-21date09:40:022
3329178993521,10cyclictest272-21dbus-daemon10:10:012
3329178993519,12cyclictest1-21systemd11:44:592
3329183993426,5cyclictest3329290-21latency_hist07:10:023
3329183993419,11cyclictest363-21snmpd08:38:473
332917899346,4cyclictest131rcu_preempt11:25:012
332917899346,4cyclictest131rcu_preempt11:25:002
332917899344,6cyclictest131rcu_preempt08:00:002
3329178993427,4cyclictest3343719-21latency_hist11:50:022
3329178993425,7cyclictest30-21ksoftirqd/207:32:342
3329178993425,6cyclictest3340480-21latency_hist10:45:022
3329178993425,5cyclictest307-21in:imklog10:25:022
3329178993425,5cyclictest307-21in:imklog10:25:022
3329178993425,5cyclictest306-21in:imuxsock12:35:002
3329178993424,7cyclictest3336187-21cron09:24:592
3329178993424,7cyclictest3335940-21sendmail09:20:002
3329178993423,8cyclictest30-21ksoftirqd/209:15:012
3329178993423,8cyclictest30-21ksoftirqd/209:15:002
3329178993422,9cyclictest30-21ksoftirqd/212:00:012
332917899334,3cyclictest131rcu_preempt08:45:012
3329178993321,9cyclictest235-21systemd-journal08:14:262
3329178993319,11cyclictest3329329-21sshd07:10:332
3329178993319,11cyclictest3329329-21sshd07:10:322
3329178993319,11cyclictest30-21ksoftirqd/211:40:022
3329178993319,11cyclictest30-21ksoftirqd/211:40:022
3329178993319,10cyclictest272-21dbus-daemon08:25:002
332916999331,28cyclictest0-21swapper/008:11:260
332917899324,3cyclictest131rcu_preempt11:00:002
3329178993224,5cyclictest3341890-21latency_hist11:15:022
332917899322,3cyclictest131rcu_preempt11:30:022
3329178993223,7cyclictest3331273-21sed07:50:012
3329178993216,12cyclictest3332237-21cut08:05:032
3329178993215,14cyclictest272-21dbus-daemon11:55:012
332916999321,28cyclictest0-21swapper/008:15:210
332917899314,4cyclictest131rcu_preempt11:35:002
332917899313,5cyclictest131rcu_preempt10:29:332
3329178993119,9cyclictest306-21in:imuxsock08:15:342
332917899305,3cyclictest131rcu_preempt12:24:322
332917899304,3cyclictest131rcu_preempt09:54:332
332917899296,3cyclictest131rcu_preempt10:35:002
3329174992921,6cyclictest23-21ksoftirqd/107:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional