You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-25 - 18:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Sat Oct 25, 2025 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
349108996719,17cyclictest351773-21sed08:00:033
349108996719,17cyclictest351773-21sed08:00:023
34909399606,13cyclictest12-21ksoftirqd/009:10:000
349101995921,32cyclictest48-21kcompactd010:45:241
349108995815,10cyclictest37-21ksoftirqd/312:15:023
34909399556,8cyclictest12-21ksoftirqd/007:30:010
34909399556,8cyclictest12-21ksoftirqd/007:30:010
34909399555,11cyclictest12-21ksoftirqd/010:40:010
34909399544,7cyclictest12-21ksoftirqd/009:20:010
349093995418,6cyclictest12-21ksoftirqd/010:01:220
34910899535,6cyclictest37-21ksoftirqd/309:20:013
34909399534,7cyclictest12-21ksoftirqd/011:40:030
34909399534,7cyclictest12-21ksoftirqd/011:40:030
349093995338,11cyclictest366190-21systemctl12:40:010
34909399526,9cyclictest12-21ksoftirqd/008:25:000
34910899517,11cyclictest37-21ksoftirqd/308:35:013
349108995134,13cyclictest1-21systemd09:05:013
349101995134,13cyclictest272-21dbus-daemon11:30:001
34909399515,7cyclictest12-21ksoftirqd/009:55:020
349108995035,11cyclictest1-21systemd12:35:003
34910199507,10cyclictest23-21ksoftirqd/110:25:021
34909399496,8cyclictest12-21ksoftirqd/011:00:000
34909399496,8cyclictest12-21ksoftirqd/007:40:010
34909399495,8cyclictest12-21ksoftirqd/011:20:010
34909399494,9cyclictest12-21ksoftirqd/008:55:000
34909399492,6cyclictest12-21ksoftirqd/009:40:010
34910899489,9cyclictest0-21swapper/310:20:013
349108994837,8cyclictest361749-21sshd11:13:223
349108994831,14cyclictest1-21systemd12:25:003
349105994829,10cyclictest30-21ksoftirqd/211:45:002
349101994812,7cyclictest23-21ksoftirqd/111:15:021
349093994831,12cyclictest353724-21tr08:40:000
34910899478,9cyclictest356067-21sh09:25:023
34910899478,9cyclictest356067-21sh09:25:023
34910899477,6cyclictest37-21ksoftirqd/311:08:223
349105994730,13cyclictest235-21systemd-journal11:35:012
34910199479,7cyclictest23-21ksoftirqd/109:20:021
34910199476,7cyclictest357046-21cat09:40:021
349101994733,10cyclictest235-21systemd-journal07:25:001
349101994730,13cyclictest272-21dbus-daemon08:20:011
34909399477,9cyclictest12-21ksoftirqd/012:25:000
34909399475,8cyclictest12-21ksoftirqd/011:45:000
349093994738,6cyclictest19050irq/34-mmc110:30:010
349093994738,6cyclictest19050irq/34-mmc110:30:000
34910899467,8cyclictest37-21ksoftirqd/310:45:013
34910899466,7cyclictest37-21ksoftirqd/310:35:013
349108994632,10cyclictest272-21dbus-daemon09:25:003
349105994627,14cyclictest235-21systemd-journal10:10:012
349105994622,19cyclictest30-21ksoftirqd/211:15:012
349101994626,14cyclictest1-21systemd08:35:011
34909399466,8cyclictest12-21ksoftirqd/007:25:010
34909399466,10cyclictest12-21ksoftirqd/010:50:000
34909399465,7cyclictest12-21ksoftirqd/012:00:030
34909399465,7cyclictest12-21ksoftirqd/008:10:020
34909399465,5cyclictest12-21ksoftirqd/009:35:010
34909399464,10cyclictest12-21ksoftirqd/012:30:010
34909399464,10cyclictest12-21ksoftirqd/012:30:010
349093994631,11cyclictest12-21ksoftirqd/009:30:000
349093994631,11cyclictest12-21ksoftirqd/009:30:000
34909399462,7cyclictest12-21ksoftirqd/009:45:010
34909399462,16cyclictest362814-21grep11:35:000
349108994534,9cyclictest355236-21sshd09:05:223
349108994530,11cyclictest357852-21cron10:00:003
349105994529,12cyclictest358867-21sshd10:15:222
34910199452,37cyclictest357169-21cat09:45:011
34909399459,10cyclictest12-21ksoftirqd/010:15:080
34909399457,6cyclictest12-21ksoftirqd/008:15:020
34909399455,8cyclictest12-21ksoftirqd/011:30:010
349093994537,5cyclictest151rcuc/009:14:220
349093994530,11cyclictest12-21ksoftirqd/007:45:000
349093994530,11cyclictest12-21ksoftirqd/007:45:000
34910899446,7cyclictest37-21ksoftirqd/307:35:003
34910899446,33cyclictest48-21kcompactd012:35:153
34910899444,7cyclictest37-21ksoftirqd/310:50:013
34910599449,10cyclictest30-21ksoftirqd/209:43:232
34910199446,9cyclictest23-21ksoftirqd/110:35:021
34910199446,24cyclictest355262-21modprobe09:10:001
349101994428,13cyclictest23-21ksoftirqd/107:35:021
34910199442,6cyclictest131rcu_preempt09:30:021
34910199442,6cyclictest131rcu_preempt09:30:011
34909399447,7cyclictest12-21ksoftirqd/011:05:000
34909399446,9cyclictest12-21ksoftirqd/009:50:000
34909399446,6cyclictest12-21ksoftirqd/012:05:010
34909399446,6cyclictest12-21ksoftirqd/007:55:070
34909399446,6cyclictest12-21ksoftirqd/007:55:070
34909399445,10cyclictest12-21ksoftirqd/012:10:000
349093994414,11cyclictest12-21ksoftirqd/012:13:370
34909399441,18cyclictest12-21ksoftirqd/009:55:220
34910899439,5cyclictest0-21swapper/308:25:073
34910899434,6cyclictest37-21ksoftirqd/310:29:223
34910899434,6cyclictest37-21ksoftirqd/310:29:223
34910899432,7cyclictest37-21ksoftirqd/307:45:023
34910899432,7cyclictest37-21ksoftirqd/307:45:023
349105994334,6cyclictest30-21ksoftirqd/212:10:002
349105994331,8cyclictest30-21ksoftirqd/210:15:012
34910599431,6cyclictest0-21swapper/208:43:222
349105994310,8cyclictest30-21ksoftirqd/210:20:222
34910199436,10cyclictest23-21ksoftirqd/111:50:011
34910199435,5cyclictest131rcu_preempt08:55:021
349101994330,8cyclictest272-21dbus-daemon08:40:001
349101994326,13cyclictest272-21dbus-daemon12:15:001
349101994312,4cyclictest0-21swapper/108:49:381
349101994310,6cyclictest0-21swapper/112:32:081
34909399437,7cyclictest12-21ksoftirqd/011:25:020
349093994336,4cyclictest16950irq/224-800000010:25:000
349093994330,10cyclictest12-21ksoftirqd/009:00:010
349093994330,10cyclictest12-21ksoftirqd/009:00:000
34909399432,12cyclictest12-21ksoftirqd/010:15:010
34910899426,8cyclictest37-21ksoftirqd/311:30:023
34910899425,6cyclictest37-21ksoftirqd/307:27:003
34910899425,6cyclictest37-21ksoftirqd/307:27:003
34910899423,4cyclictest131rcu_preempt07:25:013
349108994229,9cyclictest272-21dbus-daemon09:35:003
349108994227,11cyclictest37-21ksoftirqd/308:40:003
349108994223,14cyclictest235-21systemd-journal10:25:013
349108994223,14cyclictest1-21systemd11:05:013
34910599426,11cyclictest30-21ksoftirqd/210:32:222
349105994234,6cyclictest30-21ksoftirqd/211:55:032
349105994234,6cyclictest30-21ksoftirqd/211:55:022
349105994226,12cyclictest235-21systemd-journal11:46:222
34910199425,6cyclictest23-21ksoftirqd/112:03:461
34910199424,8cyclictest23-21ksoftirqd/107:27:521
34910199424,8cyclictest23-21ksoftirqd/107:27:521
349101994224,11cyclictest0-21swapper/108:10:001
34909399425,11cyclictest12-21ksoftirqd/010:51:230
34909399424,8cyclictest12-21ksoftirqd/011:25:000
349093994235,4cyclictest16950irq/224-800000008:29:220
34909399422,8cyclictest355013-21modprobe09:04:590
34909399422,6cyclictest12-21ksoftirqd/008:46:220
34909399421,9cyclictest365317-21ls12:20:010
349108994130,8cyclictest1-21systemd11:30:013
349108994129,9cyclictest1-21systemd11:20:013
34910899412,4cyclictest131rcu_preempt08:20:013
349108994122,15cyclictest358123-21sshd10:02:233
34910899411,4cyclictest131rcu_preempt08:50:003
34910899411,14cyclictest358733-21cut10:15:023
34910599416,7cyclictest131rcu_preempt07:50:012
34910599415,11cyclictest30-21ksoftirqd/211:26:212
34910599414,11cyclictest30-21ksoftirqd/211:20:232
34910599414,11cyclictest30-21ksoftirqd/211:20:222
349105994130,8cyclictest30-21ksoftirqd/211:16:592
349105994129,9cyclictest30-21ksoftirqd/211:00:022
349105994129,7cyclictest30-21ksoftirqd/210:00:022
349105994128,10cyclictest366215-21systemd-run12:40:002
349105994126,11cyclictest357348-21sshd09:49:212
34910599412,5cyclictest131rcu_preempt10:40:002
349105994123,6cyclictest131rcu_preempt08:40:012
34910199416,4cyclictest131rcu_preempt09:01:221
34910199415,4cyclictest131rcu_preempt08:13:221
34910199412,9cyclictest358513-21sed10:10:021
349101994127,11cyclictest351932-21sed08:05:021
34910199412,5cyclictest23-21ksoftirqd/110:04:231
34909399416,10cyclictest12-21ksoftirqd/007:20:010
34909399414,7cyclictest12-21ksoftirqd/009:25:000
349093994128,10cyclictest359666-21sshd10:30:580
349093994115,17cyclictest12-21ksoftirqd/007:33:330
349093994110,5cyclictest131rcu_preempt08:04:220
34910899406,8cyclictest37-21ksoftirqd/311:42:223
34910899406,23cyclictest364098-21modprobe12:00:003
349108994021,15cyclictest360702-21sshd10:51:053
34910899401,5cyclictest0-21swapper/311:00:013
34910899401,4cyclictest0-21swapper/310:40:013
349105994030,6cyclictest30-21ksoftirqd/209:35:002
349105994029,8cyclictest354710-21sshd08:55:472
349105994029,8cyclictest354710-21sshd08:55:472
34910199409,26cyclictest358923-21systemctl10:20:001
34910199408,5cyclictest131rcu_preempt08:22:221
34910199404,8cyclictest23-21ksoftirqd/107:12:501
34910199403,8cyclictest23-21ksoftirqd/109:57:221
34910199403,4cyclictest131rcu_preempt12:16:231
349101994030,7cyclictest349527-21sshd07:17:221
349101994029,8cyclictest272-21dbus-daemon11:40:021
349101994029,8cyclictest272-21dbus-daemon11:40:011
349101994028,10cyclictest272-21dbus-daemon07:40:011
349101994027,9cyclictest306-21in:imuxsock09:50:011
349101994026,11cyclictest23-21ksoftirqd/110:45:011
349101994024,12cyclictest272-21dbus-daemon11:20:001
34910199401,9cyclictest23-21ksoftirqd/107:50:001
349101994011,6cyclictest0-21swapper/107:42:371
349101994011,6cyclictest0-21swapper/107:42:361
349101994010,6cyclictest0-21swapper/111:42:451
349101994010,16cyclictest359946-21modprobe10:40:001
34909399406,5cyclictest12-21ksoftirqd/007:55:020
349093994034,3cyclictest16950irq/224-800000011:49:230
349093994033,4cyclictest16950irq/224-800000011:55:000
349093994033,4cyclictest16950irq/224-800000011:55:000
349093994029,9cyclictest12-21ksoftirqd/011:10:020
349093994026,10cyclictest363-21snmpd07:12:490
349093994025,12cyclictest12-21ksoftirqd/007:50:010
34910899394,6cyclictest37-21ksoftirqd/308:02:583
34910899394,6cyclictest37-21ksoftirqd/307:55:013
349108993921,12cyclictest1-21systemd12:30:023
349108993921,12cyclictest1-21systemd12:30:013
34910899391,7cyclictest0-21swapper/309:15:023
34910899391,4cyclictest131rcu_preempt09:40:023
34910899391,3cyclictest131rcu_preempt07:37:233
349108993913,7cyclictest37-21ksoftirqd/308:23:373
349108993910,5cyclictest131rcu_preempt12:19:233
349108993910,5cyclictest131rcu_preempt11:49:233
34910599399,18cyclictest30-21ksoftirqd/209:10:012
34910599398,5cyclictest131rcu_preempt09:38:172
349105993926,9cyclictest30-21ksoftirqd/210:25:222
349105993926,9cyclictest30-21ksoftirqd/210:25:222
349105993924,11cyclictest356046-21sshd09:21:232
349105993922,13cyclictest30-21ksoftirqd/211:40:032
349105993922,13cyclictest30-21ksoftirqd/211:40:022
34910199398,5cyclictest0-21swapper/109:53:221
34910199397,3cyclictest131rcu_preempt11:58:451
34910199393,7cyclictest23-21ksoftirqd/110:28:281
34910199393,7cyclictest23-21ksoftirqd/110:28:271
349101993924,11cyclictest235-21systemd-journal12:40:001
349101993910,7cyclictest0-21swapper/108:59:461
349101993910,7cyclictest0-21swapper/108:59:461
34909399394,8cyclictest12-21ksoftirqd/011:14:210
349093993933,4cyclictest0-21swapper/008:35:010
349093993928,7cyclictest358397-21modprobe10:10:000
349093993920,11cyclictest12-21ksoftirqd/012:30:310
349093993912,23cyclictest151rcuc/008:45:020
34910899389,5cyclictest131rcu_preempt09:43:173
34910899385,4cyclictest131rcu_preempt11:53:223
34910899385,4cyclictest131rcu_preempt11:53:223
349108993823,12cyclictest235-21systemd-journal11:25:003
349108993823,12cyclictest235-21systemd-journal11:25:003
34910599388,5cyclictest131rcu_preempt08:20:012
34910599385,9cyclictest30-21ksoftirqd/207:13:222
349105993828,8cyclictest30-21ksoftirqd/207:40:022
349105993826,8cyclictest30-21ksoftirqd/207:55:032
349105993826,8cyclictest30-21ksoftirqd/207:55:022
349105993825,9cyclictest235-21systemd-journal10:51:042
349105993823,12cyclictest235-21systemd-journal09:05:012
349105993811,5cyclictest131rcu_preempt08:25:022
34910199386,4cyclictest131rcu_preempt11:01:061
34910199385,7cyclictest23-21ksoftirqd/107:57:571
34910199385,7cyclictest23-21ksoftirqd/107:57:571
34910199385,4cyclictest131rcu_preempt11:50:231
34910199385,4cyclictest131rcu_preempt11:50:221
34910199383,7cyclictest23-21ksoftirqd/108:44:251
34910199382,4cyclictest131rcu_preempt12:28:501
34910199382,4cyclictest131rcu_preempt12:28:491
349101993823,12cyclictest23-21ksoftirqd/109:35:011
349101993810,9cyclictest0-21swapper/111:06:131
349101993810,5cyclictest0-21swapper/109:20:511
34910899375,8cyclictest37-21ksoftirqd/309:45:223
349108993720,13cyclictest235-21systemd-journal12:10:013
349108993711,23cyclictest1-21systemd12:05:013
34910599377,6cyclictest131rcu_preempt12:25:222
34910599377,6cyclictest131rcu_preempt12:25:212
349105993727,7cyclictest30-21ksoftirqd/208:30:002
34910599372,6cyclictest0-21swapper/208:10:362
349105993725,8cyclictest30-21ksoftirqd/207:45:012
349105993725,8cyclictest30-21ksoftirqd/207:45:002
349105993722,11cyclictest30-21ksoftirqd/207:20:002
349105993722,10cyclictest30-21ksoftirqd/210:04:222
34910599371,9cyclictest30-21ksoftirqd/208:03:222
349101993724,10cyclictest358613-21sshd10:10:221
34909399375,6cyclictest12-21ksoftirqd/008:18:220
34910899362,8cyclictest37-21ksoftirqd/308:55:333
34910899362,8cyclictest37-21ksoftirqd/308:55:323
34910899362,7cyclictest37-21ksoftirqd/308:08:233
349108993623,9cyclictest349255-21sshd07:10:133
349108993623,8cyclictest306-21in:imuxsock07:50:003
34910899361,6cyclictest37-21ksoftirqd/307:15:223
349108993616,14cyclictest272-21dbus-daemon10:10:013
349108993614,15cyclictest37-21ksoftirqd/308:43:543
34910599366,6cyclictest131rcu_preempt11:00:222
34910599362,9cyclictest30-21ksoftirqd/207:50:312
349105993629,4cyclictest360349-21latency_hist10:45:032
349105993627,7cyclictest30-21ksoftirqd/212:19:592
349105993622,9cyclictest30-21ksoftirqd/212:00:002
34910599361,7cyclictest30-21ksoftirqd/208:53:222
349105993612,19cyclictest131rcu_preempt08:06:222
349105993611,5cyclictest131rcu_preempt09:10:482
349105993610,5cyclictest131rcu_preempt07:28:232
349105993610,5cyclictest131rcu_preempt07:28:222
349105993610,4cyclictest131rcu_preempt12:30:472
34910199367,7cyclictest131rcu_preempt10:50:021
34910199367,4cyclictest131rcu_preempt11:21:081
34910199367,4cyclictest131rcu_preempt11:21:071
34910199364,6cyclictest23-21ksoftirqd/112:09:481
34910199363,4cyclictest131rcu_preempt12:21:221
34910899355,5cyclictest131rcu_preempt08:53:223
34910899354,25cyclictest363077-21cut11:40:013
34910899354,25cyclictest363077-21cut11:40:013
349108993523,10cyclictest37-21ksoftirqd/308:15:033
349105993522,10cyclictest30-21ksoftirqd/209:17:442
349105993514,15cyclictest272-21dbus-daemon08:50:022
349105993510,6cyclictest131rcu_preempt12:12:232
34910199355,6cyclictest150-21pr/ttyS210:56:471
34910199352,6cyclictest23-21ksoftirqd/111:31:221
349101993526,6cyclictest353264-21latency_hist08:30:021
349101993519,12cyclictest48-21kcompactd009:10:321
349093993524,8cyclictest360196-21sshd10:40:220
34910599347,7cyclictest131rcu_preempt12:25:012
34910599347,5cyclictest131rcu_preempt07:23:222
349105993422,9cyclictest30-21ksoftirqd/209:55:002
34910199345,22cyclictest349216-21grep07:10:021
349101993422,9cyclictest351367-21sh07:55:001
34909399349,14cyclictest349087-21cyclictest07:10:010
349108993311,18cyclictest363-21snmpd09:53:263
34910599339,5cyclictest131rcu_preempt08:34:222
34910599339,4cyclictest131rcu_preempt10:48:222
34910599336,6cyclictest131rcu_preempt12:05:022
349105993324,6cyclictest349119-21latency_hist07:10:022
34910599331,6cyclictest361480-21sendmail-mta11:05:082
34910599326,5cyclictest131rcu_preempt07:34:222
34910599321,8cyclictest30-21ksoftirqd/209:25:312
34910599321,8cyclictest30-21ksoftirqd/209:25:302
349108992821,4cyclictest349031-21latency_hist07:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional