You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 01:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Nov 20, 2025 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3578218998473,8cyclictest23-21ksoftirqd/111:30:021
3578213997872,3cyclictest0-21swapper/008:40:020
357821399657,10cyclictest3584433-21cron09:10:000
357821399653,13cyclictest3593551-21munin-run12:05:010
357821399652,8cyclictest3588067-21sshd10:19:290
3578213996522,39cyclictest3590410-21sshd11:02:290
357821399641,8cyclictest3594608-21munin-run12:25:020
357821399632,16cyclictest3591444-21sshd11:20:290
357821399632,14cyclictest3580053-21munin-run07:45:010
3578213996319,36cyclictest306-21in:imuxsock08:00:010
357821399622,15cyclictest3588579-21sshd10:25:290
357821399622,15cyclictest3588579-21sshd10:25:290
357821399621,10cyclictest3584681-21sshd09:12:290
357821399621,10cyclictest3584681-21sshd09:12:290
357821399616,42cyclictest16950irq/224-800000010:10:020
357821399612,12cyclictest3589653-21munin-run10:50:000
357821399611,9cyclictest3591701-21sshd11:26:290
3578213996117,37cyclictest272-21dbus-daemon08:20:000
357821399611,56cyclictest0-21swapper/007:19:290
357821399611,11cyclictest3585723-21sshd09:32:290
357822799609,6cyclictest131rcu_preempt12:10:003
3578223996014,6cyclictest131rcu_preempt11:40:012
3578223996013,6cyclictest131rcu_preempt09:55:012
357821399603,6cyclictest3590907-21sshd11:11:120
357821399603,12cyclictest3580543-21munin-run07:55:000
357821399602,19cyclictest3582764-21cat08:35:010
357821399602,13cyclictest3588868-21munin-run10:35:000
357821399601,55cyclictest0-21swapper/011:00:020
357821399601,55cyclictest0-21swapper/011:00:020
3578213996015,39cyclictest235-21systemd-journal11:45:010
357821399601,17cyclictest3589890-21cron10:55:000
357821399601,13cyclictest3594829-21sshd12:29:300
357821399592,9cyclictest3578954-21munin-run07:25:010
357821399592,6cyclictest16950irq/224-800000008:51:290
357821399592,16cyclictest3581506-21grep08:10:020
3578213995913,41cyclictest235-21systemd-journal12:00:000
3578223995813,7cyclictest131rcu_preempt10:20:022
357821399587,44cyclictest0-21swapper/010:25:010
3578213995839,14cyclictest1-21systemd09:00:010
357821399583,11cyclictest362-21munin-node10:10:020
357821399582,8cyclictest3585010-21munin-run09:20:010
357821399582,6cyclictest3579188-21modprobe07:29:300
3578213995822,30cyclictest3585469-21sshd09:26:290
3578213995812,39cyclictest0-21swapper/007:35:000
357821399581,14cyclictest363-21snmpd11:52:530
357821399581,13cyclictest363-21snmpd11:49:140
357821399581,12cyclictest3592210-21sshd11:37:290
357822399575,8cyclictest30-21ksoftirqd/212:00:002
357821399577,42cyclictest0-21swapper/012:40:010
357821399571,11cyclictest3578231-21sed07:10:020
3578227995614,11cyclictest37-21ksoftirqd/308:30:023
3578227995614,11cyclictest37-21ksoftirqd/308:30:023
357821399564,8cyclictest3589130-21munin-run10:40:000
357821399562,6cyclictest3583170-21munin-run08:45:010
357821399562,6cyclictest3583170-21munin-run08:45:010
357821399561,6cyclictest3595068-21sshd12:31:310
357821399561,13cyclictest363-21snmpd09:46:460
3578223995514,6cyclictest131rcu_preempt09:30:022
3578218995539,11cyclictest23-21ksoftirqd/108:55:001
357821399554,6cyclictest363-21snmpd07:38:010
357821399553,7cyclictest3585247-21munin-run09:25:010
357821399552,8cyclictest3584172-21sshd09:03:290
357821399552,7cyclictest3578381-21sshd07:10:280
357821399552,6cyclictest3594302-21cron12:20:000
357821399552,6cyclictest3581056-21sshd08:03:290
357821399552,15cyclictest3587019-21sshd09:58:290
357821399552,15cyclictest3581580-21munin-run08:14:590
357821399552,14cyclictest3582098-21sshd08:22:290
357821399551,10cyclictest3591950-21sshd11:31:140
357822799549,5cyclictest131rcu_preempt07:40:003
357822799547,8cyclictest37-21ksoftirqd/309:20:003
3578227995438,12cyclictest1-21systemd10:25:013
3578227995434,13cyclictest1-21systemd10:05:013
357822399548,5cyclictest131rcu_preempt08:35:002
3578223995414,8cyclictest131rcu_preempt07:40:002
3578223995412,16cyclictest131rcu_preempt09:10:012
3578227995312,7cyclictest131rcu_preempt08:40:003
3578223995315,6cyclictest131rcu_preempt09:20:002
3578213995330,20cyclictest3594242-21latency_hist12:15:020
3578213995330,20cyclictest3594242-21latency_hist12:15:020
357821399532,6cyclictest1-21systemd11:20:010
357821399532,15cyclictest363-21snmpd11:07:290
357821399531,9cyclictest3586248-21sshd09:40:290
3578223995212,6cyclictest131rcu_preempt08:40:002
357821399523,5cyclictest3580306-21munin-run07:50:010
357821399523,5cyclictest3580306-21munin-run07:50:010
3578213995233,14cyclictest235-21systemd-journal10:05:000
357821399522,14cyclictest3586052-21wc09:40:010
3578227995135,11cyclictest1-21systemd08:05:003
3578227995120,20cyclictest361rcuc/312:05:023
357822799511,22cyclictest0-21swapper/307:35:013
357822399515,6cyclictest131rcu_preempt11:10:012
3578223995114,6cyclictest131rcu_preempt11:55:002
3578223995113,7cyclictest30-21ksoftirqd/210:00:012
3578223995113,11cyclictest30-21ksoftirqd/207:30:002
3578218995132,14cyclictest3587052-21systemctl10:00:011
357821399512,6cyclictest363-21snmpd10:43:250
357821399512,6cyclictest363-21snmpd10:43:240
3578213995121,26cyclictest0-21swapper/008:29:260
3578213995121,26cyclictest0-21swapper/008:29:260
357821399511,10cyclictest363-21snmpd12:06:300
357822799507,6cyclictest131rcu_preempt09:55:003
357822399507,5cyclictest131rcu_preempt12:05:002
3578223995011,5cyclictest131rcu_preempt09:00:022
3578218995032,13cyclictest1-21systemd09:05:001
357822799495,5cyclictest131rcu_preempt08:20:023
357822399499,6cyclictest131rcu_preempt11:00:002
357822399499,6cyclictest131rcu_preempt11:00:002
357822399495,9cyclictest30-21ksoftirqd/212:40:012
357822399491,7cyclictest30-21ksoftirqd/209:50:012
3578223994914,15cyclictest131rcu_preempt10:10:002
357821399497,4cyclictest0-21swapper/009:52:010
357821399492,10cyclictest3583373-21sendmail-mta08:45:170
357822799487,5cyclictest131rcu_preempt11:00:003
357822799487,5cyclictest131rcu_preempt11:00:003
3578227994831,12cyclictest272-21dbus-daemon08:10:003
3578227994810,6cyclictest131rcu_preempt10:55:003
357822399489,6cyclictest131rcu_preempt08:20:022
357822399488,9cyclictest30-21ksoftirqd/208:00:002
3578223994812,5cyclictest131rcu_preempt12:35:012
3578223994811,6cyclictest131rcu_preempt11:15:022
3578218994832,12cyclictest1-21systemd07:25:011
357822799479,9cyclictest37-21ksoftirqd/308:25:003
357822799477,6cyclictest131rcu_preempt08:00:023
3578227994714,6cyclictest131rcu_preempt11:40:023
357822399479,6cyclictest131rcu_preempt11:32:292
357822399476,6cyclictest131rcu_preempt08:05:022
3578223994738,7cyclictest30-21ksoftirqd/207:50:012
3578223994738,7cyclictest30-21ksoftirqd/207:50:012
357822399473,6cyclictest131rcu_preempt12:20:002
357822399471,42cyclictest291rcuc/212:10:022
357822399471,42cyclictest291rcuc/212:10:022
3578223994710,5cyclictest131rcu_preempt07:11:292
357822799468,5cyclictest131rcu_preempt10:50:013
357822799468,5cyclictest131rcu_preempt07:25:023
357822799466,6cyclictest131rcu_preempt12:30:003
357822799466,11cyclictest37-21ksoftirqd/310:15:293
357822799465,10cyclictest37-21ksoftirqd/311:22:293
357822799465,10cyclictest37-21ksoftirqd/309:30:013
357822799464,7cyclictest131rcu_preempt07:43:293
357822799464,12cyclictest37-21ksoftirqd/311:07:293
3578227994629,12cyclictest1-21systemd07:55:013
357822799462,6cyclictest131rcu_preempt11:01:293
3578227994610,5cyclictest131rcu_preempt09:50:013
357822399467,5cyclictest131rcu_preempt11:25:012
357822399462,7cyclictest131rcu_preempt09:35:022
357822399462,11cyclictest30-21ksoftirqd/208:30:022
357822399462,11cyclictest30-21ksoftirqd/208:30:022
3578223994611,6cyclictest131rcu_preempt07:43:082
3578223994610,5cyclictest131rcu_preempt07:10:022
3578218994629,12cyclictest235-21systemd-journal11:00:011
3578218994629,12cyclictest235-21systemd-journal11:00:011
357822799459,6cyclictest131rcu_preempt07:30:013
357822799458,6cyclictest131rcu_preempt08:45:003
357822799458,6cyclictest131rcu_preempt08:45:003
357822799457,6cyclictest131rcu_preempt12:15:023
357822799457,6cyclictest131rcu_preempt12:15:023
357822799456,11cyclictest37-21ksoftirqd/311:54:303
357822799455,10cyclictest3581543-21sshd08:10:293
357822799454,9cyclictest37-21ksoftirqd/311:26:133
3578227994532,10cyclictest37-21ksoftirqd/312:40:013
3578227994531,11cyclictest37-21ksoftirqd/311:45:003
357822799451,14cyclictest0-21swapper/307:10:203
357822399458,6cyclictest131rcu_preempt09:40:002
357822399455,6cyclictest131rcu_preempt08:15:002
357822399453,7cyclictest30-21ksoftirqd/210:45:032
357822399453,7cyclictest30-21ksoftirqd/210:45:022
357822399452,12cyclictest30-21ksoftirqd/208:05:002
357822399451,8cyclictest0-21swapper/210:46:292
357822399451,5cyclictest131rcu_preempt10:03:292
3578218994529,12cyclictest23-21ksoftirqd/111:05:011
3578218994528,11cyclictest235-21systemd-journal12:15:011
3578218994528,11cyclictest235-21systemd-journal12:15:001
357822799449,5cyclictest131rcu_preempt12:35:023
357822799448,5cyclictest131rcu_preempt08:52:293
357822799447,5cyclictest131rcu_preempt11:20:003
3578227994428,11cyclictest306-21in:imuxsock08:50:003
3578227994412,4cyclictest131rcu_preempt12:15:303
357822399447,5cyclictest131rcu_preempt10:31:292
357822399446,4cyclictest131rcu_preempt09:25:012
3578223994434,7cyclictest30-21ksoftirqd/208:55:022
3578223994411,5cyclictest131rcu_preempt10:30:022
3578223994411,5cyclictest131rcu_preempt10:30:022
3578223994410,4cyclictest131rcu_preempt10:25:012
3578218994426,11cyclictest235-21systemd-journal12:00:001
3578218994419,20cyclictest23-21ksoftirqd/110:15:011
357821899441,5cyclictest131rcu_preempt09:55:001
357822799439,7cyclictest131rcu_preempt10:15:023
357822799438,6cyclictest131rcu_preempt07:15:293
357822799436,7cyclictest37-21ksoftirqd/310:30:023
357822799436,7cyclictest37-21ksoftirqd/310:30:023
357822799435,10cyclictest37-21ksoftirqd/310:43:303
357822799435,10cyclictest37-21ksoftirqd/310:43:293
357822799435,10cyclictest37-21ksoftirqd/307:45:273
357822799435,10cyclictest37-21ksoftirqd/307:45:273
357822399438,5cyclictest131rcu_preempt12:30:002
357822399436,9cyclictest30-21ksoftirqd/210:36:042
357822399434,7cyclictest30-21ksoftirqd/211:40:302
357822399434,11cyclictest30-21ksoftirqd/207:55:002
357822399434,10cyclictest30-21ksoftirqd/208:45:012
357822399434,10cyclictest30-21ksoftirqd/208:45:012
357822399433,9cyclictest30-21ksoftirqd/211:46:302
357822399432,7cyclictest30-21ksoftirqd/209:15:012
357822399432,7cyclictest30-21ksoftirqd/209:15:012
3578223994311,5cyclictest131rcu_preempt07:17:002
3578218994330,8cyclictest306-21in:imuxsock08:30:011
3578218994330,8cyclictest306-21in:imuxsock08:30:011
357822799425,10cyclictest37-21ksoftirqd/311:31:293
3578227994231,7cyclictest37-21ksoftirqd/310:00:013
357822799423,10cyclictest37-21ksoftirqd/310:10:013
357822799423,10cyclictest37-21ksoftirqd/309:15:013
357822799423,10cyclictest37-21ksoftirqd/309:15:013
3578227994225,13cyclictest37-21ksoftirqd/312:25:013
3578227994211,6cyclictest131rcu_preempt11:10:293
357822399428,4cyclictest131rcu_preempt12:25:012
357822399425,5cyclictest131rcu_preempt08:48:292
3578223994235,5cyclictest30-21ksoftirqd/210:13:292
3578223994229,8cyclictest30-21ksoftirqd/211:20:012
357822399422,7cyclictest30-21ksoftirqd/209:00:032
3578218994219,6cyclictest131rcu_preempt12:05:001
3578218994217,13cyclictest3578209-21cyclictest07:55:021
357822799416,10cyclictest37-21ksoftirqd/310:35:003
357822799414,11cyclictest37-21ksoftirqd/310:40:003
3578227994128,11cyclictest37-21ksoftirqd/309:45:013
3578227994127,9cyclictest3593258-21cron12:00:003
3578227994126,11cyclictest37-21ksoftirqd/309:00:013
357822799412,5cyclictest131rcu_preempt09:21:293
3578227994110,5cyclictest131rcu_preempt08:33:293
357822399414,9cyclictest30-21ksoftirqd/210:53:292
357822399414,8cyclictest30-21ksoftirqd/211:26:132
357822399414,6cyclictest30-21ksoftirqd/208:22:292
3578223994131,7cyclictest30-21ksoftirqd/211:05:012
357822399412,6cyclictest30-21ksoftirqd/207:21:292
357821899414,5cyclictest131rcu_preempt08:40:011
3578218994117,15cyclictest3579171-21sshd07:25:291
357822799406,5cyclictest131rcu_preempt09:05:023
3578227994031,6cyclictest37-21ksoftirqd/309:10:003
357821899401,5cyclictest131rcu_preempt12:40:021
3578218994015,20cyclictest23-21ksoftirqd/108:45:011
3578218994015,20cyclictest23-21ksoftirqd/108:45:011
3578218994012,14cyclictest131rcu_preempt10:06:291
357822799396,7cyclictest37-21ksoftirqd/309:31:293
357822799396,6cyclictest131rcu_preempt11:50:003
357822799392,6cyclictest131rcu_preempt09:40:003
3578218993931,4cyclictest23-21ksoftirqd/111:20:001
3578218993927,9cyclictest272-21dbus-daemon09:00:011
3578218993927,9cyclictest23-21ksoftirqd/111:43:591
3578218993924,11cyclictest3592222-21cron11:40:001
357821899391,4cyclictest131rcu_preempt12:10:001
357822399382,5cyclictest131rcu_preempt07:35:022
3578223993813,6cyclictest131rcu_preempt12:05:312
3578218993829,6cyclictest23-21ksoftirqd/111:50:021
3578218993827,7cyclictest3584677-21sshd09:10:541
3578218993827,7cyclictest3584677-21sshd09:10:541
3578218993826,9cyclictest3592998-21sshd11:53:301
3578218993826,9cyclictest23-21ksoftirqd/110:05:001
3578218993825,9cyclictest235-21systemd-journal09:30:031
3578218993822,12cyclictest23-21ksoftirqd/110:35:011
3578218993818,14cyclictest235-21systemd-journal08:15:001
357821899381,6cyclictest131rcu_preempt12:35:001
3578218993726,6cyclictest306-21in:imuxsock10:40:011
3578218993725,10cyclictest23-21ksoftirqd/111:15:031
3578218993723,10cyclictest23-21ksoftirqd/110:50:001
3578218993722,9cyclictest306-21in:imuxsock12:25:011
3578218993722,10cyclictest272-21dbus-daemon09:25:011
3578218993715,18cyclictest23-21ksoftirqd/108:50:001
3578218993714,8cyclictest131rcu_preempt10:45:021
3578218993714,8cyclictest131rcu_preempt10:45:021
357822399361,8cyclictest0-21swapper/209:41:292
3578218993627,4cyclictest23-21ksoftirqd/109:30:001
357821899362,5cyclictest131rcu_preempt10:20:001
3578218993625,7cyclictest23-21ksoftirqd/112:30:031
357821899361,6cyclictest0-21swapper/107:33:291
357821899361,5cyclictest131rcu_preempt12:20:001
357821899354,3cyclictest131rcu_preempt11:09:301
3578218993528,4cyclictest3578157-21latency_hist07:10:011
3578218993526,6cyclictest363-21snmpd07:18:031
3578218993523,9cyclictest23-21ksoftirqd/108:35:001
3578218993520,11cyclictest235-21systemd-journal08:20:401
357821899351,5cyclictest131rcu_preempt07:40:001
3578218993423,7cyclictest23-21ksoftirqd/109:40:021
3578218993421,10cyclictest1-21systemd11:35:011
357821899341,5cyclictest131rcu_preempt08:00:021
3578218993412,5cyclictest131rcu_preempt07:43:281
3578218993323,6cyclictest306-21in:imuxsock09:45:011
3578218993321,9cyclictest3589868-21sshd10:50:291
3578218993321,9cyclictest23-21ksoftirqd/107:50:011
3578218993321,9cyclictest23-21ksoftirqd/107:50:011
3578218993320,10cyclictest306-21in:imuxsock09:10:011
357821899331,5cyclictest131rcu_preempt11:25:001
357821899328,16cyclictest23-21ksoftirqd/108:18:161
357821899324,4cyclictest131rcu_preempt10:24:111
357821899323,4cyclictest131rcu_preempt09:45:171
3578218993225,4cyclictest3588612-21latency_hist10:30:021
3578218993225,4cyclictest3588612-21latency_hist10:30:021
3578218993220,9cyclictest3585072-21sed09:20:011
3578218993220,7cyclictest3578548-21wc07:15:021
3578218993121,8cyclictest23-21ksoftirqd/108:10:011
3578218993017,9cyclictest272-21dbus-daemon08:05:001
3578227992921,5cyclictest3578229-21latency_hist07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional