You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 05:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Nov 18, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32670299910268,8cyclictest23-21ksoftirqd/100:15:021
326702599961,86cyclictest0-21swapper/021:10:020
3267033997833,12cyclictest235-21systemd-journal21:30:012
3267025997610,47cyclictest0-21swapper/023:10:010
3267025997225,43cyclictest3271284-21cat20:30:010
3267025997218,25cyclictest3269065-21latency_hist19:45:020
3267025997218,25cyclictest3269065-21latency_hist19:45:020
3267029997032,28cyclictest23-21ksoftirqd/121:10:021
3267025997020,14cyclictest3275875-21latency_hist22:00:020
3267025996816,19cyclictest3280806-21latency_hist23:35:010
3267025996719,44cyclictest3273599-21latency_hist21:15:010
3267037996528,18cyclictest37-21ksoftirqd/319:50:023
3267025996318,41cyclictest0-21swapper/019:50:020
3267025996222,35cyclictest363-21snmpd21:20:360
3267025996211,15cyclictest3272006-21systemd-run20:45:010
3267025996151,7cyclictest1250ksoftirqd/020:25:020
3267025996121,35cyclictest0-21swapper/023:15:020
3267025996121,35cyclictest0-21swapper/023:15:010
3267025996116,40cyclictest3276511-21cat22:10:010
326703399586,32cyclictest235-21systemd-journal23:50:022
3267025995854,2cyclictest0-21swapper/020:15:010
3267025995850,4cyclictest3281340-21munin-run23:45:000
3267025995822,11cyclictest0-21swapper/022:55:570
3267025995821,9cyclictest1-21systemd23:47:340
3267025995740,14cyclictest3271426-21sshd20:30:340
3267025995732,21cyclictest3280295-21cron23:25:000
3267025995729,9cyclictest363-21snmpd20:55:530
3267025995721,6cyclictest3277691-21latency_hist22:35:020
3267025995651,3cyclictest0-21swapper/000:30:020
3267025995648,4cyclictest3282652-21munin-run00:10:010
3267025995542,10cyclictest3274308-21sshd21:28:340
3267025995541,10cyclictest3277910-21sshd22:35:340
3267025995540,12cyclictest3280007-21sshd23:18:590
3267025995535,16cyclictest3270138-21sshd20:07:340
3267025995535,16cyclictest3270138-21sshd20:07:340
3267025995440,11cyclictest3272464-21sshd20:51:340
3267025995439,11cyclictest3271757-21munin-run20:40:010
3267025995437,14cyclictest3273774-21sshd21:18:330
3267025995437,14cyclictest3268564-21sshd19:36:340
3267025995418,31cyclictest3273165-21cat21:05:020
3267025995417,5cyclictest3274878-21systemd-run21:40:010
3267025995416,33cyclictest3267181-21cat19:10:020
3267025995414,35cyclictest3267994-21cat19:25:020
326703799536,7cyclictest37-21ksoftirqd/321:15:003
326703799535,27cyclictest3282140-40mandb00:00:033
3267025995348,3cyclictest0-21swapper/021:45:020
3267025995340,10cyclictest3274549-21sshd21:30:340
3267025995339,10cyclictest3277203-21munin-run22:25:010
3267025995338,10cyclictest3275601-21sshd21:54:340
3267025995337,13cyclictest3276124-21sshd22:03:350
3267025995337,13cyclictest3276124-21sshd22:03:340
326702599533,41cyclictest3269626-21modprobe20:00:000
3267025995318,6cyclictest12-21ksoftirqd/019:10:020
3267029995233,15cyclictest235-21systemd-journal23:25:011
3267025995240,9cyclictest3267488-21sshd19:17:340
3267025995240,8cyclictest3275338-21sshd21:47:330
3267025995240,8cyclictest3272220-21sshd20:48:270
3267025995238,10cyclictest3279237-21sshd23:03:350
3267025995233,15cyclictest363-21snmpd00:18:420
3267025995138,9cyclictest3280530-21sshd23:27:340
3267025995138,10cyclictest3270654-21sshd20:18:190
3267025995137,10cyclictest3282359-21sshd00:02:340
3267025995136,12cyclictest3268316-21sshd19:32:340
3267025995117,12cyclictest3277574-21latency_hist22:30:020
3267025995038,8cyclictest3268059-21sshd19:25:350
3267025995038,8cyclictest3268059-21sshd19:25:340
3267025995028,17cyclictest3283403-21sshd00:22:340
3267025995028,17cyclictest3283403-21sshd00:22:340
326703799496,7cyclictest37-21ksoftirqd/323:29:593
326703799495,8cyclictest37-21ksoftirqd/323:00:023
326703799494,10cyclictest37-21ksoftirqd/322:30:023
3267037994937,7cyclictest37-21ksoftirqd/320:55:003
326703799492,11cyclictest37-21ksoftirqd/322:00:003
3267037994912,6cyclictest131rcu_preempt20:41:343
3267033994925,18cyclictest1-21systemd19:24:192
3267029994935,10cyclictest1-21systemd19:40:011
3267029994932,13cyclictest1-21systemd19:55:021
3267029994932,13cyclictest1-21systemd19:55:011
3267025994941,4cyclictest3283667-21sshd00:29:340
3267025994935,10cyclictest3282064-21sshd23:56:350
3267025994935,10cyclictest3281858-21munin-run23:55:020
3267025994935,10cyclictest3281858-21munin-run23:55:010
3267025994915,30cyclictest3283058-21latency_hist00:15:030
3267037994839,6cyclictest37-21ksoftirqd/322:45:023
3267037994838,7cyclictest37-21ksoftirqd/323:10:023
3267037994813,5cyclictest131rcu_preempt21:35:013
3267033994828,13cyclictest272-21dbus-daemon20:20:002
3267029994833,11cyclictest3280014-21cron23:20:001
3267025994840,4cyclictest3278479-21munin-run22:50:010
326703799476,11cyclictest37-21ksoftirqd/321:00:003
3267037994736,9cyclictest37-21ksoftirqd/322:20:003
3267037994713,6cyclictest131rcu_preempt23:25:023
3267037994712,6cyclictest131rcu_preempt20:30:013
3267033994733,10cyclictest235-21systemd-journal00:38:332
3267033994733,10cyclictest235-21systemd-journal00:38:332
3267029994731,12cyclictest23-21ksoftirqd/100:35:021
3267029994725,15cyclictest235-21systemd-journal23:15:011
3267029994725,15cyclictest235-21systemd-journal23:15:001
326702999471,5cyclictest0-21swapper/122:55:001
3267025994734,10cyclictest3278705-21sshd22:53:340
3267025994729,12cyclictest3281058-21modprobe23:40:010
3267025994727,16cyclictest3284188-21cron00:40:010
3267025994727,16cyclictest3284188-21cron00:40:000
326703799465,6cyclictest131rcu_preempt20:10:013
326703799465,6cyclictest131rcu_preempt20:10:013
3267037994636,6cyclictest37-21ksoftirqd/323:15:023
3267037994636,6cyclictest37-21ksoftirqd/323:15:023
3267033994626,15cyclictest30-21ksoftirqd/220:00:002
3267029994630,12cyclictest235-21systemd-journal21:40:001
3267025994635,8cyclictest363-21snmpd20:03:180
3267025994633,8cyclictest363-21snmpd19:51:360
3267025994633,8cyclictest363-21snmpd19:51:360
3267025994625,18cyclictest3276935-21tr22:20:000
326703799459,6cyclictest131rcu_preempt19:35:003
3267037994534,6cyclictest306-21in:imuxsock00:30:013
3267037994532,11cyclictest37-21ksoftirqd/319:45:013
3267037994532,11cyclictest37-21ksoftirqd/319:45:013
3267033994528,13cyclictest272-21dbus-daemon23:45:012
3267029994531,9cyclictest235-21systemd-journal00:05:001
3267025994534,8cyclictest48-21kcompactd022:10:320
3267025994517,26cyclictest0-21swapper/022:40:540
3267037994436,6cyclictest37-21ksoftirqd/319:37:343
3267037994433,9cyclictest37-21ksoftirqd/319:15:023
3267037994433,7cyclictest361rcuc/320:36:343
3267037994429,11cyclictest3281308-21sshd23:41:333
3267037994426,13cyclictest37-21ksoftirqd/321:25:013
326703799442,12cyclictest37-21ksoftirqd/320:04:343
3267033994434,8cyclictest3271182-21sshd20:27:342
3267033994432,7cyclictest3273786-21cron21:20:002
3267033994428,10cyclictest272-21dbus-daemon22:20:012
3267029994427,13cyclictest1-21systemd20:00:001
3267029994419,21cyclictest221rcuc/121:15:011
326703799438,6cyclictest131rcu_preempt00:25:013
326703799438,6cyclictest131rcu_preempt00:25:013
326703799434,9cyclictest37-21ksoftirqd/300:40:003
326703799434,9cyclictest37-21ksoftirqd/300:40:003
3267037994329,9cyclictest235-21systemd-journal22:40:003
3267037994328,10cyclictest235-21systemd-journal22:50:013
3267037994312,6cyclictest131rcu_preempt21:10:023
326703399435,32cyclictest3275103-21cron21:45:002
3267033994331,8cyclictest3282386-21munin-plugin-st00:05:012
3267033994330,9cyclictest306-21in:imuxsock21:33:372
3267033994328,11cyclictest235-21systemd-journal22:25:032
3267033994326,12cyclictest272-21dbus-daemon20:50:022
3267029994337,3cyclictest231ksoftirqd/123:54:071
3267029994337,3cyclictest231ksoftirqd/123:54:061
3267029994330,10cyclictest23-21ksoftirqd/123:40:021
3267029994328,10cyclictest1-21systemd20:35:011
326703799429,10cyclictest131rcu_preempt22:05:033
326703799429,10cyclictest131rcu_preempt22:05:023
3267037994228,9cyclictest37-21ksoftirqd/320:50:003
326703799421,5cyclictest0-21swapper/321:18:333
3267037994213,6cyclictest131rcu_preempt19:55:003
3267037994213,6cyclictest131rcu_preempt19:55:003
3267037994212,6cyclictest131rcu_preempt22:30:023
326703799421,13cyclictest0-21swapper/322:53:273
3267033994232,6cyclictest3267517-21sendmail19:20:002
3267033994230,8cyclictest3280328-21cat23:25:012
3267033994225,12cyclictest408-21lldpd20:35:012
326703399422,27cyclictest3278694-21kworker/u8:223:00:022
3267029994228,10cyclictest23-21ksoftirqd/122:40:001
3267029994222,15cyclictest235-21systemd-journal22:46:341
3267029994222,14cyclictest272-21dbus-daemon00:40:021
3267029994222,14cyclictest272-21dbus-daemon00:40:011
326703799418,6cyclictest131rcu_preempt19:20:003
3267037994133,6cyclictest37-21ksoftirqd/300:15:033
3267037994130,8cyclictest37-21ksoftirqd/321:40:003
326703799412,8cyclictest37-21ksoftirqd/320:00:013
3267037994127,11cyclictest37-21ksoftirqd/321:50:003
3267037994124,13cyclictest3283921-21sshd00:33:513
3267037994122,14cyclictest272-21dbus-daemon21:45:013
3267033994128,10cyclictest30-21ksoftirqd/200:00:002
3267033994127,10cyclictest3272737-21cron21:00:002
3267033994124,13cyclictest272-21dbus-daemon20:40:022
326702999417,30cyclictest3267526-21munin-plugin-st19:20:001
3267029994125,12cyclictest3271733-21tr20:40:001
3267029994124,8cyclictest3282637-21munin-plugin-st00:10:011
3267029994124,12cyclictest235-21systemd-journal23:10:001
3267029994123,14cyclictest23-21ksoftirqd/120:05:001
3267029994121,14cyclictest235-21systemd-journal23:35:011
326703799405,5cyclictest131rcu_preempt23:35:013
3267037994031,6cyclictest37-21ksoftirqd/321:55:013
3267037994030,7cyclictest37-21ksoftirqd/300:10:023
3267037994030,7cyclictest361rcuc/323:39:353
3267037994029,6cyclictest3270957-21munin-plugin-st20:25:003
326703799401,8cyclictest0-21swapper/321:28:343
3267037994011,6cyclictest131rcu_preempt23:01:353
3267033994028,9cyclictest30-21ksoftirqd/223:35:022
3267033994027,9cyclictest235-21systemd-journal22:50:002
3267033994022,12cyclictest235-21systemd-journal22:45:012
3267029994025,12cyclictest23-21ksoftirqd/120:15:001
326703799397,6cyclictest131rcu_preempt22:10:343
3267037993929,8cyclictest37-21ksoftirqd/319:20:023
3267037993929,7cyclictest3273002-21sshd21:03:303
3267037993927,8cyclictest37-21ksoftirqd/322:25:013
3267037993911,5cyclictest131rcu_preempt23:50:003
3267033993925,10cyclictest235-21systemd-journal00:25:022
3267033993925,10cyclictest235-21systemd-journal00:25:012
326703399391,7cyclictest3269131-21latency_hist19:50:022
3267029993928,8cyclictest23-21ksoftirqd/121:45:021
3267029993923,12cyclictest1-21systemd22:10:001
326703799389,6cyclictest131rcu_preempt20:15:013
3267037993817,16cyclictest272-21dbus-daemon20:35:013
3267033993829,7cyclictest30-21ksoftirqd/221:05:022
3267033993828,7cyclictest3279826-21cat23:15:012
3267033993828,7cyclictest3279826-21cat23:15:012
3267033993828,7cyclictest3273509-21sshd21:11:342
3267033993826,9cyclictest30-21ksoftirqd/222:35:012
3267033993825,10cyclictest30-21ksoftirqd/221:25:012
3267033993823,11cyclictest235-21systemd-journal22:36:342
3267033993823,10cyclictest306-21in:imuxsock00:35:012
3267033993823,10cyclictest272-21dbus-daemon20:05:012
3267033993820,13cyclictest235-21systemd-journal20:25:012
3267029993831,4cyclictest3267760-21sshd19:20:341
3267029993829,7cyclictest23-21ksoftirqd/120:20:021
3267029993824,10cyclictest23-21ksoftirqd/119:15:021
3267029993821,13cyclictest3282119-21sendmail00:00:001
3267029993821,13cyclictest23-21ksoftirqd/122:00:021
3267029993821,12cyclictest235-21systemd-journal20:10:001
3267029993821,12cyclictest235-21systemd-journal20:10:001
326703799373,9cyclictest37-21ksoftirqd/323:54:063
326703799373,9cyclictest37-21ksoftirqd/323:54:063
3267037993730,5cyclictest37-21ksoftirqd/320:20:023
3267033993729,5cyclictest1-21systemd19:35:002
3267033993724,9cyclictest272-21dbus-daemon21:55:492
3267033993724,10cyclictest3267261-21systemd-run19:15:012
3267029993728,5cyclictest307-21in:imklog23:46:351
3267029993725,8cyclictest3276899-21cron22:20:011
3267029993724,9cyclictest272-21dbus-daemon22:25:001
3267029993721,9cyclictest3273795-21modprobe21:20:001
3267029993719,13cyclictest272-21dbus-daemon22:35:021
3267037993626,5cyclictest306-21in:imuxsock23:15:343
3267033993628,6cyclictest30-21ksoftirqd/223:29:532
3267033993628,6cyclictest30-21ksoftirqd/221:10:012
3267033993628,6cyclictest271-21cron23:55:002
3267033993628,6cyclictest271-21cron23:55:002
3267033993627,7cyclictest30-21ksoftirqd/219:10:022
3267033993627,5cyclictest306-21in:imuxsock22:05:002
3267033993627,5cyclictest306-21in:imuxsock22:05:002
3267033993626,7cyclictest30-21ksoftirqd/200:15:022
3267033993626,6cyclictest3268192-21latency_hist19:30:022
3267033993626,6cyclictest3268192-21latency_hist19:30:022
3267033993624,10cyclictest30-21ksoftirqd/223:05:012
326703399362,3cyclictest131rcu_preempt20:42:342
3267033993623,9cyclictest30-21ksoftirqd/221:50:032
326702999366,24cyclictest3279150-21grep23:00:031
3267029993626,7cyclictest3274552-21sshd21:31:351
3267029993625,8cyclictest3278240-21systemd-run22:45:011
3267029993625,6cyclictest306-21in:imuxsock19:35:001
3267029993623,9cyclictest3275694-21sed21:55:011
3267029993620,12cyclictest1-21systemd00:25:021
3267029993620,12cyclictest1-21systemd00:25:011
3267029993614,5cyclictest131rcu_preempt20:45:031
3267037993527,6cyclictest37-21ksoftirqd/300:01:183
3267037993525,7cyclictest3268071-21sshd19:28:063
3267037993525,7cyclictest3268071-21sshd19:28:063
326703399353,3cyclictest131rcu_preempt21:40:002
3267033993523,9cyclictest30-21ksoftirqd/223:18:332
3267033993523,9cyclictest30-21ksoftirqd/222:11:342
3267033993520,11cyclictest272-21dbus-daemon21:50:002
3267033993519,11cyclictest272-21dbus-daemon00:30:002
3267033993518,12cyclictest235-21systemd-journal19:40:012
326702999353,6cyclictest131rcu_preempt19:50:001
3267029993523,8cyclictest23-21ksoftirqd/122:05:011
3267029993523,8cyclictest23-21ksoftirqd/122:05:011
326702999351,4cyclictest131rcu_preempt21:50:001
326703799343,6cyclictest131rcu_preempt22:10:013
3267033993424,5cyclictest306-21in:imuxsock19:45:002
3267033993424,5cyclictest306-21in:imuxsock19:45:002
3267033993419,11cyclictest1-21systemd23:35:002
3267033993419,10cyclictest30-21ksoftirqd/222:10:012
3267033993415,15cyclictest272-21dbus-daemon00:10:012
3267029993425,6cyclictest23-21ksoftirqd/120:48:271
3267029993423,7cyclictest3268095-21/usr/sbin/munin19:30:021
3267029993423,7cyclictest3268095-21/usr/sbin/munin19:30:011
3267029993421,9cyclictest306-21in:imuxsock21:25:001
3267029993420,12cyclictest23-21ksoftirqd/121:30:011
3267037993321,10cyclictest37-21ksoftirqd/300:15:023
3267033993326,4cyclictest3283156-21latency_hist00:20:032
3267033993324,7cyclictest30-21ksoftirqd/220:55:032
3267033993324,7cyclictest30-21ksoftirqd/219:51:352
3267033993324,7cyclictest30-21ksoftirqd/219:51:342
3267029993323,8cyclictest3276619-21sshd22:11:341
3267029993323,7cyclictest23-21ksoftirqd/122:28:341
3267029993320,10cyclictest3283162-21systemctl00:20:001
3267029993318,10cyclictest0-21swapper/121:00:011
326702999331,6cyclictest131rcu_preempt23:30:001
326702999331,5cyclictest131rcu_preempt21:03:341
326703399324,3cyclictest131rcu_preempt22:55:012
326703399322,3cyclictest131rcu_preempt20:15:002
3267033993214,13cyclictest235-21systemd-journal22:25:002
3267029993225,5cyclictest3281336-21latency_hist23:45:011
3267029993221,8cyclictest23-21ksoftirqd/123:05:021
3267029993221,7cyclictest23-21ksoftirqd/119:10:021
3267029993218,11cyclictest235-21systemd-journal00:29:131
3267029993212,17cyclictest0-21swapper/120:25:021
3267033993120,8cyclictest30-21ksoftirqd/223:10:022
326702999313,4cyclictest131rcu_preempt20:55:021
3267029993124,4cyclictest3271207-21latency_hist20:30:021
3267029993123,5cyclictest3268874-21latency_hist19:45:021
3267033993024,4cyclictest3270198-21latency_hist20:10:012
3267033993024,4cyclictest3270198-21latency_hist20:10:012
3267029993022,5cyclictest3269053-21latency_hist19:45:021
3267037992818,7cyclictest37-21ksoftirqd/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional