You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri May 02, 2025 12:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3663873996915,19cyclictest3678863-21sed12:00:033
3663873996915,19cyclictest3678863-21sed12:00:023
3663859996932,7cyclictest3671507-21sendmail09:40:000
3663859996827,10cyclictest272-21dbus-daemon10:50:010
3663873996617,14cyclictest3666632-21munin-run08:05:003
3663873996615,15cyclictest3671548-21munin-run09:40:003
366385999651,40cyclictest16950irq/224-800000009:05:030
3663873996314,14cyclictest3669772-21latency_hist09:05:023
3663873996314,14cyclictest3669772-21latency_hist09:05:013
3663859996243,16cyclictest0-21swapper/008:35:020
3663859996239,16cyclictest235-21systemd-journal09:00:010
3663859996140,16cyclictest1-21systemd09:15:000
366387399604,33cyclictest235-21systemd-journal10:45:023
366387399598,37cyclictest37-21ksoftirqd/309:50:033
3663873995915,15cyclictest3672506-21sshd09:58:333
3663873995911,15cyclictest1-21systemd10:32:323
3663859995941,13cyclictest1-21systemd11:45:010
3663859995941,13cyclictest1-21systemd11:45:000
3663873995838,16cyclictest3673520-21sshd10:16:333
3663859995520,9cyclictest12-21ksoftirqd/010:16:340
3663859995520,31cyclictest12-21ksoftirqd/011:50:020
3663873995441,10cyclictest3668700-21munin-run08:45:013
3663873995436,14cyclictest3672779-21sshd10:02:333
3663869995438,12cyclictest1-21systemd11:15:012
3663869995438,12cyclictest1-21systemd11:15:002
3663859995425,26cyclictest12-21ksoftirqd/008:30:030
3663873995338,11cyclictest3665827-21sshd07:46:333
3663873995336,14cyclictest3672258-21sshd09:53:333
3663873995334,14cyclictest3675079-21sshd10:47:323
3663873995240,9cyclictest363-21snmpd11:13:413
3663873995240,9cyclictest363-21snmpd11:13:403
3663873995238,10cyclictest3673315-21munin-run10:15:023
3663873995237,11cyclictest3679177-21sshd12:06:333
3663873995237,11cyclictest3679177-21sshd12:06:323
3663873995233,14cyclictest363-21snmpd09:28:343
3663864995238,11cyclictest3675570-21sshd10:55:321
3663873995138,8cyclictest3676128-21munin-run11:10:013
3663873995137,10cyclictest3677409-21munin-run11:35:013
3663873995137,10cyclictest3676602-21cron11:17:003
3663873995136,11cyclictest3680219-21sshd12:28:333
3663873995134,12cyclictest37-21ksoftirqd/308:25:003
366385999512,17cyclictest3671260-21cut09:35:010
3663873995037,10cyclictest3675336-21sshd10:54:333
3663873995036,11cyclictest3673029-21sshd10:07:333
3663873995035,12cyclictest3666326-21sshd07:56:333
3663869995035,12cyclictest235-21systemd-journal08:09:332
3663873994937,9cyclictest3670737-21sshd09:23:333
3663873994937,9cyclictest3665303-21sshd07:36:333
3663873994935,11cyclictest3671752-21sshd09:41:333
3663873994935,10cyclictest3680489-21munin-run12:35:013
3663873994935,10cyclictest3669151-21sshd08:50:323
3663873994935,10cyclictest3667367-21sshd08:19:323
3663873994934,12cyclictest3668377-21sshd08:35:343
3663873994934,10cyclictest48-21kcompactd011:44:523
3663873994934,10cyclictest48-21kcompactd011:44:513
3663873994932,13cyclictest3680713-21sshd12:37:333
3663873994924,19cyclictest3665057-21sshd07:32:333
3663873994837,8cyclictest3678410-21sshd11:53:333
3663873994836,9cyclictest3674062-21sshd10:28:343
3663873994835,9cyclictest3665623-21munin-run07:45:023
3663873994835,10cyclictest3674554-21sshd10:37:343
3663873994835,10cyclictest3670459-21sshd09:18:333
3663873994835,10cyclictest3667887-21sshd08:27:333
366387399483,26cyclictest3663763-21kworker/u8:007:25:013
3663873994831,13cyclictest3679680-21cron12:17:013
3663873994831,13cyclictest3679680-21cron12:17:013
3663873994822,19cyclictest3664343-21true07:20:013
3663864994831,12cyclictest235-21systemd-journal12:00:011
3663864994831,12cyclictest235-21systemd-journal12:00:011
3663864994830,13cyclictest272-21dbus-daemon10:50:011
3663873994735,9cyclictest3667106-21sshd08:13:343
3663873994734,9cyclictest48-21kcompactd010:58:153
3663873994734,10cyclictest3664023-21sshd07:11:333
3663873994733,9cyclictest48-21kcompactd011:20:263
3663873994733,9cyclictest48-21kcompactd008:31:463
3663873994719,24cyclictest3666979-21latency_hist08:10:013
3663869994732,12cyclictest3666395-21systemd-run08:00:002
3663864994736,7cyclictest23-21ksoftirqd/111:25:011
3663859994736,9cyclictest12-21ksoftirqd/011:55:010
3663873994636,7cyclictest3666073-21sshd07:50:333
3663873994634,8cyclictest363-21snmpd12:22:153
3663873994633,9cyclictest48-21kcompactd012:00:243
3663873994633,9cyclictest48-21kcompactd011:39:513
3663873994633,9cyclictest48-21kcompactd008:46:463
3663873994633,9cyclictest48-21kcompactd008:46:453
3663873994632,10cyclictest363-21snmpd07:28:123
366386999465,6cyclictest131rcu_preempt10:39:592
366386999465,5cyclictest131rcu_preempt07:45:002
3663864994630,13cyclictest1-21systemd09:45:011
3663864994630,12cyclictest1-21systemd07:15:021
3663864994629,13cyclictest23-21ksoftirqd/111:50:011
3663873994535,7cyclictest3670199-21sshd09:12:323
3663873994532,9cyclictest363-21snmpd10:23:363
3663873994530,11cyclictest37-21ksoftirqd/309:35:013
366386999454,4cyclictest131rcu_preempt09:00:002
3663869994531,11cyclictest30-21ksoftirqd/208:30:032
3663869994531,11cyclictest1-21systemd09:30:012
366386499454,3cyclictest131rcu_preempt11:05:041
3663864994531,10cyclictest235-21systemd-journal09:40:001
3663864994529,11cyclictest23-21ksoftirqd/107:30:021
3663859994530,12cyclictest235-21systemd-journal10:07:330
3663859994530,10cyclictest1-21systemd12:35:000
3663859994525,15cyclictest235-21systemd-journal07:50:000
3663873994432,8cyclictest48-21kcompactd011:26:333
3663869994433,8cyclictest235-21systemd-journal11:05:032
3663869994427,14cyclictest3677445-21sed11:35:022
3663869994425,14cyclictest1-21systemd08:55:012
366386499444,3cyclictest131rcu_preempt12:25:021
366386499441,5cyclictest131rcu_preempt08:45:001
3663873994336,4cyclictest3669929-21sshd09:05:343
3663873994330,9cyclictest48-21kcompactd011:45:253
3663873994321,17cyclictest3679443-21cron12:14:593
3663869994334,5cyclictest3668437-21sendmail-msp08:40:002
3663869994331,8cyclictest3672792-21cron10:05:012
3663869994329,9cyclictest3677663-21cat11:40:012
3663869994328,11cyclictest272-21dbus-daemon12:20:012
3663869994328,11cyclictest272-21dbus-daemon12:20:002
366386999431,4cyclictest131rcu_preempt10:27:332
3663864994336,4cyclictest3663866-21latency_hist07:10:011
3663864994331,8cyclictest3677395-21cron11:34:591
3663864994331,8cyclictest3671245-21cron09:35:011
3663859994328,11cyclictest1-21systemd09:50:020
3663859994324,12cyclictest235-21systemd-journal07:30:000
3663873994228,9cyclictest3669421-21cron09:00:003
366386999424,3cyclictest131rcu_preempt12:25:022
3663869994226,12cyclictest272-21dbus-daemon11:00:012
366386499426,3cyclictest131rcu_preempt09:00:011
3663864994227,11cyclictest272-21dbus-daemon07:45:011
3663859994235,4cyclictest16950irq/224-800000008:10:000
3663859994228,10cyclictest1-21systemd12:10:010
3663859994228,10cyclictest1-21systemd12:10:000
3663859994227,11cyclictest12-21ksoftirqd/010:40:010
366385999421,17cyclictest3670466-21kworker/u8:309:20:020
3663873994125,11cyclictest3675861-21sendmail-mta11:04:223
366386999414,3cyclictest131rcu_preempt11:55:022
3663869994132,5cyclictest3669948-21cron09:09:592
3663869994130,7cyclictest3665070-21cron07:34:592
3663869994126,12cyclictest3678330-21sed11:50:012
366386999412,4cyclictest131rcu_preempt07:19:592
3663864994128,10cyclictest3674943-21cat10:45:011
3663864994122,14cyclictest272-21dbus-daemon09:30:001
366386499411,7cyclictest3667900-21modprobe08:30:001
3663859994134,4cyclictest16950irq/224-800000007:45:000
3663859994121,14cyclictest272-21dbus-daemon12:40:000
366385999411,3cyclictest0-21swapper/010:05:030
366386999408,5cyclictest131rcu_preempt09:56:142
366386999408,28cyclictest3670748-21cron09:24:592
3663869994030,6cyclictest1-21systemd11:10:002
3663869994029,8cyclictest3666183-21sed07:55:012
3663869994026,11cyclictest30-21ksoftirqd/211:43:122
3663869994026,11cyclictest30-21ksoftirqd/211:43:112
366386999402,5cyclictest131rcu_preempt12:15:022
366386999402,5cyclictest0-21swapper/209:01:332
366386999402,5cyclictest0-21swapper/209:01:322
366386999402,4cyclictest131rcu_preempt11:20:002
3663869994022,13cyclictest272-21dbus-daemon09:45:002
366386999401,3cyclictest131rcu_preempt09:40:002
3663864994029,8cyclictest3672174-21ls09:50:021
3663864994028,9cyclictest23-21ksoftirqd/109:09:591
3663864994024,10cyclictest3677661-21sh11:40:011
366386499402,3cyclictest131rcu_preempt07:40:001
3663859994028,7cyclictest306-21in:imuxsock07:40:000
3663873993927,10cyclictest37-21ksoftirqd/307:10:023
366386999394,5cyclictest131rcu_preempt11:25:012
3663869993929,7cyclictest3673765-21cat10:20:032
3663869993928,8cyclictest30-21ksoftirqd/212:10:022
3663869993928,8cyclictest30-21ksoftirqd/212:10:022
3663869993928,8cyclictest235-21systemd-journal07:40:012
3663869993927,8cyclictest1-21systemd08:25:002
366386999392,4cyclictest131rcu_preempt12:00:012
366386999392,4cyclictest131rcu_preempt12:00:002
3663869993924,12cyclictest272-21dbus-daemon08:35:012
366386999391,5cyclictest131rcu_preempt12:25:332
366386499395,4cyclictest131rcu_preempt10:05:021
3663864993931,5cyclictest3665069-21cron07:34:591
366386499391,3cyclictest131rcu_preempt08:15:001
3663859993933,4cyclictest16950irq/224-800000010:40:340
366385999392,5cyclictest3668935-21/usr/sbin/munin08:50:020
366385999392,5cyclictest3668935-21/usr/sbin/munin08:50:010
3663859993918,16cyclictest12-21ksoftirqd/009:05:020
3663859993918,16cyclictest12-21ksoftirqd/009:05:010
366385999391,29cyclictest3656752-21kworker/u8:107:10:020
366385999391,11cyclictest3673355-21latency_hist10:15:020
3663869993828,6cyclictest3667373-21cron08:20:012
3663869993817,10cyclictest272-21dbus-daemon10:10:012
366386499385,3cyclictest131rcu_preempt10:40:001
366386499384,4cyclictest131rcu_preempt08:20:021
3663864993827,9cyclictest23-21ksoftirqd/111:30:021
3663864993825,9cyclictest3679225-21wc12:10:021
3663864993825,9cyclictest3679225-21wc12:10:011
3663864993823,12cyclictest3668398-21cron08:39:591
366386499381,3cyclictest131rcu_preempt09:15:001
366385999389,24cyclictest0-21swapper/010:28:310
366385999385,30cyclictest3663852-21cyclictest08:55:000
3663859993832,3cyclictest16950irq/224-800000011:25:000
3663859993830,4cyclictest16950irq/224-800000010:00:000
3663859993824,9cyclictest0-21swapper/011:05:020
3663859993824,10cyclictest235-21systemd-journal12:20:010
3663859993824,10cyclictest235-21systemd-journal12:20:010
366385999381,34cyclictest0-21swapper/011:59:200
366385999381,34cyclictest0-21swapper/011:59:190
366385999381,10cyclictest3677143-21sshd11:29:330
366386999375,3cyclictest131rcu_preempt10:45:032
366386999374,3cyclictest131rcu_preempt08:41:332
366386999373,4cyclictest131rcu_preempt10:12:322
3663869993727,5cyclictest306-21in:imuxsock08:15:002
366386999372,4cyclictest131rcu_preempt10:30:342
366386999372,3cyclictest131rcu_preempt10:45:332
366386999372,3cyclictest131rcu_preempt09:50:012
3663869993723,11cyclictest272-21dbus-daemon11:30:012
3663869993722,11cyclictest235-21systemd-journal09:15:012
366386499374,4cyclictest131rcu_preempt09:50:321
366386499374,4cyclictest131rcu_preempt08:54:591
366386499373,3cyclictest131rcu_preempt08:25:011
366386499373,3cyclictest131rcu_preempt08:00:011
3663864993727,7cyclictest3679437-21sshd12:14:331
3663864993725,9cyclictest3666936-21wc08:10:021
366386499372,4cyclictest131rcu_preempt08:35:011
366386499372,3cyclictest131rcu_preempt07:55:011
3663864993722,12cyclictest272-21dbus-daemon10:55:011
366386499371,4cyclictest131rcu_preempt12:04:321
366386499371,3cyclictest131rcu_preempt09:22:331
366385999379,4cyclictest0-21swapper/011:20:010
3663859993731,4cyclictest0-21swapper/010:24:240
3663859993728,6cyclictest12-21ksoftirqd/008:20:020
3663859993728,6cyclictest12-21ksoftirqd/007:25:010
3663859993725,9cyclictest12-21ksoftirqd/011:00:020
3663859993724,9cyclictest12-21ksoftirqd/009:55:000
3663859993724,10cyclictest408-21lldpd11:06:200
3663859993723,11cyclictest12-21ksoftirqd/008:45:020
366385999371,32cyclictest3679970-21sshd12:23:330
3663859993710,3cyclictest131rcu_preempt08:25:020
366386999363,3cyclictest131rcu_preempt09:35:012
3663869993629,4cyclictest3664051-21munin-plugin-st07:15:002
3663869993623,9cyclictest30-21ksoftirqd/212:35:022
366386999361,4cyclictest131rcu_preempt10:24:592
3663864993627,6cyclictest3664318-21/usr/sbin/munin07:20:001
3663864993625,8cyclictest23-21ksoftirqd/112:20:011
3663864993625,8cyclictest23-21ksoftirqd/112:20:001
3663864993625,8cyclictest1-21systemd10:10:011
366386499362,3cyclictest131rcu_preempt10:15:011
366386499362,3cyclictest131rcu_preempt10:00:011
3663864993622,11cyclictest23-21ksoftirqd/108:05:011
366386499361,3cyclictest131rcu_preempt11:19:591
366385999363,29cyclictest0-21swapper/008:39:100
3663859993629,4cyclictest16950irq/224-800000008:10:340
366385999362,30cyclictest3679114-21latency_hist12:05:020
366385999362,29cyclictest235-21systemd-journal09:44:220
3663859993622,10cyclictest12-21ksoftirqd/011:40:000
366385999361,5cyclictest3679468-21systemd-run12:15:000
366385999361,14cyclictest3674333-21munin-run10:35:000
366386999354,3cyclictest131rcu_preempt07:30:022
366386999353,5cyclictest131rcu_preempt12:40:002
366386999353,3cyclictest131rcu_preempt09:53:342
366386999353,3cyclictest131rcu_preempt07:25:012
3663869993525,7cyclictest3675357-21modprobe10:54:592
366386499354,3cyclictest131rcu_preempt10:35:011
366386499353,3cyclictest131rcu_preempt11:43:331
366386499353,3cyclictest131rcu_preempt11:43:321
3663864993527,6cyclictest23-21ksoftirqd/112:30:011
3663864993527,5cyclictest3674173-21latency_hist10:30:011
3663864993523,9cyclictest23-21ksoftirqd/112:40:021
3663864993517,14cyclictest272-21dbus-daemon10:20:011
366386499351,4cyclictest131rcu_preempt09:20:011
3663859993523,9cyclictest3666480-21wc08:00:020
3663859993523,9cyclictest12-21ksoftirqd/012:30:020
366385999351,31cyclictest3663980-21latency_hist07:10:030
366385999351,31cyclictest1-21systemd09:22:340
366386999342,3cyclictest131rcu_preempt12:04:332
366386499344,3cyclictest131rcu_preempt10:25:021
366386499343,3cyclictest131rcu_preempt11:14:341
366386499343,3cyclictest131rcu_preempt11:14:331
366386499343,3cyclictest131rcu_preempt08:47:341
366386499343,3cyclictest131rcu_preempt08:47:331
366386499343,3cyclictest131rcu_preempt07:20:321
366386499342,3cyclictest131rcu_preempt09:01:331
366386499342,3cyclictest131rcu_preempt09:01:321
3663864993422,8cyclictest0-21swapper/112:35:011
3663864993421,10cyclictest3666058-21sed07:50:011
366386499341,4cyclictest131rcu_preempt11:52:321
3663859993428,4cyclictest0-21swapper/011:32:540
366385999341,29cyclictest0-21swapper/010:50:320
3663869993318,11cyclictest235-21systemd-journal08:05:012
3663859993326,4cyclictest3665133-21latency_hist07:35:020
3663859993322,8cyclictest12-21ksoftirqd/011:15:010
3663859993322,8cyclictest12-21ksoftirqd/011:15:010
366385999331,29cyclictest0-21swapper/007:18:250
366385999331,28cyclictest0-21swapper/008:01:330
3663869993217,11cyclictest235-21systemd-journal09:20:002
366385999321,28cyclictest0-21swapper/007:51:380
3663869993116,12cyclictest272-21dbus-daemon08:50:002
3663869993116,12cyclictest272-21dbus-daemon08:50:002
3663859993125,4cyclictest0-21swapper/009:25:550
3663869993023,5cyclictest30-21ksoftirqd/207:50:022
366386499302,3cyclictest131rcu_preempt11:07:321
366386999284,3cyclictest131rcu_preempt07:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional