You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 20:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri May 22, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3024385999027,11cyclictest3084233-21ssh23:00:560
3024385998327,13cyclictest3117751-21ssh00:10:270
302438599821,37cyclictest3110727-21ssh23:55:500
302438599821,37cyclictest3110727-21ssh23:55:500
3024385997627,12cyclictest3047874-21ssh21:45:250
3024385997524,13cyclictest3093222-21run-parts23:20:000
3024385997423,17cyclictest3131872-21ssh00:40:010
3024385997235,4cyclictest16950irq/224-800000022:55:000
302438599722,11cyclictest3090951-21ls23:15:020
3024385997042,24cyclictest16950irq/224-800000021:40:000
3024385997029,12cyclictest3088429-21ssh23:09:590
3024385996726,12cyclictest3051552-21ssh21:53:200
3024385996726,10cyclictest3120859-21ssh00:16:550
3024390996624,37cyclictest3062130-21sh22:15:041
3024385996629,12cyclictest3116145-21ssh00:07:170
3024385996629,12cyclictest3116145-21ssh00:07:170
302439099653,13cyclictest3059516-21munin-run22:10:011
3024390996525,35cyclictest3090671-21sh23:14:411
3024385996518,25cyclictest16950irq/224-800000023:45:030
3024395996426,27cyclictest37-21ksoftirqd/323:55:023
302439099642,14cyclictest3027204-21munin-run20:05:011
3024385996421,11cyclictest3064865-21ssh22:20:430
3024385996415,14cyclictest3061539-21ssh22:14:090
3024385996414,29cyclictest12-21ksoftirqd/022:46:010
3024390996326,33cyclictest3038657-21sh21:26:251
3024390996324,35cyclictest23-21ksoftirqd/100:01:031
302439099632,14cyclictest3074012-21munin-run22:40:011
302439099631,15cyclictest3034319-21ssh21:17:361
302439099631,14cyclictest364-21snmpd23:09:341
302439099621,16cyclictest3024588-21munin-run19:15:021
302439099621,16cyclictest3024588-21munin-run19:15:011
302439099620,25cyclictest3049170-21ssh21:48:251
302439099612,18cyclictest3026923-21sendmail20:00:001
3024390996121,36cyclictest23-21ksoftirqd/123:55:171
3024390996121,36cyclictest23-21ksoftirqd/123:55:171
3024390996121,35cyclictest3068626-21sh22:28:421
302439099611,15cyclictest48-21kcompactd022:50:371
302439099611,13cyclictest3467602-21systemd-journal22:24:411
302439099609,4cyclictest0-21swapper/121:59:501
3024390996018,36cyclictest275-21dbus-daemon00:15:011
3024390996017,39cyclictest3467562-21sshd21:38:071
302439099601,16cyclictest3050741-21ssh21:51:261
302439599597,6cyclictest131rcu_preempt22:56:323
3024392995950,6cyclictest3011910-21kworker/2:120:55:002
302439099592,17cyclictest3030036-21munin-run21:00:001
302439099592,15cyclictest3130888-21ssh00:37:521
302439099592,14cyclictest364-21snmpd21:00:051
302439099591,16cyclictest3120248-21ssh00:15:351
302439099591,16cyclictest3081511-21ssh22:55:101
302439099591,13cyclictest3031881-21sh21:12:241
3024390995910,4cyclictest0-21swapper/121:08:461
302439099589,4cyclictest0-21swapper/120:42:541
302439099589,3cyclictest0-21swapper/123:46:481
302439099582,9cyclictest3076393-21modprobe22:45:001
3024390995825,29cyclictest23-21ksoftirqd/123:17:221
302439099582,14cyclictest3084971-21sshd23:02:401
302439099582,11cyclictest3026420-21munin-run19:50:011
3024390995814,39cyclictest3055385-21sshd22:01:081
3024390995813,12cyclictest23-21ksoftirqd/122:33:221
302439099581,14cyclictest48-21kcompactd000:25:191
302439099581,12cyclictest3037602-21sh21:24:301
302439099572,14cyclictest3104789-21ssh23:43:411
3024390995721,33cyclictest23-21ksoftirqd/119:29:591
3024390995719,33cyclictest3128530-21sh00:32:591
302439099571,16cyclictest3467602-21systemd-journal23:39:531
302439099571,16cyclictest3026295-21latency_hist19:45:011
3024385995733,11cyclictest12-21ksoftirqd/022:40:010
3024395995639,12cyclictest3467602-21systemd-journal22:10:013
302439099565,8cyclictest0-21swapper/120:15:011
302439099563,19cyclictest3047458-21ssh21:45:001
302439099562,8cyclictest3027945-21cron20:20:001
302439099562,8cyclictest3027945-21cron20:19:591
302439099562,17cyclictest3040493-21ssh21:30:051
302439099562,15cyclictest3060005-21ssh22:10:371
3024390995613,6cyclictest3030561-21diskmemload23:26:511
302439099561,11cyclictest3100528-21cron23:35:001
302439099561,10cyclictest3093647-21ssh23:20:241
302439099561,10cyclictest3093647-21ssh23:20:231
3024385995626,15cyclictest3054150-21ssh21:58:440
302438599562,49cyclictest3037969-21cat21:25:020
3024395995536,14cyclictest3467602-21systemd-journal00:31:373
3024392995546,5cyclictest3011910-21kworker/2:121:10:012
3024392995545,6cyclictest3011910-21kworker/2:121:15:182
3024390995512,37cyclictest275-21dbus-daemon20:10:001
302439099551,12cyclictest3077468-21ssh22:46:571
302439099551,11cyclictest3123324-21ssh00:22:011
302439099551,11cyclictest3123324-21ssh00:22:011
3024385995528,23cyclictest3075250-21ssh22:42:190
3024385995526,14cyclictest3097768-21ssh23:29:160
3024385995526,13cyclictest3055065-21ssh22:00:170
3024385995525,14cyclictest3062912-21ssh22:16:530
302438599551,3cyclictest131rcu_preempt00:31:140
302438599551,38cyclictest3105853-21ssh23:45:400
3024392995446,5cyclictest3045103-21kworker/2:323:31:282
3024392995445,6cyclictest3122213-21kworker/2:000:36:322
302439099542,15cyclictest3108659-21ssh23:51:401
302439099542,15cyclictest3029789-21munin-run20:55:011
302439099542,11cyclictest3467602-21systemd-journal20:37:371
3024390995410,5cyclictest0-21swapper/119:20:011
3024385995428,22cyclictest3095025-21ssh23:23:320
3024385995428,22cyclictest3095025-21ssh23:23:320
3024392995345,5cyclictest3091419-21kworker/2:123:48:542
3024392995345,5cyclictest3045103-21kworker/2:322:12:072
3024392995344,6cyclictest3045103-21kworker/2:300:05:332
3024392995344,6cyclictest3045103-21kworker/2:300:05:322
3024392995344,5cyclictest3011910-21kworker/2:121:24:222
3024392995343,6cyclictest3045103-21kworker/2:322:40:432
3024392995341,6cyclictest3045103-21kworker/2:323:15:002
3024392995336,12cyclictest1-21systemd00:30:002
3024392995334,14cyclictest1-21systemd23:20:012
3024392995334,13cyclictest275-21dbus-daemon20:35:012
3024392995334,13cyclictest275-21dbus-daemon20:35:012
302439099531,17cyclictest3115276-21ssh00:05:201
302439099531,17cyclictest3115276-21ssh00:05:201
302439099531,15cyclictest3028250-21munin-run20:25:011
3024385995327,11cyclictest3122998-21ssh00:21:200
3024385995327,11cyclictest3122998-21ssh00:21:200
3024385995318,22cyclictest3045953-21ssh21:41:340
3024385995317,32cyclictest12-21ksoftirqd/023:31:160
3024385995314,30cyclictest0-21swapper/021:16:320
3024392995245,4cyclictest3045103-21kworker/2:323:41:512
3024392995244,5cyclictest3045103-21kworker/2:322:39:292
3024392995243,5cyclictest3091419-21kworker/2:100:12:032
3024392995243,5cyclictest3011910-21kworker/2:121:29:132
3024392995240,6cyclictest3030030-21sendmail21:00:002
3024392995211,24cyclictest3086149-21ssh23:05:022
302439099522,11cyclictest3025861-21sendmail-mta19:39:441
3024390995217,31cyclictest23-21ksoftirqd/120:30:011
3024385995243,5cyclictest16950irq/224-800000020:10:010
3024385995227,8cyclictest12-21ksoftirqd/021:35:010
3024385995227,10cyclictest3108973-21ssh23:52:210
3024395995115,8cyclictest131rcu_preempt23:35:003
3024392995143,5cyclictest3045103-21kworker/2:322:08:012
3024392995143,5cyclictest3011910-21kworker/2:121:32:032
3024392995142,6cyclictest3045103-21kworker/2:322:17:152
3024392995142,6cyclictest3011910-21kworker/2:120:05:012
3024392995142,5cyclictest3122213-21kworker/2:000:31:062
3024392995133,14cyclictest1-21systemd20:15:002
302439099511,12cyclictest364-21snmpd20:33:081
302439099511,12cyclictest364-21snmpd20:33:081
3024385995126,10cyclictest12-21ksoftirqd/022:35:010
3024385995122,23cyclictest3024384-21sort19:10:020
3024395995035,6cyclictest309-21in:imuxsock21:00:013
3024395995032,13cyclictest275-21dbus-daemon00:00:033
3024395995032,13cyclictest275-21dbus-daemon00:00:033
3024392995043,4cyclictest3011910-21kworker/2:121:15:012
3024392995042,5cyclictest3045103-21kworker/2:322:59:112
3024392995042,5cyclictest3045103-21kworker/2:322:02:302
3024392995041,5cyclictest3091419-21kworker/2:100:20:252
3024392995041,5cyclictest3091419-21kworker/2:100:20:252
3024392995031,13cyclictest275-21dbus-daemon20:25:002
302439099504,8cyclictest0-21swapper/120:50:011
302439099504,8cyclictest0-21swapper/120:50:001
302439099501,13cyclictest364-21snmpd19:33:341
302439099501,11cyclictest364-21snmpd19:53:401
302439099501,11cyclictest364-21snmpd19:53:391
3024395994924,18cyclictest275-21dbus-daemon23:40:013
3024392994941,5cyclictest3045103-21kworker/2:322:53:062
3024392994941,5cyclictest3045103-21kworker/2:322:34:312
3024392994941,5cyclictest3045103-21kworker/2:322:29:272
3024392994940,6cyclictest3045103-21kworker/2:323:25:012
3024392994940,6cyclictest3045103-21kworker/2:323:25:002
3024392994940,5cyclictest3045103-21kworker/2:323:59:072
3024392994940,5cyclictest3045103-21kworker/2:323:59:072
3024392994939,6cyclictest3027218-21kworker/2:221:55:012
3024385994941,5cyclictest16950irq/224-800000023:40:000
3024395994836,8cyclictest3073989-21cron22:40:003
3024392994840,5cyclictest3091419-21kworker/2:123:52:092
3024392994840,5cyclictest3091419-21kworker/2:100:15:402
3024392994840,5cyclictest3045103-21kworker/2:323:06:142
3024392994840,5cyclictest3045103-21kworker/2:322:22:102
3024392994840,5cyclictest3011910-21kworker/2:120:19:082
3024392994840,5cyclictest3011910-21kworker/2:120:19:082
3024392994839,6cyclictest3011910-21kworker/2:121:39:002
3024392994839,5cyclictest3011910-21kworker/2:120:00:002
3024392994835,8cyclictest30-21ksoftirqd/221:43:532
302439299483,4cyclictest131rcu_preempt22:47:232
3024385994814,30cyclictest12-21ksoftirqd/000:26:220
3024395994730,13cyclictest37-21ksoftirqd/300:25:003
3024395994730,13cyclictest37-21ksoftirqd/300:25:003
3024392994739,5cyclictest3011910-21kworker/2:119:49:522
3024392994738,5cyclictest3027218-21kworker/2:221:56:492
3024392994732,11cyclictest30-21ksoftirqd/223:25:452
3024392994732,11cyclictest30-21ksoftirqd/200:03:252
3024385994740,4cyclictest16850irq/134-800000021:15:000
3024385994733,12cyclictest12-21ksoftirqd/021:30:010
302439599468,6cyclictest131rcu_preempt23:20:033
3024395994627,14cyclictest1-21systemd20:10:013
3024395994619,20cyclictest37-21ksoftirqd/322:20:413
302439599461,5cyclictest131rcu_preempt22:15:413
3024392994639,4cyclictest3011910-21kworker/2:119:30:012
3024392994638,5cyclictest3027218-21kworker/2:221:46:112
3024392994638,5cyclictest3011910-21kworker/2:120:45:072
3024392994638,5cyclictest3011910-21kworker/2:120:45:072
3024385994630,13cyclictest3114572-21ssh00:04:080
3024385994630,12cyclictest3467718-21lldpd22:30:010
3024385994625,17cyclictest3026374-21sshd19:46:070
302439599453,3cyclictest131rcu_preempt22:32:353
3024395994528,13cyclictest3045040-21cron21:40:003
3024395994524,16cyclictest3054737-21munin-run22:00:013
3024395994513,5cyclictest131rcu_preempt20:15:013
3024392994537,5cyclictest3011910-21kworker/2:121:02:402
3024392994536,6cyclictest3045103-21kworker/2:323:38:562
302439099453,8cyclictest3025110-21modprobe19:25:001
302439099453,8cyclictest3025110-21modprobe19:25:001
3024385994527,14cyclictest3057451-21ssh22:05:170
3024395994433,8cyclictest3131825-21latency_hist00:40:003
3024395994430,11cyclictest275-21dbus-daemon20:20:013
3024395994430,11cyclictest275-21dbus-daemon20:20:003
3024395994427,13cyclictest1-21systemd23:45:013
3024395994420,5cyclictest37-21ksoftirqd/320:00:013
3024395994418,18cyclictest275-21dbus-daemon19:45:013
302438599445,34cyclictest3028508-21cat20:30:020
3024385994437,4cyclictest16950irq/224-800000020:59:590
3024385994432,8cyclictest12-21ksoftirqd/020:00:010
3024385994428,12cyclictest3467602-21systemd-journal20:20:010
3024385994428,12cyclictest3467602-21systemd-journal20:20:010
3024385994427,13cyclictest3082813-21ssh22:58:090
302438599442,37cyclictest3027184-21modprobe20:05:000
302439599438,26cyclictest3076391-21ssh22:45:003
3024395994327,13cyclictest275-21dbus-daemon00:20:013
3024395994325,12cyclictest37-21ksoftirqd/321:35:003
3024392994335,5cyclictest3011910-21kworker/2:120:30:022
3024392994335,5cyclictest3011910-21kworker/2:120:05:322
3024392994334,6cyclictest3011910-21kworker/2:119:20:012
3024392994328,10cyclictest309-21in:imuxsock19:55:012
3024392994328,10cyclictest309-21in:imuxsock19:55:012
302438599432,4cyclictest3024574-21modprobe19:15:010
302438599432,4cyclictest3024574-21modprobe19:15:010
302438599431,38cyclictest0-21swapper/019:35:020
302439599428,5cyclictest131rcu_preempt00:10:013
302439599428,5cyclictest131rcu_preempt00:10:013
3024395994231,8cyclictest37-21ksoftirqd/321:20:003
3024395994229,9cyclictest3049972-21cat21:50:013
3024395994228,9cyclictest309-21in:imuxsock21:10:033
3024395994228,11cyclictest37-21ksoftirqd/322:15:013
3024395994227,11cyclictest275-21dbus-daemon22:50:013
3024392994233,5cyclictest3011910-21kworker/2:119:10:022
302439099422,13cyclictest3024509-21sed19:10:021
302438599423,18cyclictest131rcu_preempt21:10:000
302439599419,18cyclictest3024831-21modprobe19:19:593
3024395994135,3cyclictest371ksoftirqd/320:05:013
3024395994127,9cyclictest3028978-21cron20:40:003
3024395994127,10cyclictest37-21ksoftirqd/322:03:263
3024395994124,12cyclictest275-21dbus-daemon19:50:013
3024395994123,12cyclictest3467602-21systemd-journal21:25:013
302439599411,5cyclictest131rcu_preempt21:40:523
3024385994131,7cyclictest12-21ksoftirqd/020:25:010
302438599411,35cyclictest3024828-21modprobe19:20:000
302438599411,35cyclictest12-21ksoftirqd/019:54:080
302438599411,35cyclictest12-21ksoftirqd/019:54:070
302439599403,26cyclictest3467602-21systemd-journal23:25:003
302439599403,26cyclictest3467602-21systemd-journal23:25:003
3024395994028,9cyclictest3086065-21systemd-run23:05:013
3024395994028,7cyclictest3126997-21sh00:30:013
3024395994026,11cyclictest37-21ksoftirqd/323:25:303
3024395994026,10cyclictest3040259-21systemctl21:30:013
302438599403,7cyclictest3026072-21latency_hist19:40:020
3024385994032,5cyclictest19350irq/34-mmc119:20:070
3024385994032,5cyclictest19350irq/34-mmc119:20:060
3024385994015,22cyclictest12-21ksoftirqd/019:26:100
302439599396,8cyclictest37-21ksoftirqd/323:15:013
3024395993929,7cyclictest37-21ksoftirqd/300:11:333
3024395993927,9cyclictest37-21ksoftirqd/323:45:173
3024395993922,12cyclictest37-21ksoftirqd/319:10:023
3024395993917,15cyclictest275-21dbus-daemon20:45:013
3024395993911,21cyclictest0-21swapper/323:10:023
3024392993925,10cyclictest30-21ksoftirqd/219:35:002
302438599393,9cyclictest3467602-21systemd-journal19:44:270
3024385993926,9cyclictest3029090-21latency_hist20:40:010
302439599382,5cyclictest131rcu_preempt19:30:003
3024395993824,11cyclictest37-21ksoftirqd/319:35:013
302439299384,27cyclictest3029273-21grep20:45:002
3024392993822,11cyclictest309-21in:imuxsock19:45:002
3024385993830,5cyclictest19350irq/34-mmc121:04:020
302439599373,5cyclictest131rcu_preempt22:25:423
3024395993723,10cyclictest309-21in:imuxsock21:00:033
3024392993727,7cyclictest3025846-21sshd19:37:072
3024385993727,6cyclictest19350irq/34-mmc120:34:580
3024385993727,6cyclictest19350irq/34-mmc120:34:570
3024385993724,9cyclictest3029503-21sshd20:48:070
3024385993724,9cyclictest3029503-21sshd20:48:060
302439599366,19cyclictest131rcu_preempt22:55:013
3024395993627,5cyclictest3025923-21latency_hist19:40:013
3024395993624,10cyclictest37-21ksoftirqd/321:52:143
3024395993623,8cyclictest37-21ksoftirqd/320:55:003
3024395993620,12cyclictest1-21systemd20:35:013
3024395993620,12cyclictest1-21systemd20:35:013
302439599361,4cyclictest131rcu_preempt00:01:083
3024392993624,10cyclictest30-21ksoftirqd/220:38:072
3024385993617,4cyclictest3029245-21sshd20:42:170
3024395993521,10cyclictest37-21ksoftirqd/319:20:023
3024395993521,10cyclictest37-21ksoftirqd/319:20:023
3024392993522,10cyclictest275-21dbus-daemon19:25:012
3024392993522,10cyclictest275-21dbus-daemon19:25:012
302438599351,4cyclictest0-21swapper/020:55:010
302438599351,31cyclictest0-21swapper/020:11:020
3024395993424,7cyclictest0-21swapper/321:10:023
302439299345,3cyclictest131rcu_preempt19:15:032
302439299345,3cyclictest131rcu_preempt19:15:022
302439599334,23cyclictest3467602-21systemd-journal20:50:013
302439599334,23cyclictest3467602-21systemd-journal20:50:003
3024395993318,12cyclictest3026787-21sed19:55:023
3024395993318,12cyclictest3026787-21sed19:55:013
3024395993218,11cyclictest3028494-21systemd-run20:30:003
302439599321,4cyclictest131rcu_preempt19:15:023
302439599321,4cyclictest131rcu_preempt19:15:023
3024395993121,7cyclictest3028201-21sshd20:20:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional