You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-29 - 03:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Jul 29, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
401540499971,8cyclictest131rcu_preempt21:55:023
4015390998653,23cyclictest12-21ksoftirqd/021:15:030
4015404997524,46cyclictest48-21kcompactd000:05:093
4015390997030,35cyclictest1-21systemd00:40:020
4015404996722,39cyclictest4073473-21sh22:58:473
4015390996731,32cyclictest12-21ksoftirqd/022:10:000
401540499652,12cyclictest4053255-21ssh22:16:123
4015399996519,9cyclictest30-21ksoftirqd/200:00:012
4015395996529,32cyclictest221rcuc/121:15:021
4015404996431,30cyclictest4084654-21sshd23:21:583
401540499642,13cyclictest4016096-21sshd19:21:193
4015390996435,4cyclictest12-21ksoftirqd/023:14:330
401540499633,13cyclictest4033478-21munin-run21:35:023
4015404996326,32cyclictest4093141-21cron23:39:593
401540499631,10cyclictest4016850-21sshd19:36:223
4015399996313,9cyclictest30-21ksoftirqd/200:05:022
4015399996313,9cyclictest30-21ksoftirqd/200:05:012
401539099631,46cyclictest40564711ssh22:23:020
401540499622,7cyclictest4110986-21cron00:17:003
4015404996224,34cyclictest4030511-21sh21:28:463
401540499621,18cyclictest4117120-21munin-plugin-st00:30:003
401540499621,14cyclictest4018217-21munin-run20:05:013
401540499621,13cyclictest4079901-21ssh23:12:003
401540499621,13cyclictest4044837-21ssh21:58:463
401540499621,13cyclictest4044837-21ssh21:58:453
401540499621,12cyclictest4068430-21ssh22:47:593
401540499612,14cyclictest4035984-21latency_hist21:40:013
401540499612,13cyclictest4090800-21munin-run23:35:013
401540499612,13cyclictest4090800-21munin-run23:35:013
401540499612,13cyclictest4021287-21sshd21:00:543
401540499612,13cyclictest4021287-21sshd21:00:533
401540499612,12cyclictest4104233-21ssh00:02:533
401540499612,12cyclictest4104233-21ssh00:02:533
401540499611,14cyclictest364-21snmpd22:34:473
401540499611,13cyclictest4017386-21sshd19:46:223
401539999616,47cyclictest30-21ksoftirqd/220:05:012
4015399996112,8cyclictest30-21ksoftirqd/221:50:022
4015399996112,8cyclictest30-21ksoftirqd/221:50:012
401539099613,38cyclictest131rcu_preempt21:13:300
401540499603,13cyclictest4114721-21munin-run00:25:013
401540499603,13cyclictest4114721-21munin-run00:25:013
401540499602,14cyclictest4017265-21cat19:45:023
401540499601,9cyclictest364-21snmpd22:43:113
401540499601,18cyclictest4059156-21ssh22:28:473
401540499601,18cyclictest4059156-21ssh22:28:473
401540499601,11cyclictest4120549-21sshd00:37:013
401540499601,10cyclictest4108562-21ssh00:11:423
4015399996019,11cyclictest30-21ksoftirqd/222:15:012
4015395996026,14cyclictest23-21ksoftirqd/122:10:011
4015404995926,29cyclictest4015812-21sshd19:16:533
4015404995923,32cyclictest4100328-21cron23:55:013
401540499592,14cyclictest4020537-21munin-run20:50:003
401540499592,12cyclictest4102743-40mandb00:00:003
401540499591,15cyclictest4074551-21ssh23:00:393
4015399995915,9cyclictest30-21ksoftirqd/221:20:412
4015390995929,4cyclictest40916201ssh23:36:310
4015390995929,14cyclictest4032335-21ssh21:32:270
401540499583,7cyclictest4016367-21cron19:30:003
401540499582,13cyclictest4048303-21ssh22:05:373
401540499582,11cyclictest364-21snmpd20:58:273
401540499581,13cyclictest4118148-21rm00:31:493
4015399995820,31cyclictest310-21in:imklog19:30:022
4015399995816,8cyclictest30-21ksoftirqd/221:55:482
4015399995816,8cyclictest30-21ksoftirqd/221:55:482
4015399995814,10cyclictest30-21ksoftirqd/221:45:022
401539099583,37cyclictest12-21ksoftirqd/022:46:240
4015390995828,14cyclictest4113711-21ssh00:22:380
4015390995828,14cyclictest4113711-21ssh00:22:380
4015390995828,14cyclictest4039050-21ssh21:46:280
4015390995828,14cyclictest4039050-21ssh21:46:270
4015390995826,15cyclictest4089398-21ssh23:31:490
4015390995826,15cyclictest4089398-21ssh23:31:490
4015390995814,26cyclictest131rcu_preempt23:58:300
401540499573,14cyclictest4021556-21sshd21:09:533
401540499572,7cyclictest4095547-21cron23:45:003
401540499572,7cyclictest4095547-21cron23:45:003
401540499572,6cyclictest4019740-21cron20:35:003
4015404995723,29cyclictest4078830-21cron23:10:003
401540499572,19cyclictest4052553-21cut22:15:003
401540499572,14cyclictest4020969-21latency_hist20:55:033
401540499571,14cyclictest364-21snmpd20:26:213
401540499571,12cyclictest364-21snmpd22:04:263
401540499571,11cyclictest4055647-21sh22:21:063
4015390995730,17cyclictest12-21ksoftirqd/023:28:460
4015390995723,30cyclictest12-21ksoftirqd/022:43:210
401539099571,39cyclictest4084733-21ssh23:22:080
401540499569,11cyclictest4070732-21sh22:52:473
401540499563,7cyclictest4016615-21sshd19:34:523
401540499562,18cyclictest4038256-21ssh21:45:023
401540499562,16cyclictest4023882-21ssh21:15:003
401540499562,16cyclictest4017684-21munin-run19:55:013
401540499562,14cyclictest4020255-21sshd20:41:303
401540499562,11cyclictest364-21snmpd23:25:043
401540499561,10cyclictest4063085-21ssh22:36:543
401540499561,10cyclictest4063085-21ssh22:36:543
4015399995615,9cyclictest30-21ksoftirqd/221:30:022
4015399995610,8cyclictest30-21ksoftirqd/200:39:392
401539099568,30cyclictest131rcu_preempt23:50:080
4015390995620,32cyclictest12-21ksoftirqd/022:12:010
401540499552,8cyclictest4017892-21sshd19:56:223
401540499552,5cyclictest3467602-21systemd-journal23:15:333
401540499552,15cyclictest4026338-21munin-run21:20:023
4015404995513,35cyclictest4019991-21cron20:40:013
4015404995513,35cyclictest4019991-21cron20:40:003
4015399995519,8cyclictest30-21ksoftirqd/200:18:072
401539099553,32cyclictest151rcuc/020:25:010
4015390995528,23cyclictest4036645-21ssh21:41:260
4015390995527,12cyclictest4119403-21ssh00:34:450
4015390995519,18cyclictest16850irq/134-800000022:15:380
4015390995519,18cyclictest16850irq/134-800000022:03:340
4015390995518,6cyclictest16850irq/134-800000000:18:420
4015390995518,18cyclictest16850irq/134-800000022:55:400
4015390995510,5cyclictest131rcu_preempt21:58:540
4015390995510,5cyclictest131rcu_preempt21:58:530
401540499542,9cyclictest4019213-21sendmail-mta20:20:383
4015404995429,21cyclictest4027979-21sh21:23:183
401540499542,18cyclictest4040678-21grep21:50:023
401540499542,18cyclictest4040678-21grep21:50:013
401540499541,16cyclictest4097629-21ssh23:49:173
401539999547,11cyclictest30-21ksoftirqd/222:36:192
401539999547,11cyclictest30-21ksoftirqd/222:36:182
4015399995444,6cyclictest291rcuc/222:51:552
4015399995434,15cyclictest275-21dbus-daemon21:00:002
4015399995429,7cyclictest30-21ksoftirqd/221:52:212
401539099548,9cyclictest12-21ksoftirqd/021:51:100
4015390995419,18cyclictest16850irq/134-800000021:24:500
4015390995419,18cyclictest16850irq/134-800000000:29:510
4015390995419,17cyclictest16850irq/134-800000023:15:220
4015390995419,17cyclictest16850irq/134-800000021:29:220
4015390995418,18cyclictest16850irq/134-800000000:12:550
4015390995418,18cyclictest16850irq/134-800000000:05:150
4015390995415,11cyclictest12-21ksoftirqd/022:55:040
4015399995344,6cyclictest291rcuc/222:57:022
4015399995322,9cyclictest131rcu_preempt21:05:012
4015399995322,9cyclictest131rcu_preempt21:05:012
4015399995314,10cyclictest30-21ksoftirqd/222:45:012
4015390995319,17cyclictest16850irq/134-800000023:04:020
4015390995318,18cyclictest16850irq/134-800000023:42:200
4015390995318,18cyclictest16850irq/134-800000023:42:190
4015390995317,19cyclictest16850irq/134-800000022:25:270
4015390995317,19cyclictest16850irq/134-800000022:25:260
4015390995317,18cyclictest16850irq/134-800000022:31:290
4015390995310,4cyclictest131rcu_preempt21:36:320
401539999527,10cyclictest30-21ksoftirqd/221:32:242
401539999525,12cyclictest30-21ksoftirqd/222:30:352
401539999524,11cyclictest30-21ksoftirqd/223:17:552
4015399995232,13cyclictest275-21dbus-daemon22:30:012
4015399995232,13cyclictest275-21dbus-daemon22:30:002
401539999523,11cyclictest30-21ksoftirqd/223:11:452
4015399995215,8cyclictest30-21ksoftirqd/223:40:572
4015399995215,8cyclictest30-21ksoftirqd/223:40:572
4015399995212,7cyclictest30-21ksoftirqd/221:10:002
4015399995211,8cyclictest30-21ksoftirqd/222:45:032
4015390995223,14cyclictest4062418-21ssh22:35:230
4015390995223,14cyclictest4062418-21ssh22:35:220
4015390995217,17cyclictest16850irq/134-800000023:46:050
4015390995217,17cyclictest16850irq/134-800000023:09:510
401540499519,35cyclictest4018979-21munin-plugin-st20:20:003
401540499511,18cyclictest4015583-21latency_hist19:15:013
401540499511,18cyclictest4015583-21latency_hist19:15:013
401539999516,9cyclictest30-21ksoftirqd/223:29:142
401539999515,10cyclictest30-21ksoftirqd/200:32:432
4015399995113,11cyclictest30-21ksoftirqd/221:10:542
4015390995115,18cyclictest16850irq/134-800000000:00:120
4015390995115,18cyclictest16850irq/134-800000000:00:120
401539999508,11cyclictest4055771-21ssh22:21:232
401539999505,9cyclictest30-21ksoftirqd/221:17:112
401539999505,10cyclictest30-21ksoftirqd/222:07:152
4015399995042,6cyclictest30-21ksoftirqd/222:01:512
4015399995013,8cyclictest30-21ksoftirqd/223:05:002
4015395995026,4cyclictest41028231ssh00:00:021
401540499491,8cyclictest1-21systemd20:15:003
401540499491,7cyclictest1-21systemd20:10:003
401539999499,10cyclictest30-21ksoftirqd/200:12:152
401539999494,9cyclictest30-21ksoftirqd/221:39:252
401539999494,6cyclictest30-21ksoftirqd/200:25:112
401539999492,6cyclictest30-21ksoftirqd/223:34:322
401539999492,6cyclictest30-21ksoftirqd/223:34:322
401539999492,10cyclictest30-21ksoftirqd/223:36:562
4015395994933,12cyclictest3467602-21systemd-journal23:49:591
401539099492,14cyclictest4015879-21modprobe19:20:010
401540499485,8cyclictest0-21swapper/319:10:013
401539999488,7cyclictest30-21ksoftirqd/223:22:532
401539999486,8cyclictest30-21ksoftirqd/200:23:132
401539999486,8cyclictest30-21ksoftirqd/200:23:122
4015399994830,12cyclictest3467602-21systemd-journal19:40:002
4015395994834,11cyclictest4075559-21sshd23:02:581
4015395994834,10cyclictest275-21dbus-daemon23:25:011
4015395994833,11cyclictest4039121-21sshd21:46:371
4015395994833,11cyclictest4039121-21sshd21:46:371
4015395994828,14cyclictest275-21dbus-daemon21:30:001
401539099482,41cyclictest4017916-21modprobe20:00:010
4015390994813,15cyclictest16850irq/134-800000020:56:300
401539999477,11cyclictest30-21ksoftirqd/222:15:422
401539999475,7cyclictest30-21ksoftirqd/223:07:392
401539999474,9cyclictest30-21ksoftirqd/200:05:042
4015399994729,12cyclictest3467602-21systemd-journal19:55:002
401539999471,10cyclictest30-21ksoftirqd/220:40:022
401539999471,10cyclictest30-21ksoftirqd/220:40:022
4015399994710,9cyclictest0-21swapper/220:10:002
401539599475,5cyclictest131rcu_preempt23:33:571
401539599475,5cyclictest131rcu_preempt23:33:571
4015395994731,12cyclictest1-21systemd21:45:011
4015395994728,13cyclictest275-21dbus-daemon20:55:011
401539599471,5cyclictest131rcu_preempt21:51:091
401539999466,8cyclictest30-21ksoftirqd/223:48:102
401539999464,9cyclictest30-21ksoftirqd/220:15:022
4015395994632,12cyclictest23-21ksoftirqd/119:30:011
4015395994626,13cyclictest3467602-21systemd-journal21:40:001
401539599461,15cyclictest0-21swapper/121:11:251
401539999455,7cyclictest30-21ksoftirqd/220:50:012
401539999453,7cyclictest30-21ksoftirqd/220:30:022
401539999453,10cyclictest30-21ksoftirqd/223:54:182
4015399994529,12cyclictest3467602-21systemd-journal19:19:542
4015395994515,20cyclictest131rcu_preempt22:45:441
4015390994537,4cyclictest16950irq/224-800000019:25:010
4015390994531,11cyclictest3467602-21systemd-journal20:54:530
4015390994512,6cyclictest131rcu_preempt20:30:010
401539999441,11cyclictest30-21ksoftirqd/220:45:002
4015395994428,11cyclictest275-21dbus-daemon21:35:021
4015395994424,14cyclictest275-21dbus-daemon20:10:011
4015395994416,6cyclictest131rcu_preempt22:20:011
4015395994415,22cyclictest131rcu_preempt22:31:171
4015395994412,7cyclictest131rcu_preempt00:40:001
4015390994435,6cyclictest19350irq/34-mmc121:10:000
4015390994428,13cyclictest4018688-21sshd20:13:540
4015390994428,11cyclictest1-21systemd20:05:010
4015399994323,14cyclictest275-21dbus-daemon20:25:012
401539999431,11cyclictest30-21ksoftirqd/219:45:032
401539599434,6cyclictest23-21ksoftirqd/120:15:001
401539599434,11cyclictest23-21ksoftirqd/123:15:001
4015395994331,8cyclictest23-21ksoftirqd/100:10:001
4015395994330,9cyclictest3467602-21systemd-journal20:25:001
4015395994328,11cyclictest4074001-21cron23:00:001
4015395994326,12cyclictest275-21dbus-daemon22:05:011
4015390994328,11cyclictest1-21systemd20:20:000
4015390994327,11cyclictest1-21systemd19:15:020
4015390994327,11cyclictest1-21systemd19:15:010
4015399994234,5cyclictest4017908-21cron20:00:012
4015399994210,5cyclictest0-21swapper/219:15:022
4015399994210,5cyclictest0-21swapper/219:15:012
4015395994232,7cyclictest23-21ksoftirqd/123:39:591
4015395994228,11cyclictest4017570-21sed19:50:011
4015395994227,11cyclictest4028929-21cut21:25:021
4015395994226,12cyclictest4052551-21munin-run22:14:591
4015395994215,6cyclictest131rcu_preempt19:25:011
4015395994213,18cyclictest23-21ksoftirqd/122:27:051
4015395994213,18cyclictest23-21ksoftirqd/122:27:041
401539099424,6cyclictest3467602-21systemd-journal20:40:010
401539099424,6cyclictest3467602-21systemd-journal20:40:000
401539099421,38cyclictest0-21swapper/019:40:000
4015395994133,6cyclictest23-21ksoftirqd/119:58:421
4015395994129,9cyclictest23-21ksoftirqd/123:45:021
4015395994129,9cyclictest23-21ksoftirqd/123:45:021
4015395994127,11cyclictest1-21systemd00:05:011
4015395994127,11cyclictest1-21systemd00:05:011
4015395994124,12cyclictest3467602-21systemd-journal22:00:011
4015395994124,12cyclictest3467602-21systemd-journal22:00:001
4015395994115,6cyclictest131rcu_preempt23:50:441
4015395994115,6cyclictest131rcu_preempt22:40:131
401539099412,5cyclictest4016380-21munin-run19:30:000
4015390994124,14cyclictest12-21ksoftirqd/021:05:020
4015390994124,14cyclictest12-21ksoftirqd/021:05:020
4015390994122,13cyclictest3467602-21systemd-journal19:45:000
401539999409,6cyclictest0-21swapper/219:50:002
4015399994031,7cyclictest30-21ksoftirqd/219:35:022
4015399994031,7cyclictest30-21ksoftirqd/219:25:002
4015399994029,8cyclictest4018944-21sshd20:17:532
401539599404,14cyclictest131rcu_preempt00:18:151
4015395994030,5cyclictest2-21kthreadd00:35:011
4015395994028,9cyclictest23-21ksoftirqd/100:30:011
4015395994028,8cyclictest1-21systemd19:40:011
4015395994027,9cyclictest23-21ksoftirqd/122:40:021
4015395994027,9cyclictest23-21ksoftirqd/122:40:021
4015395994026,11cyclictest23-21ksoftirqd/120:20:011
4015395994026,10cyclictest23-21ksoftirqd/123:27:391
4015395994025,11cyclictest1-21systemd22:25:011
4015395994013,19cyclictest23-21ksoftirqd/122:54:571
4015399993924,11cyclictest4019946-21sed20:35:022
401539599393,5cyclictest131rcu_preempt23:05:351
4015395993925,11cyclictest23-21ksoftirqd/100:21:591
4015395993925,11cyclictest23-21ksoftirqd/100:21:581
401539099392,6cyclictest4016632-21munin-run19:35:000
401539099392,6cyclictest364-21snmpd19:45:360
4015390993924,11cyclictest12-21ksoftirqd/020:44:590
4015390993923,12cyclictest4017654-21sshd19:54:530
401539099392,2cyclictest0-21swapper/020:50:020
4015399993829,6cyclictest30-21ksoftirqd/219:10:032
4015395993828,8cyclictest23-21ksoftirqd/119:50:031
4015395993826,9cyclictest23-21ksoftirqd/120:50:031
4015395993826,9cyclictest23-21ksoftirqd/120:45:001
4015395993816,19cyclictest221rcuc/119:45:021
4015390993827,8cyclictest12-21ksoftirqd/019:10:010
4015399993713,5cyclictest0-21swapper/220:52:072
401539599376,5cyclictest131rcu_preempt23:16:051
4015395993714,16cyclictest23-21ksoftirqd/100:12:031
401539099372,30cyclictest4019948-21latency_hist20:35:020
401539099371,32cyclictest0-21swapper/020:05:200
4015395993626,7cyclictest23-21ksoftirqd/120:38:101
4015395993626,7cyclictest23-21ksoftirqd/120:38:091
401539599362,5cyclictest131rcu_preempt19:15:011
401539599362,5cyclictest131rcu_preempt19:15:001
4015395993624,8cyclictest4019737-21sshd20:34:531
4015395993623,11cyclictest23-21ksoftirqd/121:10:011
4015395993622,11cyclictest23-21ksoftirqd/121:05:011
4015395993622,11cyclictest23-21ksoftirqd/121:05:001
4015395993622,11cyclictest23-21ksoftirqd/120:30:011
4015395993619,4cyclictest23-21ksoftirqd/120:05:001
4015395993611,14cyclictest23-21ksoftirqd/120:55:491
401539599354,5cyclictest131rcu_preempt19:17:531
4015395993319,10cyclictest4015532-21cat19:10:021
4015395993121,8cyclictest23-21ksoftirqd/119:31:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional