You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 06:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 29, 2026 00:45:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3912300998627,26cyclictest3961830-21sh22:47:091
391230599808,7cyclictest131rcu_preempt00:25:023
391229699801,68cyclictest3999712-21sh00:15:020
3912296997715,9cyclictest12-21ksoftirqd/021:50:020
3912305997438,31cyclictest1-21systemd22:35:013
3912296997423,29cyclictest4004332-21rm00:25:220
3912296997326,43cyclictest3986651-21latency_hist23:45:020
391230599691,16cyclictest3929710-21ssh21:32:493
391230099681,38cyclictest4009249-21ssh00:36:481
3912300996731,32cyclictest3916303-21sshd20:01:511
391230099671,41cyclictest3968321-21ssh23:02:181
391230099671,41cyclictest3968321-21ssh23:02:181
391229699676,10cyclictest121ksoftirqd/000:15:220
3912305996622,40cyclictest37-21ksoftirqd/323:55:013
3912305996622,39cyclictest309-21in:imuxsock22:50:003
391230099664,52cyclictest120-21kswapd000:00:031
3912296996662,2cyclictest0-21swapper/020:05:020
3912305996525,36cyclictest3978255-21sh23:25:053
3912305996412,47cyclictest275-21dbus-daemon22:15:013
391230599641,20cyclictest3971037-21ssh23:08:403
391230599632,18cyclictest3933979-21ssh21:42:393
391230599632,16cyclictest3927650-21ssh21:27:593
3912305996315,43cyclictest275-21dbus-daemon00:05:013
391230599631,17cyclictest3981065-21ssh23:31:353
3912305996310,49cyclictest3914286-21sshd19:26:093
391230099631,29cyclictest3946563-21ssh22:11:491
3912296996329,30cyclictest12-21ksoftirqd/021:31:050
391230599622,19cyclictest3942844-21ssh22:03:213
391230599622,17cyclictest3921079-21ssh21:12:483
391230599622,15cyclictest3945410-21ssh22:09:283
391230599621,19cyclictest3923973-21ssh21:19:433
391230599621,16cyclictest3987925-21ssh23:47:413
391230599612,15cyclictest3960355-21ssh22:44:013
391230599612,15cyclictest3960355-21ssh22:44:013
391230599611,8cyclictest3976220-21ssh23:20:193
3912300996146,9cyclictest23-21ksoftirqd/123:45:011
391230099613,52cyclictest3984568-21munin-run23:40:021
3912300996129,29cyclictest3916884-21latency_hist20:15:011
3912300996120,37cyclictest4004001-21latency_hist00:25:011
391230099611,32cyclictest0-21swapper/123:17:281
391230099611,31cyclictest3977409-21ssh23:23:281
391230599602,8cyclictest4008392-21cat00:35:023
391230599602,8cyclictest4003963-21ssh00:25:003
391230599602,7cyclictest3957747-21ssh22:37:493
391230599602,16cyclictest3995809-21ssh00:05:463
391230599601,8cyclictest3931033-21ssh21:35:373
391230599601,16cyclictest3975135-21ssh23:18:093
391230099602,53cyclictest3928577-21sort21:30:021
391229699601,3cyclictest0-21swapper/023:00:020
391230599593,12cyclictest48-21kcompactd019:45:003
391230599592,6cyclictest3925815-21sshd21:23:503
3912305995922,33cyclictest3919758-21diskmemload21:46:083
391230599592,19cyclictest3918374-21latency_hist20:40:023
391230599592,16cyclictest3950901-21ssh22:21:573
391230599591,8cyclictest3965477-21ssh22:55:343
391230599591,8cyclictest3948852-21ssh22:17:113
391230599591,7cyclictest3983072-21ssh23:36:183
391230599591,7cyclictest3938279-21ssh21:52:423
3912302995931,7cyclictest3965053-21kworker/2:223:20:032
3912300995929,26cyclictest3970818-21sh23:08:081
391229699594,8cyclictest121ksoftirqd/021:39:070
3912296995926,4cyclictest1-21systemd22:20:010
391230599583,10cyclictest3918464-21modprobe20:45:003
391230599581,7cyclictest4010401-21ssh00:39:503
391230599581,7cyclictest3998181-21ssh00:11:183
391230599581,7cyclictest3972332-21ssh23:11:323
3912305995815,40cyclictest37-21ksoftirqd/323:43:433
3912302995848,7cyclictest3919260-21kworker/2:322:48:312
3912302995848,6cyclictest3919260-21kworker/2:321:35:012
391230099582,51cyclictest3982328-21sshd23:34:531
3912296995820,34cyclictest12-21ksoftirqd/022:11:430
391230599572,14cyclictest3999837-21modprobe00:15:033
391230599571,6cyclictest3991647-21ssh23:56:133
391230599571,17cyclictest3968152-21ssh23:01:443
391230599571,17cyclictest3968152-21ssh23:01:433
3912302995746,8cyclictest3919260-21kworker/2:321:50:132
3912302995744,8cyclictest3919260-21kworker/2:300:30:262
3912302995739,13cyclictest1-21systemd19:25:012
391230099571,52cyclictest3933253-21ssh21:40:471
391230099571,51cyclictest3918752-21latency_hist20:50:021
3912296995738,16cyclictest16850irq/134-800000023:40:010
3912296995725,9cyclictest12-21ksoftirqd/021:29:370
391230599565,43cyclictest3914909-21systemctl19:40:013
391230599561,16cyclictest3912371-21latency_hist19:10:013
3912302995648,5cyclictest3919260-21kworker/2:300:27:282
3912302995647,6cyclictest3965053-21kworker/2:223:56:162
3912302995647,5cyclictest3919260-21kworker/2:321:45:212
3912302995636,16cyclictest3467602-21systemd-journal21:00:002
3912300995624,26cyclictest275-21dbus-daemon19:26:221
391230099561,51cyclictest3957965-21ssh22:38:271
391230099561,49cyclictest3916654-21cat20:10:021
3912296995627,26cyclictest12-21ksoftirqd/022:40:140
3912296995627,26cyclictest12-21ksoftirqd/022:40:140
3912296995625,15cyclictest3954287-21ssh22:30:000
391230599552,7cyclictest3963680-21ssh22:51:283
391230599552,7cyclictest3963680-21ssh22:51:273
391230599552,6cyclictest3915966-21sshd19:56:083
391230599552,5cyclictest3916344-21munin-run20:05:013
391230599551,5cyclictest3467602-21systemd-journal22:28:463
3912302995546,6cyclictest3965053-21kworker/2:223:46:082
3912302995546,6cyclictest3919260-21kworker/2:300:00:222
3912302995545,6cyclictest3919260-21kworker/2:322:10:492
391230099552,48cyclictest3920495-21ssh21:11:131
3912300995524,26cyclictest3987075-21sh23:45:301
391230099551,50cyclictest3915455-21latency_hist19:50:021
391230099551,26cyclictest3919758-21diskmemload23:52:171
391229699557,8cyclictest12-21ksoftirqd/019:25:430
391229699554,7cyclictest121ksoftirqd/023:54:120
391230599547,5cyclictest3919758-21diskmemload21:58:253
391230599543,45cyclictest3918855-21grep20:50:023
3912305995428,22cyclictest37-21ksoftirqd/321:10:023
391230599542,5cyclictest3919502-21munin-run21:05:003
391230599542,5cyclictest364-21snmpd19:51:213
3912302995446,5cyclictest3905203-21kworker/2:220:40:012
3912302995445,5cyclictest3965053-21kworker/2:223:35:082
3912302995445,5cyclictest3919260-21kworker/2:323:08:192
3912302995445,5cyclictest3919260-21kworker/2:323:04:292
3912302995445,5cyclictest3919260-21kworker/2:323:04:292
3912302995444,7cyclictest3905203-21kworker/2:219:30:002
3912302995443,6cyclictest3905203-21kworker/2:221:12:202
3912302995432,16cyclictest275-21dbus-daemon22:00:012
391230099543,46cyclictest3917873-21sed20:30:011
3912300995420,29cyclictest3996953-21sh00:08:381
3912300995416,33cyclictest0-21swapper/122:01:261
391230099541,50cyclictest3925425-21ssh21:22:441
391230099541,49cyclictest3972475-21ssh23:11:541
391230099541,49cyclictest3923407-21ssh21:18:131
391230099541,48cyclictest3963518-21ssh22:50:571
391230099541,48cyclictest3963518-21ssh22:50:571
3912296995428,11cyclictest3978764-21ssh23:26:240
391229699542,16cyclictest3993110-40mandb00:00:030
3912296995421,29cyclictest12-21ksoftirqd/021:55:020
391230599532,9cyclictest3914772-21latency_hist19:35:023
391230599532,10cyclictest3919146-21grep20:55:033
3912302995345,5cyclictest3932755-21kworker/2:122:00:592
3912302995345,5cyclictest3919260-21kworker/2:300:17:352
3912302995345,5cyclictest3919260-21kworker/2:300:07:342
3912302995344,6cyclictest3965053-21kworker/2:223:22:192
3912302995344,6cyclictest3919260-21kworker/2:300:13:152
3912302995344,5cyclictest3919260-21kworker/2:300:35:462
3912302995343,6cyclictest3919260-21kworker/2:322:33:422
3912302995334,13cyclictest1-21systemd22:30:012
391230099532,47cyclictest3993110-40mandb00:00:041
391230099532,47cyclictest3954961-21ssh22:31:161
391230099532,47cyclictest3937949-21ssh21:51:531
391230099532,47cyclictest3915141-21modprobe19:40:031
391230099532,45cyclictest4000315-21ssh00:16:101
3912300995321,28cyclictest275-21dbus-daemon19:35:001
391230099531,48cyclictest3931313-21ssh21:36:271
391230099531,48cyclictest3914900-21latency_hist19:40:031
391230099531,47cyclictest3950354-21ssh22:20:301
391230099531,47cyclictest3935278-21ssh21:45:341
391230099531,46cyclictest48-21kcompactd022:40:071
391230099531,46cyclictest48-21kcompactd022:40:061
391229699539,5cyclictest131rcu_preempt23:45:300
391229699537,9cyclictest12-21ksoftirqd/000:02:060
3912296995327,11cyclictest3941769-21ssh22:00:320
3912296995326,12cyclictest3933693-21ssh21:41:570
3912296995324,25cyclictest3923250-21sshd21:17:510
391229699531,7cyclictest121ksoftirqd/023:15:250
391229699531,37cyclictest4002954-21ssh00:22:250
3912296995310,4cyclictest131rcu_preempt22:07:500
391230599521,14cyclictest3916552-21time-dir20:05:363
3912302995244,5cyclictest3919260-21kworker/2:323:28:192
3912302995244,5cyclictest3919260-21kworker/2:322:55:402
3912302995244,5cyclictest3919260-21kworker/2:322:16:382
3912302995243,6cyclictest3919260-21kworker/2:322:35:392
3912302995243,5cyclictest3919260-21kworker/2:321:28:382
3912302995242,6cyclictest3965053-21kworker/2:223:45:012
3912302995242,6cyclictest3919260-21kworker/2:322:22:352
3912300995225,23cyclictest3912481-21latency_hist19:15:031
3912300995225,23cyclictest3912481-21latency_hist19:15:031
391230099522,46cyclictest3939553-21ssh21:55:311
391230099522,45cyclictest275-21dbus-daemon21:35:011
3912300995216,32cyclictest3920168-1kworker/u9:022:06:201
391230099521,48cyclictest3997960-21ssh00:10:401
391230099521,47cyclictest4005455-21ssh00:28:181
391230099521,47cyclictest3948946-21ssh22:17:241
391230099521,46cyclictest364-21snmpd23:27:211
391229699528,7cyclictest12-21ksoftirqd/000:36:450
391229699527,10cyclictest12-21ksoftirqd/022:48:310
391229699526,9cyclictest12-21ksoftirqd/022:52:320
391229699526,9cyclictest12-21ksoftirqd/022:52:320
3912296995234,12cyclictest3467602-21systemd-journal21:10:010
391229699522,8cyclictest12-21ksoftirqd/023:35:010
3912305995126,21cyclictest3917821-21latency_hist20:30:023
3912302995143,5cyclictest3965053-21kworker/2:223:12:182
3912302995143,5cyclictest3932755-21kworker/2:122:05:162
3912302995142,5cyclictest3919260-21kworker/2:300:20:552
3912300995125,22cyclictest3912941-21latency_hist19:20:011
391230099511,46cyclictest4006719-21ssh00:30:531
391230099511,46cyclictest3965806-21ssh22:56:201
391230099511,46cyclictest3918944-21sshd20:52:511
391230099511,45cyclictest3919209-21cron21:00:001
3912296995142,6cyclictest16850irq/134-800000022:20:070
3912296995126,10cyclictest3925695-21ssh21:23:320
3912296995123,13cyclictest3972541-21ssh23:12:030
3912296995114,9cyclictest131rcu_preempt00:07:050
391229699511,44cyclictest3467602-21systemd-journal21:02:500
3912296995113,9cyclictest12-21ksoftirqd/023:04:170
3912296995113,9cyclictest12-21ksoftirqd/023:04:170
391230599503,13cyclictest3917904-21modprobe20:31:503
391230599502,9cyclictest3917391-21sshd20:20:503
391230599502,15cyclictest364-21snmpd20:10:453
3912302995042,5cyclictest3919260-21kworker/2:322:41:302
3912302995042,5cyclictest3919260-21kworker/2:322:41:302
3912302995042,5cyclictest3905203-21kworker/2:221:05:502
3912302995042,4cyclictest3919260-21kworker/2:321:24:152
3912302995041,6cyclictest3919260-21kworker/2:321:42:572
3912302995041,6cyclictest3919260-21kworker/2:321:35:462
3912302995041,6cyclictest3912056-21kworker/2:019:10:022
391230099501,45cyclictest3952728-21ssh22:26:041
3912296995041,5cyclictest16950irq/224-800000023:09:590
3912296995018,28cyclictest16850irq/134-800000022:31:000
391230599492,5cyclictest364-21snmpd20:56:103
391230599491,7cyclictest3912991-21sshd19:20:463
3912302994941,5cyclictest3919260-21kworker/2:322:50:062
3912302994941,5cyclictest3919260-21kworker/2:322:50:062
3912302994941,5cyclictest3912056-21kworker/2:020:25:032
3912302994941,5cyclictest3905203-21kworker/2:221:16:512
3912302994941,5cyclictest3905203-21kworker/2:220:30:002
3912302994939,5cyclictest3919260-21kworker/2:323:30:142
391230099492,42cyclictest3467602-21systemd-journal21:08:301
391230099492,41cyclictest3918226-21munin-run20:40:021
391230099491,45cyclictest364-21snmpd19:50:031
391229699495,10cyclictest12-21ksoftirqd/023:21:090
391229699493,8cyclictest12-21ksoftirqd/000:30:340
391229699491,7cyclictest3941363-21sendmail-msp22:00:010
391230599483,4cyclictest364-21snmpd20:17:093
391230599482,6cyclictest3912447-21modprobe19:10:033
391230599482,6cyclictest3912447-21modprobe19:10:033
3912302994833,11cyclictest1-21systemd19:50:012
3912300994819,25cyclictest3913209-21cat19:25:021
391230099481,43cyclictest3915971-21sshd19:56:491
391230099481,42cyclictest48-21kcompactd021:00:281
3912296994838,6cyclictest16950irq/224-800000019:50:000
3912296994837,7cyclictest151rcuc/021:14:020
3912296994828,15cyclictest3467718-21lldpd20:20:010
391229699482,14cyclictest12-21ksoftirqd/022:39:460
391230599472,5cyclictest48-21kcompactd019:45:353
391230599472,5cyclictest364-21snmpd19:17:443
3912302994740,4cyclictest3905203-21kworker/2:219:35:012
3912302994739,5cyclictest3912056-21kworker/2:020:16:112
3912302994738,5cyclictest3965053-21kworker/2:223:51:062
391230099472,42cyclictest3917101-21modprobe20:15:021
391230099472,41cyclictest3917453-21sort20:25:021
391230099471,41cyclictest364-21snmpd20:44:071
3912296994739,4cyclictest16950irq/224-800000021:00:000
3912302994640,3cyclictest3912056-21kworker/2:019:11:292
3912302994640,3cyclictest3912056-21kworker/2:019:11:292
3912302994638,5cyclictest3912056-21kworker/2:020:14:322
3912302994638,5cyclictest3912056-21kworker/2:019:18:002
3912302994638,5cyclictest3905203-21kworker/2:220:31:572
3912300994612,30cyclictest3918105-21latency_hist20:35:021
3912302994537,5cyclictest3905203-21kworker/2:220:50:012
3912302994536,6cyclictest3912056-21kworker/2:020:10:012
3912302994536,6cyclictest3905203-21kworker/2:220:04:432
3912302994533,7cyclictest3914901-21sendmail19:40:002
391229699454,5cyclictest121ksoftirqd/020:11:420
391229699451,12cyclictest3912678-21sed19:15:020
3912302994428,12cyclictest3467602-21systemd-journal19:59:432
391229699443,10cyclictest275-21dbus-daemon20:05:460
3912296994417,23cyclictest12-21ksoftirqd/019:35:010
3912302994336,4cyclictest3905203-21kworker/2:221:00:092
3912302994335,5cyclictest3905203-21kworker/2:220:51:102
3912302994335,5cyclictest3905203-21kworker/2:219:50:192
3912302994335,5cyclictest3905203-21kworker/2:219:41:372
391230099432,35cyclictest3912370-21latency_hist19:10:011
3912296994335,5cyclictest16850irq/134-800000019:51:580
391229699433,10cyclictest12-21ksoftirqd/020:55:020
3912296994324,13cyclictest3914897-21sh19:40:010
3912296994229,10cyclictest12-21ksoftirqd/019:13:570
3912302994134,4cyclictest3905203-21kworker/2:220:44:062
3912296994126,12cyclictest12-21ksoftirqd/020:40:010
391229699412,34cyclictest0-21swapper/020:50:000
3912296994023,12cyclictest275-21dbus-daemon20:45:010
391229699396,8cyclictest12-21ksoftirqd/020:26:130
3912296993931,4cyclictest3912754-21munin-plugin-st19:20:000
391229699392,32cyclictest3917944-21unin-run20:35:010
391229699382,7cyclictest3913027-21munin-run19:25:000
391229699382,5cyclictest12-21ksoftirqd/019:40:500
3912296993629,4cyclictest16950irq/224-800000020:25:010
391229699362,5cyclictest3467602-21systemd-journal19:59:430
391229699293,18cyclictest3912299-21latency_hist19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional