You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-29 - 09:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Aug 29, 2026 00:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30851079910312,9cyclictest131rcu_preempt00:30:020
308510799891,69cyclictest3142872-21ssh22:59:310
3085107998721,15cyclictest3176443-21ssh00:10:210
3085107998721,15cyclictest3175397-21ssh00:08:300
3085115998628,22cyclictest3122885-21ssh22:16:411
3085115998628,22cyclictest3122885-21ssh22:16:401
3085115998534,13cyclictest3145257-21ssh23:04:351
3085115998534,13cyclictest3145257-21ssh23:04:341
3085107998543,38cyclictest16950irq/224-800000023:15:000
3085107998543,38cyclictest16950irq/224-800000023:15:000
3085121998436,15cyclictest3467602-21systemd-journal23:30:013
3085115998323,20cyclictest364-21snmpd23:06:511
3085115998321,13cyclictest275-21dbus-daemon20:15:021
3085115998321,13cyclictest275-21dbus-daemon20:15:011
3085107998327,51cyclictest3137035-21ssh22:46:570
3085107998326,53cyclictest3111021-21ssh21:51:260
3085115998217,25cyclictest3105881-21ssh21:40:321
3085115998126,19cyclictest3161460-21ssh23:39:061
3085115998126,19cyclictest3161460-21ssh23:39:061
3085107998133,43cyclictest16850irq/134-800000023:30:010
3085115998025,19cyclictest3118908-21ssh22:08:281
3085115998022,24cyclictest3189366-21ssh00:38:081
3085107998026,50cyclictest3108409-21ssh21:45:480
3085115997925,17cyclictest3094150-21ssh21:15:331
3085115997826,14cyclictest3111873-21ssh21:53:251
3085115997822,18cyclictest3093304-21ssh21:14:051
308510799782,71cyclictest3112987-21ssh21:55:290
3085107997822,37cyclictest3161466-21ssh23:39:070
3085107997822,37cyclictest3161466-21ssh23:39:060
308510799781,5cyclictest3095542-21ssh21:18:530
3085115997723,13cyclictest3178410-21munin-run00:15:021
308510799771,37cyclictest3152466-21latency_hist23:20:020
3085115997630,12cyclictest3164852-21ssh23:45:511
3085115997630,12cyclictest3164852-21ssh23:45:511
3085115997527,15cyclictest3174944-21ssh00:07:281
3085115997524,13cyclictest3167355-21ssh23:51:091
3085115997522,15cyclictest3088360-21cut20:10:021
3085107997517,39cyclictest16850irq/134-800000022:05:190
308510799751,56cyclictest0-21swapper/022:16:400
308510799751,56cyclictest0-21swapper/022:16:400
3085115997437,13cyclictest3163614-21ssh23:43:251
3085115997426,14cyclictest3091285-21diskmemload23:30:131
3085115997423,15cyclictest3102195-21ssh21:32:571
3085115997422,16cyclictest3141417-21ssh22:56:091
3085115997422,15cyclictest3137497-21ssh22:48:041
3085115997420,15cyclictest3116479-21ssh22:03:151
3085107997429,41cyclictest16950irq/224-800000022:50:540
3085107997429,41cyclictest16950irq/224-800000022:50:530
3085107997427,44cyclictest0-21swapper/022:39:040
3085107997426,31cyclictest3146201-21ssh23:06:180
3085107997423,34cyclictest3184594-21ssh00:27:560
308510799741,57cyclictest3154241-21ssh23:23:320
3085115997327,15cyclictest3183989-21sh00:26:291
3085115997326,19cyclictest3187420-21modprobe00:34:011
3085115997326,14cyclictest3170392-21ssh23:57:461
3085107997327,42cyclictest3169631-21ssh23:56:020
3085107997327,41cyclictest3166163-21ssh23:48:580
3085107997327,41cyclictest3166163-21ssh23:48:570
3085107997321,11cyclictest3106671-21ssh21:42:180
3085107997315,12cyclictest3173117-21ssh00:03:360
3085115997225,14cyclictest3113420-21ssh21:56:281
3085115997222,16cyclictest3120409-21ssh22:11:301
3085107997220,9cyclictest3154680-21kworker/u8:023:34:290
3085115997124,15cyclictest3090508-21systemd-run20:55:001
3085115997123,14cyclictest3098236-21ssh21:24:431
3085115997123,13cyclictest3179131-21ssh00:16:161
3085115997122,14cyclictest3124899-21ssh22:20:541
3085115997122,14cyclictest3124899-21ssh22:20:541
308510799711,66cyclictest0-21swapper/021:34:240
3085115997024,14cyclictest3087824-21sed20:00:031
3085115997024,14cyclictest3087824-21sed20:00:021
308510799702,5cyclictest3098390-21munin-run21:25:000
3085107997021,16cyclictest3116718-21ssh22:03:470
308510799701,65cyclictest0-21swapper/022:30:560
308510799701,65cyclictest0-21swapper/022:30:560
308510799701,64cyclictest0-21swapper/021:38:160
308510799701,64cyclictest0-21swapper/021:38:160
3085107997016,49cyclictest3179121-21ssh00:16:150
3085115996924,12cyclictest0-21swapper/122:27:321
3085115996923,14cyclictest3100373-21ssh21:29:151
3085115996923,14cyclictest3100373-21ssh21:29:141
3085115996921,15cyclictest3172013-21ssh00:01:031
3085115996921,14cyclictest3155954-21ssh23:27:021
3085115996920,14cyclictest3133094-21ssh22:38:521
308510799693,4cyclictest312056450ssh22:11:490
3085107996926,28cyclictest3135162-21ssh22:43:120
3085107996924,28cyclictest3168716-21ssh23:54:260
3085115996824,15cyclictest3103824-21sh21:36:161
3085115996824,15cyclictest3103824-21sh21:36:161
3085115996823,11cyclictest0-21swapper/121:49:271
308510799681,2cyclictest0-21swapper/022:24:540
308510799681,2cyclictest0-21swapper/022:24:530
3085115996726,10cyclictest150-21pr/ttyS222:40:371
3085115996723,16cyclictest3138965-21ssh22:51:021
3085115996723,16cyclictest3138965-21ssh22:51:011
3085107996724,39cyclictest0-21swapper/022:27:310
3085115996622,15cyclictest3182604-21ssh00:23:531
3085115996622,15cyclictest3182604-21ssh00:23:531
3085115996618,14cyclictest3091285-21diskmemload23:18:411
3085107996515,14cyclictest3093149-21ssh21:13:440
3085115996451,10cyclictest3148983-21ssh23:12:101
3085115996451,10cyclictest3148983-21ssh23:12:101
3085107996424,36cyclictest3190315-21cat00:40:030
3085115996336,22cyclictest3086319-21sendmail-mta19:31:021
3085115996336,22cyclictest3086319-21sendmail-mta19:31:011
3085115996321,12cyclictest0-21swapper/123:20:061
3085107996320,37cyclictest3091270-21cron21:10:010
3085107996317,13cyclictest3144954-21ssh23:03:490
3085107996317,13cyclictest3144954-21ssh23:03:480
3085115996244,15cyclictest3129268-21ssh22:30:231
3085115996244,15cyclictest3129268-21ssh22:30:231
3085107996236,24cyclictest0-21swapper/019:15:020
3085107996222,2cyclictest0-21swapper/023:44:270
308510799611,2cyclictest0-21swapper/000:21:080
308510799611,2cyclictest0-21swapper/000:21:080
3085115996045,11cyclictest3089479-21munin-run20:35:011
3085107996038,15cyclictest3467602-21systemd-journal20:04:590
308510799601,55cyclictest0-21swapper/021:30:030
3085115995944,12cyclictest3090217-21sshd20:46:291
3085115995942,13cyclictest3467602-21systemd-journal20:21:191
3085115995845,10cyclictest3088723-21munin-run20:20:001
3085115995844,11cyclictest3087383-21munin-run19:55:011
3085115995843,10cyclictest3086096-21munin-run19:30:011
308510799582,13cyclictest3087625-21modprobe20:00:010
308510799582,13cyclictest3087625-21modprobe20:00:010
3085107995817,5cyclictest19350irq/34-mmc120:10:010
308512099577,25cyclictest3110408-21ssh21:50:032
3085120995745,5cyclictest3096051-21kworker/2:323:43:102
3085120995648,5cyclictest3096051-21kworker/2:300:07:052
3085120995645,7cyclictest291rcuc/200:20:012
3085115995632,18cyclictest3085553-21cron19:20:001
308510799562,15cyclictest3090638-21latency_hist20:55:020
308510799561,51cyclictest0-21swapper/019:48:300
308512099551,9cyclictest0-21swapper/200:31:152
3085115995542,10cyclictest3091046-21munin-run21:05:001
3085115995523,10cyclictest364-21snmpd19:13:161
3085115995435,15cyclictest364-21snmpd19:40:121
3085115995433,16cyclictest3091303-21systemd-run21:10:011
3085107995415,6cyclictest16850irq/134-800000021:28:280
308510799541,23cyclictest0-21swapper/019:30:030
3085121995334,14cyclictest275-21dbus-daemon20:50:013
308512099534,7cyclictest131rcu_preempt20:20:002
3085120995344,6cyclictest3147814-21kworker/2:223:30:262
3085120995344,5cyclictest3090467-21kworker/2:121:28:322
3085120995344,5cyclictest3090467-21kworker/2:121:28:322
3085120995340,9cyclictest30-21ksoftirqd/223:21:552
3085115995342,8cyclictest3467602-21systemd-journal20:56:081
3085115995341,9cyclictest3086570-21sshd19:36:291
3085115995341,8cyclictest3087892-21sshd20:02:301
3085115995340,10cyclictest3467602-21systemd-journal20:43:381
3085115995332,17cyclictest3089698-21sshd20:39:291
3085107995311,10cyclictest12-21ksoftirqd/020:30:020
3085107995311,10cyclictest12-21ksoftirqd/020:30:020
3085121995216,11cyclictest131rcu_preempt22:40:013
3085120995244,5cyclictest3096051-21kworker/2:322:09:232
3085120995243,6cyclictest3096051-21kworker/2:321:57:062
3085120995243,6cyclictest3090467-21kworker/2:122:35:162
3085120995242,6cyclictest3096051-21kworker/2:323:04:112
3085120995242,6cyclictest3096051-21kworker/2:323:04:112
3085121995136,11cyclictest275-21dbus-daemon22:00:013
3085120995143,5cyclictest3147814-21kworker/2:200:20:212
3085120995143,5cyclictest3147814-21kworker/2:200:20:212
3085120995143,5cyclictest3096051-21kworker/2:323:54:422
3085120995143,5cyclictest3096051-21kworker/2:323:08:322
3085120995143,5cyclictest3096051-21kworker/2:322:20:482
3085120995143,5cyclictest3096051-21kworker/2:322:20:482
3085120995143,5cyclictest3096051-21kworker/2:321:30:322
3085120995143,5cyclictest3090467-21kworker/2:122:29:202
3085120995143,5cyclictest3090467-21kworker/2:122:17:262
3085120995143,5cyclictest3090467-21kworker/2:122:17:252
3085120995143,5cyclictest3090467-21kworker/2:121:23:152
3085120995142,6cyclictest3147814-21kworker/2:200:37:512
3085120995142,6cyclictest3147814-21kworker/2:200:28:142
3085120995141,7cyclictest3096051-21kworker/2:322:54:122
3085120995141,7cyclictest3096051-21kworker/2:322:54:122
3085120995141,6cyclictest3096051-21kworker/2:322:04:552
3085115995139,9cyclictest3089214-21munin-run20:30:011
3085115995139,9cyclictest3089214-21munin-run20:30:001
308510799513,16cyclictest3087385-21cut19:54:590
3085121995042,5cyclictest3085160-21latency_hist19:10:013
3085120995043,5cyclictest3147814-21kworker/2:223:36:452
3085120995043,5cyclictest3147814-21kworker/2:223:36:452
3085120995042,5cyclictest3147814-21kworker/2:223:25:392
3085120995042,5cyclictest3147814-21kworker/2:223:17:072
3085120995042,5cyclictest3147814-21kworker/2:200:11:282
3085120995042,5cyclictest3096051-21kworker/2:323:55:052
3085120995042,5cyclictest3096051-21kworker/2:323:45:232
3085120995042,5cyclictest3096051-21kworker/2:323:45:232
3085120995042,5cyclictest3096051-21kworker/2:323:11:572
3085120995042,5cyclictest3096051-21kworker/2:323:11:572
3085120995042,5cyclictest3096051-21kworker/2:322:48:072
3085120995042,5cyclictest3096051-21kworker/2:300:01:132
3085120995042,5cyclictest3090467-21kworker/2:122:11:282
3085120995042,5cyclictest3090467-21kworker/2:121:47:282
3085120995042,5cyclictest3090467-21kworker/2:121:15:432
3085120995042,5cyclictest2843130-21kworker/2:119:31:052
3085120995042,5cyclictest2843130-21kworker/2:119:31:052
3085120995041,6cyclictest3085316-21kworker/2:020:06:292
3085107995012,11cyclictest12-21ksoftirqd/019:45:010
308510799501,16cyclictest3086025-21latency_hist19:25:030
3085121994940,5cyclictest3090755-21sh21:00:013
3085121994926,15cyclictest275-21dbus-daemon21:10:013
3085120994942,4cyclictest3085316-21kworker/2:019:49:222
3085120994941,5cyclictest3096051-21kworker/2:322:56:142
3085120994941,5cyclictest3090467-21kworker/2:122:40:592
3085120994941,5cyclictest3090467-21kworker/2:122:34:392
3085120994941,5cyclictest3090467-21kworker/2:122:34:392
3085120994941,5cyclictest3090467-21kworker/2:121:40:112
3085120994941,5cyclictest3090467-21kworker/2:121:37:172
3085120994941,5cyclictest3090467-21kworker/2:121:37:172
3085120994941,5cyclictest3085316-21kworker/2:021:09:052
3085120994941,5cyclictest3085316-21kworker/2:020:50:262
308512099492,6cyclictest0-21swapper/221:10:152
3085115994929,15cyclictest311-21rs:main1
3085107994910,9cyclictest1-21systemd20:15:010
3085107994910,9cyclictest1-21systemd20:15:000
308512199484,3cyclictest131rcu_preempt22:44:433
3085121994830,12cyclictest3088693-21systemctl20:20:003
3085120994840,5cyclictest3085316-21kworker/2:021:03:262
3085120994839,5cyclictest3085316-21kworker/2:020:13:082
3085120994839,5cyclictest3085316-21kworker/2:020:13:072
3085115994831,13cyclictest3467562-21sshd19:22:281
308512199475,32cyclictest3085544-21kworker/u8:319:20:013
308512199473,3cyclictest131rcu_preempt23:47:263
308512199473,3cyclictest131rcu_preempt23:47:263
308512099477,6cyclictest131rcu_preempt19:10:022
3085120994739,5cyclictest3086903-21kworker/2:219:59:182
3085120994739,5cyclictest3086903-21kworker/2:219:59:172
3085120994739,5cyclictest3085316-21kworker/2:020:40:092
3085120994739,5cyclictest3085316-21kworker/2:020:31:212
3085120994739,5cyclictest3085316-21kworker/2:020:25:492
3085120994739,5cyclictest3085316-21kworker/2:020:25:482
3085120994739,5cyclictest2843130-21kworker/2:119:17:022
308512099471,12cyclictest30-21ksoftirqd/219:39:592
308510799473,16cyclictest3467602-21systemd-journal19:39:590
3085107994726,15cyclictest3467602-21systemd-journal20:40:010
3085121994637,7cyclictest37-21ksoftirqd/322:05:013
3085121994630,12cyclictest1-21systemd23:35:003
3085120994638,5cyclictest3086903-21kworker/2:220:01:432
3085120994638,5cyclictest3085316-21kworker/2:020:56:022
3085120994638,5cyclictest3085316-21kworker/2:020:45:332
3085120994638,5cyclictest3085316-21kworker/2:019:42:072
3085120994638,5cyclictest2843130-21kworker/2:119:26:132
3085120994636,6cyclictest2843130-21kworker/2:119:21:152
308510799462,10cyclictest3467602-21systemd-journal20:56:450
308512199453,3cyclictest131rcu_preempt22:17:373
308512199453,3cyclictest131rcu_preempt22:17:363
308512199452,39cyclictest37-21ksoftirqd/323:04:413
308512199452,39cyclictest37-21ksoftirqd/323:04:413
308512199452,39cyclictest1-21systemd21:25:013
3085120994538,4cyclictest3085316-21kworker/2:020:37:412
3085120994538,4cyclictest3085316-21kworker/2:019:51:302
3085120994537,5cyclictest3085316-21kworker/2:020:20:062
3085120994537,5cyclictest2843130-21kworker/2:119:10:032
3085121994434,7cyclictest3106334-21sshd21:41:293
3085121994428,12cyclictest37-21ksoftirqd/321:20:003
3085115994423,16cyclictest3085216-21latency_hist19:10:011
3085107994434,5cyclictest12-21ksoftirqd/021:05:010
308510799442,6cyclictest3467602-21systemd-journal19:19:590
3085107994417,4cyclictest1-21systemd20:50:000
308510799441,39cyclictest3086331-21sshd19:34:290
308510799441,39cyclictest3086331-21sshd19:34:280
308510799441,39cyclictest0-21swapper/020:44:090
308512199435,3cyclictest131rcu_preempt00:03:513
3085121994328,11cyclictest3467602-21systemd-journal23:39:293
3085121994328,11cyclictest3467602-21systemd-journal23:39:293
308512199431,3cyclictest131rcu_preempt21:27:503
308512199431,3cyclictest131rcu_preempt21:27:493
308510799433,12cyclictest3089450-21modprobe20:34:280
3085121994225,13cyclictest3467602-21systemd-journal00:40:013
3085121994225,12cyclictest3129066-21date22:30:033
3085121994222,16cyclictest275-21dbus-daemon23:20:003
308512199421,37cyclictest37-21ksoftirqd/322:23:463
308512199421,37cyclictest37-21ksoftirqd/322:23:453
308510799421,37cyclictest3088737-21latency_hist20:20:010
308512199414,3cyclictest0-21swapper/321:11:423
3085121994135,4cyclictest3175949-1kworker/3:0H00:15:033
3085121994132,6cyclictest309-21in:imuxsock00:25:033
3085121994131,7cyclictest37-21ksoftirqd/323:43:213
3085121994131,7cyclictest37-21ksoftirqd/323:15:003
3085121994131,7cyclictest37-21ksoftirqd/323:15:003
3085121994131,6cyclictest3143054-21sh23:00:003
3085121994130,7cyclictest37-21ksoftirqd/321:49:523
3085121994129,9cyclictest3103112-21diskmemload21:35:003
3085121994128,10cyclictest275-21dbus-daemon20:25:483
3085121994128,10cyclictest275-21dbus-daemon20:25:473
3085121994126,12cyclictest37-21ksoftirqd/323:55:003
3085121994126,11cyclictest37-21ksoftirqd/323:21:333
3085121994123,14cyclictest1-21systemd19:15:013
3085121994122,12cyclictest37-21ksoftirqd/321:39:383
3085121994122,12cyclictest37-21ksoftirqd/321:39:373
308512199411,4cyclictest131rcu_preempt00:24:323
308512199411,4cyclictest131rcu_preempt00:24:323
308510799411,35cyclictest3467718-21lldpd20:22:590
3085121994029,9cyclictest37-21ksoftirqd/300:30:333
3085121994028,9cyclictest37-21ksoftirqd/320:00:013
3085121994028,9cyclictest37-21ksoftirqd/320:00:003
3085121994027,9cyclictest3087883-21sshd20:00:483
308512199401,3cyclictest131rcu_preempt21:53:493
3085121994011,15cyclictest3085101-21cyclictest20:25:023
308512199394,3cyclictest131rcu_preempt22:05:303
3085121993928,9cyclictest37-21ksoftirqd/323:06:113
3085121993927,10cyclictest37-21ksoftirqd/323:55:103
3085121993918,15cyclictest275-21dbus-daemon22:15:013
308512199391,11cyclictest3140872-21cut22:55:033
308512199391,11cyclictest3140872-21cut22:55:023
3085121993826,9cyclictest37-21ksoftirqd/322:30:263
3085121993826,9cyclictest37-21ksoftirqd/322:30:253
3085121993816,15cyclictest275-21dbus-daemon19:30:013
3085121993812,5cyclictest131rcu_preempt22:47:173
3085121993728,5cyclictest3086875-21sh19:45:003
3085121993726,8cyclictest3089962-21sshd20:40:593
3085121993721,10cyclictest275-21dbus-daemon19:25:003
3085121993720,13cyclictest275-21dbus-daemon00:20:013
3085121993711,6cyclictest131rcu_preempt00:09:333
308512199365,5cyclictest131rcu_preempt20:10:003
308512199365,3cyclictest131rcu_preempt20:35:013
3085121993627,5cyclictest309-21in:imuxsock19:54:593
308512199362,4cyclictest131rcu_preempt20:40:003
3085121993523,9cyclictest3087107-21sshd19:47:293
308512199344,3cyclictest131rcu_preempt19:40:023
3085121993421,10cyclictest37-21ksoftirqd/320:55:013
308510799341,13cyclictest3085146-21latency_hist19:10:020
3085121993318,12cyclictest275-21dbus-daemon19:35:013
3085121993318,12cyclictest275-21dbus-daemon19:35:003
3085121993220,9cyclictest37-21ksoftirqd/320:15:013
3085121993220,9cyclictest37-21ksoftirqd/320:15:003
3085121993116,10cyclictest275-21dbus-daemon21:05:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional