You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-07 - 20:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Jun 07, 2026 12:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40994299911561,44cyclictest131rcu_preempt10:50:011
40994259910887,6cyclictest16850irq/134-800000011:50:020
40994259910290,3cyclictest0-21swapper/008:15:020
40994259910290,3cyclictest0-21swapper/008:15:010
4099425999774,21cyclictest12-21ksoftirqd/008:50:020
4099425998673,3cyclictest0-21swapper/008:25:020
409942599832,37cyclictest4136629-21cat10:20:020
4099425998230,11cyclictest4163895-21ssh11:22:100
4099425998228,13cyclictest4185292-21ssh12:10:320
4099425998218,17cyclictest16850irq/134-800000009:38:470
4099432997428,41cyclictest3467602-21systemd-journal10:55:002
4099425997436,4cyclictest16850irq/134-800000011:39:590
409942599742,17cyclictest4143174-21cat10:35:020
409942599734,7cyclictest4112166-21modprobe09:24:590
4099425997318,6cyclictest16850irq/134-800000009:51:210
4099425997318,18cyclictest16850irq/134-800000010:47:440
4099425997314,17cyclictest16850irq/134-800000010:02:300
4099425997222,42cyclictest3467602-21systemd-journal11:45:010
4099432997126,40cyclictest4188866-21ssh12:18:552
4099425997116,5cyclictest16850irq/134-800000011:25:440
4099425997115,17cyclictest16850irq/134-800000012:26:520
4099425997024,13cyclictest4137214-21ssh10:21:170
4099425996815,5cyclictest16850irq/134-800000012:19:310
409942599681,36cyclictest4147431-21unin-run10:45:000
4099425996715,18cyclictest16850irq/134-800000011:50:180
4099432996626,36cyclictest3467602-21systemd-journal09:20:032
4099432996625,5cyclictest309-21in:imuxsock12:30:002
4099425996633,6cyclictest16950irq/224-800000010:00:010
4099425996633,6cyclictest16950irq/224-800000010:00:000
4099425996510,18cyclictest16850irq/134-800000009:12:380
4099432996422,13cyclictest30-21ksoftirqd/209:36:482
409943299633,19cyclictest4182774-21wc12:05:022
4099425996331,11cyclictest4168998-21ssh11:33:580
4099425996328,13cyclictest4109749-21ssh09:19:370
4099432996120,8cyclictest30-21ksoftirqd/207:40:012
4099432996120,4cyclictest4113068-21sh09:26:422
4099432996023,9cyclictest30-21ksoftirqd/210:43:492
4099432996021,5cyclictest4174575-21ssh11:46:182
4099432996019,4cyclictest4141916-21sh10:32:162
4099425996033,11cyclictest4181957-21ssh12:03:120
4099432995927,28cyclictest30-21ksoftirqd/212:11:322
4099432995924,13cyclictest30-21ksoftirqd/210:04:302
4099432995921,5cyclictest4173323-21sh11:43:362
4099432995920,7cyclictest30-21ksoftirqd/209:49:372
4099432995920,7cyclictest30-21ksoftirqd/209:49:362
409942599598,32cyclictest19350irq/34-mmc111:00:030
4099432995828,27cyclictest30-21ksoftirqd/208:35:012
4099432995826,8cyclictest30-21ksoftirqd/210:10:372
4099432995824,5cyclictest4108615-21sh09:16:412
4099432995819,6cyclictest4168356-21sh11:32:212
4099432995817,9cyclictest30-21ksoftirqd/212:10:012
4099425995820,8cyclictest4150899-21ssh10:52:490
4099432995723,4cyclictest4105601-21diskmemload11:05:032
4099432995722,4cyclictest4105601-21diskmemload11:25:022
4099432995718,8cyclictest30-21ksoftirqd/212:35:392
4099432995718,8cyclictest30-21ksoftirqd/212:35:382
4099432995718,7cyclictest4116365-21ssh09:34:242
4099425995738,5cyclictest16850irq/134-800000011:20:000
4099432995622,4cyclictest4154307-21sh11:00:072
4099432995620,4cyclictest4105409-21latency_hist09:05:012
4099425995619,6cyclictest16850irq/134-800000012:00:010
4099432995527,24cyclictest4161520-21sh11:16:362
409943299552,23cyclictest4180087-21ssh11:59:022
4099432995520,8cyclictest30-21ksoftirqd/210:08:072
4099432995520,8cyclictest30-21ksoftirqd/210:08:072
4099425995525,14cyclictest4139699-21ssh10:27:070
4099425995523,28cyclictest12-21ksoftirqd/012:36:440
4099425995523,28cyclictest12-21ksoftirqd/012:36:440
4099432995426,7cyclictest30-21ksoftirqd/210:37:572
4099432995421,8cyclictest30-21ksoftirqd/211:10:312
4099432995421,8cyclictest30-21ksoftirqd/211:10:302
4099432995417,12cyclictest275-21dbus-daemon10:30:022
4099432995415,6cyclictest0-21swapper/208:20:002
4099425995419,31cyclictest16850irq/134-800000012:31:010
4099425995418,6cyclictest16850irq/134-800000010:38:490
4099425995418,32cyclictest16850irq/134-800000009:44:410
4099425995417,19cyclictest16850irq/134-800000009:30:150
4099425995410,40cyclictest151rcuc/009:10:020
409943299532,21cyclictest4154026-21munin-run11:00:002
4099432995319,7cyclictest4059112-1kworker/u9:009:13:372
4099429995334,15cyclictest275-21dbus-daemon08:00:021
4099429995332,15cyclictest3467602-21systemd-journal12:25:001
4099425995318,17cyclictest16850irq/134-800000009:46:180
4099425995318,17cyclictest16850irq/134-800000009:46:170
4099425995317,6cyclictest16850irq/134-800000010:59:250
4099425995317,18cyclictest16850irq/134-800000012:22:260
4099435995233,14cyclictest3467602-21systemd-journal11:55:013
4099435995231,16cyclictest3467602-21systemd-journal11:19:593
4099435995216,6cyclictest131rcu_preempt12:00:003
4099425995218,18cyclictest16850irq/134-800000011:12:090
4099425995218,18cyclictest16850irq/134-800000011:12:090
4099425995217,6cyclictest16850irq/134-800000010:06:490
4099425995217,6cyclictest16850irq/134-800000010:06:480
4099425995216,17cyclictest16850irq/134-800000012:05:080
4099425995215,6cyclictest16850irq/134-800000009:27:140
4099432995135,8cyclictest4112209-21kworker/2:010:23:292
4099432995122,5cyclictest4105601-21diskmemload11:52:212
4099432995120,6cyclictest30-21ksoftirqd/208:56:392
4099429995113,6cyclictest131rcu_preempt10:59:591
4099435995036,11cyclictest4109373-21sshd09:18:363
4099435995036,10cyclictest309-21in:imuxsock11:45:033
4099432995028,18cyclictest4166047-21sh11:26:562
4099432995027,7cyclictest30-21ksoftirqd/209:59:282
4099432995027,7cyclictest30-21ksoftirqd/209:59:272
4099432995020,9cyclictest30-21ksoftirqd/212:25:022
4099432995019,8cyclictest30-21ksoftirqd/209:54:592
4099429995036,11cyclictest3467602-21systemd-journal07:49:361
4099429995014,17cyclictest4105601-21diskmemload11:30:031
4099432994939,6cyclictest131rcu_preempt08:25:032
4099432994939,6cyclictest131rcu_preempt08:25:022
4099432994920,6cyclictest30-21ksoftirqd/209:45:022
4099432994920,4cyclictest4105601-21diskmemload10:48:472
4099429994910,5cyclictest131rcu_preempt11:15:021
4099429994910,5cyclictest131rcu_preempt11:15:021
4099425994922,11cyclictest4133951-21ssh10:14:170
4099435994831,13cyclictest275-21dbus-daemon10:50:013
4099435994830,14cyclictest4104048-21tr08:40:003
409942999486,9cyclictest23-21ksoftirqd/109:15:001
4099429994830,14cyclictest275-21dbus-daemon09:00:011
4099429994824,6cyclictest23-21ksoftirqd/111:58:511
4099435994732,11cyclictest3467602-21systemd-journal12:17:383
4099435994730,12cyclictest3467602-21systemd-journal07:20:013
409943299479,11cyclictest30-21ksoftirqd/207:45:352
4099432994739,5cyclictest4112209-21kworker/2:011:38:092
4099432994737,6cyclictest4116607-21kworker/2:210:17:012
4099432994734,10cyclictest3467602-21systemd-journal12:32:362
409942999475,35cyclictest4100410-21grep07:30:011
409942999475,35cyclictest4100410-21grep07:30:001
4099429994728,15cyclictest23-21ksoftirqd/109:40:021
409943599463,3cyclictest131rcu_preempt10:44:233
4099435994628,12cyclictest275-21dbus-daemon11:30:003
409943299462,6cyclictest131rcu_preempt07:25:012
4099432994618,8cyclictest30-21ksoftirqd/207:55:012
409942999464,3cyclictest131rcu_preempt09:45:581
409942999464,3cyclictest131rcu_preempt09:45:571
409942999463,37cyclictest275-21dbus-daemon07:20:021
4099429994630,11cyclictest1-21systemd08:40:001
4099429994626,15cyclictest1-21systemd07:40:011
4099429994625,16cyclictest275-21dbus-daemon07:35:001
4099429994625,15cyclictest275-21dbus-daemon12:10:011
409942599463,39cyclictest0-21swapper/008:40:010
4099425994629,13cyclictest4156629-21ssh11:05:260
4099425994629,13cyclictest4100899-21cron07:40:000
4099425994623,8cyclictest0-21swapper/008:00:010
4099432994537,5cyclictest4098615-21kworker/2:007:30:022
4099432994537,5cyclictest4098615-21kworker/2:007:30:012
4099432994530,11cyclictest1-21systemd08:45:002
4099432994530,11cyclictest1-21systemd08:45:002
4099429994536,5cyclictest4109956-21latency_hist09:20:011
4099429994534,8cyclictest23-21ksoftirqd/110:34:081
4099429994529,12cyclictest1-21systemd08:10:001
409942999452,5cyclictest131rcu_preempt07:55:021
4099425994538,4cyclictest16950irq/224-800000008:30:020
4099425994538,4cyclictest16950irq/224-800000008:30:020
409943599446,6cyclictest131rcu_preempt12:15:013
409943599444,3cyclictest131rcu_preempt10:11:543
4099435994432,10cyclictest37-21ksoftirqd/307:40:003
4099435994431,8cyclictest309-21in:imuxsock09:35:023
4099435994426,14cyclictest3467602-21systemd-journal08:20:003
4099435994426,13cyclictest1-21systemd12:40:013
4099435994426,13cyclictest1-21systemd12:40:003
4099432994436,5cyclictest4100673-21kworker/2:207:42:362
4099432994436,5cyclictest4100673-21kworker/2:207:42:352
4099429994433,7cyclictest3467562-21sshd11:41:351
4099429994432,8cyclictest23-21ksoftirqd/112:20:011
4099429994427,12cyclictest275-21dbus-daemon11:50:011
4099429994426,14cyclictest3467602-21systemd-journal11:40:011
409942999442,3cyclictest131rcu_preempt11:21:111
4099429994420,20cyclictest23-21ksoftirqd/112:28:521
4099429994410,5cyclictest131rcu_preempt12:14:281
409943599434,3cyclictest131rcu_preempt09:31:453
4099435994331,9cyclictest4156490-21sshd11:05:093
4099435994329,11cyclictest37-21ksoftirqd/307:25:013
4099435994329,10cyclictest1-21systemd09:30:023
4099435994327,11cyclictest4193868-21date12:30:023
4099435994326,13cyclictest275-21dbus-daemon10:20:003
409943299432,3cyclictest131rcu_preempt08:05:002
4099429994327,12cyclictest275-21dbus-daemon10:00:011
4099429994327,12cyclictest275-21dbus-daemon10:00:001
4099429994326,13cyclictest3467602-21systemd-journal10:20:001
4099425994336,4cyclictest16950irq/224-800000007:49:590
4099425994334,5cyclictest4104296-21cron08:44:590
4099425994334,5cyclictest4104296-21cron08:44:590
4099425994334,4cyclictest16850irq/134-800000007:35:000
4099435994232,5cyclictest4140880-21sshd10:30:013
4099435994230,10cyclictest37-21ksoftirqd/310:57:483
4099435994229,9cyclictest4160688-21systemd-run11:15:013
4099435994229,9cyclictest4160688-21systemd-run11:15:003
4099435994229,10cyclictest1-21systemd09:50:013
4099435994229,10cyclictest1-21systemd09:50:003
4099435994226,11cyclictest309-21in:imuxsock09:00:033
4099435994225,12cyclictest0-21swapper/312:25:003
4099435994218,12cyclictest4107619-21ssh09:14:423
409943299421,4cyclictest131rcu_preempt08:54:592
4099429994229,10cyclictest23-21ksoftirqd/110:36:591
4099429994220,5cyclictest23-21ksoftirqd/111:00:251
4099429994218,4cyclictest23-21ksoftirqd/111:30:341
4099429994210,4cyclictest131rcu_preempt08:30:471
4099425994233,6cyclictest19350irq/34-mmc109:02:510
4099425994227,10cyclictest3467602-21systemd-journal09:00:000
4099425994212,26cyclictest12-21ksoftirqd/008:35:010
4099435994129,9cyclictest3467602-21systemd-journal10:22:373
4099435994126,12cyclictest1-21systemd09:00:013
4099435994125,11cyclictest37-21ksoftirqd/311:20:273
409943599412,3cyclictest131rcu_preempt12:07:523
409943599411,3cyclictest131rcu_preempt10:51:433
409943299413,4cyclictest131rcu_preempt09:10:002
4099432994126,12cyclictest3467602-21systemd-journal08:10:032
4099432994126,12cyclictest3467602-21systemd-journal08:10:022
4099429994129,10cyclictest23-21ksoftirqd/110:25:011
4099429994128,9cyclictest23-21ksoftirqd/110:50:431
4099429994127,11cyclictest1-21systemd12:35:021
4099429994125,12cyclictest3467602-21systemd-journal08:54:361
409942999411,5cyclictest4129848-21ssh10:05:001
4099425994136,2cyclictest0-21swapper/007:25:020
4099425994136,2cyclictest0-21swapper/007:25:020
409943599404,3cyclictest131rcu_preempt11:35:203
4099435994027,10cyclictest37-21ksoftirqd/308:49:593
4099432994024,11cyclictest309-21in:imuxsock08:10:002
409942999404,3cyclictest131rcu_preempt12:37:541
409942999404,3cyclictest131rcu_preempt12:37:531
4099429994030,8cyclictest23-21ksoftirqd/109:21:141
4099429994026,11cyclictest23-21ksoftirqd/110:43:531
4099429994021,14cyclictest0-21swapper/109:10:011
4099429994013,4cyclictest23-21ksoftirqd/109:26:171
409942599402,4cyclictest4102992-21modprobe08:20:000
409942599401,6cyclictest4101191-21munin-run07:45:010
409942599401,6cyclictest4101191-21munin-run07:45:000
4099435993928,7cyclictest309-21in:imuxsock08:00:023
4099435993926,11cyclictest37-21ksoftirqd/312:00:033
4099435993925,11cyclictest37-21ksoftirqd/310:08:253
4099435993925,11cyclictest37-21ksoftirqd/310:08:243
409943599392,4cyclictest131rcu_preempt09:55:013
409943599391,3cyclictest131rcu_preempt10:00:013
409943599391,3cyclictest131rcu_preempt10:00:013
4099432993930,6cyclictest4099184-21kworker/2:307:15:012
4099432993927,9cyclictest30-21ksoftirqd/208:25:002
4099432993925,11cyclictest30-21ksoftirqd/207:19:532
409943299391,3cyclictest131rcu_preempt07:10:022
4099429993929,7cyclictest4139526-21sshd10:26:371
4099429993924,11cyclictest4134392-21latency_hist10:15:021
409942999392,3cyclictest131rcu_preempt10:05:481
409942999392,3cyclictest131rcu_preempt10:05:471
4099429993923,12cyclictest23-21ksoftirqd/111:15:511
4099429993915,4cyclictest23-21ksoftirqd/112:04:541
409942599394,30cyclictest0-21swapper/008:01:430
4099425993927,9cyclictest12-21ksoftirqd/008:55:010
4099435993826,9cyclictest3467602-21systemd-journal07:13:373
4099435993826,10cyclictest37-21ksoftirqd/308:35:003
4099435993824,11cyclictest37-21ksoftirqd/310:36:223
4099435993824,10cyclictest37-21ksoftirqd/309:43:473
409943599382,3cyclictest131rcu_preempt12:31:253
4099435993823,11cyclictest37-21ksoftirqd/311:43:053
4099432993824,10cyclictest275-21dbus-daemon08:40:022
409942999385,25cyclictest4104303-21modprobe08:45:001
409942999385,25cyclictest4104303-21modprobe08:44:591
409942999384,3cyclictest131rcu_preempt09:32:061
4099425993829,6cyclictest4102670-21latency_hist08:10:030
4099425993826,9cyclictest3467718-21lldpd07:13:580
4099425993822,11cyclictest3467602-21systemd-journal07:20:000
4099435993726,8cyclictest37-21ksoftirqd/307:55:003
4099435993725,9cyclictest37-21ksoftirqd/307:10:013
4099435993724,9cyclictest37-21ksoftirqd/311:00:243
4099435993724,10cyclictest37-21ksoftirqd/311:32:083
409943299372,32cyclictest275-21dbus-daemon08:50:012
4099429993727,7cyclictest23-21ksoftirqd/109:41:561
4099429993726,8cyclictest23-21ksoftirqd/108:20:011
4099429993726,6cyclictest309-21in:imuxsock11:50:361
4099429993725,10cyclictest23-21ksoftirqd/111:06:021
409943599364,3cyclictest131rcu_preempt10:04:303
4099435993626,8cyclictest37-21ksoftirqd/309:23:013
4099435993625,9cyclictest37-21ksoftirqd/307:43:363
4099435993625,9cyclictest37-21ksoftirqd/307:43:353
4099435993624,10cyclictest37-21ksoftirqd/308:30:023
4099435993624,10cyclictest37-21ksoftirqd/308:30:013
4099429993627,5cyclictest4104548-21cron08:49:591
4099429993626,7cyclictest4102918-21sed08:15:031
4099429993626,7cyclictest4102918-21sed08:15:021
4099429993618,13cyclictest1-21systemd07:25:001
4099429993615,16cyclictest275-21dbus-daemon08:05:001
4099425993626,7cyclictest4101670-21sshd07:52:370
409943599354,3cyclictest131rcu_preempt10:34:153
4099435993526,7cyclictest37-21ksoftirqd/307:35:023
4099435993522,10cyclictest37-21ksoftirqd/308:15:013
4099435993522,10cyclictest37-21ksoftirqd/308:15:003
409943299352,4cyclictest131rcu_preempt07:31:352
4099432993523,9cyclictest30-21ksoftirqd/207:55:092
409942999359,5cyclictest131rcu_preempt07:15:011
409942999355,3cyclictest131rcu_preempt09:50:491
4099429993525,7cyclictest309-21in:imuxsock08:30:001
4099429993525,7cyclictest309-21in:imuxsock08:30:001
4099429993524,9cyclictest23-21ksoftirqd/109:04:431
4099425993518,11cyclictest275-21dbus-daemon07:25:010
4099435993423,9cyclictest37-21ksoftirqd/307:50:003
4099435993422,9cyclictest284-21systemd-logind07:28:003
4099435993422,9cyclictest284-21systemd-logind07:27:593
4099435993419,12cyclictest4104277-21sshd08:40:363
4099435993419,12cyclictest4104277-21sshd08:40:353
4099429993419,11cyclictest1-21systemd08:25:021
4099435993324,7cyclictest37-21ksoftirqd/308:03:343
4099429993320,10cyclictest23-21ksoftirqd/107:45:001
4099429993320,10cyclictest23-21ksoftirqd/107:45:001
409943599323,3cyclictest131rcu_preempt08:55:013
409943599323,21cyclictest4105588-21cron09:10:003
4099435993221,6cyclictest0-21swapper/308:25:013
409943599312,3cyclictest131rcu_preempt08:10:023
4099429993121,7cyclictest23-21ksoftirqd/107:10:011
409942599303,3cyclictest131rcu_preempt07:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional