You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 23:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Sep 11, 2026 12:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
99300899992,8cyclictest23-21ksoftirqd/109:00:031
993005998682,2cyclictest0-21swapper/010:05:020
99300599851,80cyclictest1042639-21ssh10:43:300
993013998215,47cyclictest1050424-21ssh11:00:032
99300599811,76cyclictest1029964-21ssh10:15:570
993005998013,6cyclictest16850irq/134-800000011:17:380
993005997929,45cyclictest1034849-21ssh10:26:340
993005997929,45cyclictest1034849-21ssh10:26:340
993005997921,40cyclictest1063880-21ssh11:29:310
993005997919,11cyclictest1009078-21ssh09:31:060
993005997915,13cyclictest1020974-21kworker/u8:310:10:000
993005997829,32cyclictest1060113-21ssh11:21:020
993005997726,47cyclictest1040733-21ssh10:39:260
993005997726,47cyclictest1040733-21ssh10:39:250
993005997726,32cyclictest1020640-21ssh09:56:040
993005997623,36cyclictest1014065-21ssh09:42:010
993005997622,37cyclictest1028459-21ssh10:12:540
993005997524,34cyclictest1055344-21ssh11:10:420
993005997518,13cyclictest1018723-21ssh09:51:520
993005997425,31cyclictest1077492-21ssh11:58:520
993005997424,34cyclictest0-21swapper/011:30:540
99300599741,56cyclictest1011993-21ssh09:37:370
993005997322,34cyclictest1037032-21ssh10:31:140
99300599731,68cyclictest0-21swapper/010:58:380
993005997315,11cyclictest1047400-21ssh10:54:070
993005997231,36cyclictest0-21swapper/011:08:560
993005997225,30cyclictest1002605-21ssh09:17:220
993005997220,13cyclictest1032956-21ssh10:22:290
993005997119,18cyclictest1004443-21ssh09:21:040
993005997119,17cyclictest1016850-21ssh09:48:030
993005997115,12cyclictest1067875-21ssh11:38:100
993005997031,34cyclictest16950irq/224-800000009:10:010
993005997022,43cyclictest0-21swapper/010:49:530
99300599701,54cyclictest0-21swapper/012:37:540
993013996956,8cyclictest995593-21kworker/2:209:39:592
99300599691,2cyclictest0-21swapper/012:30:320
993005996828,36cyclictest1080790-21ssh12:05:340
993005996822,42cyclictest1090791-21ssh12:27:480
993005996822,42cyclictest1090791-21ssh12:27:480
993005996822,42cyclictest0-21swapper/009:11:470
993005996817,14cyclictest1070950-21ssh11:44:470
99300599681,62cyclictest0-21swapper/012:15:010
993005996624,38cyclictest1074477-21ssh11:52:040
99300599652,58cyclictest1071428-21ssh11:45:180
993005996518,6cyclictest16850irq/134-800000009:26:230
993005996418,14cyclictest995990-21cut08:05:010
993005996319,16cyclictest1088206-21ssh12:21:370
993005996314,42cyclictest0-21swapper/011:05:020
993005996314,42cyclictest0-21swapper/011:05:010
993005996312,11cyclictest1078977-21ssh12:01:520
993005996215,10cyclictest275-21dbus-daemon08:25:010
99300599619,10cyclictest1085341-21ssh12:15:200
99300599619,10cyclictest1085341-21ssh12:15:200
993005996117,5cyclictest993101-21latency_hist07:10:020
993005995940,15cyclictest16950irq/224-800000007:40:010
993005995910,15cyclictest996345-21grep08:15:020
993018995617,4cyclictest1027059-21latency_hist10:10:023
993013995647,6cyclictest995593-21kworker/2:209:27:272
993005995613,5cyclictest0-21swapper/007:50:000
993008995537,14cyclictest1-21systemd08:15:011
993005995537,14cyclictest3467602-21systemd-journal08:55:010
99300599553,16cyclictest994264-21munin-run07:35:000
993005995529,22cyclictest994153-21sed07:30:020
993005995518,9cyclictest275-21dbus-daemon08:45:010
993005995518,9cyclictest275-21dbus-daemon08:45:010
993018995437,14cyclictest1050216-21systemctl11:00:013
99301899543,6cyclictest131rcu_preempt12:10:003
993013995446,5cyclictest1027022-21kworker/2:310:21:332
99300899542,6cyclictest131rcu_preempt11:15:021
993005995422,28cyclictest12-21ksoftirqd/007:45:010
993005995420,30cyclictest995628-21latency_hist08:00:020
993013995337,12cyclictest3467602-21systemd-journal10:45:022
993013995244,5cyclictest1027022-21kworker/2:310:52:312
993008995236,12cyclictest1051397-21sshd11:02:231
993008995236,12cyclictest1051397-21sshd11:02:231
993008995235,13cyclictest1-21systemd10:50:011
99300899522,6cyclictest131rcu_preempt10:40:011
99300899522,6cyclictest131rcu_preempt10:40:011
99301899519,7cyclictest131rcu_preempt09:10:013
993013995144,4cyclictest995593-21kworker/2:211:28:012
993013995143,5cyclictest995593-21kworker/2:209:58:552
993013995142,5cyclictest1027022-21kworker/2:311:21:062
99300899516,8cyclictest23-21ksoftirqd/112:15:001
993008995138,9cyclictest1-21systemd10:05:001
993005995122,25cyclictest998149-21latency_hist08:50:030
993005995111,7cyclictest997633-21sendmail08:39:590
99301899509,6cyclictest131rcu_preempt10:35:003
99301899503,10cyclictest37-21ksoftirqd/311:11:523
993013995042,5cyclictest995593-21kworker/2:212:30:222
993013995042,5cyclictest1027022-21kworker/2:310:17:432
993013995032,6cyclictest0-21swapper/209:55:022
993008995026,15cyclictest1093563-21ssh12:33:541
993005995020,8cyclictest12-21ksoftirqd/008:55:220
993005995015,6cyclictest12-21ksoftirqd/007:22:240
99301899495,6cyclictest131rcu_preempt10:00:003
993013994941,5cyclictest995593-21kworker/2:210:40:302
993013994941,5cyclictest995593-21kworker/2:209:32:432
993013994941,5cyclictest995593-21kworker/2:209:22:052
993013994941,5cyclictest995593-21kworker/2:208:20:572
993013994941,5cyclictest1027022-21kworker/2:311:33:512
99300899497,5cyclictest131rcu_preempt09:45:011
99300899494,6cyclictest131rcu_preempt10:30:021
99300899494,6cyclictest131rcu_preempt10:30:021
99300599493,40cyclictest0-21swapper/008:20:010
993005994923,23cyclictest0-21swapper/008:30:020
993005994923,23cyclictest0-21swapper/008:30:020
99301899487,5cyclictest131rcu_preempt11:49:213
993018994832,12cyclictest1068679-21cat11:40:003
993013994842,3cyclictest1027022-21kworker/2:311:09:452
993013994841,5cyclictest1027022-21kworker/2:311:12:422
993013994840,5cyclictest995593-21kworker/2:211:45:112
993013994840,5cyclictest995593-21kworker/2:211:44:342
993013994840,5cyclictest995593-21kworker/2:209:15:462
993013994840,5cyclictest1008351-21kworker/2:109:41:232
993013994839,6cyclictest995593-21kworker/2:212:37:202
993013994839,6cyclictest995593-21kworker/2:211:58:452
993013994839,6cyclictest995593-21kworker/2:209:46:432
99301399482,4cyclictest131rcu_preempt12:25:002
993013994811,5cyclictest131rcu_preempt08:30:012
993013994811,5cyclictest131rcu_preempt08:30:012
993008994829,13cyclictest275-21dbus-daemon10:35:011
993005994816,10cyclictest1-21systemd07:55:010
99301899478,5cyclictest131rcu_preempt10:43:203
99301899472,6cyclictest131rcu_preempt07:15:013
993018994724,6cyclictest37-21ksoftirqd/310:46:283
993013994739,5cyclictest995593-21kworker/2:210:36:002
993013994739,5cyclictest995593-21kworker/2:210:36:002
993013994739,5cyclictest1027022-21kworker/2:312:02:112
993013994739,5cyclictest1027022-21kworker/2:310:14:442
993013994739,5cyclictest1008351-21kworker/2:110:08:122
993013994739,5cyclictest1008351-21kworker/2:110:02:362
993013994738,6cyclictest985962-21kworker/2:107:40:472
993013994738,5cyclictest995593-21kworker/2:210:34:362
993018994630,12cyclictest1096394-21cron12:40:003
993018994622,5cyclictest37-21ksoftirqd/311:08:213
993018994622,5cyclictest37-21ksoftirqd/310:15:003
993013994639,4cyclictest995593-21kworker/2:212:28:082
993013994639,4cyclictest995593-21kworker/2:212:28:072
993013994638,5cyclictest995593-21kworker/2:212:10:132
993013994638,5cyclictest995593-21kworker/2:211:51:452
993013994638,5cyclictest1082658-21kworker/2:112:18:232
993013994638,5cyclictest1082658-21kworker/2:112:18:232
99300899464,5cyclictest131rcu_preempt11:30:001
993008994635,9cyclictest23-21ksoftirqd/108:30:021
993008994635,9cyclictest23-21ksoftirqd/108:30:021
993008994629,11cyclictest309-21in:imuxsock08:25:011
993005994615,9cyclictest998975-21sed09:05:010
99300599461,41cyclictest0-21swapper/008:30:030
99301899454,5cyclictest131rcu_preempt11:31:513
993018994521,5cyclictest37-21ksoftirqd/310:54:463
993018994521,18cyclictest1-21systemd09:15:013
993018994511,5cyclictest131rcu_preempt10:30:023
993018994511,5cyclictest131rcu_preempt10:30:023
993013994538,5cyclictest1027022-21kworker/2:310:28:232
993013994538,5cyclictest1027022-21kworker/2:310:28:232
993013994537,5cyclictest995593-21kworker/2:209:15:012
993013994536,5cyclictest985962-21kworker/2:107:55:012
993008994528,11cyclictest1077993-21systemctl12:00:001
993008994520,20cyclictest23-21ksoftirqd/109:34:041
99300899451,10cyclictest999192-21modprobe09:10:011
99300599451,2cyclictest0-21swapper/008:07:240
993018994432,8cyclictest1-21systemd09:50:013
993018994429,10cyclictest1-21systemd09:20:013
993018994427,13cyclictest37-21ksoftirqd/308:40:003
993018994421,5cyclictest37-21ksoftirqd/312:16:583
993018994421,5cyclictest37-21ksoftirqd/312:16:583
99301899441,10cyclictest37-21ksoftirqd/311:22:333
993018994410,5cyclictest131rcu_preempt12:34:313
993013994437,4cyclictest1027022-21kworker/2:312:06:422
993008994433,8cyclictest996584-21cron08:20:001
993008994430,10cyclictest3467602-21systemd-journal11:50:031
993008994422,19cyclictest1043284-21kworker/u8:211:15:031
99301899438,6cyclictest131rcu_preempt09:30:203
993018994329,10cyclictest1-21systemd12:25:003
993018994329,10cyclictest1-21systemd07:35:013
993018994321,6cyclictest37-21ksoftirqd/309:37:463
993018994319,5cyclictest37-21ksoftirqd/311:41:403
993018994318,20cyclictest37-21ksoftirqd/311:51:493
993018994312,5cyclictest131rcu_preempt12:28:363
993018994312,5cyclictest131rcu_preempt12:28:363
993013994336,5cyclictest1027022-21kworker/2:311:02:002
993013994336,5cyclictest1027022-21kworker/2:311:02:002
993013994335,5cyclictest985962-21kworker/2:108:07:002
99300899433,6cyclictest131rcu_preempt10:00:001
993008994329,11cyclictest1013037-21sed09:40:021
993008994328,11cyclictest1054929-21date11:10:021
993008994327,12cyclictest1-21systemd10:20:011
99300899432,5cyclictest131rcu_preempt11:00:011
99300899432,4cyclictest131rcu_preempt08:55:011
993008994322,12cyclictest1067301-21ssh11:36:451
99300899431,5cyclictest131rcu_preempt11:25:021
99301899426,7cyclictest37-21ksoftirqd/310:16:523
99301899423,5cyclictest131rcu_preempt10:01:233
993018994229,10cyclictest37-21ksoftirqd/309:29:373
993018994226,12cyclictest37-21ksoftirqd/310:24:323
993018994222,16cyclictest997889-21cron08:45:013
993018994222,16cyclictest997889-21cron08:45:013
993018994220,5cyclictest37-21ksoftirqd/311:03:433
993018994220,5cyclictest37-21ksoftirqd/311:03:423
993018994219,5cyclictest37-21ksoftirqd/311:56:303
993018994213,18cyclictest131rcu_preempt10:39:223
993018994213,18cyclictest131rcu_preempt10:39:223
99301399428,7cyclictest131rcu_preempt08:20:002
993013994234,5cyclictest1027022-21kworker/2:311:39:042
993013994231,8cyclictest993423-21sshd07:15:232
993013994229,10cyclictest30-21ksoftirqd/208:15:022
99300899424,5cyclictest131rcu_preempt12:40:001
99300899423,6cyclictest131rcu_preempt07:20:001
993008994228,10cyclictest1003784-21tr09:20:001
993008994224,8cyclictest23-21ksoftirqd/107:40:001
993008994221,9cyclictest23-21ksoftirqd/111:42:411
993008994218,10cyclictest23-21ksoftirqd/110:07:131
993008994216,6cyclictest1091760-21cron12:30:001
993008994216,6cyclictest1091760-21cron12:30:001
99300899421,5cyclictest131rcu_preempt12:17:231
99300899421,5cyclictest131rcu_preempt12:17:231
99300899421,3cyclictest131rcu_preempt09:47:241
99301899418,5cyclictest131rcu_preempt09:44:293
99301899413,5cyclictest131rcu_preempt11:15:323
99301899413,3cyclictest131rcu_preempt09:24:243
993018994129,8cyclictest995032-21cron07:49:593
993018994127,11cyclictest997051-21sed08:25:023
993018994126,11cyclictest995840-21wc08:05:023
993018994125,11cyclictest275-21dbus-daemon08:00:013
993018994110,7cyclictest131rcu_preempt08:11:123
993018994110,5cyclictest131rcu_preempt11:30:003
993013994127,11cyclictest30-21ksoftirqd/208:51:452
993013994127,10cyclictest1-21systemd09:10:012
993013994126,11cyclictest995023-21cron07:50:002
99300899412,5cyclictest131rcu_preempt10:55:021
993008994125,8cyclictest23-21ksoftirqd/111:46:101
993008994125,8cyclictest23-21ksoftirqd/110:14:151
993008994125,11cyclictest3467602-21systemd-journal08:40:001
99300899411,4cyclictest131rcu_preempt09:14:491
99300899411,3cyclictest131rcu_preempt11:32:241
99301899408,5cyclictest131rcu_preempt09:00:013
993018994019,4cyclictest37-21ksoftirqd/309:53:003
993013994032,5cyclictest985962-21kworker/2:108:00:272
993013994032,5cyclictest1027022-21kworker/2:311:15:402
993013994029,7cyclictest994502-21modprobe07:40:012
99300899408,6cyclictest131rcu_preempt09:30:011
99300899406,5cyclictest131rcu_preempt09:53:231
993008994028,9cyclictest23-21ksoftirqd/110:44:041
993008994028,7cyclictest1082643-21latency_hist12:10:021
993008994026,10cyclictest309-21in:imuxsock07:50:001
99300899401,6cyclictest131rcu_preempt10:25:021
99300899401,5cyclictest131rcu_preempt12:25:001
99300899401,3cyclictest131rcu_preempt09:22:181
99301899398,5cyclictest131rcu_preempt12:04:233
993013993927,8cyclictest30-21ksoftirqd/207:25:032
993013993925,11cyclictest30-21ksoftirqd/208:50:012
993013993916,18cyclictest275-21dbus-daemon07:35:012
99300899394,6cyclictest131rcu_preempt08:35:021
993008993929,5cyclictest993988-21cron07:30:001
993008993928,8cyclictest998129-21sshd08:48:231
993008993928,7cyclictest23-21ksoftirqd/112:00:201
993008993922,12cyclictest275-21dbus-daemon07:45:011
99300599391,22cyclictest0-21swapper/007:10:080
993018993828,6cyclictest309-21in:imuxsock08:19:593
99301899382,6cyclictest131rcu_preempt12:10:403
993018993821,13cyclictest997291-21wc08:30:033
993018993821,13cyclictest997291-21wc08:30:023
993013993822,11cyclictest3467602-21systemd-journal09:05:012
99301399381,11cyclictest997625-21cron08:40:002
993008993827,8cyclictest23-21ksoftirqd/107:14:001
99300599381,4cyclictest993427-21kworker/u8:007:17:000
993018993727,8cyclictest37-21ksoftirqd/307:45:013
993018993720,4cyclictest37-21ksoftirqd/308:33:563
99301399379,5cyclictest131rcu_preempt09:00:022
99300899379,14cyclictest23-21ksoftirqd/109:05:001
993008993728,5cyclictest993712-21sshd07:20:241
993008993726,9cyclictest23-21ksoftirqd/107:55:011
99300899372,5cyclictest131rcu_preempt08:45:011
99300899372,5cyclictest131rcu_preempt08:45:001
99300899371,5cyclictest131rcu_preempt07:35:001
993008993713,6cyclictest131rcu_preempt08:10:011
993018993625,9cyclictest37-21ksoftirqd/308:50:013
993018993623,10cyclictest37-21ksoftirqd/307:53:483
993018993610,5cyclictest131rcu_preempt07:40:023
993013993627,6cyclictest30-21ksoftirqd/207:25:032
99301399362,5cyclictest30-21ksoftirqd/207:10:242
993013993622,9cyclictest30-21ksoftirqd/208:33:562
993013993622,10cyclictest995549-21systemctl07:59:592
99300899367,8cyclictest131rcu_preempt08:00:021
99300899363,5cyclictest131rcu_preempt07:10:031
99301899353,6cyclictest131rcu_preempt09:05:013
993018993526,5cyclictest309-21in:imuxsock07:30:003
993018993518,13cyclictest1-21systemd07:25:003
993008993513,17cyclictest131rcu_preempt08:05:011
993018993424,6cyclictest998542-21latency_hist08:55:023
993018993326,3cyclictest37-21ksoftirqd/308:08:503
993013993325,5cyclictest997903-21/usr/sbin/munin08:45:012
993013993325,5cyclictest997903-21/usr/sbin/munin08:45:002
993013993220,9cyclictest993064-21sed07:10:022
99301899303,5cyclictest131rcu_preempt07:16:233
99301899271,4cyclictest131rcu_preempt07:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional