You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 19:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu May 28, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
963323991021,84cyclictest0-21swapper/011:20:020
96332899703,6cyclictest131rcu_preempt09:30:161
96332399679,34cyclictest131rcu_preempt09:30:340
963334996634,9cyclictest37-21ksoftirqd/309:50:473
963334996633,10cyclictest37-21ksoftirqd/312:36:393
963332996617,10cyclictest30-21ksoftirqd/211:30:402
96332399663,8cyclictest121ksoftirqd/012:37:110
963323996629,33cyclictest12-21ksoftirqd/009:17:160
963323996514,23cyclictest12-21ksoftirqd/011:07:090
963334996427,11cyclictest37-21ksoftirqd/311:26:593
96333299648,18cyclictest1002439-21ssh10:23:052
96333299648,18cyclictest1002439-21ssh10:23:042
963323996415,44cyclictest16850irq/134-800000010:10:390
963323996415,44cyclictest16850irq/134-800000010:10:390
963332996321,8cyclictest30-21ksoftirqd/210:30:252
963332996321,8cyclictest30-21ksoftirqd/210:30:252
963332996316,11cyclictest30-21ksoftirqd/212:25:292
963332996316,10cyclictest30-21ksoftirqd/210:42:482
963323996312,10cyclictest12-21ksoftirqd/011:03:370
963323996310,27cyclictest12-21ksoftirqd/010:44:100
963334996222,11cyclictest37-21ksoftirqd/311:25:003
963332996219,8cyclictest30-21ksoftirqd/209:36:272
963332996218,11cyclictest30-21ksoftirqd/211:52:332
963332996217,8cyclictest30-21ksoftirqd/212:07:512
963332996217,8cyclictest30-21ksoftirqd/210:17:272
963332996217,8cyclictest30-21ksoftirqd/210:07:172
963332996215,10cyclictest30-21ksoftirqd/211:23:282
963332996215,10cyclictest30-21ksoftirqd/211:08:362
963332996212,14cyclictest973366-21ssh09:18:342
96332399629,34cyclictest131rcu_preempt12:27:420
96332399627,36cyclictest12-21ksoftirqd/009:10:130
96333499617,10cyclictest37-21ksoftirqd/312:30:003
96332399619,11cyclictest12-21ksoftirqd/011:32:080
963323996113,23cyclictest12-21ksoftirqd/009:37:540
96333299608,15cyclictest993996-21ssh10:04:232
96333299606,21cyclictest1060159-21ssh12:31:322
963332996018,8cyclictest30-21ksoftirqd/212:24:222
963332996018,10cyclictest30-21ksoftirqd/211:30:022
963332996015,8cyclictest30-21ksoftirqd/207:45:022
963332996015,8cyclictest30-21ksoftirqd/207:45:022
963332996015,10cyclictest30-21ksoftirqd/211:57:252
963332996015,10cyclictest30-21ksoftirqd/208:20:012
963332996015,10cyclictest30-21ksoftirqd/207:20:002
963323996013,14cyclictest131rcu_preempt11:26:570
96333299597,15cyclictest970749-21ssh09:12:522
963332995917,10cyclictest30-21ksoftirqd/212:16:082
963332995916,10cyclictest30-21ksoftirqd/209:42:552
963332995915,10cyclictest30-21ksoftirqd/211:36:212
963332995914,10cyclictest30-21ksoftirqd/210:37:022
96332399597,11cyclictest12-21ksoftirqd/012:19:180
96332399592,4cyclictest131rcu_preempt12:07:360
963323995917,37cyclictest16850irq/134-800000009:30:000
963323995917,37cyclictest16850irq/134-800000009:30:000
96333499583,13cyclictest37-21ksoftirqd/312:20:233
963334995812,14cyclictest37-21ksoftirqd/312:10:023
963332995839,15cyclictest1-21systemd10:15:012
963332995839,15cyclictest1-21systemd10:15:012
963332995817,7cyclictest30-21ksoftirqd/208:10:012
963332995815,8cyclictest30-21ksoftirqd/211:00:012
963332995815,8cyclictest30-21ksoftirqd/210:25:392
963332995812,15cyclictest1052100-21ssh12:13:582
963332995810,10cyclictest30-21ksoftirqd/211:18:272
963323995851,4cyclictest150-21pr/ttyS210:55:020
963323995819,10cyclictest12-21ksoftirqd/011:40:390
963332995715,10cyclictest30-21ksoftirqd/209:30:172
963332995712,9cyclictest30-21ksoftirqd/209:23:272
963332995712,10cyclictest30-21ksoftirqd/210:50:432
96332899576,6cyclictest131rcu_preempt11:26:221
963328995717,6cyclictest131rcu_preempt12:35:001
96332399573,26cyclictest12-21ksoftirqd/011:38:440
963323995719,33cyclictest16850irq/134-800000011:55:010
963334995635,17cyclictest275-21dbus-daemon10:45:013
963334995616,8cyclictest37-21ksoftirqd/309:48:313
963334995611,11cyclictest37-21ksoftirqd/312:18:503
96333299566,18cyclictest966139-21systemd-run08:05:012
96333299564,14cyclictest1020119-21ssh11:02:272
963332995618,9cyclictest30-21ksoftirqd/210:46:392
963332995617,8cyclictest30-21ksoftirqd/212:03:012
96332399568,5cyclictest131rcu_preempt11:56:170
96332399565,33cyclictest12-21ksoftirqd/009:23:310
963323995644,8cyclictest12-21ksoftirqd/010:40:000
96333499557,9cyclictest37-21ksoftirqd/310:38:593
963334995519,10cyclictest37-21ksoftirqd/308:35:023
963334995512,11cyclictest37-21ksoftirqd/312:05:013
963332995511,8cyclictest30-21ksoftirqd/211:11:222
963328995516,5cyclictest131rcu_preempt07:40:001
963328995511,5cyclictest131rcu_preempt09:20:001
96332399552,8cyclictest121ksoftirqd/011:23:160
96333499544,10cyclictest37-21ksoftirqd/310:06:353
963334995413,9cyclictest37-21ksoftirqd/311:09:403
96333499541,17cyclictest1043837-21ssh11:55:153
963334995410,8cyclictest37-21ksoftirqd/309:12:563
963332995414,10cyclictest30-21ksoftirqd/209:28:072
963332995414,10cyclictest30-21ksoftirqd/209:28:072
963332995413,9cyclictest30-21ksoftirqd/211:45:062
963328995435,13cyclictest3467602-21systemd-journal08:00:001
96332399546,7cyclictest12-21ksoftirqd/010:00:010
963323995417,33cyclictest16850irq/134-800000010:23:160
963323995417,33cyclictest16850irq/134-800000010:23:160
96332399541,3cyclictest131rcu_preempt11:47:070
96333499539,9cyclictest37-21ksoftirqd/310:30:013
96333499536,7cyclictest37-21ksoftirqd/311:45:403
96333499534,9cyclictest37-21ksoftirqd/311:39:223
963334995346,5cyclictest361rcuc/309:20:593
96333499533,14cyclictest37-21ksoftirqd/309:29:193
96333499533,14cyclictest37-21ksoftirqd/309:29:193
963334995323,11cyclictest37-21ksoftirqd/307:15:013
96333499532,13cyclictest37-21ksoftirqd/310:59:053
96332399537,27cyclictest12-21ksoftirqd/010:46:130
963323995338,12cyclictest12-21ksoftirqd/011:11:390
963323995325,13cyclictest996061-21ssh10:09:030
963323995317,20cyclictest16850irq/134-800000012:30:550
96333499527,8cyclictest37-21ksoftirqd/309:41:173
96333499525,7cyclictest37-21ksoftirqd/310:51:593
96333499522,14cyclictest37-21ksoftirqd/307:45:023
963332995243,6cyclictest983492-21kworker/2:009:49:062
96332899529,5cyclictest131rcu_preempt12:08:551
963328995242,6cyclictest23-21ksoftirqd/109:40:001
963328995232,15cyclictest3467602-21systemd-journal10:05:021
963328995216,6cyclictest131rcu_preempt09:30:011
963328995216,6cyclictest131rcu_preempt09:30:011
963328995210,5cyclictest131rcu_preempt11:58:361
963328995210,4cyclictest131rcu_preempt09:54:201
96332399524,27cyclictest16850irq/134-800000009:50:020
963323995238,11cyclictest12-21ksoftirqd/009:10:020
963323995217,30cyclictest16850irq/134-800000010:01:170
963323995216,6cyclictest16850irq/134-800000012:04:130
96333499516,11cyclictest37-21ksoftirqd/311:31:033
96333499516,10cyclictest37-21ksoftirqd/311:16:133
96333499512,14cyclictest37-21ksoftirqd/312:13:013
963334995112,11cyclictest37-21ksoftirqd/310:20:013
96333499511,11cyclictest37-21ksoftirqd/312:30:223
963332995136,11cyclictest968193-21cron08:45:002
96332899519,5cyclictest131rcu_preempt10:50:271
963328995140,9cyclictest23-21ksoftirqd/110:00:031
963328995138,9cyclictest23-21ksoftirqd/112:05:011
963328995134,13cyclictest1-21systemd11:50:011
963328995129,8cyclictest23-21ksoftirqd/112:13:171
963328995114,5cyclictest131rcu_preempt10:40:221
963328995113,6cyclictest131rcu_preempt11:34:011
963328995111,6cyclictest131rcu_preempt09:46:341
963328995111,5cyclictest131rcu_preempt11:05:121
963323995117,19cyclictest16850irq/134-800000010:52:590
963323995116,31cyclictest16850irq/134-800000010:34:540
963323995116,31cyclictest16850irq/134-800000010:34:540
96333499507,9cyclictest37-21ksoftirqd/310:48:363
96333499507,9cyclictest37-21ksoftirqd/310:35:013
96333499507,9cyclictest37-21ksoftirqd/310:35:013
96333499507,9cyclictest37-21ksoftirqd/310:13:103
96333499507,9cyclictest37-21ksoftirqd/310:13:103
96333499507,9cyclictest37-21ksoftirqd/310:01:133
96333499507,8cyclictest964083-21latency_hist07:25:023
96333499506,11cyclictest37-21ksoftirqd/311:54:063
96333499506,11cyclictest37-21ksoftirqd/309:18:003
96333499504,7cyclictest37-21ksoftirqd/311:45:013
96333499503,6cyclictest37-21ksoftirqd/308:00:013
963334995027,18cyclictest3467602-21systemd-journal09:40:003
963334995012,5cyclictest131rcu_preempt11:13:183
963332995013,9cyclictest991125-21ssh09:57:572
96332899508,5cyclictest131rcu_preempt12:26:391
96332899507,12cyclictest1038317-21ssh11:43:141
96332899506,5cyclictest131rcu_preempt12:37:271
963328995041,6cyclictest23-21ksoftirqd/110:00:011
96332899502,7cyclictest23-21ksoftirqd/111:54:041
96332899502,5cyclictest131rcu_preempt10:30:001
963328995021,6cyclictest221rcuc/111:20:001
963328995021,15cyclictest0-21swapper/109:24:531
96332399505,8cyclictest12-21ksoftirqd/012:23:180
963323995022,13cyclictest983944-21ssh09:41:500
963323995020,11cyclictest12-21ksoftirqd/008:25:010
963323995014,32cyclictest16850irq/134-800000010:29:240
96332399501,3cyclictest131rcu_preempt08:25:220
963323995013,11cyclictest12-21ksoftirqd/012:13:480
963323995012,5cyclictest131rcu_preempt09:52:440
96333499498,8cyclictest37-21ksoftirqd/308:45:013
96333499495,13cyclictest37-21ksoftirqd/310:24:583
96333499495,13cyclictest37-21ksoftirqd/310:24:583
96333499494,12cyclictest37-21ksoftirqd/307:20:013
96333499492,8cyclictest37-21ksoftirqd/309:57:573
96333499491,10cyclictest37-21ksoftirqd/308:20:003
96333299495,7cyclictest30-21ksoftirqd/212:35:282
96332899499,6cyclictest131rcu_preempt11:38:001
96332899499,5cyclictest131rcu_preempt08:41:541
96332899497,6cyclictest131rcu_preempt10:45:341
963328994911,5cyclictest131rcu_preempt09:11:501
963328994910,6cyclictest131rcu_preempt10:40:001
963323994941,4cyclictest16950irq/224-800000010:20:000
963334994830,14cyclictest275-21dbus-daemon08:15:013
96333299487,5cyclictest131rcu_preempt09:10:012
963332994830,14cyclictest3467602-21systemd-journal07:15:002
963332994811,8cyclictest0-21swapper/211:43:072
96332899489,6cyclictest131rcu_preempt08:05:011
96332899488,4cyclictest131rcu_preempt10:56:421
96332899487,8cyclictest23-21ksoftirqd/110:11:411
96332899487,8cyclictest23-21ksoftirqd/110:11:401
96332899487,5cyclictest131rcu_preempt10:34:051
96332899487,5cyclictest131rcu_preempt10:34:051
96332899487,14cyclictest23-21ksoftirqd/107:25:581
963328994838,6cyclictest23-21ksoftirqd/109:10:011
963328994830,8cyclictest23-21ksoftirqd/108:20:011
963328994813,8cyclictest131rcu_preempt11:13:171
963328994810,4cyclictest131rcu_preempt11:22:261
963328994810,4cyclictest131rcu_preempt11:03:201
963323994830,14cyclictest275-21dbus-daemon09:05:000
96332399482,39cyclictest965850-21sh08:00:000
96333499479,5cyclictest131rcu_preempt07:35:023
96333499476,8cyclictest37-21ksoftirqd/311:01:473
96333499475,10cyclictest37-21ksoftirqd/308:10:013
963334994738,7cyclictest37-21ksoftirqd/309:33:083
963334994733,11cyclictest3467602-21systemd-journal07:53:223
963332994738,5cyclictest957194-21kworker/2:107:30:012
963332994727,14cyclictest3467602-21systemd-journal07:55:002
96333299470,14cyclictest967319-21latency_hist08:25:022
963328994714,3cyclictest131rcu_preempt12:19:211
963328994712,5cyclictest131rcu_preempt10:22:331
963328994712,5cyclictest131rcu_preempt10:22:331
963328994711,5cyclictest131rcu_preempt07:25:011
963323994716,12cyclictest12-21ksoftirqd/008:55:000
96333499467,6cyclictest37-21ksoftirqd/307:10:023
96333499463,9cyclictest37-21ksoftirqd/307:50:023
96333499463,9cyclictest37-21ksoftirqd/307:50:023
96332899469,6cyclictest131rcu_preempt12:20:061
96332899464,10cyclictest23-21ksoftirqd/110:15:591
96332899461,5cyclictest131rcu_preempt07:55:001
963328994611,7cyclictest131rcu_preempt08:36:371
963328994610,5cyclictest131rcu_preempt09:42:161
963323994630,10cyclictest275-21dbus-daemon07:25:010
963334994516,8cyclictest37-21ksoftirqd/308:25:013
96333299457,5cyclictest131rcu_preempt08:15:002
96333299454,14cyclictest964550-21cron07:35:022
963332994526,11cyclictest309-21in:imuxsock09:00:002
963332994526,11cyclictest309-21in:imuxsock09:00:002
963332994524,16cyclictest987923-21ssh09:50:332
96332899457,5cyclictest131rcu_preempt07:20:001
96332399452,8cyclictest12-21ksoftirqd/007:15:220
963334994433,7cyclictest309-21in:imuxsock08:05:013
96333499441,7cyclictest969514-21ssh09:10:013
96332899443,10cyclictest23-21ksoftirqd/108:55:001
963323994430,11cyclictest275-21dbus-daemon08:20:010
96332399442,9cyclictest3467602-21systemd-journal07:40:000
96332399442,8cyclictest3467602-21systemd-journal07:45:000
96333499435,11cyclictest37-21ksoftirqd/308:55:003
96333499432,8cyclictest0-21swapper/308:45:353
963334994324,13cyclictest275-21dbus-daemon08:40:013
963332994332,9cyclictest30-21ksoftirqd/207:25:022
963332994326,9cyclictest967933-21cat08:40:002
963328994329,10cyclictest3467602-21systemd-journal09:05:001
963328994310,5cyclictest131rcu_preempt07:12:141
963323994336,4cyclictest16950irq/224-800000009:00:010
963323994336,4cyclictest16950irq/224-800000009:00:010
963323994336,4cyclictest16950irq/224-800000008:05:270
963323994334,6cyclictest151rcuc/008:02:350
963323994334,6cyclictest12-21ksoftirqd/007:50:010
963323994334,6cyclictest12-21ksoftirqd/007:50:010
96333499425,9cyclictest37-21ksoftirqd/307:30:023
963334994230,9cyclictest37-21ksoftirqd/309:00:013
963334994230,9cyclictest37-21ksoftirqd/309:00:013
96333499421,7cyclictest0-21swapper/309:00:103
963332994228,10cyclictest967401-21sshd08:28:222
96332899423,5cyclictest131rcu_preempt08:15:021
963323994236,4cyclictest16950irq/224-800000007:30:010
96333499411,36cyclictest964997-21sed07:40:023
963332994131,7cyclictest965812-21sshd07:55:222
963332994127,10cyclictest965073-21sshd07:44:222
96332899419,5cyclictest131rcu_preempt08:10:011
96332899418,5cyclictest131rcu_preempt08:23:241
96332899413,12cyclictest23-21ksoftirqd/107:32:271
963323994129,9cyclictest964691-21sed07:35:020
963332994031,6cyclictest969212-21sshd09:01:222
963332994026,10cyclictest964576-21munin-run07:35:012
96332899406,7cyclictest23-21ksoftirqd/107:41:211
96332899402,8cyclictest23-21ksoftirqd/108:31:221
96332399408,28cyclictest0-21swapper/008:38:340
963323994012,6cyclictest131rcu_preempt07:10:020
96332899392,8cyclictest23-21ksoftirqd/107:47:221
96332899392,8cyclictest23-21ksoftirqd/107:47:221
963323993917,10cyclictest12-21ksoftirqd/007:52:220
963323993916,19cyclictest16850irq/134-800000008:13:230
963323993912,6cyclictest131rcu_preempt08:35:020
963332993827,8cyclictest967636-21sshd08:30:082
963332993810,19cyclictest968452-21modprobe08:50:002
96332899382,9cyclictest23-21ksoftirqd/108:49:401
96332899381,5cyclictest150-21pr/ttyS208:27:411
963328993811,5cyclictest131rcu_preempt07:10:021
96332899381,12cyclictest23-21ksoftirqd/108:55:401
96332899381,12cyclictest23-21ksoftirqd/108:55:401
963323993810,8cyclictest131rcu_preempt08:46:130
963332993720,12cyclictest309-21in:imuxsock08:55:002
96332399376,5cyclictest131rcu_preempt08:42:210
963323993718,7cyclictest12-21ksoftirqd/007:13:330
96333499351,5cyclictest0-21swapper/308:29:393
963332993423,8cyclictest963329-21latency_hist07:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional