You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-20 - 16:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Aug 20, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269677099822,48cyclictest12-21ksoftirqd/011:47:570
2696779997530,36cyclictest2759433-21kworker/u8:211:50:012
269677099726,44cyclictest12-21ksoftirqd/011:54:240
269678499711,12cyclictest37-21ksoftirqd/308:15:013
269677099704,35cyclictest12-21ksoftirqd/012:05:020
269677099693,31cyclictest12-21ksoftirqd/011:25:000
2696770996911,7cyclictest0-21swapper/010:57:310
269677099675,29cyclictest12-21ksoftirqd/010:13:260
269677099674,11cyclictest12-21ksoftirqd/012:20:010
2696770996651,12cyclictest16850irq/134-800000010:05:320
2696770996625,9cyclictest12-21ksoftirqd/010:30:130
2696770996620,8cyclictest12-21ksoftirqd/010:45:490
2696770996620,8cyclictest12-21ksoftirqd/010:45:490
2696770996555,5cyclictest16850irq/134-800000009:40:020
2696770996455,5cyclictest16850irq/134-800000009:15:010
269677099634,31cyclictest12-21ksoftirqd/011:40:150
2696779996246,7cyclictest2701676-21kworker/2:209:12:472
2696779996245,7cyclictest2689919-21kworker/2:310:38:002
269677099624,9cyclictest12-21ksoftirqd/009:05:020
2696770996247,11cyclictest19350irq/34-mmc112:25:010
2696770996247,11cyclictest19350irq/34-mmc112:25:010
2696770996239,18cyclictest2753597-21cron10:55:000
2696770996220,34cyclictest16850irq/134-800000009:23:490
2696770996220,33cyclictest16850irq/134-800000012:37:460
269677099621,25cyclictest12-21ksoftirqd/011:34:270
269677099619,27cyclictest131rcu_preempt10:22:150
2696770996125,9cyclictest12-21ksoftirqd/009:56:390
2696770996124,28cyclictest271737650rm09:40:040
2696770996115,12cyclictest12-21ksoftirqd/012:30:010
269677099611,27cyclictest2748883-21latency_hist10:45:030
269677099611,12cyclictest12-21ksoftirqd/011:27:120
2696784996011,6cyclictest131rcu_preempt10:00:023
2696779996052,5cyclictest2749172-21kworker/2:011:04:382
2696779996041,7cyclictest2701676-21kworker/2:210:29:442
269677599607,7cyclictest23-21ksoftirqd/107:30:001
269677099604,8cyclictest12-21ksoftirqd/010:01:530
2696770996019,32cyclictest16850irq/134-800000009:18:490
269677099601,39cyclictest12-21ksoftirqd/012:31:110
2696779995944,7cyclictest2749172-21kworker/2:010:49:182
2696779995944,7cyclictest2749172-21kworker/2:010:49:182
269677099597,11cyclictest12-21ksoftirqd/009:29:460
2696770995919,31cyclictest16850irq/134-800000011:36:440
2696770995914,16cyclictest3467602-21systemd-journal08:05:010
269678499589,5cyclictest131rcu_preempt09:35:013
269678499587,6cyclictest131rcu_preempt11:00:003
2696779995846,8cyclictest2689919-21kworker/2:310:42:202
2696779995842,8cyclictest2689919-21kworker/2:309:46:432
2696775995832,10cyclictest23-21ksoftirqd/109:46:081
2696775995818,9cyclictest23-21ksoftirqd/108:05:011
2696775995810,5cyclictest131rcu_preempt11:40:011
2696770995854,2cyclictest0-21swapper/008:35:010
269677099585,12cyclictest12-21ksoftirqd/008:20:010
269677099583,8cyclictest12-21ksoftirqd/008:40:020
269677099583,8cyclictest12-21ksoftirqd/008:40:010
2696770995813,6cyclictest16850irq/134-800000011:03:150
2696770995813,5cyclictest0-21swapper/011:20:010
2696770995811,12cyclictest12-21ksoftirqd/010:28:390
269678499578,6cyclictest131rcu_preempt12:10:023
269678499578,6cyclictest131rcu_preempt09:29:593
269678499575,9cyclictest37-21ksoftirqd/307:30:023
2696779995742,7cyclictest2701676-21kworker/2:209:56:312
2696779995742,7cyclictest2689919-21kworker/2:312:26:192
2696779995742,6cyclictest2749172-21kworker/2:011:22:432
2696779995741,7cyclictest2775574-21kworker/2:111:40:132
2696779995741,7cyclictest2701676-21kworker/2:209:23:002
2696779995739,7cyclictest2749172-21kworker/2:010:54:432
2696775995711,6cyclictest131rcu_preempt11:11:231
2696775995710,6cyclictest131rcu_preempt09:05:021
2696770995746,8cyclictest2702946-21modprobe09:10:010
2696770995726,15cyclictest2720211-21ssh09:45:540
269678499566,5cyclictest131rcu_preempt11:40:033
269678499564,5cyclictest131rcu_preempt12:00:013
269678499563,9cyclictest37-21ksoftirqd/311:50:273
2696784995636,15cyclictest275-21dbus-daemon10:20:003
2696784995628,9cyclictest37-21ksoftirqd/310:07:103
2696784995619,15cyclictest37-21ksoftirqd/310:48:563
2696784995619,15cyclictest37-21ksoftirqd/310:48:563
2696784995613,6cyclictest131rcu_preempt07:40:013
2696784995611,6cyclictest131rcu_preempt10:03:563
2696779995640,7cyclictest2689919-21kworker/2:312:09:462
269677599569,6cyclictest131rcu_preempt12:15:131
269677599565,12cyclictest23-21ksoftirqd/112:25:011
269677599565,12cyclictest23-21ksoftirqd/112:25:011
2696775995614,5cyclictest131rcu_preempt11:25:021
2696775995611,5cyclictest131rcu_preempt11:05:011
2696770995614,32cyclictest16850irq/134-800000011:11:110
2696770995613,7cyclictest12-21ksoftirqd/009:50:360
269678499554,10cyclictest37-21ksoftirqd/310:53:203
2696784995522,19cyclictest37-21ksoftirqd/310:13:563
2696779995545,6cyclictest2701676-21kworker/2:210:03:182
269677599555,7cyclictest23-21ksoftirqd/112:00:011
269677599555,7cyclictest23-21ksoftirqd/111:49:281
269677599555,12cyclictest23-21ksoftirqd/111:34:471
2696775995525,17cyclictest23-21ksoftirqd/110:26:151
2696775995520,9cyclictest23-21ksoftirqd/109:35:061
2696775995515,6cyclictest131rcu_preempt12:02:071
2696775995513,5cyclictest131rcu_preempt11:10:021
2696775995511,6cyclictest131rcu_preempt12:39:391
2696775995510,5cyclictest131rcu_preempt09:28:271
2696770995524,15cyclictest2714516-21ssh09:34:390
269677099552,38cyclictest2784022-21ssh11:57:210
269677099551,8cyclictest12-21ksoftirqd/007:15:010
2696770995517,33cyclictest16850irq/134-800000010:37:420
2696770995515,7cyclictest12-21ksoftirqd/010:15:180
2696770995515,30cyclictest16850irq/134-800000012:13:140
2696770995513,9cyclictest12-21ksoftirqd/011:07:560
2696784995433,7cyclictest37-21ksoftirqd/310:23:203
2696784995412,6cyclictest131rcu_preempt09:00:003
2696779995446,5cyclictest2689919-21kworker/2:311:29:012
2696779995445,6cyclictest2689919-21kworker/2:310:21:152
2696779995445,6cyclictest2689919-21kworker/2:309:52:222
2696779995441,6cyclictest2689919-21kworker/2:312:19:192
2696779995439,7cyclictest2749172-21kworker/2:011:10:172
269677599548,7cyclictest23-21ksoftirqd/111:19:351
269677599547,32cyclictest2698758-21kworker/u8:007:50:031
269677599547,32cyclictest2698758-21kworker/u8:007:50:031
2696775995414,6cyclictest131rcu_preempt12:05:161
2696775995414,5cyclictest131rcu_preempt09:25:011
2696775995412,6cyclictest131rcu_preempt12:32:291
2696775995412,6cyclictest131rcu_preempt08:45:001
2696775995412,5cyclictest131rcu_preempt11:00:011
2696775995411,5cyclictest131rcu_preempt07:40:001
2696770995412,9cyclictest12-21ksoftirqd/009:00:020
2696770995410,7cyclictest12-21ksoftirqd/007:35:030
269678499539,6cyclictest131rcu_preempt12:25:023
269678499539,6cyclictest131rcu_preempt12:25:023
269678499532,5cyclictest131rcu_preempt11:35:023
2696784995316,6cyclictest2702953-21diskmemload11:41:043
2696784995315,5cyclictest131rcu_preempt10:35:023
2696784995314,5cyclictest131rcu_preempt10:40:013
2696779995345,5cyclictest2749172-21kworker/2:011:33:482
2696779995339,9cyclictest30-21ksoftirqd/212:25:022
2696779995339,9cyclictest30-21ksoftirqd/212:25:022
2696779995317,10cyclictest30-21ksoftirqd/211:53:292
269677599539,5cyclictest131rcu_preempt11:53:001
269677599538,5cyclictest131rcu_preempt11:26:451
269677599538,5cyclictest131rcu_preempt09:12:431
269677599534,10cyclictest23-21ksoftirqd/110:37:011
2696775995317,15cyclictest131rcu_preempt10:04:331
2696775995315,10cyclictest23-21ksoftirqd/108:25:011
2696775995311,5cyclictest131rcu_preempt09:54:261
269677099534,16cyclictest12-21ksoftirqd/012:06:100
269678499528,6cyclictest131rcu_preempt09:15:003
269678499527,7cyclictest131rcu_preempt08:04:593
269678499526,5cyclictest131rcu_preempt12:37:483
269678499524,8cyclictest37-21ksoftirqd/311:08:443
2696784995234,14cyclictest2792702-21cut12:15:033
2696784995225,11cyclictest37-21ksoftirqd/312:00:143
2696784995215,10cyclictest131rcu_preempt09:50:023
2696784995211,6cyclictest131rcu_preempt08:25:023
2696784995210,6cyclictest131rcu_preempt07:45:003
269677999528,6cyclictest131rcu_preempt11:40:022
2696779995244,5cyclictest2775574-21kworker/2:112:31:012
2696779995244,5cyclictest2775574-21kworker/2:111:55:332
2696779995238,6cyclictest2689919-21kworker/2:308:16:322
2696779995215,23cyclictest30-21ksoftirqd/212:12:192
269677599529,5cyclictest131rcu_preempt09:32:591
269677599528,5cyclictest131rcu_preempt07:50:031
269677599528,4cyclictest131rcu_preempt10:48:271
269677599528,4cyclictest131rcu_preempt10:48:271
269677599526,6cyclictest131rcu_preempt12:26:041
269677599524,9cyclictest23-21ksoftirqd/110:45:021
2696775995215,5cyclictest131rcu_preempt09:59:541
2696775995212,6cyclictest131rcu_preempt07:20:001
2696775995212,6cyclictest131rcu_preempt07:20:001
2696775995211,5cyclictest131rcu_preempt10:24:581
2696775995211,5cyclictest131rcu_preempt10:15:021
2696775995210,5cyclictest131rcu_preempt10:53:341
269677099524,12cyclictest12-21ksoftirqd/008:10:010
269677099523,7cyclictest12-21ksoftirqd/008:45:010
2696770995235,12cyclictest12-21ksoftirqd/007:20:000
2696770995235,12cyclictest12-21ksoftirqd/007:19:590
2696770995220,28cyclictest12-21ksoftirqd/008:25:150
2696770995220,28cyclictest12-21ksoftirqd/008:25:150
269678499517,6cyclictest131rcu_preempt10:43:383
269678499515,7cyclictest37-21ksoftirqd/311:10:303
2696784995123,20cyclictest37-21ksoftirqd/311:27:583
2696784995115,6cyclictest131rcu_preempt09:55:013
2696784995114,5cyclictest131rcu_preempt11:20:173
2696784995113,6cyclictest131rcu_preempt09:15:473
2696784995111,6cyclictest131rcu_preempt09:35:553
2696784995111,5cyclictest131rcu_preempt12:32:543
2696779995144,4cyclictest2689919-21kworker/2:309:40:502
2696779995143,5cyclictest2775574-21kworker/2:112:39:182
2696779995143,5cyclictest2701676-21kworker/2:209:30:562
2696779995143,5cyclictest2689919-21kworker/2:311:16:312
2696779995143,5cyclictest2689919-21kworker/2:310:08:242
2696779995142,6cyclictest2689919-21kworker/2:310:13:422
2696779995142,5cyclictest2701676-21kworker/2:209:39:052
2696779995142,5cyclictest2689919-21kworker/2:310:17:322
2696779995138,6cyclictest2701676-21kworker/2:209:18:502
269677599518,6cyclictest131rcu_preempt10:17:031
269677599513,9cyclictest23-21ksoftirqd/111:43:051
269677599513,6cyclictest131rcu_preempt12:15:021
269677599512,22cyclictest23-21ksoftirqd/109:00:001
269677099519,9cyclictest0-21swapper/007:30:030
269677099519,9cyclictest0-21swapper/007:30:030
269678499508,6cyclictest131rcu_preempt09:40:403
269678499508,5cyclictest131rcu_preempt08:45:013
269678499504,7cyclictest37-21ksoftirqd/309:10:023
269678499504,12cyclictest37-21ksoftirqd/307:20:003
269678499504,12cyclictest37-21ksoftirqd/307:20:003
2696784995013,6cyclictest131rcu_preempt09:23:593
2696784995013,5cyclictest131rcu_preempt12:26:453
2696784995011,6cyclictest131rcu_preempt11:50:023
2696784995011,4cyclictest131rcu_preempt12:17:123
2696784995010,5cyclictest131rcu_preempt10:29:373
269677999504,8cyclictest30-21ksoftirqd/211:10:012
2696779995041,6cyclictest2749172-21kworker/2:010:55:112
2696779995041,6cyclictest2689919-21kworker/2:308:40:002
2696779995041,6cyclictest2689919-21kworker/2:308:40:002
2696779995034,8cyclictest2701676-21kworker/2:209:28:222
2696779995034,12cyclictest3467602-21systemd-journal08:51:562
2696779995034,12cyclictest3467602-21systemd-journal08:51:562
269677599505,7cyclictest23-21ksoftirqd/109:41:221
269677599503,9cyclictest23-21ksoftirqd/109:19:011
2696775995014,15cyclictest131rcu_preempt10:07:181
2696775995011,5cyclictest131rcu_preempt09:10:011
2696775995010,5cyclictest131rcu_preempt10:31:241
269677099509,5cyclictest0-21swapper/008:49:370
269677099501,12cyclictest12-21ksoftirqd/007:30:020
269678499498,5cyclictest131rcu_preempt08:00:013
269678499494,11cyclictest37-21ksoftirqd/307:50:013
269678499494,11cyclictest37-21ksoftirqd/307:50:013
269678499492,6cyclictest131rcu_preempt08:50:013
2696784994913,5cyclictest131rcu_preempt07:55:013
2696784994913,5cyclictest131rcu_preempt07:55:003
2696779994941,5cyclictest2775574-21kworker/2:112:03:472
2696779994941,5cyclictest2701676-21kworker/2:210:31:072
2696779994941,5cyclictest2689919-21kworker/2:307:57:552
2696779994940,6cyclictest2689919-21kworker/2:308:35:012
269677999491,3cyclictest131rcu_preempt07:20:022
269677999491,3cyclictest131rcu_preempt07:20:022
269677599498,4cyclictest131rcu_preempt07:21:551
2696775994913,5cyclictest131rcu_preempt08:29:551
2696775994913,5cyclictest131rcu_preempt08:29:551
2696775994911,5cyclictest131rcu_preempt08:00:011
269677099491,9cyclictest2699309-21munin-run08:00:000
269677099491,11cyclictest12-21ksoftirqd/008:55:010
269677099491,11cyclictest12-21ksoftirqd/008:55:000
269678499489,5cyclictest131rcu_preempt11:16:103
269678499488,6cyclictest131rcu_preempt08:30:023
269678499488,6cyclictest131rcu_preempt08:30:013
269678499481,9cyclictest2756747-21sh11:00:593
2696779994839,5cyclictest2701676-21kworker/2:208:55:062
269677599487,5cyclictest131rcu_preempt07:35:021
269677599487,5cyclictest131rcu_preempt07:35:011
269677599485,9cyclictest23-21ksoftirqd/108:40:011
269677599485,9cyclictest23-21ksoftirqd/108:40:011
269677099485,12cyclictest12-21ksoftirqd/007:50:030
269677099485,12cyclictest12-21ksoftirqd/007:50:030
2696770994810,34cyclictest0-21swapper/007:23:510
269678499478,5cyclictest131rcu_preempt07:10:023
2696779994739,5cyclictest2701676-21kworker/2:209:06:362
2696779994739,5cyclictest2701676-21kworker/2:208:48:592
2696779994739,5cyclictest2696538-21kworker/2:107:51:212
2696779994739,5cyclictest2696538-21kworker/2:107:51:212
2696779994739,5cyclictest2689919-21kworker/2:308:43:392
2696779994739,5cyclictest2689919-21kworker/2:308:04:012
269677999472,5cyclictest131rcu_preempt07:15:022
269677599476,9cyclictest23-21ksoftirqd/108:35:021
269677099479,8cyclictest0-21swapper/008:11:490
269677099477,6cyclictest12-21ksoftirqd/007:55:020
269677099477,6cyclictest12-21ksoftirqd/007:55:020
2696770994710,9cyclictest0-21swapper/008:24:480
269678499469,6cyclictest131rcu_preempt07:15:013
269678499465,6cyclictest131rcu_preempt08:20:013
2696784994611,5cyclictest131rcu_preempt07:25:003
2696779994639,4cyclictest2696538-21kworker/2:107:35:542
2696779994638,5cyclictest2701676-21kworker/2:209:02:452
2696779994638,5cyclictest2696538-21kworker/2:107:40:192
2696779994638,5cyclictest2696538-21kworker/2:107:33:252
2696779994638,5cyclictest2696538-21kworker/2:107:33:252
2696779994638,5cyclictest2696538-21kworker/2:107:27:552
2696779994637,5cyclictest2689919-21kworker/2:308:25:592
2696779994637,5cyclictest2689919-21kworker/2:308:25:592
269677599465,7cyclictest23-21ksoftirqd/107:13:551
269678499458,6cyclictest131rcu_preempt07:35:013
269678499458,6cyclictest131rcu_preempt07:35:013
2696784994511,5cyclictest131rcu_preempt08:55:023
2696784994511,5cyclictest131rcu_preempt08:55:023
2696779994538,4cyclictest2689919-21kworker/2:308:05:192
2696775994513,5cyclictest131rcu_preempt08:17:001
2696775994513,5cyclictest131rcu_preempt08:15:001
269678499449,5cyclictest131rcu_preempt08:10:013
269678499446,6cyclictest131rcu_preempt08:40:013
269678499446,6cyclictest131rcu_preempt08:40:013
269678499444,11cyclictest37-21ksoftirqd/309:05:013
2696779994437,4cyclictest2696538-21kworker/2:107:46:452
2696779994437,4cyclictest2696538-21kworker/2:107:46:442
2696779994437,4cyclictest2689919-21kworker/2:308:23:442
2696779994437,4cyclictest2689919-21kworker/2:308:10:142
269677599447,3cyclictest131rcu_preempt07:50:011
269677599445,8cyclictest23-21ksoftirqd/108:05:551
269677599445,5cyclictest131rcu_preempt07:40:111
269677599439,5cyclictest131rcu_preempt08:47:591
269677599437,4cyclictest131rcu_preempt08:50:561
269677599437,4cyclictest131rcu_preempt08:50:561
269677099431,11cyclictest12-21ksoftirqd/007:45:020
269678499415,8cyclictest37-21ksoftirqd/308:35:013
269677599419,5cyclictest131rcu_preempt07:10:021
2696779994033,4cyclictest2696538-21kworker/2:107:21:262
269677099386,6cyclictest12-21ksoftirqd/007:10:010
2696779993423,8cyclictest30-21ksoftirqd/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional