You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 02:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Apr 18, 2026 12:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6867039910294,5cyclictest37-21ksoftirqd/310:35:023
68669799977,7cyclictest131rcu_preempt12:30:021
686694999084,4cyclictest12-21ksoftirqd/007:35:020
686694997874,2cyclictest0-21swapper/007:55:020
686694997874,2cyclictest0-21swapper/007:55:020
686694997874,2cyclictest0-21swapper/007:25:030
686697997225,15cyclictest275-21dbus-daemon11:05:001
686694997116,49cyclictest131rcu_preempt08:50:000
686694997116,49cyclictest131rcu_preempt08:50:000
686694997033,28cyclictest151rcuc/010:35:020
686694997021,12cyclictest275-21dbus-daemon10:15:020
68669799684,23cyclictest705036-21kworker/u8:211:25:031
686701996530,4cyclictest768264-21latency_hist12:05:032
686697996520,12cyclictest23-21ksoftirqd/112:35:461
686697996427,5cyclictest705834-21sendmail09:40:011
686694996441,5cyclictest16850irq/134-800000012:15:010
686694996441,5cyclictest16850irq/134-800000012:15:000
68670399627,31cyclictest37-21ksoftirqd/307:40:023
68670399627,31cyclictest37-21ksoftirqd/307:40:023
686701996232,5cyclictest729545-21sh10:35:002
68669799622,7cyclictest131rcu_preempt10:35:031
686694996218,33cyclictest12-21ksoftirqd/011:02:270
686703996014,6cyclictest131rcu_preempt08:15:023
686701996023,11cyclictest30-21ksoftirqd/211:38:522
686697996029,4cyclictest761836-21latency_hist11:50:011
686697996024,8cyclictest23-21ksoftirqd/110:11:091
686694996010,33cyclictest131rcu_preempt11:10:500
686703995918,7cyclictest37-21ksoftirqd/308:25:023
686701995926,5cyclictest781483-21sh12:35:062
686697995928,8cyclictest23-21ksoftirqd/111:13:591
686697995927,9cyclictest23-21ksoftirqd/112:05:021
68669499591,5cyclictest131rcu_preempt11:35:020
686701995840,13cyclictest30-21ksoftirqd/207:20:002
686694995849,6cyclictest736156-21latency_hist10:50:020
686694995849,6cyclictest736156-21latency_hist10:50:020
686694995840,4cyclictest729970-21kworker/u8:410:38:450
686694995828,4cyclictest6985781ssh09:22:590
686703995712,9cyclictest131rcu_preempt11:25:023
686703995712,6cyclictest131rcu_preempt10:20:023
686694995727,15cyclictest748225-21ssh11:18:190
686703995613,7cyclictest131rcu_preempt12:20:013
686701995628,13cyclictest275-21dbus-daemon10:15:022
686701995621,10cyclictest753422-21sed11:30:032
686694995636,17cyclictest16850irq/134-800000012:00:000
686703995520,8cyclictest37-21ksoftirqd/312:10:023
686703995518,7cyclictest37-21ksoftirqd/310:40:013
686697995526,7cyclictest23-21ksoftirqd/109:54:261
686697995524,11cyclictest23-21ksoftirqd/110:15:551
686694995525,15cyclictest737069-21ssh10:52:100
686703995411,6cyclictest131rcu_preempt08:50:023
686703995411,6cyclictest131rcu_preempt08:50:023
686694995427,11cyclictest750743-21ssh11:24:110
686694995426,13cyclictest782660-21ssh12:38:130
686694995416,33cyclictest16850irq/134-800000011:30:000
686703995343,6cyclictest37-21ksoftirqd/312:00:003
686703995311,8cyclictest131rcu_preempt10:25:013
686703995310,6cyclictest131rcu_preempt12:30:283
686703995310,6cyclictest131rcu_preempt12:30:273
686703995310,6cyclictest131rcu_preempt11:30:003
686703995310,5cyclictest131rcu_preempt09:38:213
68669499535,39cyclictest12-21ksoftirqd/009:40:140
686694995345,4cyclictest0-21swapper/007:45:020
686694995331,17cyclictest3467602-21systemd-journal09:50:010
686694995330,4cyclictest131rcu_preempt11:50:010
686694995327,11cyclictest727184-21ssh10:29:290
686694995326,11cyclictest779609-21ssh12:30:530
686694995326,11cyclictest779609-21ssh12:30:520
686694995325,13cyclictest700431-21ssh09:27:100
686694995324,14cyclictest763736-21ssh11:54:300
68670399529,5cyclictest131rcu_preempt09:30:523
68670199528,26cyclictest0-21swapper/212:26:102
686694995235,13cyclictest774719-21cron12:19:590
686694995227,9cyclictest722740-21ssh10:19:020
686694995227,11cyclictest744562-21ssh11:09:510
686694995224,16cyclictest131rcu_preempt11:36:030
686694995224,12cyclictest778968-21ssh12:29:530
686694995216,20cyclictest16850irq/134-800000009:35:140
68670399518,7cyclictest37-21ksoftirqd/311:05:013
68670399518,5cyclictest131rcu_preempt12:28:483
686703995133,14cyclictest3467602-21systemd-journal09:00:003
68670399513,11cyclictest37-21ksoftirqd/308:39:023
686703995114,5cyclictest131rcu_preempt08:45:023
686703995111,5cyclictest131rcu_preempt10:15:023
686703995110,6cyclictest131rcu_preempt11:40:013
686701995143,5cyclictest731758-21kworker/2:111:58:492
686701995143,5cyclictest695283-21kworker/2:011:20:002
686701995143,5cyclictest689014-21kworker/2:008:20:012
686701995143,5cyclictest689014-21kworker/2:008:20:012
686701995110,6cyclictest131rcu_preempt10:27:352
686697995110,6cyclictest131rcu_preempt11:30:021
68669499515,9cyclictest12-21ksoftirqd/007:15:020
686694995134,11cyclictest1-21systemd10:05:010
686694995125,22cyclictest711230-21ssh09:52:140
686694995116,31cyclictest16850irq/134-800000012:07:000
686694995115,6cyclictest16850irq/134-800000010:45:020
686694995115,32cyclictest16850irq/134-800000009:15:220
68670399509,5cyclictest131rcu_preempt09:15:203
68670399509,5cyclictest131rcu_preempt08:05:013
68670399507,6cyclictest131rcu_preempt09:10:013
686703995015,5cyclictest131rcu_preempt10:02:003
686701995042,5cyclictest772548-21kworker/2:312:31:352
686701995042,5cyclictest772548-21kworker/2:312:31:342
686701995042,5cyclictest731758-21kworker/2:111:49:352
68669799503,7cyclictest131rcu_preempt11:35:021
686694995015,31cyclictest16850irq/134-800000009:57:300
68670399498,5cyclictest131rcu_preempt07:28:023
68670399497,6cyclictest131rcu_preempt09:53:243
68670399494,12cyclictest37-21ksoftirqd/310:00:003
686703994911,5cyclictest131rcu_preempt12:05:013
686703994910,5cyclictest131rcu_preempt10:05:063
68670199494,8cyclictest30-21ksoftirqd/212:12:132
68670199494,8cyclictest30-21ksoftirqd/212:12:122
686701994941,5cyclictest714512-21kworker/2:110:23:382
686701994941,5cyclictest690859-21kworker/2:309:55:522
686701994941,5cyclictest690859-21kworker/2:309:49:022
686701994941,5cyclictest690859-21kworker/2:309:35:532
686701994941,5cyclictest690859-21kworker/2:309:34:532
686701994941,5cyclictest690859-21kworker/2:309:12:332
686701994940,6cyclictest727417-21kworker/2:210:37:392
686701994940,6cyclictest695283-21kworker/2:009:43:172
686701994940,6cyclictest689014-21kworker/2:008:15:002
686701994940,5cyclictest731758-21kworker/2:111:44:452
686701994940,5cyclictest689014-21kworker/2:008:23:022
68670199493,6cyclictest3467602-21systemd-journal09:10:002
686697994934,11cyclictest3467602-21systemd-journal09:45:021
68669499494,42cyclictest12-21ksoftirqd/010:56:110
686694994915,30cyclictest16850irq/134-800000009:13:260
686694994914,31cyclictest16850irq/134-800000012:00:150
68669499491,13cyclictest777005-21ssh12:25:030
68669499491,13cyclictest777005-21ssh12:25:020
68670399487,5cyclictest131rcu_preempt09:46:533
686703994811,4cyclictest131rcu_preempt08:31:023
686703994811,4cyclictest131rcu_preempt08:31:023
686703994810,5cyclictest131rcu_preempt08:00:013
686701994842,4cyclictest695283-21kworker/2:010:00:562
686701994840,5cyclictest731758-21kworker/2:110:54:162
686701994840,5cyclictest695283-21kworker/2:009:50:042
686701994840,5cyclictest689014-21kworker/2:008:05:022
686701994828,16cyclictest3467602-21systemd-journal08:35:012
686701994828,16cyclictest3467602-21systemd-journal08:35:002
68669799487,8cyclictest23-21ksoftirqd/109:30:001
68669799487,6cyclictest0-21swapper/108:00:011
68669499483,10cyclictest725449-21grep10:25:020
68670399479,6cyclictest131rcu_preempt11:35:013
68670399479,6cyclictest131rcu_preempt09:30:003
68670399479,5cyclictest131rcu_preempt11:06:293
68670399476,9cyclictest37-21ksoftirqd/312:39:023
686703994738,6cyclictest361rcuc/309:43:113
686703994712,11cyclictest37-21ksoftirqd/311:49:063
686697994738,6cyclictest23-21ksoftirqd/111:10:001
686697994733,11cyclictest687151-21cron07:17:001
686694994732,11cyclictest1-21systemd07:20:010
68669499472,5cyclictest131rcu_preempt11:45:020
686694994713,30cyclictest16850irq/134-800000010:07:080
68670399468,6cyclictest131rcu_preempt11:20:023
68670399468,6cyclictest131rcu_preempt11:00:003
68670399466,6cyclictest131rcu_preempt08:10:003
68670399461,6cyclictest0-21swapper/312:12:103
68670399461,6cyclictest0-21swapper/312:12:103
686703994611,6cyclictest131rcu_preempt11:53:023
686703994610,5cyclictest131rcu_preempt10:29:253
68670199468,6cyclictest131rcu_preempt09:05:012
68670199463,9cyclictest701712-21sort09:30:022
686701994637,6cyclictest695283-21kworker/2:009:22:222
68669499467,25cyclictest19350irq/34-mmc107:45:010
686703994512,5cyclictest131rcu_preempt08:15:333
686703994512,5cyclictest131rcu_preempt08:15:333
686703994511,6cyclictest131rcu_preempt09:25:033
686703994511,5cyclictest131rcu_preempt10:45:013
686703994511,4cyclictest131rcu_preempt08:55:003
686701994538,5cyclictest695283-21kworker/2:011:12:462
686701994537,5cyclictest731758-21kworker/2:110:47:162
686701994537,5cyclictest731758-21kworker/2:110:47:152
686701994537,5cyclictest690859-21kworker/2:308:37:122
686701994512,5cyclictest131rcu_preempt10:08:312
68669799456,4cyclictest131rcu_preempt12:00:021
68669799454,5cyclictest774889-21sed12:20:021
686697994531,9cyclictest3467602-21systemd-journal09:10:001
68669499452,7cyclictest692391-21sendmail-msp09:00:000
686694994510,30cyclictest16850irq/134-800000009:31:030
68670399449,6cyclictest131rcu_preempt10:45:583
68670399449,6cyclictest131rcu_preempt10:45:583
68670399444,7cyclictest37-21ksoftirqd/307:55:023
68670399444,7cyclictest37-21ksoftirqd/307:55:023
68670399444,6cyclictest131rcu_preempt07:20:143
68670399444,6cyclictest131rcu_preempt07:20:143
686703994410,6cyclictest131rcu_preempt11:45:013
686703994410,5cyclictest131rcu_preempt10:51:213
68670199448,5cyclictest131rcu_preempt11:06:162
68670199444,3cyclictest131rcu_preempt11:21:522
686701994436,5cyclictest772548-21kworker/2:312:23:372
686701994436,5cyclictest772548-21kworker/2:312:23:362
686701994436,5cyclictest731758-21kworker/2:110:41:052
686701994436,5cyclictest690859-21kworker/2:309:17:472
686701994435,5cyclictest714512-21kworker/2:110:20:022
68670199443,4cyclictest131rcu_preempt08:45:012
686701994429,11cyclictest1-21systemd07:45:002
68670199442,7cyclictest739501-21ssh10:57:532
686697994429,11cyclictest23-21ksoftirqd/112:06:551
68669799441,8cyclictest0-21swapper/111:35:511
68669799441,11cyclictest692303-21latency_hist08:55:021
68669499449,5cyclictest131rcu_preempt07:40:010
68669499449,5cyclictest131rcu_preempt07:40:000
686694994434,7cyclictest12-21ksoftirqd/008:54:310
686694994428,11cyclictest3467602-21systemd-journal08:20:000
686694994428,11cyclictest3467602-21systemd-journal08:20:000
68669499442,38cyclictest693053-21latency_hist09:10:020
68670399438,5cyclictest131rcu_preempt12:20:543
68670399438,5cyclictest131rcu_preempt12:20:543
68670399438,5cyclictest131rcu_preempt07:10:023
68670399431,9cyclictest0-21swapper/311:14:043
68670399431,8cyclictest0-21swapper/309:14:563
68670199434,6cyclictest131rcu_preempt08:00:002
686701994335,5cyclictest731758-21kworker/2:112:08:352
68670199432,11cyclictest742188-21ssh11:04:192
686701994311,5cyclictest131rcu_preempt12:18:372
68669799437,7cyclictest23-21ksoftirqd/110:00:011
68669799435,5cyclictest131rcu_preempt09:20:001
686697994325,14cyclictest748881-21cron11:20:001
68669799431,6cyclictest0-21swapper/110:25:091
68669799431,5cyclictest0-21swapper/112:14:121
68669799431,5cyclictest0-21swapper/112:14:111
68669499437,26cyclictest686693-21latency_hist07:10:010
686694994336,4cyclictest16950irq/224-800000008:45:010
686694994317,20cyclictest131rcu_preempt08:00:000
686703994213,4cyclictest131rcu_preempt07:35:013
686701994232,7cyclictest30-21ksoftirqd/207:50:012
68670199421,11cyclictest763234-21ssh11:53:082
68669799426,6cyclictest131rcu_preempt08:58:011
68669799423,3cyclictest131rcu_preempt09:25:031
686697994231,7cyclictest309-21in:imuxsock07:45:001
68669799422,9cyclictest0-21swapper/110:55:271
686697994225,13cyclictest3467602-21systemd-journal08:25:001
68669799421,9cyclictest724007-21ssh10:21:431
68669799421,5cyclictest0-21swapper/110:06:081
68669799421,5cyclictest0-21swapper/109:40:581
68669799421,4cyclictest131rcu_preempt10:48:161
68669799421,4cyclictest131rcu_preempt10:48:151
686697994211,5cyclictest131rcu_preempt10:37:041
68669799421,12cyclictest733582-21ssh10:44:181
68669499429,29cyclictest131rcu_preempt08:15:010
686694994231,7cyclictest19350irq/34-mmc109:05:020
686703994127,10cyclictest3467602-21systemd-journal07:10:133
68670199416,6cyclictest131rcu_preempt07:35:152
68670199416,6cyclictest131rcu_preempt07:35:152
686701994132,6cyclictest731758-21kworker/2:111:30:352
686697994131,7cyclictest690120-21sed08:15:011
68669499416,6cyclictest131rcu_preempt08:35:010
68669499416,6cyclictest131rcu_preempt08:35:010
686694994134,4cyclictest16950irq/224-800000008:30:000
68670399403,29cyclictest688757-21cut07:50:003
68670199409,4cyclictest131rcu_preempt07:35:002
68669799407,5cyclictest131rcu_preempt10:02:431
68669799405,5cyclictest131rcu_preempt11:55:001
68669799404,4cyclictest131rcu_preempt07:10:021
686697994031,6cyclictest309-21in:imuxsock09:10:481
68669799402,9cyclictest781104-21sh12:34:441
68669799402,9cyclictest781104-21sh12:34:441
68669799402,9cyclictest23-21ksoftirqd/108:29:091
686697994026,11cyclictest23-21ksoftirqd/112:22:491
686697994026,11cyclictest23-21ksoftirqd/112:22:491
686697994025,9cyclictest309-21in:imuxsock07:35:011
686697994023,13cyclictest275-21dbus-daemon09:35:011
68669799401,4cyclictest0-21swapper/111:42:121
68669499405,5cyclictest131rcu_preempt08:10:020
68669499404,7cyclictest12-21ksoftirqd/008:05:020
68670399392,10cyclictest37-21ksoftirqd/308:30:003
68670199395,6cyclictest131rcu_preempt08:47:012
68670199395,6cyclictest131rcu_preempt08:47:012
686701993931,5cyclictest679852-21kworker/2:107:29:342
68669799391,5cyclictest131rcu_preempt07:55:031
68669799391,5cyclictest131rcu_preempt07:55:021
68669799391,10cyclictest687516-21sed07:25:021
68669799391,10cyclictest687516-21sed07:25:021
68669499391,34cyclictest691374-21unin-run08:40:000
686694993912,7cyclictest131rcu_preempt07:25:030
686694993912,7cyclictest131rcu_preempt07:25:020
68670399386,5cyclictest131rcu_preempt07:20:033
686701993830,5cyclictest30-21ksoftirqd/207:10:022
68670199381,5cyclictest131rcu_preempt08:10:012
686701993810,8cyclictest131rcu_preempt08:55:502
68669799388,5cyclictest131rcu_preempt10:54:051
68669799386,6cyclictest131rcu_preempt08:10:021
686697993828,6cyclictest309-21in:imuxsock08:05:011
686697993819,13cyclictest275-21dbus-daemon08:50:021
686697993819,13cyclictest275-21dbus-daemon08:50:011
68669499382,8cyclictest690574-21latency_hist08:25:020
68670399379,5cyclictest131rcu_preempt07:45:013
686703993723,9cyclictest275-21dbus-daemon09:05:013
68669799376,21cyclictest690798-21kworker/u8:108:45:021
68669799372,6cyclictest0-21swapper/108:20:001
68669799372,6cyclictest0-21swapper/108:20:001
686697993726,7cyclictest687865-21sshd07:30:031
686697993722,11cyclictest23-21ksoftirqd/108:40:001
686701993623,9cyclictest30-21ksoftirqd/207:50:272
686701993623,9cyclictest30-21ksoftirqd/207:50:262
68669799361,4cyclictest131rcu_preempt09:05:021
68669799361,4cyclictest131rcu_preempt07:40:021
68669799361,4cyclictest131rcu_preempt07:40:011
686701993525,7cyclictest30-21ksoftirqd/208:30:012
686701993524,8cyclictest30-21ksoftirqd/208:53:262
68670199351,4cyclictest131rcu_preempt07:24:302
68670199351,4cyclictest131rcu_preempt07:24:302
68669799353,3cyclictest131rcu_preempt08:35:011
68669799353,3cyclictest131rcu_preempt08:35:011
68669799351,4cyclictest131rcu_preempt07:50:021
68670199342,3cyclictest131rcu_preempt07:15:012
68669799333,5cyclictest131rcu_preempt07:15:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional