You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 19:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Mon Apr 13, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
205927321330,3sleep00-21swapper/007:55:160
2040768210848,50sleep20-21swapper/207:05:102
204200827636,13sleep10-21swapper/107:07:191
204192527555,13sleep00-21swapper/007:06:130
139099740,61rtkit-daemon0-21swapper/307:06:123
21511412590,11sleep234-21ksoftirqd/212:15:162
21331432440,2sleep00-21swapper/011:25:130
21533582430,3sleep32153356-21python312:20:223
20610862420,4sleep2331rcuc/208:00:172
2042372994012,5cyclictest436-20systemd-journal08:50:313
2042358993633,2cyclictest161rcuc/007:35:120
2042372993113,3cyclictest2156712-21ntp_states12:30:193
2042372993010,4cyclictest2086769-21find09:15:113
204237299300,27cyclictest2119512-21ntp_states10:45:203
204237299290,6cyclictest2112997-21seq10:29:393
204235899289,10cyclictest13-21ksoftirqd/009:20:010
2042358992810,9cyclictest13-21ksoftirqd/011:05:000
204235899279,7cyclictest13-21ksoftirqd/008:05:000
204235899278,10cyclictest13-21ksoftirqd/012:10:010
204235899278,10cyclictest13-21ksoftirqd/010:40:010
204235899274,8cyclictest13-21ksoftirqd/008:05:170
204235899274,7cyclictest13-21ksoftirqd/007:30:170
139099270,4rtkit-daemon1389-21rtkit-daemon12:31:472
20598642260,6sleep20-21swapper/207:55:232
2042372992619,6cyclictest0-21swapper/310:32:493
2042372992617,8cyclictest0-21swapper/308:05:203
2042372992616,1cyclictest3056-21PK-Backend09:05:233
2042372992616,1cyclictest3056-21PK-Backend09:05:233
204237299260,1cyclictest3056-21PK-Backend09:40:303
2042364992617,7cyclictest2138216-21cat11:40:132
204235899264,6cyclictest2120243-21perf10:50:010
204235899262,7cyclictest13-21ksoftirqd/012:30:220
21467332250,4chrt3056-21PK-Backend12:05:002
2042372992518,6cyclictest0-21swapper/308:56:243
204235899256,7cyclictest13-21ksoftirqd/010:50:150
204235899256,7cyclictest13-21ksoftirqd/008:40:010
204235899254,8cyclictest13-21ksoftirqd/011:40:000
204235899254,4cyclictest2102041-21python309:55:210
204235899251,9cyclictest2115037-21perf10:35:000
20659442240,4sleep33056-21PK-Backend08:15:133
20472502240,9chrt0-21swapper/207:20:192
2042364992422,1cyclictest0-21swapper/207:26:562
2042362992414,3cyclictest26-21ksoftirqd/111:20:001
204235899245,7cyclictest1095-21runrttasks07:40:190
204235899245,10cyclictest0-21swapper/009:11:560
204235899244,7cyclictest13-21ksoftirqd/010:45:020
204235899243,12cyclictest13-21ksoftirqd/009:20:190
204235899242,6cyclictest13-21ksoftirqd/008:45:000
2042358992421,2cyclictest161rcuc/010:09:300
204235899241,7cyclictest13-21ksoftirqd/010:10:180
21332842230,12sleep234-21ksoftirqd/211:25:152
2042372992320,2cyclictest3056-21PK-Backend10:05:383
2042364992321,1cyclictest0-21swapper/210:09:262
2042364992316,6cyclictest2056852-21wc07:50:012
2042362992318,3cyclictest2047824-21seq07:23:131
2042362992317,4cyclictest2141108-21chrt11:47:431
204236299231,2cyclictest141rcu_preempt08:55:011
204235899233,6cyclictest13-21ksoftirqd/008:55:230
204235899232,7cyclictest13-21ksoftirqd/009:10:000
204235899232,7cyclictest13-21ksoftirqd/009:09:590
204235899232,7cyclictest13-21ksoftirqd/008:10:200
204235899232,7cyclictest13-21ksoftirqd/008:10:200
204235899232,6cyclictest13-21ksoftirqd/012:35:020
204235899232,6cyclictest13-21ksoftirqd/007:30:000
2042372992220,1cyclictest0-21swapper/312:38:213
2042372992219,2cyclictest3056-21PK-Backend11:58:143
2042372992219,2cyclictest3056-21PK-Backend11:58:133
204237299220,21cyclictest3056-21PK-Backend09:57:013
2042364992220,1cyclictest0-21swapper/212:05:322
2042364992220,1cyclictest0-21swapper/208:40:382
2042364992214,6cyclictest2113089-21wc10:30:012
2042364992214,6cyclictest2108972-21python310:15:222
204236499220,2cyclictest2055227-21latency_hist07:45:012
204236499220,21cyclictest0-21swapper/211:59:242
204236499220,21cyclictest0-21swapper/211:59:242
204236499220,21cyclictest0-21swapper/210:45:212
204236299224,2cyclictest141rcu_preempt11:05:011
204236299224,2cyclictest141rcu_preempt07:35:001
204236299223,2cyclictest141rcu_preempt07:55:011
2042362992217,4cyclictest26-21ksoftirqd/111:15:011
204235899229,9cyclictest883-21dbus-daemon09:30:010
204235899223,8cyclictest2098790-21chrt09:48:370
204235899223,7cyclictest13-21ksoftirqd/007:45:120
204235899222,6cyclictest13-21ksoftirqd/011:10:180
204235899222,6cyclictest13-21ksoftirqd/007:55:000
2042358992220,1cyclictest0-21swapper/012:19:150
204235899221,8cyclictest13-21ksoftirqd/012:10:130
2042358992218,3cyclictest161rcuc/008:20:100
2042358992218,3cyclictest161rcuc/007:10:170
139099220,3rtkit-daemon0-21swapper/208:45:172
2042372992119,1cyclictest3056-21PK-Backend12:18:013
2042372992119,1cyclictest3056-21PK-Backend12:12:063
2042372992119,1cyclictest3056-21PK-Backend11:21:233
2042372992119,1cyclictest3056-21PK-Backend07:50:203
2042372992119,1cyclictest3056-21PK-Backend07:25:523
2042372992119,1cyclictest3056-21PK-Backend07:15:183
2042372992119,1cyclictest0-21swapper/311:51:183
2042372992119,1cyclictest0-21swapper/310:42:433
2042372992119,1cyclictest0-21swapper/309:51:473
2042372992119,1cyclictest0-21swapper/308:26:503
2042372992118,2cyclictest3056-21PK-Backend12:00:163
2042372992118,2cyclictest3056-21PK-Backend10:03:023
2042372992118,2cyclictest3056-21PK-Backend10:03:013
2042372992118,2cyclictest3056-21PK-Backend09:25:513
2042372992118,2cyclictest0-21swapper/311:10:003
2042372992116,3cyclictest2061381-21python308:00:203
204237299210,20cyclictest0-21swapper/311:45:233
204237299210,20cyclictest0-21swapper/309:11:583
204237299210,19cyclictest3056-21PK-Backend11:00:543
204237299210,19cyclictest3056-21PK-Backend07:24:173
2042364992119,1cyclictest0-21swapper/211:22:402
2042364992119,1cyclictest0-21swapper/211:08:132
2042364992119,1cyclictest0-21swapper/210:41:432
2042364992119,1cyclictest0-21swapper/209:35:472
2042364992119,1cyclictest0-21swapper/209:25:042
2042364992119,1cyclictest0-21swapper/209:18:012
2042364992119,1cyclictest0-21swapper/208:36:392
2042364992119,1cyclictest0-21swapper/208:20:442
2042364992119,1cyclictest0-21swapper/207:50:012
2042364992119,1cyclictest0-21swapper/207:37:032
2042364992119,1cyclictest0-21swapper/207:14:462
204236499210,20cyclictest0-21swapper/211:39:232
204236499210,20cyclictest0-21swapper/211:15:432
204236499210,20cyclictest0-21swapper/210:36:172
204236499210,20cyclictest0-21swapper/207:33:042
204236499210,19cyclictest0-21swapper/211:45:172
204236299218,6cyclictest26-21ksoftirqd/108:20:111
2042362992114,6cyclictest2105506-21python310:05:211
204236299211,19cyclictest0-21swapper/112:05:281
204236299210,20cyclictest0-21swapper/109:09:131
204236299210,20cyclictest0-21swapper/109:09:121
204236299210,19cyclictest3056-21PK-Backend08:55:271
204235899213,8cyclictest2140702-21python311:45:210
204235899213,7cyclictest13-21ksoftirqd/012:00:120
204235899213,6cyclictest13-21ksoftirqd/010:25:110
204235899213,5cyclictest13-21ksoftirqd/012:25:000
204235899212,5cyclictest13-21ksoftirqd/011:40:140
2042358992117,3cyclictest161rcuc/010:19:110
204235899210,8cyclictest13-21ksoftirqd/011:05:190
204235899210,8cyclictest13-21ksoftirqd/008:50:190
204235899210,20cyclictest2069553-21df_inode08:25:130
204235899210,20cyclictest0-21swapper/010:59:580
204235899210,20cyclictest0-21swapper/010:02:260
204235899210,20cyclictest0-21swapper/010:02:260
204235899210,20cyclictest0-21swapper/009:35:080
139099210,14rtkit-daemon0-21swapper/308:33:433
204237299202,16cyclictest2128363-21tail11:10:193
2042372992018,1cyclictest3056-21PK-Backend11:17:313
2042372992018,1cyclictest3056-21PK-Backend07:10:393
2042372992018,1cyclictest0-21swapper/311:44:223
2042372992018,1cyclictest0-21swapper/311:37:383
2042372992018,1cyclictest0-21swapper/310:57:173
2042372992018,1cyclictest0-21swapper/307:56:073
2042372992013,5cyclictest2055029-21cron07:45:013
204237299200,1cyclictest3056-21PK-Backend11:29:553
204237299200,19cyclictest3056-21PK-Backend08:13:453
204237299200,19cyclictest3056-21PK-Backend08:13:453
204237299200,19cyclictest0-21swapper/310:37:153
204237299200,19cyclictest0-21swapper/309:20:373
204237299200,18cyclictest3056-21PK-Backend08:22:233
2042364992018,1cyclictest3056-21PK-Backend11:30:382
2042364992018,1cyclictest0-21swapper/211:00:572
2042364992018,1cyclictest0-21swapper/209:48:162
2042364992018,1cyclictest0-21swapper/209:12:012
2042364992018,1cyclictest0-21swapper/208:16:182
2042364992018,1cyclictest0-21swapper/208:09:132
204236499200,1cyclictest0-21swapper/211:53:162
204236499200,1cyclictest0-21swapper/210:58:072
204236499200,19cyclictest0-21swapper/212:21:412
204236499200,19cyclictest0-21swapper/211:13:402
204236499200,19cyclictest0-21swapper/210:52:392
204236499200,19cyclictest0-21swapper/210:11:152
204236499200,19cyclictest0-21swapper/210:03:112
204236499200,19cyclictest0-21swapper/210:03:112
204236499200,19cyclictest0-21swapper/209:42:022
204236499200,19cyclictest0-21swapper/208:32:252
204236499200,19cyclictest0-21swapper/208:26:582
2042362992018,1cyclictest3056-21PK-Backend08:47:291
2042362992018,1cyclictest0-21swapper/110:13:551
2042362992018,1cyclictest0-21swapper/109:47:111
2042362992018,1cyclictest0-21swapper/109:41:581
2042362992017,2cyclictest3056-21PK-Backend10:20:571
2042362992017,2cyclictest3056-21PK-Backend07:59:371
2042362992013,5cyclictest2108813-21ntpq10:15:191
2042362992012,6cyclictest2094754-21awk09:35:191
204236299200,1cyclictest3056-21PK-Backend10:39:161
204236299200,1cyclictest0-21swapper/112:20:211
204236299200,19cyclictest3056-21PK-Backend11:42:031
204236299200,19cyclictest3056-21PK-Backend11:36:211
204236299200,19cyclictest3056-21PK-Backend10:53:351
204236299200,19cyclictest3056-21PK-Backend09:54:011
204236299200,19cyclictest3056-21PK-Backend09:21:021
204236299200,19cyclictest0-21swapper/112:00:351
204236299200,18cyclictest3056-21PK-Backend08:42:421
204235899208,2cyclictest141rcu_preempt08:45:150
204235899203,8cyclictest13-21ksoftirqd/011:50:150
2042358992019,1cyclictest0-21swapper/009:40:220
2042358992018,1cyclictest0-21swapper/011:24:120
2042358992018,1cyclictest0-21swapper/007:22:390
2042358992016,3cyclictest161rcuc/011:55:190
2042358992016,3cyclictest161rcuc/011:55:190
2042358992012,5cyclictest436-20systemd-journal07:20:000
204235899200,19cyclictest0-21swapper/008:16:200
139099200,3rtkit-daemon1389-21rtkit-daemon07:15:092
139099200,14rtkit-daemon3056-21PK-Backend11:31:073
139099200,14rtkit-daemon0-21swapper/209:20:322
2042372991917,1cyclictest0-21swapper/312:27:473
2042372991917,1cyclictest0-21swapper/307:32:013
2042372991916,2cyclictest3056-21PK-Backend10:16:073
2042372991914,3cyclictest2052104-21df_inode07:35:143
2042372991912,5cyclictest2056041-21ntpq07:45:193
2042372991910,7cyclictest2121084-21memory10:50:173
204237299190,1cyclictest3056-21PK-Backend09:32:413
204237299190,1cyclictest0-21swapper/310:22:093
204237299190,1cyclictest0-21swapper/310:11:413
204237299190,1cyclictest0-21swapper/309:45:453
2042364991917,1cyclictest3056-21PK-Backend08:11:512
2042364991917,1cyclictest3056-21PK-Backend08:11:512
2042364991917,1cyclictest0-21swapper/209:50:052
2042364991912,5cyclictest2100637-21apt-get09:55:012
2042364991911,6cyclictest436-20systemd-journal12:30:012
2042364991911,6cyclictest2149412-21grep12:10:172
204236499190,1cyclictest0-21swapper/210:21:572
204236499190,1cyclictest0-21swapper/209:34:452
204236499190,1cyclictest0-21swapper/209:00:592
2042362991917,1cyclictest3056-21PK-Backend12:36:381
2042362991917,1cyclictest3056-21PK-Backend12:32:461
2042362991917,1cyclictest3056-21PK-Backend12:25:141
2042362991917,1cyclictest3056-21PK-Backend10:49:091
2042362991917,1cyclictest0-21swapper/112:19:201
2042362991917,1cyclictest0-21swapper/111:26:471
2042362991917,1cyclictest0-21swapper/108:19:511
2042362991917,1cyclictest0-21swapper/107:26:591
2042362991916,2cyclictest3056-21PK-Backend11:05:191
2042362991912,5cyclictest26-21ksoftirqd/109:00:101
204236299190,8cyclictest3056-21PK-Backend10:30:241
204236299190,1cyclictest3056-21PK-Backend10:55:241
204236299190,1cyclictest3056-21PK-Backend10:41:051
204236299190,1cyclictest3056-21PK-Backend08:39:311
204236299190,1cyclictest0-21swapper/112:12:231
204236299190,1cyclictest0-21swapper/111:22:551
204236299190,1cyclictest0-21swapper/108:26:071
204236299190,18cyclictest0-21swapper/111:52:361
204236299190,18cyclictest0-21swapper/107:14:481
204235899191,8cyclictest2099576-21hddtemp_smartct09:50:160
2042358991916,2cyclictest161rcuc/009:03:060
204235899191,4cyclictest2154966-21ntp_states12:25:180
2042358991914,3cyclictest2135129-21sed11:30:160
2042358991914,3cyclictest2093190-21threads09:30:220
2042358991911,6cyclictest2110237-21NetworkManager10:20:140
204235899190,5cyclictest13-21ksoftirqd/008:30:200
204237299181,14cyclictest2075846-21apt-get08:45:033
204236499189,7cyclictest2158711-21ntp_states12:35:192
2042364991816,1cyclictest0-21swapper/209:09:242
2042364991816,1cyclictest0-21swapper/209:09:242
2042364991812,4cyclictest1095-21runrttasks08:50:192
2042364991811,6cyclictest2114835-21seq10:32:362
204236299189,7cyclictest2044743-21ntp_states07:15:031
2042362991816,1cyclictest0-21swapper/110:02:391
2042362991816,1cyclictest0-21swapper/110:02:381
2042362991816,1cyclictest0-21swapper/109:56:371
2042362991815,2cyclictest3056-21PK-Backend09:12:371
204235899181,5cyclictest13-21ksoftirqd/011:20:010
2042372991712,3cyclictest1095-21runrttasks08:40:213
2042372991710,5cyclictest2147004-21/usr/sbin/munin12:05:213
204237299170,15cyclictest2074109-21cron08:40:003
204236299179,6cyclictest2055045-21cron07:45:001
204236299178,8cyclictest918-21rs:main1
204236299174,2cyclictest141rcu_preempt08:10:141
204236299174,2cyclictest141rcu_preempt08:10:141
204236299173,2cyclictest141rcu_preempt08:00:151
204236299172,2cyclictest141rcu_preempt10:25:171
2042362991710,5cyclictest2134960-21cat11:30:141
204236299170,4cyclictest3056-21PK-Backend07:38:091
204236299169,2cyclictest26-21ksoftirqd/109:15:201
204236299161,13cyclictest2092011-20unin-run09:30:001
204236299160,9cyclictest141rcu_preempt08:31:271
204236299160,2cyclictest3056-21PK-Backend07:45:201
2042372991513,1cyclictest3056-21PK-Backend09:37:063
204236499158,6cyclictest2080056-21cat08:55:132
204236299154,2cyclictest141rcu_preempt08:05:151
2042362991513,1cyclictest0-21swapper/111:59:331
2042362991513,1cyclictest0-21swapper/111:59:321
2042372991410,3cyclictest3056-21PK-Backend09:00:453
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional