You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-28 - 19:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Oct 28, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
97002242166,24sleep30-21swapper/307:06:073
98072241164,25sleep10-21swapper/107:07:291
99282239163,24sleep20-21swapper/207:09:032
99262237161,25sleep00-21swapper/007:09:010
10109999793,3cyclictest0-21swapper/211:56:082
11822630,4sleep232-21ksoftirqd/211:40:192
1011599581,29cyclictest0-21swapper/310:58:253
1011599581,29cyclictest0-21swapper/310:58:253
321132490,1sleep00-21swapper/010:25:150
83342470,4sleep011-21rcuc/009:30:150
296532460,4sleep3401ktimersoftd/309:05:163
10109994517,5cyclictest8881-21kworker/2:011:06:472
273812440,1sleep10-21swapper/109:00:141
1011599431,29cyclictest0-21swapper/310:32:013
1011599431,27cyclictest0-21swapper/310:28:513
10115994230,3cyclictest0-21swapper/308:08:453
1011599421,32cyclictest0-21swapper/310:21:313
1011599421,28cyclictest0-21swapper/308:25:293
1011599421,27cyclictest0-21swapper/307:31:593
1011599411,33cyclictest0-21swapper/307:13:053
1011599411,28cyclictest0-21swapper/310:48:333
1011599410,27cyclictest0-21swapper/311:48:593
1011599401,33cyclictest0-21swapper/310:00:313
1011599401,27cyclictest0-21swapper/309:38:573
1011599401,26cyclictest0-21swapper/311:19:533
1011599401,25cyclictest0-21swapper/309:48:033
10115993925,8cyclictest0-21swapper/312:39:013
10115993924,9cyclictest0-21swapper/312:04:373
1011599391,26cyclictest0-21swapper/312:09:233
1011599391,26cyclictest0-21swapper/311:36:253
1011599391,15cyclictest0-21swapper/310:19:533
10115993826,3cyclictest0-21swapper/308:19:473
1011599381,31cyclictest0-21swapper/309:14:013
1011599381,28cyclictest0-21swapper/310:12:413
1011599381,22cyclictest0-21swapper/310:42:493
1011599381,22cyclictest0-21swapper/308:42:493
1011599380,29cyclictest0-21swapper/312:29:153
1010999385,5cyclictest780-21rngd07:29:062
1009999381,12cyclictest3149-21kworker/0:209:19:550
1009999380,12cyclictest25155-21kworker/0:110:22:310
1011599371,27cyclictest0-21swapper/309:41:073
1011599371,27cyclictest0-21swapper/308:52:073
1011599371,26cyclictest0-21swapper/310:08:233
1011599371,22cyclictest0-21swapper/312:22:273
1011599371,17cyclictest0-21swapper/311:10:093
1009999370,12cyclictest16788-21kworker/0:307:15:160
10115993628,3cyclictest0-21swapper/307:51:093
1011599361,26cyclictest0-21swapper/309:56:033
1011599361,26cyclictest0-21swapper/309:16:073
1011599361,16cyclictest0-21swapper/311:29:393
1011599361,12cyclictest0-21swapper/311:08:533
1011599361,10cyclictest0-21swapper/312:33:213
1011599360,13cyclictest0-21swapper/311:57:393
1009999361,6cyclictest25849-21kworker/0:211:30:240
1009999360,12cyclictest21761-21kworker/0:212:31:120
1009999360,11cyclictest0-21swapper/012:13:580
1011599351,24cyclictest0-21swapper/311:42:093
1011599351,12cyclictest0-21swapper/311:30:073
1011599351,12cyclictest0-21swapper/311:21:113
1011599351,11cyclictest0-21swapper/310:50:233
1009999351,26cyclictest0-21swapper/011:15:470
1009999350,10cyclictest16685-21kworker/0:210:00:490
1011599341,11cyclictest0-21swapper/311:04:173
1011599341,11cyclictest0-21swapper/309:22:273
1011599340,15cyclictest0-21swapper/308:55:073
1011599340,14cyclictest0-21swapper/309:31:473
1011599340,11cyclictest0-21swapper/308:48:553
1010999345,2cyclictest32-21ksoftirqd/207:10:252
1009999342,13cyclictest9196-21hddtemp_smartct10:45:180
1009999341,6cyclictest24033-21kworker/0:112:39:080
1009999341,16cyclictest5751-21kworker/0:210:39:270
1009999340,13cyclictest0-21swapper/011:27:020
199552330,4sleep21010999cyclictest11:10:162
1011599331,16cyclictest0-21swapper/308:22:333
1011599331,11cyclictest0-21swapper/312:17:193
1011599331,10cyclictest0-21swapper/307:55:193
1011599330,21cyclictest0-21swapper/308:38:413
1011599330,12cyclictest0-21swapper/307:38:393
1010999335,2cyclictest30-21rcuc/210:04:312
1009999331,11cyclictest25155-21kworker/0:110:09:450
1009999330,12cyclictest7338-21kworker/0:009:28:010
1009999330,12cyclictest20095-21kworker/0:308:48:490
1009999330,12cyclictest12538-21kworker/0:110:58:580
1009999330,12cyclictest12538-21kworker/0:110:58:580
1009999330,11cyclictest4731-21kworker/0:308:11:390
10115993216,10cyclictest0-21swapper/308:00:253
1011599321,16cyclictest0-21swapper/310:37:073
1011599321,15cyclictest0-21swapper/312:13:193
1011599321,13cyclictest0-21swapper/307:21:233
10109993212,3cyclictest6082-21ntp_states08:10:222
1009999321,24cyclictest0-21swapper/012:07:540
1009999321,24cyclictest0-21swapper/011:49:270
1009999321,13cyclictest0-21swapper/011:20:100
1009999321,11cyclictest0-21swapper/008:58:370
1009999320,5cyclictest4316-21kworker/0:112:22:280
1009999320,18cyclictest6545-21kworker/0:011:55:540
1009999320,14cyclictest17833-21kworker/0:008:41:590
1009999320,12cyclictest4316-21kworker/0:112:15:300
242912310,1sleep10-21swapper/111:20:161
1011599311,15cyclictest0-21swapper/309:28:153
1011599311,14cyclictest0-21swapper/307:18:313
1011599311,13cyclictest0-21swapper/311:51:373
1009999311,2cyclictest14076-21kworker/0:009:52:280
1009999311,2cyclictest14076-21kworker/0:009:52:270
1009999311,24cyclictest0-21swapper/011:37:270
1009999311,13cyclictest14838-21kworker/0:211:07:220
1009999311,12cyclictest0-21swapper/007:40:130
1009999310,7cyclictest23967-21kworker/0:107:45:310
1009999310,7cyclictest23967-21kworker/0:107:45:310
1009999310,6cyclictest21761-21kworker/0:212:29:160
1009999310,14cyclictest3242-21kworker/0:309:23:130
1009999310,11cyclictest11839-21kworker/0:209:38:120
1009999310,10cyclictest14838-21kworker/0:211:12:260
277982300,3sleep123-21ksoftirqd/110:15:121
10115993017,3cyclictest0-21swapper/307:48:193
10115993017,3cyclictest0-21swapper/307:48:193
10115993014,3cyclictest0-21swapper/309:53:013
10115993014,3cyclictest0-21swapper/309:53:013
1011599301,27cyclictest0-21swapper/307:26:553
1011599301,16cyclictest0-21swapper/309:00:193
1011599301,12cyclictest0-21swapper/307:42:313
1011599301,10cyclictest0-21swapper/308:10:573
1010999302,4cyclictest32-21ksoftirqd/212:06:272
1009999301,6cyclictest29556-21kworker/0:010:32:020
1009999301,24cyclictest0-21swapper/011:02:170
1009999301,23cyclictest0-21swapper/012:04:550
1009999301,13cyclictest25155-21kworker/0:110:17:550
1009999301,13cyclictest0-21swapper/009:05:330
1009999301,12cyclictest12538-21kworker/0:110:53:240
1009999300,6cyclictest8003-21kworker/0:110:42:500
1009999300,6cyclictest25849-21kworker/0:211:41:240
1009999300,11cyclictest16788-21kworker/0:307:12:500
1009999300,11cyclictest15618-21kworker/0:108:33:030
1010999291,23cyclictest0-21swapper/211:49:412
1009999291,22cyclictest0-21swapper/011:50:490
1009999290,8cyclictest31054-21kworker/0:109:14:010
1009999290,12cyclictest19777-21kworker/0:007:39:070
1009999290,12cyclictest16788-21kworker/0:307:23:150
10109992816,2cyclictest8881-21kworker/2:008:46:102
10109992815,11cyclictest86850irq/129-enp1s0-07:23:362
10109992811,2cyclictest8881-21kworker/2:007:17:012
1010999281,12cyclictest10770-21hddtemp_smartct09:35:172
1010999281,11cyclictest8881-21kworker/2:011:16:232
1010399281,20cyclictest0-21swapper/111:31:581
1009999281,10cyclictest20095-21kworker/0:309:01:310
1009999280,13cyclictest11097-21kworker/0:208:25:310
1010999278,9cyclictest0-21swapper/211:22:132
1010999278,9cyclictest0-21swapper/209:30:212
1010999276,7cyclictest8881-21kworker/2:011:51:472
1010999275,3cyclictest8881-21kworker/2:011:01:592
1010999271,13cyclictest0-21swapper/210:28:052
1010999271,12cyclictest24990-21irqstats07:40:202
1010999270,14cyclictest0-21swapper/209:52:062
1010999270,14cyclictest0-21swapper/209:52:052
1009999271,10cyclictest20508-21kworker/0:109:58:150
1009999270,9cyclictest16788-21kworker/0:307:30:330
1009999270,11cyclictest14076-21kworker/0:009:45:240
1009999270,11cyclictest11097-21kworker/0:208:24:310
1010999269,2cyclictest86750irq/128-enp1s0-12:30:392
1010999261,12cyclictest0-21swapper/208:23:502
10109992610,3cyclictest0-21swapper/209:46:132
1010399261,19cyclictest0-21swapper/112:22:291
1010399260,21cyclictest0-21swapper/109:31:291
1009999266,4cyclictest30394-21kworker/0:307:53:230
1009999261,11cyclictest32604-21kworker/0:007:59:470
1009999261,11cyclictest0-21swapper/010:13:170
1009999260,10cyclictest32604-21kworker/0:008:00:170
1009999260,10cyclictest11097-21kworker/0:208:36:120
1011599251,14cyclictest0-21swapper/308:34:573
1010999258,2cyclictest10218-21ntpq12:00:212
1010999257,2cyclictest86750irq/128-enp1s0-12:20:172
1010999251,12cyclictest0-21swapper/211:32:132
1010399251,14cyclictest0-21swapper/107:54:521
1010399251,10cyclictest0-21swapper/112:26:031
1009999251,8cyclictest16788-21kworker/0:307:28:350
10099992511,3cyclictest9584-21kworker/0:109:43:280
1009999251,13cyclictest7382-21kworker/0:008:18:170
1009999251,12cyclictest32604-21kworker/0:008:08:010
1010999249,3cyclictest0-21swapper/207:50:142
1010999246,6cyclictest32-21ksoftirqd/209:11:482
1010999246,4cyclictest0-21swapper/208:35:582
1010999246,3cyclictest8881-21kworker/2:009:24:162
1010999244,2cyclictest8881-21kworker/2:009:25:232
1010999241,10cyclictest0-21swapper/208:34:462
1010999240,7cyclictest86850irq/129-enp1s0-07:32:202
1010399243,10cyclictest780-21rngd09:57:061
1010399242,19cyclictest20848-21ntp_states07:30:251
1010399242,13cyclictest2459-21sshd08:03:311
10103992415,3cyclictest0-21swapper/108:45:171
10103992415,3cyclictest0-21swapper/108:13:421
1010399241,17cyclictest0-21swapper/109:15:211
1010399241,17cyclictest0-21swapper/108:56:571
1010399241,12cyclictest0-21swapper/108:40:321
1010999239,2cyclictest0-21swapper/211:25:172
1010999233,7cyclictest8881-21kworker/2:011:35:112
1010999233,2cyclictest86550irq/126-enp1s0-09:58:192
1010999231,7cyclictest0-21swapper/210:44:072
1010999231,14cyclictest0-21swapper/208:06:542
1010999231,12cyclictest0-21swapper/210:05:342
1010999231,10cyclictest929-21df_inode08:00:162
1010999231,10cyclictest0-21swapper/212:15:092
1010999231,10cyclictest0-21swapper/209:09:262
1010999230,9cyclictest30-21rcuc/210:46:572
1010399232,15cyclictest0-21swapper/110:50:051
1010399232,15cyclictest0-21swapper/108:22:391
1010399231,20cyclictest0-21swapper/107:20:201
1010399231,15cyclictest31226-21diskmemload11:25:511
1010399231,10cyclictest0-21swapper/108:07:141
1009999231,10cyclictest22630-21/usr/sbin/munin08:50:240
1010999228,3cyclictest0-21swapper/207:37:422
1010999228,11cyclictest86750irq/128-enp1s0-08:59:402
1010999227,3cyclictest86750irq/128-enp1s0-08:18:202
1010999224,6cyclictest32-21ksoftirqd/210:37:232
10109992211,3cyclictest8881-21kworker/2:010:13:062
1010999221,10cyclictest0-21swapper/207:45:582
1010999221,10cyclictest0-21swapper/207:45:582
1010399225,5cyclictest86850irq/129-enp1s0-10:10:341
1010399223,16cyclictest26993-21grep07:45:171
1010399223,16cyclictest26993-21grep07:45:171
1010399222,4cyclictest23066-21munin-run10:05:021
1010399222,3cyclictest836-21runrttasks11:38:181
1010399222,15cyclictest23-21ksoftirqd/110:41:261
1010399222,15cyclictest0-21swapper/109:45:261
10103992214,3cyclictest0-21swapper/109:06:441
1010399221,16cyclictest0-21swapper/110:56:341
1010399221,16cyclictest0-21swapper/110:56:341
1010399221,15cyclictest0-21swapper/109:53:351
1010399221,15cyclictest0-21swapper/109:53:351
10103992211,3cyclictest16388-21smart_sda11:00:271
1010999219,10cyclictest86650irq/127-enp1s0-07:56:082
1010999216,6cyclictest18232-21cpu08:40:142
1010999214,2cyclictest86750irq/128-enp1s0-10:57:432
1010999214,2cyclictest86750irq/128-enp1s0-10:57:432
1010999211,4cyclictest0-21swapper/212:36:132
1010999211,14cyclictest0-21swapper/210:33:252
1010999211,13cyclictest0-21swapper/212:28:452
1010999210,10cyclictest0-21swapper/210:23:172
1010399218,11cyclictest16876-21kworker/u8:111:10:031
1010399213,2cyclictest86750irq/128-enp1s0-12:36:141
1010399212,2cyclictest86850irq/129-enp1s0-11:40:161
1010399212,18cyclictest0-21swapper/112:08:311
10103992117,3cyclictest0-21swapper/108:54:141
10103992114,3cyclictest0-21swapper/110:27:471
1010399211,18cyclictest31226-21diskmemload12:17:001
1010399211,18cyclictest0-21swapper/110:35:241
1010399211,15cyclictest0-21swapper/111:06:411
1010399211,15cyclictest0-21swapper/110:30:371
1010399211,15cyclictest0-21swapper/110:22:261
1010399211,15cyclictest0-21swapper/109:36:071
1010399211,15cyclictest0-21swapper/109:13:401
1010399211,11cyclictest14637-21ntp_states12:10:231
1010399210,9cyclictest24882-21/usr/sbin/munin10:05:321
1010399210,15cyclictest0-21swapper/108:15:041
1010999203,6cyclictest30-21rcuc/210:18:182
1010999201,7cyclictest0-21swapper/209:44:112
1010999201,12cyclictest0-21swapper/208:51:262
1010999200,8cyclictest30-21rcuc/209:00:102
1010399205,4cyclictest0-21swapper/112:32:471
10103992016,3cyclictest0-21swapper/107:36:481
1010399201,5cyclictest0-21swapper/111:53:411
1010399201,3cyclictest0-21swapper/108:30:461
1010399201,3cyclictest0-21swapper/107:10:161
10103992013,3cyclictest0-21swapper/109:27:181
10103992013,2cyclictest0-21swapper/109:40:241
1010399201,18cyclictest0-21swapper/111:49:271
1010399201,18cyclictest0-21swapper/108:36:121
1010399201,17cyclictest0-21swapper/107:26:161
1010399201,11cyclictest0-21swapper/111:57:341
1010399200,15cyclictest0-21swapper/109:23:021
1010999190,3cyclictest32-21ksoftirqd/210:51:432
1010399195,3cyclictest0-21swapper/110:45:431
1010399192,16cyclictest0-21swapper/107:56:001
1010399191,2cyclictest0-21swapper/107:18:261
1010399191,16cyclictest0-21swapper/112:00:101
1010399191,10cyclictest26233-21cat07:45:021
32172181,3sleep20-21swapper/209:19:352
1010999181,6cyclictest0-21swapper/212:10:072
10103991814,3cyclictest0-21swapper/111:16:341
1010399181,3cyclictest13714-21cat08:30:011
1010999160,13cyclictest0-21swapper/208:28:462
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional