You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 15:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Nov 14, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
200152321110,7sleep32197014-21PK-Backend07:40:203
20043622950,5sleep20-21swapper/207:50:012
198963028768,13sleep30-21swapper/307:09:113
20381652860,7sleep32197014-21PK-Backend09:25:023
162399860,8rtkit-daemon1622-21rtkit-daemon08:03:321
21011692830,7sleep32197014-21PK-Backend12:20:153
162399810,63rtkit-daemon0-21swapper/207:07:412
20868552760,5sleep30-21swapper/311:40:143
20142212760,5sleep10-21swapper/108:15:291
20992192730,15sleep0131ksoftirqd/012:15:130
198939027340,27sleep10-21swapper/107:06:021
198940426645,15sleep00-21swapper/007:06:130
162399650,8rtkit-daemon1622-21rtkit-daemon11:30:423
198982899594,5cyclictest141rcu_preempt09:30:190
1989831995819,22cyclictest26-21ksoftirqd/110:42:271
162399580,9rtkit-daemon1622-21rtkit-daemon11:16:372
162399580,8rtkit-daemon1622-21rtkit-daemon08:25:301
162399570,9rtkit-daemon1622-21rtkit-daemon08:25:570
1989836995620,20cyclictest34-21ksoftirqd/212:35:012
1989831995622,19cyclictest26-21ksoftirqd/107:15:011
198983199561,23cyclictest26-21ksoftirqd/109:30:171
162399560,8rtkit-daemon1622-21rtkit-daemon10:26:043
162399560,8rtkit-daemon1622-21rtkit-daemon07:33:371
1989836995517,20cyclictest34-21ksoftirqd/211:05:002
198983199558,29cyclictest26-21ksoftirqd/109:15:141
198983199554,48cyclictest1622-21rtkit-daemon09:13:111
1989831995511,16cyclictest26-21ksoftirqd/111:28:051
1989831995511,16cyclictest26-21ksoftirqd/111:28:051
19940072540,7chrt2197014-21PK-Backend07:20:133
198983699546,11cyclictest34-21ksoftirqd/209:15:152
1989836995432,7cyclictest34-21ksoftirqd/209:45:542
20136632520,6sleep32197014-21PK-Backend08:15:143
1989831995228,9cyclictest26-21ksoftirqd/110:25:011
1989831995220,11cyclictest26-21ksoftirqd/112:10:011
1989831995215,10cyclictest26-21ksoftirqd/110:00:011
1989828995212,5cyclictest141rcu_preempt10:30:020
198983699519,19cyclictest2061323-21tr10:30:132
198983699512,16cyclictest34-21ksoftirqd/208:50:022
198983199516,41cyclictest1622-21rtkit-daemon07:17:551
162399510,8rtkit-daemon1622-21rtkit-daemon12:00:233
162399510,8rtkit-daemon1622-21rtkit-daemon07:26:450
20918702500,8chrt2197014-21PK-Backend11:55:032
20918702500,8chrt2197014-21PK-Backend11:55:022
19924512500,6sleep02197014-21PK-Backend07:15:140
198983699504,9cyclictest34-21ksoftirqd/210:55:022
198983699503,24cyclictest1753-21gnome-shell08:30:122
1989836995019,13cyclictest34-21ksoftirqd/210:09:302
1989836995014,18cyclictest34-21ksoftirqd/211:40:152
1989831995016,18cyclictest26-21ksoftirqd/112:20:011
1989831995015,20cyclictest26-21ksoftirqd/111:15:011
1989836994926,7cyclictest34-21ksoftirqd/209:40:282
1989836994917,9cyclictest34-21ksoftirqd/208:55:032
1989836994916,19cyclictest34-21ksoftirqd/208:35:152
1989836994914,19cyclictest34-21ksoftirqd/208:15:012
198983199496,14cyclictest26-21ksoftirqd/108:50:011
198983199493,9cyclictest261ksoftirqd/110:50:161
198983199491,44cyclictest1622-21rtkit-daemon07:36:011
198983199491,44cyclictest1622-21rtkit-daemon07:36:001
1989838994818,26cyclictest2060627-21apt-get10:30:093
1989836994823,12cyclictest34-21ksoftirqd/212:35:202
1989831994815,18cyclictest26-21ksoftirqd/110:30:021
198983199481,33cyclictest26-21ksoftirqd/108:05:131
162399480,42rtkit-daemon1989821-21cyclictest10:10:123
162399480,34rtkit-daemon2197014-21PK-Backend08:44:192
198983699478,17cyclictest34-21ksoftirqd/207:40:022
198983699478,17cyclictest34-21ksoftirqd/207:40:012
198983699477,22cyclictest2179-21gvfsd-metadata08:05:022
1989836994722,7cyclictest34-21ksoftirqd/211:32:552
1989836994721,23cyclictest34-21ksoftirqd/209:20:272
1989836994710,16cyclictest34-21ksoftirqd/210:40:592
1989836994710,15cyclictest34-21ksoftirqd/208:16:382
198983199474,8cyclictest26-21ksoftirqd/110:40:011
1989831994717,9cyclictest26-21ksoftirqd/109:51:461
1989831994712,8cyclictest26-21ksoftirqd/110:00:371
198983199471,19cyclictest26-21ksoftirqd/112:00:301
162399470,4rtkit-daemon2197014-21PK-Backend11:11:343
198983699465,8cyclictest34-21ksoftirqd/208:25:032
198983699464,10cyclictest34-21ksoftirqd/210:00:162
1989836994615,8cyclictest34-21ksoftirqd/211:50:012
1989836994610,19cyclictest34-21ksoftirqd/209:55:152
1989836994610,18cyclictest2197014-21PK-Backend07:56:312
1989836994610,18cyclictest2197014-21PK-Backend07:56:302
198983199467,21cyclictest26-21ksoftirqd/108:30:131
198983199466,20cyclictest26-21ksoftirqd/110:30:271
198983199462,8cyclictest26-21ksoftirqd/109:40:001
198983199461,17cyclictest20877622timerandwakeup11:40:321
1989828994613,14cyclictest141rcu_preempt09:20:200
162399460,37rtkit-daemon1989821-21cyclictest07:59:291
162399460,37rtkit-daemon1989821-21cyclictest07:59:291
198983699459,18cyclictest34-21ksoftirqd/212:15:012
198983699453,7cyclictest34-21ksoftirqd/211:00:012
1989836994519,9cyclictest34-21ksoftirqd/209:05:002
1989836994517,10cyclictest141rcu_preempt10:36:102
1989836994514,16cyclictest34-21ksoftirqd/209:11:572
198983699451,23cyclictest34-21ksoftirqd/210:20:242
1989836994511,17cyclictest34-21ksoftirqd/210:50:032
198983199458,11cyclictest26-21ksoftirqd/109:00:021
198983199454,8cyclictest26-21ksoftirqd/110:50:021
198983199452,9cyclictest26-21ksoftirqd/109:30:011
198983199452,9cyclictest26-21ksoftirqd/109:30:001
198983199451,17cyclictest1753-21gnome-shell11:49:581
198982899459,7cyclictest20111302apt-check08:06:270
1989828994517,23cyclictest2104674-40apt-check12:30:130
1989828994514,7cyclictest141rcu_preempt11:05:280
1989828994514,6cyclictest141rcu_preempt11:12:220
198983899448,12cyclictest141rcu_preempt12:20:013
198983899444,23cyclictest2055864-21tr10:15:143
1989838994415,11cyclictest141rcu_preempt10:25:023
1989838994412,12cyclictest141rcu_preempt07:10:193
198983699447,21cyclictest34-21ksoftirqd/207:15:022
198983699446,21cyclictest34-21ksoftirqd/212:18:192
198983699446,16cyclictest34-21ksoftirqd/209:26:142
198983699446,16cyclictest34-21ksoftirqd/209:26:142
198983699443,9cyclictest34-21ksoftirqd/210:25:182
198983699443,19cyclictest2197014-21PK-Backend09:31:062
1989836994419,8cyclictest34-21ksoftirqd/207:24:272
1989836994417,8cyclictest34-21ksoftirqd/209:36:152
1989836994413,10cyclictest34-21ksoftirqd/210:18:512
1989836994410,19cyclictest34-21ksoftirqd/211:05:142
198983199449,14cyclictest26-21ksoftirqd/109:00:171
198983199448,19cyclictest26-21ksoftirqd/112:30:301
1989831994424,15cyclictest3242253-21nautilus11:25:021
1989831994415,8cyclictest26-21ksoftirqd/109:20:321
198983199441,20cyclictest26-21ksoftirqd/109:40:241
198983199441,11cyclictest26-21ksoftirqd/112:20:161
198982899449,16cyclictest141rcu_preempt11:47:120
198982899446,5cyclictest141rcu_preempt07:10:230
1989828994412,7cyclictest141rcu_preempt10:10:000
198983899439,30cyclictest1753-21gnome-shell08:39:433
1989838994325,13cyclictest3242253-21nautilus11:05:023
1989838994312,11cyclictest141rcu_preempt11:21:213
198983699437,10cyclictest34-21ksoftirqd/209:10:002
198983699435,21cyclictest34-21ksoftirqd/212:05:142
198983699434,20cyclictest34-21ksoftirqd/208:27:212
1989836994333,7cyclictest331rcuc/212:30:002
1989836994333,7cyclictest331rcuc/212:30:002
1989836994327,12cyclictest855-21dbus-daemon08:10:002
1989836994318,8cyclictest34-21ksoftirqd/211:11:222
1989836994317,13cyclictest34-21ksoftirqd/212:24:372
198983699431,22cyclictest34-21ksoftirqd/211:25:372
198983699431,22cyclictest34-21ksoftirqd/211:25:372
198983699431,21cyclictest34-21ksoftirqd/207:40:232
1989836994312,10cyclictest34-21ksoftirqd/212:00:012
198983199438,20cyclictest26-21ksoftirqd/107:45:211
198983199433,20cyclictest26-21ksoftirqd/109:05:111
198983199433,16cyclictest26-21ksoftirqd/110:20:011
1989831994328,11cyclictest2074363-21ntpq11:05:241
1989831994314,9cyclictest26-21ksoftirqd/110:10:221
1989831994314,8cyclictest26-21ksoftirqd/108:40:391
1989831994313,10cyclictest26-21ksoftirqd/112:40:011
198983199431,19cyclictest26-21ksoftirqd/111:15:041
1989831994311,9cyclictest26-21ksoftirqd/107:50:141
1989828994313,25cyclictest2061769-40lsb_release10:30:220
1989828994312,6cyclictest141rcu_preempt11:35:270
1989828994311,13cyclictest141rcu_preempt12:05:160
162399430,36rtkit-daemon1753-21gnome-shell07:49:000
162399430,29rtkit-daemon0-21swapper/207:15:382
1989838994213,24cyclictest42-21ksoftirqd/309:35:223
198983699423,19cyclictest34-21ksoftirqd/212:00:132
198983199426,18cyclictest26-21ksoftirqd/110:05:201
198983199424,26cyclictest26-21ksoftirqd/108:39:381
1989831994232,7cyclictest251rcuc/108:20:121
198983199421,16cyclictest26-21ksoftirqd/108:10:301
1989831994211,10cyclictest26-21ksoftirqd/111:55:021
1989831994211,10cyclictest26-21ksoftirqd/111:55:021
1989828994224,14cyclictest855-21dbus-daemon08:40:010
198982899422,35cyclictest1753-21gnome-shell11:15:230
1989828994213,6cyclictest141rcu_preempt10:50:170
1989828994211,7cyclictest141rcu_preempt08:24:000
198982899421,16cyclictest21013772if_enp2s012:20:210
20196172410,7sleep3411rcuc/308:30:203
198983899413,21cyclictest42-21ksoftirqd/307:47:093
1989838994131,7cyclictest1069-21runrttasks07:36:473
1989838994131,7cyclictest1069-21runrttasks07:36:473
1989838994123,13cyclictest42-21ksoftirqd/311:35:313
1989838994121,15cyclictest42-21ksoftirqd/310:05:273
1989838994113,11cyclictest141rcu_preempt11:55:023
1989838994113,11cyclictest141rcu_preempt11:55:013
198983699418,18cyclictest2054373-21smartctl10:10:202
198983699418,13cyclictest2197014-21PK-Backend07:30:102
198983699411,16cyclictest1995742-21seq07:25:122
198983699411,15cyclictest34-21ksoftirqd/211:40:012
198983199419,15cyclictest26-21ksoftirqd/111:05:021
198983199417,19cyclictest26-21ksoftirqd/112:25:161
198983199417,19cyclictest26-21ksoftirqd/112:25:151
198983199417,16cyclictest26-21ksoftirqd/111:56:431
198983199416,19cyclictest26-21ksoftirqd/109:50:031
198983199414,9cyclictest26-21ksoftirqd/107:23:191
1989831994133,6cyclictest251rcuc/107:40:151
1989831994124,13cyclictest2098421-21cat12:10:301
1989831994123,13cyclictest2084011-21ntpq11:30:281
198983199411,21cyclictest2071304-21seq10:55:481
198983199411,21cyclictest1996840-21taskset07:25:351
198982899418,17cyclictest141rcu_preempt09:03:580
1989828994125,12cyclictest2024171-21cron08:45:000
1989828994124,12cyclictest438-20systemd-journal08:05:010
1989828994115,7cyclictest141rcu_preempt11:54:250
1989828994115,7cyclictest141rcu_preempt11:54:240
1989828994112,6cyclictest141rcu_preempt10:42:270
162399410,33rtkit-daemon1991551-21munin-node09:54:030
20140492400,6sleep0161rcuc/008:15:230
1989838994029,7cyclictest2034853-21apt09:15:123
1989838994023,13cyclictest1998254-21proc_pri07:30:283
1989838994021,15cyclictest2018716-21unin-run08:30:003
198983899401,9cyclictest2197014-21PK-Backend12:07:103
1989838994011,6cyclictest141rcu_preempt07:55:013
1989838994011,24cyclictest42-21ksoftirqd/309:05:213
1989838994011,11cyclictest141rcu_preempt11:05:293
1989838994010,13cyclictest141rcu_preempt09:30:183
198983699407,18cyclictest34-21ksoftirqd/209:50:112
198983699406,16cyclictest34-21ksoftirqd/211:20:062
198983199403,18cyclictest26-21ksoftirqd/108:50:141
198982899404,16cyclictest141rcu_preempt10:22:470
1989828994028,8cyclictest2042784-21xargs09:35:300
1989828994023,13cyclictest2039795-21/usr/sbin/munin09:30:020
1989828994023,13cyclictest2039795-21/usr/sbin/munin09:30:020
1989828994021,6cyclictest13-21ksoftirqd/008:10:040
162399400,32rtkit-daemon13-21ksoftirqd/007:54:300
162399400,21rtkit-daemon42-21ksoftirqd/308:41:313
1989838993926,9cyclictest2039780-21sh09:30:013
1989838993926,9cyclictest2039780-21sh09:30:003
1989838993913,21cyclictest42-21ksoftirqd/309:45:173
1989838993913,13cyclictest141rcu_preempt08:20:133
198983899391,16cyclictest2064832-21apt-config10:40:043
198983699392,18cyclictest34-21ksoftirqd/207:53:242
1989836993914,8cyclictest34-21ksoftirqd/208:55:342
198983199397,18cyclictest26-21ksoftirqd/111:35:281
198982899392,8cyclictest1753-21gnome-shell08:47:530
1989828993928,7cyclictest1938-21gsd-housekeepin10:56:180
1989828993922,13cyclictest1753-21gnome-shell12:37:060
1989828993921,13cyclictest438-20systemd-journal12:00:000
1989828993913,5cyclictest141rcu_preempt11:00:380
1989828993912,5cyclictest141rcu_preempt09:55:160
198982899391,11cyclictest2001955-21grep07:40:300
162399390,32rtkit-daemon42-21ksoftirqd/311:55:443
20983862380,5sleep00-21swapper/012:10:300
198983899389,11cyclictest141rcu_preempt10:40:013
198983899387,5cyclictest141rcu_preempt11:15:263
1989838993820,14cyclictest42-21ksoftirqd/309:40:243
1989838993820,13cyclictest42-21ksoftirqd/308:50:203
1989838993814,13cyclictest141rcu_preempt10:55:023
1989838993813,5cyclictest141rcu_preempt08:57:273
1989838993812,21cyclictest0-21swapper/308:05:183
1989838993811,11cyclictest141rcu_preempt07:55:323
1989838993811,11cyclictest141rcu_preempt07:55:323
198982899389,13cyclictest141rcu_preempt08:30:160
198982899389,11cyclictest141rcu_preempt11:40:230
1989828993826,8cyclictest1753-21gnome-shell07:23:030
1989828993825,9cyclictest1997474-21apt07:30:140
1989828993820,14cyclictest2080098-21df11:20:150
198982899381,9cyclictest1753-21gnome-shell09:10:000
1989828993814,6cyclictest141rcu_preempt10:39:380
1989828993813,7cyclictest141rcu_preempt10:01:070
198982899381,21cyclictest2035445-21grep09:15:220
198982899381,10cyclictest1753-21gnome-shell09:42:490
1989828993810,6cyclictest141rcu_preempt11:35:000
162399380,30rtkit-daemon921-21thermald07:58:460
162399380,30rtkit-daemon921-21thermald07:58:460
198983899379,11cyclictest141rcu_preempt11:47:363
198983899376,6cyclictest141rcu_preempt12:40:003
198983899372,18cyclictest2068175-21cron10:50:003
198983899371,4cyclictest141rcu_preempt09:55:173
1989838993710,10cyclictest141rcu_preempt10:59:323
198982899378,5cyclictest141rcu_preempt10:13:380
198982899372,7cyclictest1753-21gnome-shell12:02:550
198982899372,31cyclictest0-21swapper/012:26:040
198982899372,31cyclictest0-21swapper/012:26:030
1989828993721,12cyclictest2033844-21ntp_kernel_pll_09:10:240
1989828993711,6cyclictest141rcu_preempt09:50:010
198983899368,6cyclictest141rcu_preempt12:30:163
198983899363,11cyclictest42-21ksoftirqd/311:25:423
198983899363,11cyclictest42-21ksoftirqd/311:25:413
198983899361,9cyclictest2197014-21PK-Backend12:11:503
1989838993619,13cyclictest42-21ksoftirqd/310:05:023
1989828993619,6cyclictest13-21ksoftirqd/010:46:290
1989828993613,6cyclictest141rcu_preempt07:37:320
1989828993613,6cyclictest141rcu_preempt07:37:320
198983899359,11cyclictest141rcu_preempt09:50:213
198983899357,15cyclictest2197014-21PK-Backend12:26:193
198983899357,15cyclictest2197014-21PK-Backend12:26:183
198983899353,10cyclictest42-21ksoftirqd/307:25:323
198983899351,30cyclictest1991891-21cpuspeed07:15:033
1989828993518,13cyclictest884-21gdbus11:28:070
1989828993518,13cyclictest884-21gdbus11:28:060
1989828993517,14cyclictest2028701-21awk08:55:270
1989828993517,13cyclictest2027268-21users08:50:310
1989828993514,17cyclictest2055909-21cat10:15:140
198983899347,11cyclictest141rcu_preempt08:45:263
198983899341,30cyclictest2197014-21PK-Backend09:00:123
1989838993318,11cyclictest2033599-21irqstats09:10:213
198983899331,4cyclictest2197014-21PK-Backend08:14:513
1989838993225,4cyclictest2197014-21PK-Backend08:03:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional