You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 16:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Mon Oct 20, 2025 12:45:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
37476121220,5sleep3411rcuc/312:25:003
3069132970,5sleep32197014-21PK-Backend09:10:233
263422999385,5cyclictest0-21swapper/111:24:051
263422999284,5cyclictest0-21swapper/111:27:451
263422999183,5cyclictest0-21swapper/111:16:371
26342299912,85cyclictest0-21swapper/111:30:211
263422998678,5cyclictest0-21swapper/111:13:361
26303028666,14sleep30-21swapper/307:06:363
26343099811,56cyclictest1753-21gnome-shell11:17:043
162399810,6rtkit-daemon0-21swapper/110:30:181
162399790,6rtkit-daemon0-21swapper/110:15:261
26342099782,55cyclictest1753-21gnome-shell11:21:060
162399770,5rtkit-daemon0-21swapper/107:14:221
26343099761,14cyclictest1753-21gnome-shell11:25:133
162399760,63rtkit-daemon0-21swapper/207:05:432
162399760,5rtkit-daemon0-21swapper/110:41:431
162399760,5rtkit-daemon0-21swapper/109:51:351
26342599755,6cyclictest141rcu_preempt11:24:022
26342299753,45cyclictest1753-21gnome-shell10:40:021
162399740,5rtkit-daemon0-21swapper/107:48:101
162399740,5rtkit-daemon0-21swapper/107:48:101
26342099725,7cyclictest141rcu_preempt11:25:300
26343099712,49cyclictest1753-21gnome-shell11:20:223
26343099712,11cyclictest1753-21gnome-shell11:30:123
26182427146,16sleep10-21swapper/107:05:141
26342099701,7cyclictest141rcu_preempt11:15:540
162399700,10rtkit-daemon3242254-21gmain12:25:461
162399690,8rtkit-daemon1622-21rtkit-daemon11:52:401
26318626847,14sleep00-21swapper/007:08:380
162399680,9rtkit-daemon1622-21rtkit-daemon12:07:530
26342099662,7cyclictest141rcu_preempt11:14:550
3274222650,21sleep126-21ksoftirqd/110:10:021
3274222650,21sleep126-21ksoftirqd/110:10:021
26342599652,7cyclictest0-21swapper/211:27:172
26342599632,20cyclictest1753-21gnome-shell11:15:202
26343099622,56cyclictest1753-21gnome-shell11:13:523
162399620,8rtkit-daemon1622-21rtkit-daemon12:03:361
26342599613,52cyclictest1753-21gnome-shell11:30:042
26342599611,55cyclictest1753-21gnome-shell11:14:202
162399610,8rtkit-daemon1622-21rtkit-daemon12:10:551
162399610,8rtkit-daemon1622-21rtkit-daemon11:38:290
162399600,50rtkit-daemon42-21ksoftirqd/312:33:193
263422995915,40cyclictest308566-21idleruntime09:15:211
263422995915,40cyclictest308566-21idleruntime09:15:211
3697452580,5sleep20-21swapper/212:10:142
263422995819,35cyclictest273201-21which07:35:211
26342099582,13cyclictest1753-21gnome-shell11:30:510
162399580,9rtkit-daemon1753-21gnome-shell11:08:232
162399580,9rtkit-daemon1753-21gnome-shell11:08:222
162399580,8rtkit-daemon1622-21rtkit-daemon11:42:451
26342299572,50cyclictest1753-21gnome-shell08:55:341
263422995718,34cyclictest297468-21cron08:45:001
263422995717,34cyclictest327639-21cpuspeed_turbos10:10:151
26342299571,51cyclictest270944-21date07:30:021
162399570,6rtkit-daemon1622-21rtkit-daemon12:37:410
263425995626,25cyclictest372302-40lsb_release12:15:272
26342299562,48cyclictest340249-21memory10:45:231
26342299562,48cyclictest340249-21memory10:45:231
263422995619,32cyclictest362513-21latency_hist11:50:021
263422995613,38cyclictest341608-21update-motd-upd10:50:051
26342299552,8cyclictest1753-21gnome-shell09:55:251
263422995521,30cyclictest295398-40apt-check08:36:221
263422995521,30cyclictest295398-40apt-check08:36:221
263422995516,35cyclictest298470-21ntp_kernel_pll_08:45:241
263422995516,34cyclictest438-20systemd-journal10:25:001
263422995515,35cyclictest308025-21cat09:15:021
263422995513,36cyclictest317534-21which09:40:211
26342299542,48cyclictest855-21dbus-daemon08:20:011
26342299542,48cyclictest855-21dbus-daemon08:20:011
263422995416,33cyclictest373786-21cut12:20:221
263422995416,33cyclictest292358-21cat08:30:151
263422995416,33cyclictest292358-21cat08:30:151
263422995415,35cyclictest288925-21tr08:20:151
162399540,8rtkit-daemon1622-21rtkit-daemon07:59:212
162399540,6rtkit-daemon1622-21rtkit-daemon12:23:040
3172262530,6sleep00-21swapper/009:40:150
26342299534,44cyclictest305485-21grep09:05:271
26342299533,45cyclictest285160-21grep08:10:151
26342299532,6cyclictest314238-21memory09:30:191
26342299532,46cyclictest333594-21python310:25:281
26342299532,46cyclictest333594-21python310:25:281
26342299532,46cyclictest280672-21python307:55:291
263422995319,29cyclictest368531-21sh12:05:251
263422995314,33cyclictest357760-21idleruntime11:35:221
3122772520,13sleep0131ksoftirqd/009:25:180
26342299522,10cyclictest275536-21switchtime07:40:311
263422995219,29cyclictest1069-21runrttasks09:46:471
263422995214,34cyclictest300978-21pool-org.gnome.08:55:011
26342299521,13cyclictest303884-21cat09:00:301
26342299521,13cyclictest303884-21cat09:00:301
26342299521,11cyclictest284047-21proc_pri08:05:281
3136032510,8chrt1753-21gnome-shell09:30:032
26342299519,37cyclictest1753-21gnome-shell11:55:181
26342299512,44cyclictest266341-21irqstats07:15:191
26342299511,11cyclictest348165-21seq11:07:291
26342299511,11cyclictest348165-21seq11:07:291
26342299511,11cyclictest343972-21sed10:55:231
26342599502,20cyclictest860997-21snmpd12:29:082
263422995018,27cyclictest371464-21update-motd-upd12:15:041
263422995017,29cyclictest281502-21/usr/sbin/munin08:00:141
263422995016,30cyclictest1-21systemd12:40:001
26342299491,44cyclictest272535-21munin-run07:35:011
263422994912,33cyclictest377701-21cat12:30:331
263422994910,34cyclictest267961-21sh07:20:161
162399490,5rtkit-daemon0-21swapper/009:14:090
263430994825,19cyclictest303565-21/usr/sbin/munin09:00:233
263430994825,19cyclictest303565-21/usr/sbin/munin09:00:233
26342299489,34cyclictest324265-21cat10:00:141
26342299487,36cyclictest438-20systemd-journal07:54:321
26342299487,36cyclictest438-20systemd-journal07:54:321
263422994816,27cyclictest315604-21cstates09:35:141
263422994814,29cyclictest26-21ksoftirqd/108:25:301
26342299481,41cyclictest310635-21uname09:20:241
263420994817,11cyclictest1753-21gnome-shell12:02:470
162399480,6rtkit-daemon0-21swapper/008:12:250
263425994732,10cyclictest361396-21sed11:45:222
263425994722,20cyclictest342813-21apt-check10:51:172
26342299472,15cyclictest345647-21/usr/sbin/munin11:00:211
26342299471,9cyclictest313097-21chrt09:26:271
3755232460,5sleep02197014-21PK-Backend12:25:210
263430994620,21cyclictest42-21ksoftirqd/307:45:593
263430994620,21cyclictest42-21ksoftirqd/307:45:593
26342599468,13cyclictest1069-21runrttasks11:52:072
263420994615,17cyclictest1753-21gnome-shell11:09:360
263420994615,17cyclictest1753-21gnome-shell11:09:360
162399460,5rtkit-daemon0-21swapper/009:09:210
3754022450,15chrt42-21ksoftirqd/312:25:163
263420994514,17cyclictest1753-21gnome-shell12:30:380
263420994512,12cyclictest13-21ksoftirqd/010:34:040
2676462440,15sleep2331rcuc/207:20:152
263430994427,13cyclictest42-21ksoftirqd/312:15:273
263420994431,9cyclictest1-21systemd07:15:010
162399440,31rtkit-daemon1753-21gnome-shell07:28:550
162399440,30rtkit-daemon0-21swapper/008:30:190
162399440,30rtkit-daemon0-21swapper/008:30:190
263430994316,23cyclictest1753-21gnome-shell09:55:383
26342099436,19cyclictest1753-21gnome-shell07:55:000
26342099436,19cyclictest1753-21gnome-shell07:55:000
263430994224,14cyclictest855-21dbus-daemon12:05:013
263430994224,13cyclictest438-20systemd-journal09:34:563
263430994214,7cyclictest141rcu_preempt07:15:033
263425994225,13cyclictest311662-21/usr/sbin/munin09:25:002
263425994211,10cyclictest1753-21gnome-shell09:30:192
26342099422,34cyclictest1753-21gnome-shell08:42:040
263420994210,11cyclictest1753-21gnome-shell10:37:060
26343099412,7cyclictest17531gnome-shell11:45:003
263430994126,11cyclictest320039-21sshd09:46:323
263430994123,13cyclictest438-20systemd-journal08:25:003
263430994116,22cyclictest1753-21gnome-shell08:32:113
263430994116,22cyclictest1753-21gnome-shell08:32:113
26343099411,11cyclictest1753-21gnome-shell07:25:343
263425994124,12cyclictest322501-21grep09:55:122
26342599412,16cyclictest127-1kworker/2:1H09:40:022
26342599411,10cyclictest2197014-21PK-Backend11:59:432
26342099416,19cyclictest0-21swapper/009:57:050
263420994125,12cyclictest295615-21sshd08:38:320
263420994125,12cyclictest295615-21sshd08:38:320
263420994120,5cyclictest320363-21latency_hist09:50:020
263420994111,10cyclictest0-21swapper/011:57:080
263430994029,7cyclictest266001-21cpuspeed_turbos07:15:123
26343099402,33cyclictest334946-21irqstats10:30:233
263430994022,13cyclictest347960-21cat11:06:063
263430994022,13cyclictest347960-21cat11:06:063
263430994018,7cyclictest272895-21ls07:35:153
26342599402,8cyclictest1753-21gnome-shell08:04:402
263425994024,11cyclictest438-20systemd-journal11:38:122
26342599402,13cyclictest1753-21gnome-shell08:31:292
26342599402,13cyclictest1753-21gnome-shell08:31:292
26342099409,17cyclictest1753-21gnome-shell09:18:360
26342099409,17cyclictest1753-21gnome-shell09:18:360
26342099406,18cyclictest0-21swapper/007:30:120
26342099404,11cyclictest13-21ksoftirqd/012:17:410
162399400,33rtkit-daemon265144-21munin-node12:11:230
263430993929,6cyclictest301900-21cut08:55:253
263430993926,9cyclictest325980-21grep10:05:143
263430993926,9cyclictest325980-21grep10:05:143
263430993923,12cyclictest291621-21sshd08:25:393
263430993922,13cyclictest357738-21fw_packets11:35:203
263430993916,20cyclictest391irq_work/311:51:083
263430993911,23cyclictest1753-21gnome-shell07:21:183
263425993924,11cyclictest267038-21sshd07:16:322
26342599392,32cyclictest1753-21gnome-shell09:49:062
263425993922,13cyclictest304360-21sshd09:04:322
263425993922,13cyclictest304360-21sshd09:04:322
263425993922,13cyclictest298869-21sshd08:45:432
26342099398,11cyclictest1753-21gnome-shell11:47:540
26342099392,7cyclictest1753-21gnome-shell08:02:080
263420993924,11cyclictest344057-21missed_timers10:55:240
263420993921,14cyclictest855-21dbus-daemon10:20:010
263420993911,10cyclictest1753-21gnome-shell07:15:200
263420993910,5cyclictest141rcu_preempt07:50:010
263420993910,5cyclictest141rcu_preempt07:50:010
2946212380,5chrt0-21swapper/308:35:183
2946212380,5chrt0-21swapper/308:35:183
26343099385,13cyclictest281939-21sort08:00:213
263430993826,9cyclictest1753-21gnome-shell10:20:003
263430993825,9cyclictest1753-21gnome-shell10:03:283
263430993824,11cyclictest42-21ksoftirqd/310:50:203
26343099382,32cyclictest1753-21gnome-shell09:21:543
263430993823,11cyclictest42-21ksoftirqd/312:15:033
263430993822,12cyclictest42-21ksoftirqd/308:05:263
263430993822,11cyclictest297844-21cpuspeed_turbos08:45:143
263430993821,13cyclictest328307-21pluginstate10:10:293
26343099381,33cyclictest320569-21/usr/sbin/munin09:50:213
263430993811,9cyclictest1753-21gnome-shell10:25:313
263430993811,9cyclictest1753-21gnome-shell10:25:313
263425993824,10cyclictest34-21ksoftirqd/210:11:452
263425993822,11cyclictest369251-21munin-run12:10:002
263425993821,5cyclictest34-21ksoftirqd/208:07:032
263425993816,18cyclictest887-21smartd10:31:562
26342099382,31cyclictest341127-21apt-get10:50:110
263420993821,12cyclictest290515-21cpu08:25:130
263420993821,12cyclictest289422-21tail08:20:240
263420993821,12cyclictest267563-21sshd07:20:080
263420993812,22cyclictest287845-31apt-check08:15:300
263420993812,22cyclictest287845-31apt-check08:15:300
263430993727,6cyclictest318590-21latency_hist09:45:003
263430993723,10cyclictest42-21ksoftirqd/308:50:323
263430993722,11cyclictest42-21ksoftirqd/307:59:443
263430993720,12cyclictest287256-21expr08:15:163
263430993720,12cyclictest287256-21expr08:15:163
26343099371,33cyclictest0-21swapper/308:14:443
26342599372,8cyclictest1753-21gnome-shell07:31:502
263425993721,11cyclictest304875-21tr09:05:142
263425993720,5cyclictest34-21ksoftirqd/208:55:292
263425993720,12cyclictest366700-21cat12:00:252
26342599371,9cyclictest2197014-21PK-Backend10:55:302
26342599371,9cyclictest0-21swapper/207:13:332
26342599371,13cyclictest2197014-21PK-Backend08:42:472
26342599371,13cyclictest0-21swapper/210:04:232
26342099378,9cyclictest13-21ksoftirqd/010:14:560
26342099374,16cyclictest0-21swapper/009:39:180
26342099372,31cyclictest1753-21gnome-shell10:46:320
26342099372,31cyclictest1753-21gnome-shell10:46:320
263420993721,11cyclictest301157-21date08:55:020
263420993720,13cyclictest13-21ksoftirqd/007:41:380
263420993719,13cyclictest13-21ksoftirqd/009:00:380
263420993719,13cyclictest13-21ksoftirqd/009:00:380
263430993622,11cyclictest42-21ksoftirqd/310:22:063
263430993621,11cyclictest312677-21ntp_states09:25:303
263430993620,12cyclictest337928-21apt10:40:113
26343099361,9cyclictest0-21swapper/309:17:493
26343099361,9cyclictest0-21swapper/309:17:493
26343099361,6cyclictest141rcu_preempt12:05:193
263430993611,8cyclictest2962346-21in:imuxsock08:42:323
26342599365,6cyclictest141rcu_preempt08:15:472
26342599365,6cyclictest141rcu_preempt08:15:472
263425993625,7cyclictest331154-21tr10:20:142
263425993623,9cyclictest3009812chrt08:55:012
263425993621,11cyclictest270995-21sort07:30:022
263425993620,12cyclictest855-21dbus-daemon08:30:012
263425993619,12cyclictest329658-21which10:15:212
263425993618,13cyclictest373487-21cstates12:20:142
26342599361,30cyclictest438-20systemd-journal08:40:002
26342599361,30cyclictest438-20systemd-journal08:40:002
26342599361,15cyclictest2197014-21PK-Backend10:35:282
26342099364,10cyclictest13-21ksoftirqd/007:59:080
26342099363,11cyclictest13-21ksoftirqd/011:51:190
26342099362,5cyclictest0-21swapper/008:08:550
263420993625,7cyclictest1753-21gnome-shell07:36:050
26342099362,30cyclictest0-21swapper/010:01:470
263420993622,10cyclictest313793-21cpuspeed_turbos09:30:110
263420993619,13cyclictest310733-21memory09:20:250
263420993619,12cyclictest339003-21cat10:40:320
162399360,29rtkit-daemon0-21swapper/011:43:060
263430993524,7cyclictest276055-21sh07:45:003
26343099352,28cyclictest1753-21gnome-shell10:45:083
26343099352,28cyclictest1753-21gnome-shell10:45:083
263430993520,11cyclictest42-21ksoftirqd/310:55:573
263430993520,11cyclictest272454-21seq07:34:173
26343099351,9cyclictest0-21swapper/311:55:213
263430993518,12cyclictest361860-21cat11:45:313
263425993524,7cyclictest310014-21date09:20:032
263425993520,11cyclictest326095-21expr10:05:142
263425993520,11cyclictest326095-21expr10:05:132
26342599351,9cyclictest2197014-21PK-Backend12:30:292
263425993518,13cyclictest2181-21gdbus08:15:022
26342599351,7cyclictest1753-21gnome-shell10:42:352
26342599351,30cyclictest2197014-21PK-Backend10:45:242
26342599351,30cyclictest2197014-21PK-Backend10:45:242
263420993524,7cyclictest345891-21taskset11:00:240
263420993520,11cyclictest301640-21basename08:55:210
263420993520,11cyclictest299242-21systemd-run08:50:000
263430993422,8cyclictest1753-21gnome-shell09:35:123
263430993420,10cyclictest42-21ksoftirqd/312:35:253
263430993419,11cyclictest305169-21kernelversion09:05:213
263430993417,13cyclictest345651-21irqstats11:00:213
263425993419,11cyclictest378751-21expr12:35:142
263425993418,12cyclictest289948-21sshd08:21:322
263425993417,13cyclictest345363-21cstates11:00:142
263425993417,13cyclictest1753-21gnome-shell09:52:122
263425993417,12cyclictest309614-21apt-get09:20:002
263425993417,12cyclictest275077-21kernelversion07:40:232
263425993415,14cyclictest333508-21proc_pri10:25:272
263425993415,14cyclictest333508-21proc_pri10:25:262
26342099349,5cyclictest141rcu_preempt09:51:460
26342099344,11cyclictest141rcu_preempt10:21:070
263420993417,12cyclictest326539-21ntpq10:05:250
263420993417,12cyclictest326539-21ntpq10:05:250
26343099332,9cyclictest1753-21gnome-shell07:50:193
26343099332,9cyclictest1753-21gnome-shell07:50:193
263430993318,12cyclictest42-21ksoftirqd/310:35:563
263425993319,10cyclictest34-21ksoftirqd/209:43:562
263425993319,10cyclictest34-21ksoftirqd/207:45:142
263425993319,10cyclictest34-21ksoftirqd/207:45:132
26342599331,11cyclictest2197014-21PK-Backend07:35:242
26342599331,10cyclictest2197014-21PK-Backend07:50:552
26342599331,10cyclictest2197014-21PK-Backend07:50:552
26342599325,22cyclictest306583-21cpuspeed_turbos09:10:162
263425993214,13cyclictest855-21dbus-daemon11:45:012
263420993123,4cyclictest2197014-21PK-Backend10:25:210
263420993123,4cyclictest2197014-21PK-Backend10:25:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional