You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-25 - 02:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Mar 24, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991280,107rtkit-daemon0-21swapper/007:09:540
162399960,20rtkit-daemon2197014-21PK-Backend11:33:282
162399960,20rtkit-daemon2197014-21PK-Backend11:33:282
162399890,33rtkit-daemon1753-21gnome-shell09:05:573
318929828166,9sleep10-21swapper/107:08:391
318925427963,9sleep20-21swapper/207:08:042
162399760,6rtkit-daemon2197014-21PK-Backend12:38:403
318788327347,16sleep30-21swapper/307:05:083
162399630,10rtkit-daemon1622-21rtkit-daemon08:33:003
162399620,10rtkit-daemon1622-21rtkit-daemon08:41:032
162399620,10rtkit-daemon1622-21rtkit-daemon07:55:052
3189546996117,15cyclictest3286261-21/usr/sbin/munin12:15:261
3189546996019,15cyclictest3270365-21taskset11:22:491
3189551995930,6cyclictest3221466-21grep08:45:013
3189546995921,33cyclictest3260415-21wc10:50:181
3189546995913,19cyclictest3206345-40lsb_release07:57:221
3189546995913,15cyclictest3264752-21munin-run11:05:011
3189546995819,16cyclictest3255662-21seq10:35:121
3189546995818,22cyclictest3228954-21diskstats09:05:161
3189546995816,25cyclictest3258523-21vmstat10:40:321
162399580,8rtkit-daemon1622-21rtkit-daemon12:05:573
162399580,8rtkit-daemon1622-21rtkit-daemon11:37:372
162399580,8rtkit-daemon1622-21rtkit-daemon08:38:492
162399580,8rtkit-daemon1622-21rtkit-daemon08:14:572
162399580,8rtkit-daemon1622-21rtkit-daemon07:38:202
3189546995717,21cyclictest3259510-21runrttasks10:49:581
162399570,9rtkit-daemon1622-21rtkit-daemon10:13:113
162399570,9rtkit-daemon1622-21rtkit-daemon08:45:322
162399570,8rtkit-daemon1622-21rtkit-daemon10:42:072
162399570,8rtkit-daemon1622-21rtkit-daemon08:46:343
318954999560,52cyclictest1622-21rtkit-daemon11:56:582
3189546995617,17cyclictest3273427-21/usr/sbin/munin11:35:281
3189546995616,17cyclictest1753-21gnome-shell11:48:171
3189546995616,15cyclictest3244908-21taskset09:56:361
3189546995614,16cyclictest3266202-21memory11:10:181
3189546995614,12cyclictest3216401-21/usr/sbin/munin08:30:251
3189546995613,15cyclictest3252383-21users10:20:351
3189546995518,32cyclictest3197587-21basename07:30:231
3189546995517,16cyclictest3231545-21seq09:12:181
3189546995517,16cyclictest3231545-21seq09:12:171
3189546995514,18cyclictest1753-21gnome-shell08:00:121
3189546995514,17cyclictest3261868-21expr10:55:141
3189546995514,17cyclictest3261868-21expr10:55:141
3189546995514,17cyclictest3253384-21munin-run10:30:001
3189546995512,16cyclictest3224000-21sed08:50:241
3189546995512,16cyclictest3224000-21sed08:50:241
3189546995511,15cyclictest3241076-21latency_hist09:45:011
162399550,9rtkit-daemon1622-21rtkit-daemon12:15:142
162399550,9rtkit-daemon1622-21rtkit-daemon11:53:473
162399550,9rtkit-daemon1622-21rtkit-daemon08:01:373
3189551995420,9cyclictest1753-21gnome-shell07:20:283
3189546995418,20cyclictest3270873-21taskset11:26:101
3189546995418,20cyclictest3250885-21seq10:18:261
3189546995417,14cyclictest3211721-21ntp_states08:15:371
3189546995412,25cyclictest3232439-21hddtemp_smartct09:15:191
3189546995411,18cyclictest854-21avahi-daemon08:41:051
162399540,9rtkit-daemon1622-21rtkit-daemon08:59:552
162399540,8rtkit-daemon1622-21rtkit-daemon11:04:273
3189546995312,16cyclictest3284300-21grep12:10:171
3189546995312,15cyclictest3254107-21if_enp2s010:30:161
3189546995312,15cyclictest3210596-21sshd08:14:041
3189546995312,14cyclictest0-21swapper/112:20:361
162399530,8rtkit-daemon1622-21rtkit-daemon07:56:413
318955199522,29cyclictest3242253-21nautilus08:40:003
3189546995217,11cyclictest1753-21gnome-shell11:42:331
3189546995216,10cyclictest3280477-21/usr/sbin/munin11:55:261
3189546995212,16cyclictest3201320-21memory07:40:271
3189546995211,26cyclictest3267658-21expr11:15:121
3189546995211,26cyclictest3267658-21expr11:15:121
3189546995211,17cyclictest3245876-21basename10:00:211
162399520,9rtkit-daemon1622-21rtkit-daemon09:58:422
162399520,8rtkit-daemon1622-21rtkit-daemon09:24:183
162399520,8rtkit-daemon1622-21rtkit-daemon08:57:453
318955199512,9cyclictest1753-21gnome-shell07:18:333
318955199512,7cyclictest3215165-21ntp_states08:25:253
3189546995118,29cyclictest1-21systemd08:40:011
3189546995113,16cyclictest3238284-21smartctl09:35:221
3189546995111,19cyclictest3235695-21taskset09:27:591
3189551995027,4cyclictest0-21swapper/309:52:133
318955199501,14cyclictest3212694-21taskset08:20:123
3189546995013,15cyclictest3236552-21idleruntime09:30:151
3189546995012,17cyclictest3214632-21cstates08:25:141
3189546995012,16cyclictest3214149-21latency_hist08:25:021
3189546995011,14cyclictest3281790-21df12:00:151
162399500,8rtkit-daemon1622-21rtkit-daemon10:24:592
32432172490,15sleep126-21ksoftirqd/109:52:071
3189546994921,23cyclictest1753-21gnome-shell12:30:041
3189546994919,25cyclictest438-20systemd-journal07:50:191
3189546994912,11cyclictest3226609-21munin-run09:00:011
32460422480,5sleep0161rcuc/010:00:230
3189549994824,21cyclictest3198649-21awk07:35:002
318954699487,13cyclictest0-21swapper/110:10:181
3189546994818,15cyclictest1753-21gnome-shell11:33:131
3189546994818,15cyclictest1753-21gnome-shell11:33:121
3189546994810,23cyclictest1753-21gnome-shell09:03:381
3189549994735,7cyclictest438-20systemd-journal10:54:542
3189546994717,25cyclictest3208726-21taskset08:07:161
3189546994712,12cyclictest1753-21gnome-shell10:09:391
3189546994711,15cyclictest3190334-21memory07:10:241
162399470,39rtkit-daemon2197014-21PK-Backend10:59:483
162399470,39rtkit-daemon2197014-21PK-Backend10:59:483
3189549994621,21cyclictest3277127-21apt-get11:47:172
318954699466,14cyclictest0-21swapper/112:37:001
3189546994621,9cyclictest3222096-21if_enp2s008:45:201
3189544994627,14cyclictest438-20systemd-journal10:00:000
3189544994620,22cyclictest1753-21gnome-shell09:33:470
32677322450,9sleep21753-21gnome-shell11:15:152
32677322450,9sleep21753-21gnome-shell11:15:142
32494202450,6sleep22197014-21PK-Backend10:15:022
3189549994524,6cyclictest34-21ksoftirqd/209:34:552
3189549994523,14cyclictest34-21ksoftirqd/210:32:012
3189549994523,14cyclictest34-21ksoftirqd/209:10:182
3189549994523,14cyclictest34-21ksoftirqd/209:10:172
3189549994523,13cyclictest34-21ksoftirqd/207:50:162
3189546994518,22cyclictest3199383-21irqstats07:35:221
3189546994518,10cyclictest3278329-21df_inode11:50:121
3189546994517,24cyclictest1753-21gnome-shell07:48:271
3189546994517,12cyclictest3235417-21sed09:25:021
3189551994433,7cyclictest1069-21runrttasks12:25:373
3189549994424,15cyclictest3242253-21nautilus08:30:032
3189549994422,14cyclictest34-21ksoftirqd/209:35:142
318954999441,11cyclictest3261490-21chrt10:55:032
318954999441,11cyclictest3261490-21chrt10:55:022
318954699446,15cyclictest0-21swapper/107:25:071
3189546994419,11cyclictest1753-21gnome-shell12:08:041
3189546994417,22cyclictest1-21systemd12:30:001
3189546994414,10cyclictest859-21NetworkManager09:49:211
3189544994429,11cyclictest3210959-21apt08:15:160
162399440,25rtkit-daemon1753-21gnome-shell12:22:233
3189549994323,6cyclictest34-21ksoftirqd/207:20:012
3189546994321,17cyclictest3265367-21perf11:10:001
162399430,5rtkit-daemon854-21avahi-daemon07:33:163
162399430,35rtkit-daemon2197014-21PK-Backend08:15:522
162399430,34rtkit-daemon3189536-21cyclictest08:51:452
162399430,34rtkit-daemon3189536-21cyclictest08:51:452
32829242420,8sleep21753-21gnome-shell12:05:022
3189551994218,6cyclictest141rcu_preempt12:05:013
3189549994231,6cyclictest34-21ksoftirqd/208:05:312
3189549994224,13cyclictest438-20systemd-journal09:53:292
3189544994229,8cyclictest1753-21gnome-shell08:10:100
318954499422,29cyclictest0-21swapper/007:14:090
162399420,35rtkit-daemon3191440-21munin-node09:13:303
162399420,35rtkit-daemon3191440-21munin-node09:13:303
3189551994125,11cyclictest3224835-21cron08:55:013
3189551994125,11cyclictest3224835-21cron08:55:013
3189551994124,14cyclictest54-21kcompactd007:45:163
318955199412,10cyclictest3271937-21irqstats11:30:213
318955199412,10cyclictest3271937-21irqstats11:30:213
3189546994120,17cyclictest3193932-21cat07:20:141
3189546994117,13cyclictest3193397-21taskset07:19:471
3189544994126,11cyclictest3274605-40apt-check11:36:170
3189544994111,8cyclictest141rcu_preempt07:55:230
162399410,33rtkit-daemon2197014-21PK-Backend11:11:273
3189551994014,21cyclictest1753-21gnome-shell10:49:033
3189551994010,6cyclictest141rcu_preempt12:00:013
3189549994029,6cyclictest34-21ksoftirqd/207:10:252
3189549994023,6cyclictest34-21ksoftirqd/207:44:132
3189549994023,13cyclictest34-21ksoftirqd/208:25:152
3189549994022,7cyclictest34-21ksoftirqd/208:00:282
3189549994022,14cyclictest3269824-21ntpq11:20:272
318954999402,13cyclictest1753-21gnome-shell07:25:402
318954499402,8cyclictest1753-21gnome-shell11:48:190
318954499402,34cyclictest1753-21gnome-shell09:03:120
3189544994015,21cyclictest3216973-40lsb_release08:30:260
3189544994010,16cyclictest1753-21gnome-shell07:21:360
162399400,33rtkit-daemon2197014-21PK-Backend11:01:102
3189551993922,13cyclictest1-21systemd10:40:013
3189549993928,8cyclictest1753-21gnome-shell11:50:162
3189549993928,6cyclictest34-21ksoftirqd/209:48:162
3189549993924,11cyclictest3213789-21sshd08:20:322
318954999392,33cyclictest1753-21gnome-shell09:08:402
3189549993920,13cyclictest438-20systemd-journal09:20:002
3189544993925,10cyclictest3215872-21cron08:30:000
3189544993922,13cyclictest1753-21gnome-shell07:44:330
318954499392,11cyclictest0-21swapper/010:42:400
318954499391,34cyclictest3255726-21cat10:35:140
32105552383,9sleep342-21ksoftirqd/308:13:323
3189551993824,11cyclictest3196913-21awk07:30:013
318955199382,13cyclictest3274636-40lsb_release11:36:243
3189551993821,13cyclictest3277417-21unattended-upgr11:47:223
3189549993826,7cyclictest3291747-21cron12:40:012
3189549993820,13cyclictest438-20systemd-journal10:06:072
3189549993820,13cyclictest3234184-21idleruntime09:20:212
3189549993814,19cyclictest1753-21gnome-shell07:46:172
318954999381,33cyclictest1753-21gnome-shell10:46:532
318954999381,32cyclictest28750irq/122-ahci[0011:10:002
318954499389,4cyclictest141rcu_preempt11:26:210
318954499389,25cyclictest3220961-21sed08:40:290
318954499389,10cyclictest141rcu_preempt07:55:000
318954499385,17cyclictest141rcu_preempt09:12:020
318954499385,17cyclictest141rcu_preempt09:12:020
318954499382,32cyclictest1753-21gnome-shell11:13:420
318954499382,13cyclictest3248563-21ntp_states10:10:210
318954499382,13cyclictest141rcu_preempt10:52:060
3189544993821,12cyclictest438-20systemd-journal07:30:000
3189544993818,15cyclictest3242253-21nautilus11:35:010
3189544993818,15cyclictest3242253-21nautilus11:35:010
3189544993810,6cyclictest141rcu_preempt11:44:280
162399380,25rtkit-daemon0-21swapper/007:16:590
318955199372,31cyclictest1753-21gnome-shell07:13:403
3189551993723,10cyclictest3251033-21sh10:20:003
3189551993720,13cyclictest855-21dbus-daemon08:10:013
318954999378,18cyclictest1753-21gnome-shell12:05:282
318954999377,14cyclictest1358-1kworker/u9:110:37:452
318954999372,31cyclictest1753-21gnome-shell09:29:382
318954999372,16cyclictest3193701-21apt07:20:112
3189549993720,13cyclictest34-21ksoftirqd/212:10:262
318954999371,15cyclictest1753-21gnome-shell11:26:372
318954999371,11cyclictest3253404-21perf10:29:592
318954499374,6cyclictest0-21swapper/009:05:100
3189544993725,8cyclictest1069-21runrttasks07:30:150
3189544993720,12cyclictest438-20systemd-journal11:55:030
3189544993719,14cyclictest908-21gdbus12:34:330
3189544993719,13cyclictest438-20systemd-journal08:59:590
318954499371,32cyclictest3268830-21turbostat11:20:010
318954499371,32cyclictest3268830-21turbostat11:20:010
3189544993713,19cyclictest3287625-21df_inode12:20:170
318954499371,18cyclictest3264212-21threads11:00:310
162399370,29rtkit-daemon1753-21gnome-shell07:35:243
3189551993624,8cyclictest3254457-21wc10:30:243
318955199362,11cyclictest3203447-21df07:50:113
3189551993620,11cyclictest42-21ksoftirqd/309:30:143
3189551993618,13cyclictest1632-21dbus-daemon09:20:003
3189551993612,21cyclictest1753-21gnome-shell09:25:223
3189551993611,6cyclictest141rcu_preempt10:55:003
3189549993625,7cyclictest1753-21gnome-shell10:01:492
3189549993625,6cyclictest34-21ksoftirqd/212:31:302
3189549993625,6cyclictest34-21ksoftirqd/212:22:492
3189549993625,6cyclictest34-21ksoftirqd/210:15:402
3189549993622,9cyclictest1753-21gnome-shell09:41:302
3189549993622,10cyclictest34-21ksoftirqd/212:25:382
3189549993618,6cyclictest34-21ksoftirqd/211:42:532
318954999361,32cyclictest34-21ksoftirqd/209:00:152
318954499365,7cyclictest141rcu_preempt09:21:470
318954499365,14cyclictest1358-1kworker/u9:109:54:540
318954499365,12cyclictest0-21swapper/012:13:550
3189544993625,7cyclictest1069-21runrttasks12:03:500
3189544993624,8cyclictest3202575-21sshd07:47:030
318954499362,31cyclictest0-21swapper/010:59:000
318954499362,31cyclictest0-21swapper/010:59:000
3189544993621,10cyclictest1753-21gnome-shell09:40:100
3189544993619,13cyclictest1753-21gnome-shell12:37:520
3189544993618,14cyclictest1-21systemd10:25:000
318954499361,10cyclictest0-21swapper/009:46:430
162399360,30rtkit-daemon0-21swapper/310:08:203
162399360,28rtkit-daemon0-21swapper/309:47:153
318955199357,9cyclictest3291038-21systemd-udevd12:34:313
3189551993522,9cyclictest1753-21gnome-shell09:02:103
3189551993519,12cyclictest3276371-21sshd11:41:063
3189551993519,11cyclictest438-20systemd-journal10:30:003
3189551993519,11cyclictest3286030-21cut12:15:233
318955199351,14cyclictest3267836-21irqstats11:15:193
318955199351,14cyclictest3267836-21irqstats11:15:193
318954999351,9cyclictest2197014-21PK-Backend11:10:372
318954499352,30cyclictest0-21swapper/010:26:100
318954499352,29cyclictest0-21swapper/011:54:240
318954499352,10cyclictest0-21swapper/008:05:160
3189544993519,12cyclictest3232997-21awk09:15:330
3189544993516,14cyclictest438-20systemd-journal10:35:000
3189544993513,5cyclictest141rcu_preempt08:00:130
162399350,29rtkit-daemon0-21swapper/311:25:083
3189551993422,8cyclictest1753-21gnome-shell11:06:313
3189551993420,10cyclictest42-21ksoftirqd/310:21:323
3189551993417,14cyclictest1825-21JS14:31:443
318955199341,10cyclictest2197014-21PK-Backend12:10:083
318954499349,21cyclictest0-21swapper/010:45:490
318954499342,9cyclictest1753-21gnome-shell08:35:280
3189544993426,5cyclictest0-21swapper/011:05:520
3189544993426,5cyclictest0-21swapper/008:53:360
3189544993426,5cyclictest0-21swapper/008:53:360
318954499342,29cyclictest0-21swapper/010:16:080
318954499342,27cyclictest1800-21upowerd07:36:390
3189544993421,9cyclictest13-21ksoftirqd/011:20:210
318954499342,13cyclictest0-21swapper/008:48:300
3189544993420,9cyclictest1753-21gnome-shell08:21:340
3189544993420,10cyclictest3286237-21ntpq12:15:260
3189544993417,13cyclictest1753-21gnome-shell12:09:300
318954499341,12cyclictest141rcu_preempt09:39:200
32694712330,7sleep32197014-21PK-Backend11:20:203
3189551993322,6cyclictest42-21ksoftirqd/308:19:403
3189551993321,8cyclictest1753-21gnome-shell10:00:303
3189551993316,13cyclictest3244496-21awk09:55:273
318954499331,6cyclictest3289351-21turbostat12:30:000
318955199328,7cyclictest0-21swapper/309:40:323
3189551993219,10cyclictest42-21ksoftirqd/309:35:193
3189544993224,4cyclictest2197014-21PK-Backend10:06:090
162399320,25rtkit-daemon2197014-21PK-Backend10:41:043
318954499317,9cyclictest13-21ksoftirqd/009:26:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional