You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 22:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Nov 17, 2025 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
370617424000,7sleep00-21swapper/007:05:190
37700412920,7sleep02197014-21PK-Backend10:05:180
37688232840,7sleep1251rcuc/110:00:311
370568028355,20sleep30-21swapper/307:05:113
37703822770,8sleep11753-21gnome-shell10:05:271
370708227661,9sleep20-21swapper/207:09:032
37962432720,5sleep1251rcuc/111:17:361
370700327156,9sleep10-21swapper/107:08:031
162399570,8rtkit-daemon1622-21rtkit-daemon10:31:473
162399570,8rtkit-daemon1622-21rtkit-daemon09:26:312
162399560,8rtkit-daemon1622-21rtkit-daemon08:57:532
162399550,8rtkit-daemon1622-21rtkit-daemon08:18:243
162399540,8rtkit-daemon1622-21rtkit-daemon07:22:022
370731299538,17cyclictest3744346-21needreboot08:50:243
370731299532,11cyclictest1753-21gnome-shell08:32:573
38149232510,8chrt1753-21gnome-shell12:10:303
370731299501,15cyclictest0-21swapper/310:40:013
370731299501,15cyclictest0-21swapper/309:25:153
370731299494,25cyclictest1069-21runrttasks08:45:163
3707305994926,19cyclictest3821776-21ntp_states12:30:282
370729699491,11cyclictest2197014-21PK-Backend08:01:160
370731299481,29cyclictest1753-21gnome-shell10:57:453
370731299481,21cyclictest3770068-21diskstats10:05:203
370731299481,15cyclictest0-21swapper/310:10:343
162399480,40rtkit-daemon1753-21gnome-shell08:24:562
38149432470,6sleep02197014-21PK-Backend12:10:320
37953512470,7chrt2197014-21PK-Backend11:15:232
37470802470,6sleep1231irq_work/109:00:031
370731299472,20cyclictest3712915-40lsb_release07:21:253
370731299472,17cyclictest1753-21gnome-shell11:45:013
3707300994717,16cyclictest26-21ksoftirqd/108:25:151
162399470,38rtkit-daemon1753-21gnome-shell09:03:003
3707312994613,14cyclictest3735747-21ntpq08:25:273
370731299461,26cyclictest1753-21gnome-shell07:30:293
370731299461,20cyclictest3746890-21munin-run09:00:003
370731299461,18cyclictest3808672-21latency_hist11:55:023
370731299461,16cyclictest3749631-21ntp_states09:05:283
37492792450,4chrt2197014-21PK-Backend09:05:210
370731299451,20cyclictest3724270-21cat07:55:183
370731299451,19cyclictest1753-21gnome-shell09:45:423
370731299451,19cyclictest1753-21gnome-shell09:40:163
370731299451,16cyclictest3729572-21diskstats08:10:173
3707305994516,8cyclictest37793312cat10:30:222
370729699452,10cyclictest0-21swapper/008:10:020
370729699451,14cyclictest438-20systemd-journal09:00:000
37876662440,7sleep0161rcuc/010:55:170
370731299449,16cyclictest1753-21gnome-shell09:55:053
370731299442,18cyclictest0-21swapper/309:10:373
370731299441,17cyclictest3791925-21ls11:05:313
370731299441,17cyclictest3777209-21expr10:25:173
370731299441,17cyclictest1753-21gnome-shell08:23:453
370731299441,16cyclictest3793655-21ntpq11:10:273
370731299441,15cyclictest854-21avahi-daemon07:35:063
370730599443,14cyclictest17531gnome-shell12:00:252
3707300994418,13cyclictest1753-21gnome-shell12:30:321
370729699447,5cyclictest141rcu_preempt08:10:170
3707296994427,12cyclictest438-20systemd-journal11:14:590
38132612430,6sleep10-21swapper/112:05:341
370731299439,18cyclictest1753-21gnome-shell07:48:233
370731299432,17cyclictest1753-21gnome-shell11:17:473
370731299431,18cyclictest3809249-21hddtemp_smartct11:55:193
370731299431,18cyclictest3710125-21hddtemp_smartct07:15:203
370731299431,17cyclictest3727360-21sed08:05:013
370731299431,17cyclictest3722218-21basename07:50:133
370731299431,17cyclictest3722218-21basename07:50:123
370731299431,17cyclictest1753-21gnome-shell12:32:313
370731299431,16cyclictest3775274-21expr10:20:153
3707305994321,14cyclictest34-21ksoftirqd/207:26:412
3707305994313,5cyclictest141rcu_preempt08:50:012
3707305994310,6cyclictest141rcu_preempt11:00:002
3707300994331,8cyclictest3809873-21grep11:55:311
3707300994327,12cyclictest3755531-21cron09:24:591
370729699436,5cyclictest141rcu_preempt12:25:000
370729699436,5cyclictest141rcu_preempt07:50:010
370729699435,6cyclictest141rcu_preempt11:54:590
370729699435,5cyclictest141rcu_preempt08:41:570
3707296994312,8cyclictest37388862smartctl08:35:210
3707312994227,11cyclictest42-21ksoftirqd/310:47:043
370731299421,19cyclictest3819557-21taskset12:25:153
370731299421,18cyclictest1753-21gnome-shell09:40:023
370731299421,15cyclictest0-21swapper/307:10:163
370730599425,6cyclictest141rcu_preempt07:20:032
3707305994227,12cyclictest3721964-21awk07:50:012
3707305994220,13cyclictest34-21ksoftirqd/212:28:002
3707305994215,13cyclictest438-20systemd-journal09:09:592
3707300994226,12cyclictest3711193-21/usr/sbin/munin07:20:001
3707300994224,13cyclictest438-20systemd-journal07:39:541
370730099422,26cyclictest884-21gdbus08:01:151
370729699425,5cyclictest141rcu_preempt10:25:260
370729699423,5cyclictest141rcu_preempt10:20:310
370729699422,5cyclictest141rcu_preempt09:45:010
3707296994222,15cyclictest3763120-21df_inode09:45:170
162399420,34rtkit-daemon2197014-21PK-Backend08:35:172
370731299411,17cyclictest860997-21snmpd10:00:323
370731299411,17cyclictest1753-21gnome-shell11:45:223
370731299411,15cyclictest854-21avahi-daemon11:20:353
3707305994125,11cyclictest3736926-21grep08:30:162
370730599412,34cyclictest1753-21gnome-shell08:25:192
370730599412,34cyclictest1753-21gnome-shell07:52:092
370730599412,34cyclictest1753-21gnome-shell07:52:092
3707305994120,13cyclictest34-21ksoftirqd/209:46:092
3707305994119,13cyclictest34-21ksoftirqd/210:51:202
370730599411,14cyclictest0-21swapper/212:00:012
370730099417,9cyclictest1-21systemd09:29:371
370729699412,9cyclictest1753-21gnome-shell10:34:430
370729699412,34cyclictest1753-21gnome-shell10:04:190
370729699412,34cyclictest0-21swapper/010:53:260
370729699411,9cyclictest2197014-21PK-Backend10:12:350
370729699411,10cyclictest2197014-21PK-Backend11:55:280
370729699411,10cyclictest2197014-21PK-Backend11:45:290
3707312994016,12cyclictest855-21dbus-daemon10:20:003
370731299401,34cyclictest438-20systemd-journal08:08:363
370731299401,17cyclictest3718759-21apt07:40:153
370731299401,16cyclictest3812722-21hddtemp_smartct12:05:233
370731299401,16cyclictest3754437-21expr09:20:183
370731299401,15cyclictest0-21swapper/312:00:153
370731299401,15cyclictest0-21swapper/310:40:143
370731299400,16cyclictest3741862-21latency_hist08:45:023
3707305994028,9cyclictest34-21ksoftirqd/212:12:212
3707305994027,8cyclictest3794567-21pool-org.gnome.11:15:022
3707305994018,14cyclictest34-21ksoftirqd/209:31:302
3707300994020,15cyclictest3822774-21cpu12:35:171
3707300994018,5cyclictest26-21ksoftirqd/109:50:381
3707300994017,5cyclictest26-21ksoftirqd/110:20:311
3707300994016,20cyclictest3778162-40apt-check10:27:131
3707300994011,8cyclictest26-21ksoftirqd/108:13:501
370729699405,12cyclictest141rcu_preempt07:30:430
3707296994029,7cyclictest1753-21gnome-shell07:11:400
370729699402,33cyclictest3785461-21munin-run10:50:010
370729699401,6cyclictest2197014-21PK-Backend09:40:030
3707296994012,24cyclictest181irq_work/012:16:510
370729699401,12cyclictest2197014-21PK-Backend08:27:060
370729699401,10cyclictest2197014-21PK-Backend11:05:280
370729699401,10cyclictest2197014-21PK-Backend09:24:230
3707296994010,6cyclictest141rcu_preempt10:35:230
370731299392,8cyclictest1753-21gnome-shell07:25:403
3707312993921,13cyclictest3787191-21pool-org.gnome.10:55:013
370730599392,32cyclictest1753-21gnome-shell10:41:402
3707305993921,13cyclictest34-21ksoftirqd/211:50:132
370730599391,9cyclictest2197014-21PK-Backend10:46:022
370730599391,18cyclictest3727189-21grep08:05:002
370730099398,15cyclictest1358-1kworker/u9:111:09:581
3707300993924,10cyclictest3781838-21sshd10:38:361
3707300993922,12cyclictest3816555-21ntpq12:15:261
3707300993917,12cyclictest26-21ksoftirqd/109:15:011
3707300993916,6cyclictest26-21ksoftirqd/109:49:591
3707300993914,10cyclictest3721502-21cat07:45:341
3707300993913,22cyclictest855-21dbus-daemon07:55:021
3707300993913,22cyclictest855-21dbus-daemon07:55:011
3707300993912,11cyclictest884-21gdbus12:12:231
370729699396,5cyclictest141rcu_preempt09:14:400
3707296993928,7cyclictest1069-21runrttasks08:52:440
3707296993927,8cyclictest3804021-21df_inode11:40:160
3707296993927,7cyclictest3732923-21apt-key08:20:020
3707296993927,7cyclictest1753-21gnome-shell08:33:400
370729699392,32cyclictest1753-21gnome-shell09:17:280
3707296993923,11cyclictest438-20systemd-journal11:00:020
3707296993921,13cyclictest438-20systemd-journal11:30:350
3707296993921,13cyclictest3799181-21ntpq11:25:260
370729699392,10cyclictest141rcu_preempt09:29:040
370729699391,9cyclictest2197014-21PK-Backend09:01:410
370729699391,6cyclictest141rcu_preempt08:21:170
38079022380,13sleep1251rcuc/111:50:281
370731299382,17cyclictest1753-21gnome-shell11:04:353
3707312993818,15cyclictest438-20systemd-journal09:55:003
370731299381,15cyclictest859-21NetworkManager09:15:133
370730599387,5cyclictest141rcu_preempt12:18:112
3707305993821,13cyclictest855-21dbus-daemon09:40:012
3707305993821,13cyclictest3805773-21forks11:45:202
3707305993821,12cyclictest3752107-21systemd-run09:15:002
3707305993820,6cyclictest34-21ksoftirqd/209:50:232
3707305993817,6cyclictest34-21ksoftirqd/211:20:062
370730099387,7cyclictest1358-1kworker/u9:107:26:131
370730099382,31cyclictest3799320-21python311:25:301
3707300993822,12cyclictest3752724-21diskstats09:15:181
3707300993820,13cyclictest3242253-21nautilus10:00:021
370730099381,8cyclictest3803160-21chrt11:37:211
3707300993814,19cyclictest3712906-40lsb_release07:21:241
3707300993810,8cyclictest26-21ksoftirqd/110:31:311
370729699386,5cyclictest141rcu_preempt09:50:460
370729699382,32cyclictest1753-21gnome-shell07:15:300
3707296993822,11cyclictest438-20systemd-journal09:33:070
3707296993820,13cyclictest1753-21gnome-shell11:15:320
370729699381,9cyclictest2197014-21PK-Backend12:32:140
3707296993818,12cyclictest13-21ksoftirqd/009:57:590
370729699381,7cyclictest2197014-21PK-Backend08:50:020
162399380,30rtkit-daemon42-21ksoftirqd/311:38:173
162399380,29rtkit-daemon42-21ksoftirqd/311:32:173
162399380,29rtkit-daemon42-21ksoftirqd/311:27:033
37120282370,5chrt0-21swapper/007:20:150
3707312993726,7cyclictest1753-21gnome-shell08:36:583
370731299371,12cyclictest3818647-21users12:20:343
3707305993726,7cyclictest3754739-21tail09:20:272
370730599372,18cyclictest1753-21gnome-shell08:05:452
3707305993721,5cyclictest34-21ksoftirqd/209:57:272
3707305993721,11cyclictest3770423-21ntpq10:05:272
3707305993720,12cyclictest3740821-21cat08:40:222
3707305993719,7cyclictest34-21ksoftirqd/209:45:022
3707305993711,5cyclictest141rcu_preempt08:15:182
370730099372,7cyclictest3731593-40lsb_release08:15:231
3707300993727,7cyclictest251rcuc/108:32:531
3707300993727,7cyclictest0-21swapper/108:45:051
3707300993726,7cyclictest64850irq/126-i91511:45:211
3707300993724,9cyclictest1753-21gnome-shell09:06:031
3707300993722,11cyclictest3757803-21cat09:30:161
3707300993721,12cyclictest3817947-21entropy12:20:201
3707300993721,11cyclictest3744767-21vmstat08:50:321
3707300993720,13cyclictest3724825-21sensors_temp07:55:301
3707300993720,12cyclictest3747874-21ntp_kernel_pll_09:00:281
3707300993718,14cyclictest3242253-21nautilus11:15:021
3707300993715,7cyclictest1753-21gnome-shell11:23:341
3707300993713,20cyclictest3773602-21hddtemp_smartct10:15:241
370730099371,31cyclictest3760735-21cron09:40:001
370730099371,31cyclictest3707718-21ls07:10:141
3707296993728,6cyclictest0-21swapper/007:29:340
38243512360,5sleep0161rcuc/012:40:030
370730599365,10cyclictest34-21ksoftirqd/209:16:132
3707305993621,11cyclictest3743677-21awk08:50:132
3707305993621,11cyclictest3708114-21cat07:10:232
3707305993618,5cyclictest34-21ksoftirqd/212:06:192
3707305993618,14cyclictest3748407-21sshd09:02:352
3707305993618,13cyclictest438-20systemd-journal10:40:012
370730099362,17cyclictest1753-21gnome-shell08:20:161
3707300993620,12cyclictest3785150-21sshd10:46:361
3707300993619,13cyclictest3820478-21sshd12:26:371
3707300993618,15cyclictest1069-21runrttasks10:45:001
3707300993617,14cyclictest3801268-21users11:30:331
3707300993616,16cyclictest3719677-21switchtime07:40:331
370729699369,6cyclictest141rcu_preempt12:30:010
370729699368,6cyclictest141rcu_preempt07:51:280
370729699368,6cyclictest141rcu_preempt07:51:280
370729699366,5cyclictest141rcu_preempt12:04:560
370729699362,5cyclictest0-21swapper/007:43:570
3707296993625,7cyclictest3773385-21grep10:15:170
3707296993620,12cyclictest3798166-21cron11:25:010
3707296993618,13cyclictest3802567-21cat11:35:240
370729699361,31cyclictest1753-21gnome-shell07:57:510
3707312993516,14cyclictest3757780-21expr09:30:143
370731299351,23cyclictest1753-21gnome-shell12:35:113
370730599353,28cyclictest1069-21runrttasks10:13:322
3707305993519,11cyclictest3715646-21basename07:30:212
3707305993518,13cyclictest3242254-21gmain10:25:002
3707305993517,5cyclictest34-21ksoftirqd/211:27:142
3707305993517,5cyclictest34-21ksoftirqd/211:07:432
3707305993514,5cyclictest141rcu_preempt11:02:272
370730099356,7cyclictest3772188-21ntpq10:10:281
3707300993519,12cyclictest3738829-21expr08:35:171
3707300993513,19cyclictest3761301-21expr09:40:151
370729699352,29cyclictest0-21swapper/010:41:160
3707296993515,16cyclictest3812478-21cut12:05:170
3707312993419,11cyclictest3816093-21expr12:15:173
370730599347,5cyclictest141rcu_preempt11:38:592
3707305993425,5cyclictest2197014-21PK-Backend07:43:062
3707305993422,8cyclictest1826-21JS14:31:442
3707305993417,5cyclictest34-21ksoftirqd/212:21:452
370730599341,6cyclictest2197014-21PK-Backend10:01:062
3707305993410,5cyclictest141rcu_preempt10:27:202
370730099346,8cyclictest1358-1kworker/u9:107:30:261
3707300993422,9cyclictest3787410-21/usr/sbin/munin10:55:361
3707300993419,11cyclictest3741726-21cat08:45:021
3707300993417,12cyclictest3811597-21cat12:00:321
3707300993417,12cyclictest3786078-21diskstats10:50:181
370729699344,4cyclictest141rcu_preempt07:37:430
3707305993321,7cyclictest3725580-21wc08:00:022
3707305993317,11cyclictest3804366-21ntpq11:40:252
370730099332,9cyclictest1753-21gnome-shell08:05:141
3707300993318,11cyclictest3804132-21awk11:40:221
370730599325,9cyclictest141rcu_preempt12:35:132
370730599321,28cyclictest0-21swapper/208:12:292
3707300993217,11cyclictest3789803-21cat11:00:251
370730599312,5cyclictest141rcu_preempt11:32:032
3707305993124,4cyclictest2197014-21PK-Backend07:35:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional