You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 19:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 03, 2025 12:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991110,9rtkit-daemon1622-21rtkit-daemon07:58:382
1623991060,35rtkit-daemon1753-21gnome-shell07:07:290
1623991030,9rtkit-daemon1622-21rtkit-daemon09:46:552
1623991030,8rtkit-daemon1622-21rtkit-daemon09:00:172
1623991000,9rtkit-daemon1622-21rtkit-daemon08:50:122
1623991000,9rtkit-daemon1622-21rtkit-daemon08:50:122
162399980,8rtkit-daemon1622-21rtkit-daemon08:47:032
162399950,9rtkit-daemon1622-21rtkit-daemon09:25:432
162399920,9rtkit-daemon1622-21rtkit-daemon09:37:132
23722129144,37sleep30-21swapper/307:09:353
162399900,9rtkit-daemon1622-21rtkit-daemon07:27:072
162399890,9rtkit-daemon1622-21rtkit-daemon08:43:212
23702127959,14sleep20-21swapper/207:07:052
3384812730,5sleep22197014-21PK-Backend11:55:032
23701526242,13sleep10-21swapper/107:07:001
23740299590,55cyclictest1622-21rtkit-daemon07:23:192
23740299590,55cyclictest1622-21rtkit-daemon07:23:192
162399590,9rtkit-daemon1622-21rtkit-daemon09:12:243
162399590,9rtkit-daemon1622-21rtkit-daemon09:05:263
162399580,9rtkit-daemon1622-21rtkit-daemon08:35:083
162399580,9rtkit-daemon1622-21rtkit-daemon08:07:293
162399570,8rtkit-daemon1622-21rtkit-daemon09:02:103
23739799566,6cyclictest0-21swapper/107:51:291
162399560,9rtkit-daemon1753-21gnome-shell08:30:243
162399560,9rtkit-daemon1622-21rtkit-daemon09:37:483
162399560,8rtkit-daemon1622-21rtkit-daemon07:54:473
162399550,9rtkit-daemon1753-21gnome-shell09:19:133
162399550,9rtkit-daemon1753-21gnome-shell09:19:133
162399550,9rtkit-daemon1622-21rtkit-daemon09:22:183
162399550,8rtkit-daemon1622-21rtkit-daemon09:42:232
162399550,7rtkit-daemon1622-21rtkit-daemon09:34:153
162399530,9rtkit-daemon1622-21rtkit-daemon08:56:163
162399530,9rtkit-daemon1622-21rtkit-daemon08:56:163
162399530,8rtkit-daemon1622-21rtkit-daemon08:00:152
162399520,8rtkit-daemon1622-21rtkit-daemon07:45:233
3215532510,13chrt321552-21wc11:05:231
23741099518,39cyclictest1753-21gnome-shell07:56:343
2582872500,5sleep2331rcuc/208:05:202
3044472490,7sleep00-21swapper/010:15:270
237394994730,13cyclictest13-21ksoftirqd/007:36:390
3141692460,6sleep1301623-21kworker/1:210:45:141
2669132460,4sleep20-21swapper/208:30:202
237410994618,23cyclictest248518-40lsb_release07:36:253
237402994623,14cyclictest34-21ksoftirqd/211:00:002
237397994618,13cyclictest26-21ksoftirqd/109:15:001
3224082450,7chrt322394-21perf11:10:010
237410994524,17cyclictest313974-21/usr/sbin/munin10:45:173
237402994520,9cyclictest141rcu_preempt09:59:562
237402994516,6cyclictest3336562cstates11:40:152
237394994528,12cyclictest320518-21sshd11:03:250
237394994510,19cyclictest1753-21gnome-shell11:10:290
162399450,34rtkit-daemon2197014-21PK-Backend11:19:203
3108972440,7sleep10-21swapper/110:35:151
237410994410,15cyclictest1753-21gnome-shell10:44:413
237402994425,6cyclictest34-21ksoftirqd/210:48:092
237402994423,12cyclictest34-21ksoftirqd/209:15:402
237402994423,12cyclictest34-21ksoftirqd/209:15:402
237402994421,14cyclictest34-21ksoftirqd/211:58:252
237402994421,14cyclictest34-21ksoftirqd/211:23:312
237402994421,14cyclictest34-21ksoftirqd/211:23:312
23739799442,38cyclictest1753-21gnome-shell11:22:051
23739799442,38cyclictest1753-21gnome-shell11:22:051
237397994417,22cyclictest345315-21date12:15:021
237397994417,22cyclictest345315-21date12:15:021
23739799441,39cyclictest325445-21sed11:15:331
23739499445,10cyclictest13-21ksoftirqd/008:30:110
237402994321,13cyclictest34-21ksoftirqd/211:45:302
237402994321,13cyclictest34-21ksoftirqd/211:06:242
23739799439,21cyclictest26-21ksoftirqd/112:09:381
237397994324,14cyclictest282880-21sh09:15:191
237397994324,14cyclictest282880-21sh09:15:191
23739499433,9cyclictest1753-21gnome-shell09:15:000
237394994325,13cyclictest438-20systemd-journal09:05:250
162399430,5rtkit-daemon0-21swapper/310:08:123
237410994222,14cyclictest42-21ksoftirqd/311:30:223
237402994223,14cyclictest34-21ksoftirqd/207:31:072
23740299422,22cyclictest1753-21gnome-shell08:12:582
237402994222,16cyclictest34-21ksoftirqd/208:22:022
237402994222,15cyclictest34-21ksoftirqd/209:05:052
237402994220,13cyclictest34-21ksoftirqd/209:24:092
23740299421,37cyclictest1753-21gnome-shell08:59:492
23740299421,37cyclictest1753-21gnome-shell08:59:492
237397994219,11cyclictest247119-21dpkg07:35:021
237397994217,20cyclictest349494-21cat12:25:151
23739499422,8cyclictest1753-21gnome-shell11:28:230
237394994223,14cyclictest438-20systemd-journal08:05:250
162399420,26rtkit-daemon1753-21gnome-shell11:57:543
3436272410,6sleep01753-21gnome-shell12:10:020
23741099411,16cyclictest311986-21latency_hist10:40:023
237402994122,14cyclictest34-21ksoftirqd/211:35:132
23740299411,9cyclictest2197014-21PK-Backend07:50:142
237402994118,14cyclictest34-21ksoftirqd/212:20:062
237402994111,14cyclictest34-21ksoftirqd/212:30:332
237402994111,14cyclictest34-21ksoftirqd/212:30:332
23739799419,8cyclictest264547-21kworker/1:008:30:001
237397994117,20cyclictest3242254-21gmain07:20:001
23739499418,5cyclictest141rcu_preempt10:45:020
23739499417,7cyclictest141rcu_preempt09:33:300
237394994127,10cyclictest306727-21sshd10:24:250
237394994127,10cyclictest306727-21sshd10:24:250
237394994124,12cyclictest438-20systemd-journal11:15:250
237394994123,13cyclictest438-20systemd-journal12:35:260
23739499412,10cyclictest1753-21gnome-shell08:47:080
162399410,33rtkit-daemon2197014-21PK-Backend08:43:293
237410994022,6cyclictest42-21ksoftirqd/309:25:143
237410994016,20cyclictest391irq_work/310:19:093
237410994012,5cyclictest141rcu_preempt08:20:183
237402994019,12cyclictest34-21ksoftirqd/210:00:522
23740299401,8cyclictest2197014-21PK-Backend12:07:112
237402994017,6cyclictest141rcu_preempt10:23:282
237402994017,6cyclictest141rcu_preempt10:23:282
237402994010,15cyclictest1753-21gnome-shell10:05:272
237397994023,12cyclictest438-20systemd-journal12:01:331
237397994022,14cyclictest292680-40lsb_release09:42:241
237397994019,6cyclictest251rcuc/110:58:471
23739799401,35cyclictest333135-21pool-org.gnome.11:40:011
237397994012,12cyclictest346238-21ntp_kernel_pll_12:15:251
23739499406,12cyclictest141rcu_preempt10:50:420
23739499406,10cyclictest1069-21runrttasks07:21:570
23739499406,10cyclictest1069-21runrttasks07:21:570
23739499402,34cyclictest1753-21gnome-shell12:13:470
23739499402,34cyclictest1753-21gnome-shell12:13:470
237394994023,13cyclictest290818-21sshd09:39:250
237394994023,12cyclictest438-20systemd-journal07:12:250
237394994021,6cyclictest13-21ksoftirqd/012:34:250
237394994021,6cyclictest13-21ksoftirqd/012:34:240
23739499402,12cyclictest264510-21grep08:25:000
237394994018,17cyclictest285940-21awk09:25:110
23741099392,8cyclictest1753-21gnome-shell11:43:173
23741099392,8cyclictest1753-21gnome-shell10:58:343
237410993924,11cyclictest338291-21cron11:55:013
237410993922,13cyclictest855-21dbus-daemon08:20:003
237410993922,13cyclictest348884-21sshd12:23:333
237410993921,14cyclictest42-21ksoftirqd/307:22:143
237410993921,14cyclictest42-21ksoftirqd/307:22:143
237410993921,14cyclictest323030-21df_inode11:10:153
237410993920,7cyclictest42-21ksoftirqd/311:25:233
23741099391,4cyclictest0-21swapper/310:23:513
23741099391,4cyclictest0-21swapper/310:23:513
23740299398,15cyclictest1753-21gnome-shell09:50:402
23740299392,16cyclictest141rcu_preempt10:15:452
237402993921,5cyclictest34-21ksoftirqd/212:00:142
237402993912,24cyclictest303350-21unin-run10:15:012
23739799398,12cyclictest309445-21needreboot10:30:241
23739799393,19cyclictest1753-21gnome-shell09:30:381
237397993928,7cyclictest1753-21gnome-shell07:57:151
237397993921,14cyclictest3242253-21nautilus11:50:011
237397993917,19cyclictest304674-21users10:15:301
237397993916,6cyclictest26-21ksoftirqd/109:55:021
237397993910,9cyclictest0-21swapper/109:01:021
23739499396,13cyclictest141rcu_preempt08:59:220
23739499396,13cyclictest141rcu_preempt08:59:220
23739499392,8cyclictest1753-21gnome-shell08:17:440
23739499392,7cyclictest0-21swapper/011:55:310
237394993924,11cyclictest255835-21munin-run08:00:000
237394993922,5cyclictest13-21ksoftirqd/009:41:460
237394993921,6cyclictest13-21ksoftirqd/009:49:230
237394993913,6cyclictest141rcu_preempt08:01:560
23739499391,35cyclictest0-21swapper/012:26:440
237394993911,6cyclictest141rcu_preempt09:16:100
237394993911,6cyclictest141rcu_preempt09:16:100
23741099383,31cyclictest1753-21gnome-shell11:02:153
237410993823,11cyclictest326785-21sshd11:20:253
237410993823,11cyclictest326785-21sshd11:20:253
237410993821,13cyclictest42-21ksoftirqd/307:40:343
237410993819,14cyclictest42-21ksoftirqd/308:10:213
237410993816,18cyclictest42-21ksoftirqd/312:38:183
23740299389,12cyclictest141rcu_preempt07:35:202
23740299388,6cyclictest141rcu_preempt12:20:032
23740299385,13cyclictest141rcu_preempt07:40:172
23740299384,16cyclictest0-21swapper/210:37:022
23740299383,11cyclictest141rcu_preempt11:11:272
237402993826,8cyclictest1753-21gnome-shell07:19:572
237402993821,12cyclictest315733-21sshd10:50:122
237402993816,19cyclictest311irq_work/211:29:552
237397993830,5cyclictest2498392sleep107:40:251
237397993824,10cyclictest323906-21sshd11:12:251
23739799382,32cyclictest1753-21gnome-shell08:32:541
237397993823,11cyclictest333758-21sshd11:40:181
237397993823,11cyclictest252350-21cron07:50:001
237397993822,12cyclictest339529-21threads11:55:311
237397993822,12cyclictest329666-21pool-org.gnome.11:30:011
237397993822,11cyclictest438-20systemd-journal08:25:001
237397993821,13cyclictest308552-21pool-org.gnome.10:30:001
237397993820,14cyclictest3242255-21gdbus08:20:001
237397993820,13cyclictest351728-21ntp_kernel_err12:30:241
237397993820,13cyclictest351728-21ntp_kernel_err12:30:241
237397993819,15cyclictest855-21dbus-daemon09:00:011
237397993819,15cyclictest855-21dbus-daemon09:00:011
23739499383,7cyclictest251484-21memory07:45:230
23739499382,8cyclictest298120-21apt10:00:000
23739499382,33cyclictest0-21swapper/010:12:350
23739499382,31cyclictest245332-21apt07:30:130
237394993821,12cyclictest271452-21/usr/sbin/munin08:45:000
23739499381,13cyclictest326854-21awk11:20:260
23739499381,13cyclictest326854-21awk11:20:260
237410993726,7cyclictest1069-21runrttasks12:06:363
237410993723,10cyclictest1753-21gnome-shell10:27:543
23741099372,15cyclictest1753-21gnome-shell10:13:533
237410993719,13cyclictest2962348-21rs:main3
237410993713,7cyclictest1069-21runrttasks11:45:533
23741099371,18cyclictest346980-21latency_hist12:20:023
237410993711,22cyclictest246858-21apt-key07:35:013
237402993722,11cyclictest855-21dbus-daemon12:11:562
237402993722,11cyclictest855-21dbus-daemon12:11:562
237402993721,5cyclictest34-21ksoftirqd/211:19:272
237402993718,11cyclictest34-21ksoftirqd/212:39:422
237402993714,9cyclictest34-21ksoftirqd/212:30:002
237402993714,6cyclictest141rcu_preempt07:14:572
237397993721,12cyclictest287131-21seq09:26:361
237397993721,12cyclictest243019-40lsb_release07:21:271
237397993721,12cyclictest243019-40lsb_release07:21:261
237397993721,11cyclictest904-21systemd-logind10:41:501
237397993712,21cyclictest306073-21ntpq10:20:251
237397993712,21cyclictest306073-21ntpq10:20:251
237397993710,11cyclictest243958-21sh07:25:161
23739499373,7cyclictest0-21swapper/012:00:360
23739499372,16cyclictest854-21avahi-daemon12:16:020
237394993721,5cyclictest13-21ksoftirqd/011:42:490
237394993721,11cyclictest243688-21cpuspeed_turbos07:25:140
23739499371,9cyclictest2197014-21PK-Backend12:20:570
23739499371,6cyclictest141rcu_preempt11:52:170
237394993716,12cyclictest13-21ksoftirqd/011:45:430
23739499371,4cyclictest0-21swapper/011:30:030
23739499371,32cyclictest285277-21cat09:20:310
23739499371,10cyclictest0-21swapper/007:15:040
23741099365,8cyclictest42-21ksoftirqd/308:49:273
237410993618,13cyclictest42-21ksoftirqd/309:50:273
237410993614,19cyclictest273415-21/usr/sbin/munin08:50:293
237410993614,19cyclictest273415-21/usr/sbin/munin08:50:293
23741099361,31cyclictest2197014-21PK-Backend12:02:593
237410993611,21cyclictest1753-21gnome-shell12:25:073
23740299369,6cyclictest141rcu_preempt09:10:322
23740299368,6cyclictest141rcu_preempt08:26:522
237402993621,11cyclictest319459-21expr11:00:152
237402993617,14cyclictest288419-21ntpq09:30:252
237402993611,22cyclictest311irq_work/211:30:232
23739799369,6cyclictest272807-21awk08:45:331
23739799367,14cyclictest259938-21df_inode08:10:161
237397993625,8cyclictest300023-21cat10:05:021
237397993619,13cyclictest337628-21awk11:50:271
237397993618,13cyclictest438-20systemd-journal11:04:591
237397993611,21cyclictest330171-21cat11:30:151
23739499368,9cyclictest1069-21runrttasks10:38:070
23739499362,5cyclictest141rcu_preempt10:00:410
237394993625,7cyclictest1069-21runrttasks10:09:030
23739499362,30cyclictest0-21swapper/008:35:370
237394993621,11cyclictest254084-21cron07:55:000
237394993620,12cyclictest295404-21awk09:50:200
23739499361,9cyclictest854-21avahi-daemon08:28:440
23739499361,9cyclictest2197014-21PK-Backend09:00:200
23739499361,10cyclictest2197014-21PK-Backend10:46:440
23741099352,29cyclictest1753-21gnome-shell12:13:033
23741099352,29cyclictest1753-21gnome-shell12:13:023
23741099352,29cyclictest1753-21gnome-shell10:00:413
23741099352,14cyclictest297378-21grep09:55:273
237410993520,11cyclictest292911-21cron09:45:013
237410993519,12cyclictest322032-21seq11:05:343
237410993514,17cyclictest42-21ksoftirqd/307:15:193
237402993519,11cyclictest994-21lldpd10:43:032
237402993519,11cyclictest261513-21cstates08:15:152
237397993523,8cyclictest64850irq/126-i91510:00:011
23739799352,20cyclictest0-21swapper/110:05:311
237397993519,12cyclictest284801-21memory09:20:221
237397993519,12cyclictest238328-21sshd07:10:251
237397993518,13cyclictest2181-21gdbus12:20:031
237397993517,13cyclictest468-21systemd-udevd08:02:501
237397993517,13cyclictest438-20systemd-journal08:40:251
237397993515,16cyclictest269223-21timerandwakeup08:35:301
237394993516,15cyclictest332231-21memory11:35:220
23739499351,4cyclictest2197014-21PK-Backend08:52:060
23739499351,4cyclictest2197014-21PK-Backend08:52:060
23739499351,30cyclictest2197014-21PK-Backend10:58:020
23739499351,10cyclictest2197014-21PK-Backend10:25:150
2484872340,9chrt26-21ksoftirqd/107:36:171
23741099344,12cyclictest0-21swapper/307:11:393
237410993424,6cyclictest244372-21ntpq07:25:423
23741099342,27cyclictest1753-21gnome-shell12:31:503
23741099342,27cyclictest1753-21gnome-shell12:31:503
237410993421,9cyclictest1753-21gnome-shell09:47:493
237410993420,11cyclictest42-21ksoftirqd/310:50:143
23740299341,9cyclictest2197014-21PK-Backend08:37:292
237402993419,11cyclictest251493-21grep07:45:232
23739799342,29cyclictest0-21swapper/110:50:301
237397993419,11cyclictest294245-21grep09:45:311
237397993419,10cyclictest302260-21smartctl10:10:191
237397993418,12cyclictest353607-21/usr/sbin/munin12:35:291
237397993418,12cyclictest258593-21ntpq08:05:261
237397993417,12cyclictest273532-21cpuspeed_turbos08:50:141
237397993417,12cyclictest273532-21cpuspeed_turbos08:50:141
23739499342,8cyclictest1753-21gnome-shell07:40:160
23739499341,5cyclictest141rcu_preempt08:10:230
237410993314,14cyclictest42-21ksoftirqd/311:35:193
237402993317,12cyclictest307054-21awk10:25:102
237402993311,18cyclictest1753-21gnome-shell10:31:542
237397993319,10cyclictest290168-21ntp_offset09:35:271
237397993318,11cyclictest279589-21cat09:05:211
237394993318,11cyclictest855-21dbus-daemon10:35:010
23741099322,9cyclictest265524-21ntp_states08:25:263
237410993221,8cyclictest308771-21/usr/sbin/munin10:30:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional