You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 18:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Dec 20, 2025 12:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28923621100,5sleep20-21swapper/208:05:142
26888928148,23sleep30-21swapper/307:07:093
26794527644,22sleep20-21swapper/207:05:162
197399750,5rtkit-daemon0-21swapper/307:32:013
197399740,6rtkit-daemon0-21swapper/307:16:293
197399740,5rtkit-daemon0-21swapper/307:22:553
3594952730,3sleep30-21swapper/311:25:163
26887126344,13sleep00-21swapper/007:06:540
26910726240,14sleep10-21swapper/107:09:561
197399580,9rtkit-daemon1972-21rtkit-daemon09:39:451
197399580,9rtkit-daemon1972-21rtkit-daemon09:39:451
197399580,8rtkit-daemon1972-21rtkit-daemon10:30:020
197399580,8rtkit-daemon1972-21rtkit-daemon10:30:010
269264995744,8cyclictest34-21ksoftirqd/208:28:592
197399570,9rtkit-daemon1972-21rtkit-daemon11:58:580
197399570,8rtkit-daemon1972-21rtkit-daemon09:10:330
197399570,8rtkit-daemon1972-21rtkit-daemon09:10:330
197399560,8rtkit-daemon1972-21rtkit-daemon09:34:321
197399550,8rtkit-daemon1972-21rtkit-daemon07:59:160
26925599542,10cyclictest2069-21gnome-shell09:40:590
197399540,9rtkit-daemon1972-21rtkit-daemon10:02:241
197399540,9rtkit-daemon1972-21rtkit-daemon10:02:241
197399540,8rtkit-daemon1972-21rtkit-daemon12:25:420
197399540,8rtkit-daemon1972-21rtkit-daemon12:25:410
197399540,8rtkit-daemon1972-21rtkit-daemon12:14:350
197399540,8rtkit-daemon1972-21rtkit-daemon09:16:301
26926499532,10cyclictest2069-21gnome-shell07:21:582
26926099534,46cyclictest231irq_work/108:05:591
26926099532,41cyclictest2069-21gnome-shell08:56:581
26926099531,48cyclictest2069-21gnome-shell08:00:591
3378922520,6chrt4497-21PK-Backend10:25:011
269264995238,10cyclictest2069-21gnome-shell07:30:592
26926499522,46cyclictest2069-21gnome-shell07:47:582
26926499522,10cyclictest2069-21gnome-shell09:29:592
269260995238,10cyclictest1137-21runrttasks07:22:581
26925599522,8cyclictest4497-21PK-Backend09:04:590
26925599521,10cyclictest244891-21kworker/0:107:32:580
197399520,8rtkit-daemon1972-21rtkit-daemon10:00:340
197399520,8rtkit-daemon1972-21rtkit-daemon10:00:330
3034002510,7chrt0-21swapper/208:45:032
269269995138,9cyclictest2069-21gnome-shell09:53:583
26926499519,37cyclictest2069-21gnome-shell08:16:592
26926499511,18cyclictest10542snmpd08:53:012
26926099514,7cyclictest231irq_work/109:46:581
26926099512,46cyclictest231irq_work/108:48:591
26926099512,46cyclictest231irq_work/108:48:581
26926099512,46cyclictest231irq_work/108:33:591
26926099512,45cyclictest2069-21gnome-shell07:15:581
26925599514,7cyclictest13-21ksoftirqd/009:33:580
26925599512,6cyclictest4497-21PK-Backend08:17:580
26926999502,44cyclictest2069-21gnome-shell08:28:583
26926099502,5cyclictest0-21swapper/107:44:581
26925599502,7cyclictest0-21swapper/007:25:580
26925599502,6cyclictest4497-21PK-Backend08:32:580
26925599502,6cyclictest4497-21PK-Backend08:14:590
26925599502,44cyclictest4497-21PK-Backend09:51:580
26926099492,44cyclictest231irq_work/109:43:581
26926099492,44cyclictest231irq_work/108:25:591
26926099492,44cyclictest231irq_work/107:52:581
26926099492,44cyclictest231irq_work/107:49:581
26926099491,45cyclictest231irq_work/107:13:591
26926099491,45cyclictest231irq_work/107:13:591
26925599492,6cyclictest4497-21PK-Backend08:41:580
26925599492,6cyclictest4497-21PK-Backend07:41:590
26925599492,6cyclictest0-21swapper/007:21:590
26925599492,43cyclictest4497-21PK-Backend08:29:580
26926999482,43cyclictest0-21swapper/308:33:583
26926999481,8cyclictest2069-21gnome-shell09:30:583
269260994839,6cyclictest231irq_work/108:37:581
269260994839,6cyclictest231irq_work/108:22:581
26926099481,45cyclictest231irq_work/108:13:581
26926099481,44cyclictest231irq_work/109:01:581
26926099481,44cyclictest231irq_work/107:34:591
26925599482,6cyclictest0-21swapper/007:35:580
26925599482,5cyclictest0-21swapper/009:08:580
26925599482,42cyclictest4497-21PK-Backend09:58:580
26925599481,43cyclictest4497-21PK-Backend07:45:590
26926999472,42cyclictest0-21swapper/308:58:593
26926499472,43cyclictest0-21swapper/209:15:582
26926499472,42cyclictest0-21swapper/207:54:592
269260994739,5cyclictest4497-21PK-Backend08:53:591
269255994740,4cyclictest0-21swapper/008:50:580
26925599471,5cyclictest4497-21PK-Backend08:23:580
26925599471,37cyclictest4497-21PK-Backend09:46:590
269269994638,5cyclictest42-21ksoftirqd/309:29:593
26926499462,41cyclictest0-21swapper/208:04:582
269260994638,5cyclictest231irq_work/109:28:581
269260994638,5cyclictest0-21swapper/107:57:591
269260994637,6cyclictest231irq_work/108:43:581
26926099461,43cyclictest231irq_work/109:22:581
269260994610,19cyclictest26-21ksoftirqd/111:20:001
269255994639,4cyclictest4497-21PK-Backend09:24:590
269255994639,4cyclictest4497-21PK-Backend09:16:590
26926999455,35cyclictest2069-21gnome-shell09:04:593
269269994535,7cyclictest269249-21cyclictest09:56:583
269269994533,8cyclictest2069-21gnome-shell09:44:593
269269994533,8cyclictest2069-21gnome-shell08:42:593
26926999452,41cyclictest0-21swapper/308:13:593
26926999452,41cyclictest0-21swapper/308:00:583
269264994539,4cyclictest0-21swapper/208:10:582
26926499452,41cyclictest0-21swapper/207:35:582
269264994523,14cyclictest34-21ksoftirqd/210:39:102
26926499451,5cyclictest4497-21PK-Backend09:39:592
26926499451,5cyclictest4497-21PK-Backend09:39:582
26926499451,5cyclictest4497-21PK-Backend09:11:592
26926499451,5cyclictest4497-21PK-Backend09:11:582
26926499451,14cyclictest2871842latency_hist08:00:022
26926099459,19cyclictest26-21ksoftirqd/110:10:011
269260994510,17cyclictest26-21ksoftirqd/110:55:011
269269994438,4cyclictest0-21swapper/307:58:583
26926999443,36cyclictest2069-21gnome-shell09:06:583
269269994325,6cyclictest42-21ksoftirqd/308:50:313
269264994337,4cyclictest0-21swapper/209:48:582
269264994337,4cyclictest0-21swapper/209:33:592
269264994337,4cyclictest0-21swapper/209:02:582
269264994330,9cyclictest2069-21gnome-shell08:35:582
269264994324,6cyclictest34-21ksoftirqd/209:25:012
26926099437,16cyclictest26-21ksoftirqd/110:20:021
26926099431,14cyclictest370655-21sed11:55:311
197399430,36rtkit-daemon1941-21systemd11:23:440
269269994231,8cyclictest42-21ksoftirqd/312:32:593
269269994219,9cyclictest337895-21sed10:25:013
26926999421,36cyclictest2069-21gnome-shell09:12:583
26926999421,36cyclictest2069-21gnome-shell09:12:583
269269994213,16cyclictest42-21ksoftirqd/310:10:253
269264994231,7cyclictest53-21kcompactd009:50:592
26926099425,8cyclictest26-21ksoftirqd/111:40:001
3444012410,5sleep3391irq_work/310:40:293
269269994128,9cyclictest42-21ksoftirqd/308:49:593
269269994128,9cyclictest42-21ksoftirqd/308:49:593
269269994128,9cyclictest2069-21gnome-shell09:15:593
26926499419,17cyclictest2221-21packagekitd10:10:092
269264994124,13cyclictest34-21ksoftirqd/210:40:102
269264994121,15cyclictest34-21ksoftirqd/211:57:092
26926099412,13cyclictest26-21ksoftirqd/111:05:301
3472662400,5chrt0-21swapper/310:50:163
3472662400,5chrt0-21swapper/310:50:153
26926999409,11cyclictest321081-21cat09:35:133
26926999409,11cyclictest321081-21cat09:35:133
269269994021,15cyclictest364567-21expr11:40:173
269269994020,15cyclictest2069-21gnome-shell07:45:593
26926999401,34cyclictest2069-21gnome-shell09:20:593
269264994030,6cyclictest2069-21gnome-shell07:25:592
269264994023,13cyclictest34-21ksoftirqd/210:50:002
269264994023,13cyclictest34-21ksoftirqd/209:58:102
26926099409,18cyclictest26-21ksoftirqd/111:10:271
26926099409,12cyclictest26-21ksoftirqd/110:40:241
26926099406,8cyclictest26-21ksoftirqd/108:15:231
26926099405,8cyclictest26-21ksoftirqd/112:15:011
26926099404,19cyclictest26-21ksoftirqd/109:10:241
26926099404,19cyclictest26-21ksoftirqd/109:10:231
26926099404,18cyclictest26-21ksoftirqd/110:35:231
26926099403,15cyclictest0-21swapper/107:37:211
26926099401,22cyclictest382808-21taskset12:30:391
26926099401,22cyclictest276256-21taskset07:25:471
3745932390,6sleep14497-21PK-Backend12:10:011
26926999399,16cyclictest366201-21cat11:45:153
26926999391,5cyclictest0-21swapper/311:51:493
26926499398,6cyclictest141rcu_preempt11:15:122
269264993921,14cyclictest34-21ksoftirqd/210:10:002
269264993921,13cyclictest34-21ksoftirqd/212:32:582
26926499391,13cyclictest0-21swapper/210:31:002
26926499391,11cyclictest361709-21proc_pri11:30:262
26926499391,11cyclictest2069-21gnome-shell07:40:582
26926099399,16cyclictest26-21ksoftirqd/111:21:061
26926099396,9cyclictest4497-21PK-Backend10:50:131
26926099396,9cyclictest4497-21PK-Backend10:50:121
269260993925,10cyclictest923-21dbus-daemon12:19:571
26926099391,22cyclictest360356-21seq11:26:341
26926099391,21cyclictest367213-21seq11:45:371
26926099391,11cyclictest351632-21seq11:00:421
269255993924,11cyclictest366914-21grep11:45:270
26926999384,16cyclictest0-21swapper/308:20:023
269269993822,11cyclictest438-20systemd-journal12:15:093
269269993816,19cyclictest1137-21runrttasks07:38:263
26926999381,33cyclictest2069-21gnome-shell07:42:593
26926999381,22cyclictest3421902df10:35:163
269264993828,6cyclictest1081-21ntpd11:40:262
269264993825,8cyclictest34-21ksoftirqd/212:04:592
269264993825,8cyclictest34-21ksoftirqd/212:04:592
269264993822,5cyclictest34-21ksoftirqd/212:38:092
269264993822,13cyclictest34-21ksoftirqd/207:10:132
269264993822,13cyclictest34-21ksoftirqd/207:10:122
269264993822,11cyclictest438-20systemd-journal08:33:092
269264993821,12cyclictest34-21ksoftirqd/212:05:112
269264993821,12cyclictest34-21ksoftirqd/211:25:322
269264993820,13cyclictest438-20systemd-journal08:47:102
269264993820,13cyclictest438-20systemd-journal08:47:092
26926499381,15cyclictest307874-21switchtime08:55:302
26926099387,13cyclictest26-21ksoftirqd/109:50:301
269260993824,10cyclictest26-21ksoftirqd/111:30:141
269260993823,11cyclictest329167-21cron10:00:011
269255993824,10cyclictest2069-21gnome-shell07:13:590
269255993824,10cyclictest2069-21gnome-shell07:13:580
26926999374,10cyclictest42-21ksoftirqd/312:25:033
26926999374,10cyclictest42-21ksoftirqd/312:25:033
269269993725,9cyclictest2069-21gnome-shell08:36:593
269269993725,8cyclictest379726-21systemd-run12:25:013
269269993722,11cyclictest353361-21tmpfiles11:05:403
269269993721,12cyclictest284561-21ntp_kernel_err07:50:243
269269993721,11cyclictest438-20systemd-journal08:24:553
269269993720,6cyclictest42-21ksoftirqd/310:50:023
269269993710,16cyclictest42-21ksoftirqd/311:30:143
269264993731,4cyclictest0-21swapper/207:18:582
269264993723,10cyclictest347728-21grep10:50:232
269264993723,10cyclictest347728-21grep10:50:232
269264993721,13cyclictest34-21ksoftirqd/208:23:002
269264993721,12cyclictest1-21systemd12:20:002
269264993719,14cyclictest34-21ksoftirqd/212:25:172
269264993719,14cyclictest34-21ksoftirqd/212:25:172
26926099377,15cyclictest4497-21PK-Backend10:10:281
26926099374,13cyclictest26-21ksoftirqd/110:47:321
26926099371,21cyclictest341260-21runrttasks10:34:451
26926099371,21cyclictest1051-21thermald12:36:101
26926099371,15cyclictest311760-21seq09:08:461
26926099371,15cyclictest26-21ksoftirqd/111:45:001
269255993721,11cyclictest438-20systemd-journal11:54:550
269255993720,13cyclictest361763-21rt-features11:30:270
197399370,31rtkit-daemon0-21swapper/011:25:460
197399370,30rtkit-daemon0-21swapper/009:29:260
197399370,30rtkit-daemon0-21swapper/007:51:130
269269993626,6cyclictest42-21ksoftirqd/310:30:283
269269993626,6cyclictest373625-21memory12:05:213
269269993626,6cyclictest371031-21cron12:00:003
269269993622,11cyclictest197663-21kworker/3:107:12:583
269269993622,11cyclictest197663-21kworker/3:107:12:583
269269993619,12cyclictest372109-21which12:00:253
269269993619,12cyclictest372109-21which12:00:253
26926999361,8cyclictest141rcu_preempt10:55:263
269264993618,14cyclictest34-21ksoftirqd/210:17:342
26926499361,31cyclictest308456-21kworker/2:109:05:592
26926499361,13cyclictest348760-21sshd10:55:092
26926099366,9cyclictest4497-21PK-Backend10:25:231
26926099366,9cyclictest4497-21PK-Backend10:25:231
26926099366,16cyclictest26-21ksoftirqd/112:26:461
26926099366,16cyclictest26-21ksoftirqd/112:26:461
26926099366,16cyclictest26-21ksoftirqd/112:24:091
26926099366,16cyclictest26-21ksoftirqd/111:50:441
269255993630,4cyclictest0-21swapper/008:39:590
269255993623,9cyclictest341306-21apt-get10:35:080
269255993619,13cyclictest372781-21/usr/sbin/munin12:05:010
269255993619,13cyclictest372781-21/usr/sbin/munin12:05:000
269255993619,13cyclictest349603-21cut10:55:280
26925599361,11cyclictest379702-21turbostat12:25:000
197399360,30rtkit-daemon0-21swapper/011:05:060
197399360,27rtkit-daemon1972-21rtkit-daemon12:34:200
197399360,27rtkit-daemon1972-21rtkit-daemon10:45:490
269269993521,10cyclictest336092-21cron10:20:013
269269993519,12cyclictest331961-21/usr/sbin/munin10:05:283
269269993518,13cyclictest354222-21expr11:10:143
269269993518,12cyclictest438-20systemd-journal07:29:083
26926999351,14cyclictest324632-21df_inode09:45:173
269264993525,6cyclictest34-21ksoftirqd/210:00:272
269264993525,6cyclictest34-21ksoftirqd/210:00:272
26926499352,13cyclictest141rcu_preempt12:20:212
269264993520,11cyclictest34-21ksoftirqd/211:08:092
269264993519,13cyclictest34-21ksoftirqd/211:55:012
26925599357,11cyclictest1137-21runrttasks10:50:240
26925599357,11cyclictest1137-21runrttasks10:50:240
269255993525,6cyclictest4497-21PK-Backend07:17:590
269255993523,9cyclictest364552-21tr11:40:160
269255993520,10cyclictest352144-21cat11:05:010
269255993518,13cyclictest340050-21wc10:30:160
269255993514,6cyclictest141rcu_preempt10:15:010
26925599351,14cyclictest3443992grep10:40:290
269269993420,10cyclictest331005-21cut10:05:013
269269993420,10cyclictest331005-21cut10:05:013
269269993419,11cyclictest363154-21cut11:35:223
269269993417,13cyclictest383541-21cat12:35:153
26926999341,11cyclictest339573-21awk10:30:003
26926999341,11cyclictest339573-21awk10:30:003
269269993410,13cyclictest289723-21/usr/sbin/munin08:05:243
26926499348,6cyclictest141rcu_preempt11:20:152
269264993424,6cyclictest323951-21NetworkManager09:44:442
269264993423,7cyclictest364087-21cron11:39:592
269264993423,6cyclictest34-21ksoftirqd/212:10:302
269264993423,6cyclictest34-21ksoftirqd/210:25:022
269264993419,11cyclictest354071-21sort11:10:132
26926499341,17cyclictest350502-21sort11:00:132
269260993417,13cyclictest372614-21sshd12:03:091
269260993417,13cyclictest372614-21sshd12:03:081
269255993424,7cyclictest0-21swapper/009:39:580
269255993424,7cyclictest0-21swapper/009:39:580
269255993418,7cyclictest1137-21runrttasks10:05:030
26925599341,29cyclictest4497-21PK-Backend08:56:590
26925599341,29cyclictest4497-21PK-Backend08:00:580
26925599341,10cyclictest4497-21PK-Backend10:18:250
197399340,25rtkit-daemon1972-21rtkit-daemon12:35:420
26926999332,27cyclictest356207-21/usr/sbin/munin11:15:223
269269993318,11cyclictest357422-21/usr/sbin/munin11:20:323
269264993323,6cyclictest34-21ksoftirqd/211:48:332
269264993323,6cyclictest34-21ksoftirqd/210:25:222
269264993323,6cyclictest34-21ksoftirqd/210:25:222
269255993315,13cyclictest289978-21cat08:05:300
26925599331,4cyclictest4497-21PK-Backend11:18:180
26925599331,29cyclictest4497-21PK-Backend11:38:560
26925599331,29cyclictest0-21swapper/012:15:120
26926999321,5cyclictest0-21swapper/312:10:093
269255993223,6cyclictest336515-21cpuspeed_turbos10:20:150
26925599321,4cyclictest4497-21PK-Backend11:12:220
26925599321,4cyclictest4497-21PK-Backend08:48:290
26925599321,4cyclictest4497-21PK-Backend08:48:290
26925599321,10cyclictest4497-21PK-Backend12:08:000
26926999311,7cyclictest351608-21chrt11:00:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional