You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-01 - 05:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Aug 01, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
314782240163,24sleep30-21swapper/319:06:533
316572237164,21sleep10-21swapper/119:09:121
313982214168,35sleep20-21swapper/219:05:522
314162203165,26sleep00-21swapper/019:06:060
31820991101,107cyclictest0-21swapper/222:05:532
31820991080,106cyclictest0-21swapper/223:29:052
61312600,4sleep230-21rcuc/221:50:142
222332570,1sleep20-21swapper/220:00:012
31820995450,3cyclictest0-21swapper/220:30:102
176672530,4sleep230-21rcuc/222:15:182
175422530,4sleep13181699cyclictest22:15:171
81652520,3sleep339-21rcuc/320:40:153
236052520,4sleep13181699cyclictest21:15:181
3182699431,30cyclictest27789-21tune2fs21:25:143
31809994211,14cyclictest24903-21munin-run21:20:010
3182099401,29cyclictest2862-21uptime00:10:212
3182099401,28cyclictest11322-21/usr/sbin/munin19:35:142
3182099371,26cyclictest927-21/usr/sbin/munin20:25:182
3182099371,25cyclictest0-21swapper/223:35:132
3182099371,21cyclictest21219-21sed22:25:012
3182099371,20cyclictest20506-21diskmemload21:16:182
3182699362,31cyclictest927-21/usr/sbin/munin20:25:193
3182699361,24cyclictest6377-21awk20:35:213
3182699360,25cyclictest0-21swapper/321:45:193
3182099361,25cyclictest0-21swapper/222:00:142
3181699361,12cyclictest20506-21diskmemload22:12:461
3180999368,13cyclictest13000-21awk23:20:010
3182699357,20cyclictest0-21swapper/319:20:263
3182699351,23cyclictest87350irq/126-enp1s0-23:07:593
3182099354,26cyclictest15892-40dnf20:57:322
3182099351,25cyclictest3260-21/usr/sbin/munin22:55:202
3182099351,23cyclictest29498-21sed22:40:242
3182099351,21cyclictest24832-21smart_sda22:30:232
3181699351,24cyclictest0-21swapper/119:12:441
3182699344,20cyclictest30796-21/usr/sbin/munin00:00:253
3182699341,29cyclictest22077-21latency_hist20:00:023
3182699341,21cyclictest827-21runrttasks23:53:163
3182099345,20cyclictest9669-21unixbench_multi21:55:272
3182099344,22cyclictest30714-21processes21:30:222
3182099341,22cyclictest20735-21/usr/sbin/munin21:10:122
3182099341,21cyclictest827-21runrttasks21:26:482
3182099341,21cyclictest26401-21/usr/sbin/munin23:50:142
31826993320,8cyclictest0-21swapper/321:57:203
3182699331,28cyclictest14130-21/usr/sbin/munin20:55:203
3182699331,23cyclictest87350irq/126-enp1s0-23:36:243
3182699331,20cyclictest1001-21sshd00:09:443
3182699331,18cyclictest0-21swapper/300:15:143
3182099332,27cyclictest31143-21diskstats00:00:142
3182099332,18cyclictest18540-21/usr/sbin/munin21:05:212
3182099331,23cyclictest26120-21df22:35:122
3182099331,19cyclictest909-21sssd_nss21:40:352
3181699330,25cyclictest0-21swapper/122:52:301
31826993217,12cyclictest4192-21sa123:00:003
3182699321,28cyclictest2539-21/usr/sbin/munin19:15:223
3182699321,28cyclictest20735-21/usr/sbin/munin21:10:243
3182699321,27cyclictest4423-21/usr/sbin/munin23:00:253
3182699321,21cyclictest13216-21/usr/sbin/munin23:20:133
3182699321,21cyclictest13216-21/usr/sbin/munin23:20:133
31826993212,17cyclictest6558-21expr21:50:173
3182699321,16cyclictest87550irq/128-enp1s0-00:30:253
3182699321,16cyclictest87550irq/128-enp1s0-00:30:253
3182099322,21cyclictest808-21snmpd20:35:222
3182099321,22cyclictest0-21swapper/222:15:012
3182099321,21cyclictest7877-21/usr/sbin/munin00:25:182
3182099321,19cyclictest26717-21/usr/sbin/munin20:10:182
3182099321,19cyclictest21744-21grep22:25:142
31820993211,18cyclictest7832-21ls00:25:012
3181699321,12cyclictest32599-21latency21:35:181
3181699320,13cyclictest0-21swapper/121:27:301
3180999324,13cyclictest8914-21idleruntime-cro19:30:000
3182699318,20cyclictest22611-21smart_sda22:25:233
3182699312,26cyclictest18057-21cpuspeed_turbos19:50:183
3182699312,26cyclictest18057-21cpuspeed_turbos19:50:173
3182699312,26cyclictest1717-21cat19:15:013
3182699312,21cyclictest1259-21/usr/sbin/munin00:10:083
3182699311,28cyclictest24542-21sshd21:15:333
3182699311,26cyclictest14759-21/usr/sbin/munin22:10:283
3182699311,26cyclictest13633-21ntp_states22:05:233
3182699311,20cyclictest11933-21/usr/sbin/munin20:50:173
3182699311,19cyclictest1583-21cpu21:40:133
3182699311,15cyclictest0-21swapper/321:30:123
31826993111,17cyclictest32077-21entropy21:35:143
3182699310,29cyclictest0-21swapper/320:15:123
3182099317,20cyclictest23279-21sshd20:00:192
3182099313,24cyclictest11222-21head00:30:222
3182099313,24cyclictest11222-21head00:30:222
3182099312,26cyclictest32222-21cut20:20:182
31820993112,8cyclictest0-21swapper/220:08:302
3182099311,21cyclictest11356-21df23:15:142
31820993110,17cyclictest18769-21latency19:50:232
31820993110,17cyclictest18769-21latency19:50:232
3181699311,16cyclictest0-21swapper/120:13:141
3181699311,12cyclictest0-21swapper/120:26:221
3181699311,11cyclictest0-21swapper/119:49:021
3180999315,12cyclictest16118-21perf21:00:010
3180999313,13cyclictest23739-21kworker/u8:023:58:460
3182699301,26cyclictest15761-21/usr/sbin/munin23:25:133
3182699301,25cyclictest31748-21cut20:20:123
3182699301,16cyclictest0-21swapper/300:25:363
31826993011,16cyclictest26820-21ntpq22:35:193
31826993010,17cyclictest10788-21latency_hist23:15:013
3182699300,19cyclictest0-21swapper/320:08:303
3182099307,12cyclictest87650irq/129-enp1s0-23:10:212
3182099302,22cyclictest1669-21munin-node19:15:022
3182099301,26cyclictest5461-21perf00:20:012
3182099301,18cyclictest25476-21swap23:45:242
3181699301,8cyclictest0-21swapper/119:36:381
3181699301,22cyclictest0-21swapper/123:12:221
3181699301,11cyclictest0-21swapper/121:58:021
3180999302,13cyclictest29301-21munin-run21:30:010
3182699299,16cyclictest5684-21/usr/sbin/munin00:20:163
3182699297,19cyclictest20365-21processes22:20:233
3182699293,23cyclictest15029-21grep19:40:253
3182699292,24cyclictest9735-21/usr/sbin/munin20:45:213
3182699292,23cyclictest32732-21diskstats22:50:143
3182699291,25cyclictest17756-21/usr/sbin/munin22:15:183
3182699291,25cyclictest16480-21cpuspeed_turbos21:00:123
31826992912,14cyclictest26295-21cat23:50:033
3182699290,27cyclictest0-21swapper/319:35:183
3182699290,26cyclictest12518-21/usr/sbin/munin23:15:303
3182099291,25cyclictest7882-21/usr/sbin/munin23:05:232
3182099291,25cyclictest13216-21/usr/sbin/munin23:20:172
3182099291,25cyclictest13216-21/usr/sbin/munin23:20:172
3182099291,15cyclictest0-21swapper/219:20:542
3182099291,13cyclictest0-21swapper/222:45:182
3181699291,12cyclictest0-21swapper/100:13:441
3181699291,11cyclictest21081-21diskstats21:10:151
3181699291,11cyclictest0-21swapper/121:46:541
3181699291,11cyclictest0-21swapper/119:23:031
3181699291,11cyclictest0-21swapper/100:21:501
3181699291,10cyclictest0-21swapper/122:58:551
3181699290,12cyclictest0-21swapper/121:44:541
3182699282,23cyclictest28564-21/usr/sbin/munin22:40:143
3182699281,24cyclictest27097-21ls21:25:013
3182699281,24cyclictest22773-21/usr/sbin/munin23:40:163
3182699281,24cyclictest18873-21/usr/sbin/munin23:30:243
3182699281,23cyclictest27331-21grep20:10:223
3182699281,23cyclictest24396-21sort20:05:013
31826992810,14cyclictest15847-21cpuspeed_turbos19:45:133
3182699280,25cyclictest6932-21/usr/sbin/munin19:25:133
3182699280,25cyclictest29183-21/usr/sbin/munin23:55:173
3182099288,17cyclictest12512-21/usr/sbin/munin20:50:162
3182099282,16cyclictest87350irq/126-enp1s0-23:41:052
3182099281,24cyclictest9472-21/usr/sbin/munin19:30:132
3182099281,24cyclictest29592-21hwlatdetect20:15:152
3182099281,24cyclictest25296-21cstates21:20:122
3182099281,24cyclictest1901-21pmu-power19:15:022
3181699281,7cyclictest0-21swapper/119:43:341
3181699281,25cyclictest0-21swapper/121:06:141
3181699280,12cyclictest0-21swapper/120:08:181
3180999289,16cyclictest26179-21munin-run23:50:010
3182699278,16cyclictest30544-21grep22:45:123
3182699272,21cyclictest3496-21diskstats20:30:143
3182699271,23cyclictest9496-21uniq19:30:133
3182699271,23cyclictest14234-21ls00:40:013
3182699271,23cyclictest10498-21cpuspeed_turbos22:00:133
3182699271,22cyclictest18662-21cpuspeed_turbos21:05:133
3182699270,24cyclictest24039-21/usr/sbin/munin22:30:143
3182099279,15cyclictest17188-21unixbench-2d19:45:272
3182099272,22cyclictest17007-21hddtemp_smartct21:00:162
3182099271,23cyclictest8208-21hddtemp_smartct20:40:162
3182099271,23cyclictest6932-21/usr/sbin/munin19:25:152
3182099271,23cyclictest4593-21cstates23:00:132
3182099271,23cyclictest4577-21/usr/sbin/munin21:45:192
3182099271,23cyclictest17727-21tr23:30:132
3182099271,23cyclictest14169-21grep19:40:162
3182099271,23cyclictest1342-21/usr/sbin/munin22:50:252
3182099271,23cyclictest10074-21/usr/sbin/munin20:45:142
3182099271,22cyclictest29707-21sed23:55:232
3182099271,15cyclictest31718-21/usr/sbin/munin21:35:142
3181699271,9cyclictest0-21swapper/121:03:431
3181699271,9cyclictest0-21swapper/120:00:581
3181699271,8cyclictest0-21swapper/123:55:381
3181699271,8cyclictest0-21swapper/123:49:261
3181699271,15cyclictest0-21swapper/123:51:501
3181699271,11cyclictest0-21swapper/119:50:121
3181699271,11cyclictest0-21swapper/119:50:121
3181699271,10cyclictest0-21swapper/100:18:541
3181699270,9cyclictest0-21swapper/120:56:061
3181699270,9cyclictest0-21swapper/119:30:381
3181699270,25cyclictest0-21swapper/100:08:101
3181699270,12cyclictest0-21swapper/122:31:301
3181699270,10cyclictest0-21swapper/123:30:581
3180999270,12cyclictest5467-21awk00:20:010
3182099269,14cyclictest827-21runrttasks00:05:402
3182099261,22cyclictest12417-21tr00:35:112
3181699261,8cyclictest0-21swapper/122:44:181
3181699261,8cyclictest0-21swapper/120:48:221
3181699261,7cyclictest0-21swapper/122:06:421
3181699261,23cyclictest0-21swapper/121:20:591
3181699261,14cyclictest30689-21ps21:30:221
3181699261,11cyclictest0-21swapper/122:37:121
3181699261,10cyclictest0-21swapper/123:42:101
3181699260,16cyclictest0-21swapper/122:49:261
3181699260,11cyclictest0-21swapper/120:50:221
3181699260,10cyclictest0-21swapper/119:56:541
3180999268,15cyclictest22084-21munin-run20:00:010
3181699251,9cyclictest0-21swapper/123:05:261
3181699251,17cyclictest13302-21ntp_kernel_pll_00:35:201
3181699251,11cyclictest0-21swapper/120:31:321
3181699251,10cyclictest0-21swapper/120:36:141
3181699251,10cyclictest0-21swapper/119:18:301
3181699250,8cyclictest0-21swapper/120:22:381
3181699250,23cyclictest0-21swapper/122:20:021
3181699250,12cyclictest0-21swapper/100:34:121
3181699250,12cyclictest0-21swapper/100:34:121
3181699250,11cyclictest0-21swapper/120:19:021
3181699250,10cyclictest0-21swapper/123:26:381
3181699250,10cyclictest0-21swapper/123:15:501
3181699250,10cyclictest0-21swapper/123:01:511
3180999256,16cyclictest9845-21munin-run00:30:010
3181699241,12cyclictest0-21swapper/122:03:441
3181699241,10cyclictest0-21swapper/123:21:121
3181699241,10cyclictest0-21swapper/123:21:121
3181699241,10cyclictest0-21swapper/120:41:141
3181699240,7cyclictest0-21swapper/119:25:381
3181699240,16cyclictest0-21swapper/100:02:461
3181699240,10cyclictest0-21swapper/123:37:581
3181699240,10cyclictest0-21swapper/122:25:101
31809992411,4cyclictest6405-21ntp_states20:35:220
3181699231,9cyclictest0-21swapper/100:29:041
3181699231,14cyclictest0-21swapper/121:53:261
3180999231,4cyclictest13144-21kworker/u8:222:15:150
3180999228,13cyclictest0-21swapper/000:05:000
31809992211,3cyclictest87550irq/128-enp1s0-23:40:150
3180999221,11cyclictest13367-21tr23:20:120
3180999221,11cyclictest13367-21tr23:20:120
3180999213,7cyclictest18245-1kworker/u9:222:50:350
3180999212,11cyclictest2222-21/usr/sbin/munin22:55:240
3180999211,6cyclictest0-21swapper/022:45:270
3180999211,5cyclictest0-21swapper/021:58:160
3180999211,4cyclictest0-21swapper/022:13:390
3180999208,4cyclictest27472-21ntp_states23:50:210
3180999208,4cyclictest0-21swapper/000:08:590
3180999204,3cyclictest21733-21fschecks_count22:25:150
3180999203,8cyclictest3180150kworker/u8:223:09:400
3180999203,3cyclictest24713-21pmu-power22:30:210
3180999202,8cyclictest18245-1kworker/u9:222:05:080
3180999201,4cyclictest0-21swapper/021:20:320
3180999201,3cyclictest0-21swapper/022:40:180
3180999201,3cyclictest0-21swapper/000:21:210
3180999201,11cyclictest808-21snmpd23:10:530
3180999201,11cyclictest32006-21/usr/sbin/munin19:10:140
3180999201,11cyclictest0-21swapper/022:23:420
3180999201,11cyclictest0-21swapper/021:53:070
3180999201,11cyclictest0-21swapper/000:36:190
3180999201,11cyclictest0-21swapper/000:31:300
3180999201,11cyclictest0-21swapper/000:31:300
3180999201,10cyclictest31801-21kworker/u8:223:00:120
254662200,3sleep011-21rcuc/020:05:210
3180999199,6cyclictest0-21swapper/019:54:560
3180999199,6cyclictest0-21swapper/019:54:560
3180999196,3cyclictest3686-21head19:15:220
3180999195,4cyclictest4364-21grep21:45:160
3180999192,7cyclictest19813-21/usr/sbin/munin23:35:150
3180999191,4cyclictest13297-21munin-run19:40:010
3180999191,16cyclictest30881-21munin-run20:20:020
3180999191,16cyclictest0-21swapper/020:40:270
3180999191,15cyclictest15633-21expr23:25:120
3180999191,11cyclictest0-21swapper/021:41:100
3180999191,10cyclictest20506-21diskmemload22:35:480
3180999191,10cyclictest17616-21/usr/sbin/munin23:30:120
3180999191,10cyclictest0-21swapper/021:36:120
3180999191,10cyclictest0-21swapper/021:30:450
3180999187,3cyclictest26717-21/usr/sbin/munin20:10:300
3180999182,6cyclictest18313-21munin-run21:05:010
3180999181,9cyclictest808-21snmpd00:11:130
3180999181,8cyclictest808-21snmpd20:25:200
3180999181,8cyclictest58150kworker/u8:221:14:590
3180999181,5cyclictest808-21snmpd20:50:250
31809991812,3cyclictest10102-21fschecks_count20:45:150
3180999181,10cyclictest0-21swapper/022:04:460
39552171,2sleep00-21swapper/020:30:180
3180999175,4cyclictest9891-21cat19:30:150
3180999171,6cyclictest20515-21perf21:10:010
3180999171,6cyclictest0-21swapper/019:20:210
3180999170,10cyclictest0-21swapper/019:40:220
3180999161,5cyclictest16710-21head19:45:200
31809991612,3cyclictest0-21swapper/020:04:580
31809991510,3cyclictest31407-21expr20:20:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional