You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 08:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Apr 13, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
86822238161,25sleep00-21swapper/019:07:590
88042230163,55sleep30-21swapper/319:09:363
86842202166,24sleep20-21swapper/219:08:022
85112201165,24sleep10-21swapper/119:05:471
1184421780,5sleep0892099cyclictest21:40:170
1085321370,5sleep0892099cyclictest00:05:170
1085321370,5sleep0892099cyclictest00:05:170
291512870,3sleep129152-21pmu-power21:05:231
8930997066,3cyclictest0-21swapper/222:41:222
318152520,1sleep30-21swapper/322:25:193
84962510,3sleep121-21rcuc/100:00:161
83872500,1sleep20-21swapper/220:20:132
201952500,1sleep00-21swapper/020:45:190
218672470,4sleep011-21rcuc/022:05:010
278492420,1sleep30-21swapper/319:50:163
139762410,4sleep013979-21timerandwakeup20:30:260
893799361,25cyclictest0-21swapper/323:30:233
893799350,16cyclictest0-21swapper/300:30:213
179022350,4sleep0892099cyclictest19:25:250
179022350,4sleep0892099cyclictest19:25:250
182842330,2sleep10-21swapper/123:10:121
893799321,19cyclictest0-21swapper/322:19:133
893799321,13cyclictest0-21swapper/323:08:093
143082320,4sleep230-21rcuc/221:45:212
143082320,4sleep230-21rcuc/221:45:212
892099317,11cyclictest31238-21perf23:40:000
893799300,12cyclictest0-21swapper/300:20:493
892099305,12cyclictest27811-21basename21:05:010
8920993014,14cyclictest7-21ksoftirqd/000:15:010
892099301,14cyclictest32205-21awk21:15:010
198672300,4sleep011-21rcuc/000:25:200
893799289,3cyclictest0-21swapper/322:10:213
893799281,14cyclictest827-21runrttasks20:42:213
893099280,12cyclictest0-21swapper/221:10:112
893099280,11cyclictest0-21swapper/220:34:472
893099280,10cyclictest0-21swapper/220:35:392
893099280,10cyclictest0-21swapper/220:35:392
8937992712,4cyclictest6967-21cat21:30:133
893799271,12cyclictest0-21swapper/321:40:573
893799270,5cyclictest0-21swapper/319:45:173
8930992721,3cyclictest7870-21grep22:45:162
893099270,3cyclictest0-21swapper/223:35:432
892099272,13cyclictest4848-21munin-run22:40:010
893799269,3cyclictest0-21swapper/322:35:013
893799269,3cyclictest0-21swapper/321:26:573
893799269,10cyclictest0-21swapper/322:06:293
893799261,14cyclictest0-21swapper/323:22:213
893799261,11cyclictest0-21swapper/300:08:373
893799261,11cyclictest0-21swapper/300:08:373
893799261,10cyclictest0-21swapper/319:27:213
893799261,10cyclictest0-21swapper/319:27:213
893799260,15cyclictest0-21swapper/323:56:053
893799260,12cyclictest0-21swapper/323:41:573
893799260,12cyclictest0-21swapper/300:19:173
893099260,9cyclictest0-21swapper/219:50:352
893099260,10cyclictest0-21swapper/223:15:472
175352261,3sleep117661-21users23:05:271
893799258,3cyclictest0-21swapper/319:24:093
893799258,3cyclictest0-21swapper/300:04:493
893799257,3cyclictest0-21swapper/323:01:493
893799257,3cyclictest0-21swapper/322:02:533
893799255,3cyclictest0-21swapper/322:22:173
893799251,9cyclictest0-21swapper/320:34:513
893799251,6cyclictest0-21swapper/321:49:093
893799251,6cyclictest0-21swapper/321:49:093
893799251,19cyclictest0-21swapper/300:27:243
893799251,10cyclictest0-21swapper/321:16:053
893099251,6cyclictest0-21swapper/222:10:172
893099250,11cyclictest0-21swapper/219:41:192
892599251,7cyclictest0-21swapper/120:22:501
892599251,6cyclictest0-21swapper/122:30:551
8925992513,3cyclictest0-21swapper/100:05:071
8925992513,3cyclictest0-21swapper/100:05:061
8925992511,12cyclictest17644-21kworker/u8:119:22:501
892599250,7cyclictest0-21swapper/123:57:021
892099251,12cyclictest20252-21awk23:15:010
892099251,12cyclictest10822-21munin-run21:40:010
276972250,4sleep230-21rcuc/222:15:232
893799248,3cyclictest14827-21/usr/sbin/munin20:35:253
893799248,3cyclictest14827-21/usr/sbin/munin20:35:253
893799246,3cyclictest0-21swapper/321:00:533
893799241,9cyclictest0-21swapper/322:54:533
893799241,9cyclictest0-21swapper/321:57:173
893799241,9cyclictest0-21swapper/300:13:093
893799241,14cyclictest25972-21cut23:25:213
893799241,10cyclictest0-21swapper/319:12:213
893799240,12cyclictest0-21swapper/319:16:253
893099241,5cyclictest0-21swapper/222:25:172
893099240,17cyclictest0-21swapper/223:04:192
893099240,10cyclictest0-21swapper/219:10:152
892599245,3cyclictest14188-21expr23:00:131
892599242,19cyclictest17317-21memory19:25:181
892599242,19cyclictest17317-21memory19:25:181
892599241,21cyclictest0-21swapper/121:44:141
892599241,20cyclictest21830-21perf22:05:011
892599240,7cyclictest0-21swapper/122:22:201
893799231,9cyclictest0-21swapper/323:19:173
8937992310,4cyclictest0-21swapper/323:35:063
893799230,7cyclictest0-21swapper/322:42:243
893099231,8cyclictest0-21swapper/221:29:172
893099231,6cyclictest0-21swapper/223:24:142
892599238,4cyclictest3697-21expr23:50:151
8925992317,3cyclictest5434-21forks22:40:151
892599231,5cyclictest0-21swapper/122:50:081
892599231,3cyclictest22184-21/usr/sbin/munin00:30:231
892599231,20cyclictest0-21swapper/122:48:091
892599231,19cyclictest14703-21latency_hist20:35:011
892599230,6cyclictest0-21swapper/123:29:251
892599230,6cyclictest0-21swapper/122:57:271
892599230,6cyclictest0-21swapper/122:57:271
892599230,6cyclictest0-21swapper/121:12:221
892099237,14cyclictest0-21swapper/022:45:010
892099230,10cyclictest0-21swapper/019:15:110
893799228,3cyclictest0-21swapper/320:00:573
893799227,8cyclictest2398-21chrt22:31:573
893799225,3cyclictest0-21swapper/322:45:493
893799224,10cyclictest2729-21fschecks_time21:20:163
893799223,3cyclictest0-21swapper/321:12:213
893799220,4cyclictest0-21swapper/300:37:513
893099221,9cyclictest29989-21diskmemload00:06:132
893099221,9cyclictest29989-21diskmemload00:06:132
893099221,6cyclictest0-21swapper/223:47:372
893099221,6cyclictest0-21swapper/223:08:322
893099221,6cyclictest0-21swapper/222:35:262
893099221,4cyclictest0-21swapper/221:21:012
8930992210,3cyclictest13642-21awk23:00:012
8930992210,3cyclictest13642-21awk23:00:012
893099220,10cyclictest0-21swapper/222:02:032
892599221,8cyclictest0-21swapper/122:17:481
8925992216,3cyclictest1424-21timerandwakeup21:15:261
892599221,11cyclictest20651-21cstates23:15:161
892599220,7cyclictest0-21swapper/122:35:121
892599220,6cyclictest0-21swapper/122:07:321
892599220,5cyclictest0-21swapper/100:18:531
892099227,3cyclictest31830-21needreboot22:25:190
892099222,11cyclictest14206-21/usr/sbin/munin19:20:210
892099221,6cyclictest0-21swapper/022:32:430
8920992215,2cyclictest0-21swapper/019:50:270
892099221,4cyclictest0-21swapper/021:56:550
892099221,11cyclictest22785-21awk19:40:010
892099221,10cyclictest0-21swapper/022:11:200
23342221,17sleep21633-21/usr/sbin/munin20:05:172
157422220,5sleep2893099cyclictest00:15:222
92342211,2sleep00-21swapper/022:50:010
893799218,3cyclictest0-21swapper/321:38:013
893799218,3cyclictest0-21swapper/320:17:093
893799217,3cyclictest0-21swapper/320:27:213
893799216,3cyclictest0-21swapper/323:45:413
893799211,9cyclictest0-21swapper/322:56:093
893799211,9cyclictest0-21swapper/322:56:093
893799211,7cyclictest0-21swapper/323:50:233
893799211,11cyclictest29590-21/usr/sbin/munin19:55:253
893799210,8cyclictest0-21swapper/319:39:453
893799210,4cyclictest0-21swapper/323:12:473
893099211,8cyclictest0-21swapper/221:43:562
893099211,6cyclictest0-21swapper/222:20:132
893099211,6cyclictest0-21swapper/200:14:052
893099211,16cyclictest17969-21grep20:40:182
893099210,9cyclictest0-21swapper/223:29:532
892599218,3cyclictest0-21swapper/100:21:191
892599216,4cyclictest12653-21sh00:10:131
892599214,5cyclictest0-21swapper/121:47:081
892599214,5cyclictest0-21swapper/121:47:071
892599214,3cyclictest31172-21df22:25:131
892599213,2cyclictest87550irq/128-enp1s0-19:51:271
892599212,7cyclictest15444-21/usr/sbin/munin21:50:181
892599211,7cyclictest0-21swapper/123:41:101
892599211,7cyclictest0-21swapper/123:22:461
892599211,3cyclictest2393-21awk23:45:221
892599211,10cyclictest0-21swapper/123:36:131
892599211,10cyclictest0-21swapper/120:15:331
892599210,6cyclictest0-21swapper/121:59:071
892599210,5cyclictest0-21swapper/121:39:171
892599210,5cyclictest0-21swapper/121:33:011
892599210,4cyclictest0-21swapper/121:25:341
892599210,4cyclictest0-21swapper/100:37:001
892099217,2cyclictest0-21swapper/022:50:240
8920992114,2cyclictest0-21swapper/019:10:140
892099211,2cyclictest89250irq/135-enp2s0-23:43:370
892099211,12cyclictest8228-21/usr/sbin/munin20:20:160
55092211,2sleep00-21swapper/021:25:210
893799208,4cyclictest24768-21seq20:55:223
893799208,3cyclictest0-21swapper/319:44:373
893799205,3cyclictest0-21swapper/321:50:373
8937992015,3cyclictest28463-21tune2fs21:05:153
893099203,3cyclictest0-21swapper/223:10:212
893099201,5cyclictest0-21swapper/221:55:242
893099201,5cyclictest0-21swapper/200:38:322
8930992014,3cyclictest1077-21grep22:30:142
893099201,3cyclictest0-21swapper/221:50:332
893099201,3cyclictest0-21swapper/200:32:582
893099201,10cyclictest0-21swapper/219:37:352
893099200,10cyclictest0-21swapper/219:57:192
892599203,2cyclictest87550irq/128-enp1s0-22:12:131
892599202,4cyclictest5811-21munin-run20:15:011
892599202,16cyclictest0-21swapper/120:45:061
8925992017,2cyclictest0-21swapper/120:40:221
892599201,5cyclictest808-21snmpd19:15:331
8925992013,4cyclictest0-21swapper/120:50:191
892599201,11cyclictest0-21swapper/121:05:021
892599200,4cyclictest0-21swapper/100:26:501
892099203,2cyclictest0-21swapper/020:50:210
892099201,6cyclictest0-21swapper/022:21:540
892099201,4cyclictest0-21swapper/000:31:590
892099201,11cyclictest15854-21sshd20:35:200
892099201,11cyclictest15854-21sshd20:35:200
892099201,10cyclictest0-21swapper/023:49:510
892099200,6cyclictest0-21swapper/019:55:210
893799193,3cyclictest0-21swapper/320:46:533
893099191,8cyclictest0-21swapper/200:26:022
893099191,7cyclictest0-21swapper/222:50:322
893099191,6cyclictest0-21swapper/223:30:482
893099191,6cyclictest0-21swapper/221:36:482
893099191,5cyclictest0-21swapper/221:31:482
893099191,3cyclictest0-21swapper/221:15:212
892599195,3cyclictest87650irq/129-enp1s0-20:27:421
892599192,3cyclictest0-21swapper/119:11:381
892599192,16cyclictest0-21swapper/119:38:171
892599191,7cyclictest0-21swapper/120:37:501
892599191,7cyclictest0-21swapper/120:37:501
892599191,5cyclictest28316-21/usr/sbin/munin23:30:241
892599191,5cyclictest22997-21/usr/sbin/munin19:40:251
892599191,4cyclictest0-21swapper/120:00:171
8925991913,3cyclictest26273-21ntpq19:45:231
8925991912,4cyclictest24863-21mailstats20:55:231
892599191,16cyclictest0-21swapper/119:30:061
892599190,18cyclictest0-21swapper/119:55:111
892099196,3cyclictest87550irq/128-enp1s0-20:55:160
892099194,2cyclictest0-21swapper/020:05:180
892099193,2cyclictest87550irq/128-enp1s0-00:35:090
892099191,6cyclictest27851-21cat23:30:160
892099191,6cyclictest17018-21/usr/sbin/munin20:40:160
892099191,6cyclictest0-21swapper/023:53:540
892099191,6cyclictest0-21swapper/000:19:300
892099191,5cyclictest15702-21sshd23:03:200
892099191,4cyclictest0-21swapper/023:05:130
892099191,10cyclictest31791-21/usr/sbin/munin20:00:220
892099191,10cyclictest17114-21sshd21:51:200
892099191,10cyclictest13849-21head21:45:160
892099191,10cyclictest13849-21head21:45:160
893799181,7cyclictest8777-21fschecks_time20:20:163
893799180,8cyclictest0-21swapper/320:55:003
893799180,10cyclictest0-21swapper/320:05:413
893099189,2cyclictest87550irq/128-enp1s0-00:20:212
893099185,3cyclictest0-21swapper/219:19:212
893099183,5cyclictest0-21swapper/223:55:212
893099181,7cyclictest27286-21cat19:50:022
893099181,6cyclictest0-21swapper/222:07:542
893099181,5cyclictest4878-21sed23:50:272
8930991813,3cyclictest11028-21tail20:25:162
893099181,10cyclictest0-21swapper/200:01:182
893099180,11cyclictest0-21swapper/220:57:392
892599181,3cyclictest0-21swapper/120:07:021
892599181,16cyclictest0-21swapper/121:20:501
892099186,3cyclictest0-21swapper/019:30:440
892099183,2cyclictest87350irq/126-enp1s0-23:21:210
892099181,6cyclictest0-21swapper/000:01:500
892099181,5cyclictest17311-21grep00:20:170
892099181,5cyclictest0-21swapper/022:15:370
892099181,5cyclictest0-21swapper/021:24:590
8920991815,2cyclictest0-21swapper/022:05:190
8920991810,2cyclictest6651-21/usr/sbin/munin21:30:170
8920991810,2cyclictest5688-21/usr/sbin/munin23:55:150
250622180,4sleep0892099cyclictest23:25:110
893099171,7cyclictest27184-21timerandwakeup21:00:252
8930991712,3cyclictest32259-21irqstats23:40:182
893099171,15cyclictest0-21swapper/220:04:582
893099171,14cyclictest19271-21grep19:30:152
893099171,14cyclictest0-21swapper/220:54:042
892099174,3cyclictest87550irq/128-enp1s0-19:40:250
892099174,3cyclictest0-21swapper/021:06:570
892099171,6cyclictest0-21swapper/023:17:300
892099171,6cyclictest0-21swapper/022:57:110
892099171,6cyclictest0-21swapper/022:57:110
892099171,13cyclictest32418-21/usr/sbin/munin21:15:210
38232171,2sleep30-21swapper/320:10:043
180212171,2sleep20-21swapper/219:25:362
180212171,2sleep20-21swapper/219:25:362
893799161,6cyclictest0-21swapper/319:30:233
893099164,4cyclictest28086-21cpuspeed_turbos21:05:132
893099164,4cyclictest15269-21cat19:20:202
893099161,7cyclictest0-21swapper/220:15:222
8930991611,3cyclictest20462-21sendmail_mailst20:45:232
892099169,4cyclictest4882-21ntpq20:10:200
892099161,4cyclictest25198-21/usr/sbin/munin19:45:150
892099161,2cyclictest0-21swapper/020:27:020
56732160,2sleep20-21swapper/220:13:282
8920991510,3cyclictest6040-21acpi20:15:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional