You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 19:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Aug 23, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
132152239163,25sleep00-21swapper/007:08:060
131042238163,24sleep10-21swapper/107:06:381
131062205169,23sleep30-21swapper/307:06:403
132802204131,24sleep20-21swapper/207:08:562
2393921620,4sleep21346299cyclictest12:25:152
13462999693,2cyclictest0-21swapper/208:47:502
13462999086,3cyclictest0-21swapper/208:36:012
13462997369,3cyclictest0-21swapper/210:20:582
13462996056,3cyclictest0-21swapper/209:58:412
318552570,5sleep01345099cyclictest10:15:190
309132540,3sleep339-21rcuc/307:45:243
13462995450,3cyclictest0-21swapper/209:24:452
13462995450,3cyclictest0-21swapper/209:24:442
280832530,1sleep30-21swapper/312:35:113
131022530,1sleep10-21swapper/108:20:141
83672520,1sleep10-21swapper/109:20:251
83672520,1sleep10-21swapper/109:20:251
216802490,5sleep123-21ksoftirqd/108:40:131
120592470,2sleep20-21swapper/209:30:182
194302450,1sleep30-21swapper/311:00:203
1345099338,12cyclictest13770-21munin-run10:50:010
13462993118,3cyclictest0-21swapper/211:35:202
13462993114,10cyclictest20347-21fschecks_count09:50:142
1345699302,18cyclictest20304-21df09:50:121
1345699301,22cyclictest0-21swapper/108:54:171
1346299291,12cyclictest0-21swapper/212:15:392
58672270,5sleep15873-21gltestperf10:30:161
58672270,5sleep15873-21gltestperf10:30:161
1345099279,15cyclictest29779-21basename12:40:010
1345099271,13cyclictest13149-21munin-run09:35:010
233982261,20sleep323433-21/usr/sbin/munin09:55:253
1346299260,13cyclictest0-21swapper/207:26:172
1345699261,16cyclictest13221-21irqstats12:00:191
1345099263,17cyclictest20553-21grep09:50:150
1346299251,10cyclictest0-21swapper/211:02:152
1345099250,12cyclictest19733-21crond09:50:010
284702240,3sleep10-21swapper/108:55:131
1346299248,4cyclictest9569-21grep09:25:152
1346299248,3cyclictest0-21swapper/208:04:592
13462992413,3cyclictest0-21swapper/211:55:512
1346299241,11cyclictest0-21swapper/211:27:402
13462992410,3cyclictest0-21swapper/207:22:342
1345699244,3cyclictest773-21NetworkManager12:08:311
1345699241,17cyclictest0-21swapper/109:56:361
318122230,4sleep341-21ksoftirqd/309:00:253
1346799231,16cyclictest0-21swapper/312:22:203
1346799231,16cyclictest0-21swapper/307:14:033
1346299236,3cyclictest0-21swapper/212:12:472
13462992313,3cyclictest0-21swapper/212:22:232
1346299231,10cyclictest0-21swapper/207:30:252
1346299230,16cyclictest0-21swapper/211:15:282
1345699231,12cyclictest27326-21tr11:20:121
1345099231,3cyclictest0-21swapper/011:58:420
1345099230,11cyclictest7184-21perf10:35:020
1345099230,11cyclictest7184-21perf10:35:020
250572220,7sleep081ktimersoftd/008:45:250
1346799225,5cyclictest0-21swapper/312:25:143
1346799223,10cyclictest18745-21/usr/sbin/munin07:20:183
1346799221,11cyclictest0-21swapper/312:11:423
1346799221,10cyclictest0-21swapper/311:53:353
1346299226,3cyclictest0-21swapper/212:09:152
1346299221,9cyclictest0-21swapper/210:54:352
1345699225,9cyclictest0-21swapper/112:30:281
1345699221,18cyclictest7737-21df_inode10:35:141
1345699221,18cyclictest29899-21pmu-power10:10:211
1345699221,15cyclictest0-21swapper/107:35:451
309232210,4sleep011-21rcuc/011:25:250
191172211,3sleep319124-21latency_hist08:35:023
1346299219,3cyclictest0-21swapper/209:15:182
1346299218,3cyclictest0-21swapper/208:05:152
1346299216,3cyclictest0-21swapper/209:03:402
1346299215,4cyclictest1926-21sshd09:07:132
1346299211,9cyclictest0-21swapper/208:25:152
1346299211,4cyclictest0-21swapper/209:49:162
1346299211,11cyclictest0-21swapper/207:55:152
1345699212,11cyclictest17497-21hddtemp_smartct12:10:161
1345699211,8cyclictest0-21swapper/110:55:301
1345699211,7cyclictest18689-21memory09:45:201
1345699211,3cyclictest5218-21sensors08:00:241
1345699211,3cyclictest3205-21irqstats09:10:191
13456992113,3cyclictest0-21swapper/109:08:371
13456992113,3cyclictest0-21swapper/108:49:141
13456992113,3cyclictest0-21swapper/108:37:371
13456992113,3cyclictest0-21swapper/107:46:541
13456992113,3cyclictest0-21swapper/107:10:061
1345699211,2cyclictest0-21swapper/110:43:071
1345699211,18cyclictest0-21swapper/111:38:411
1345699211,17cyclictest29060-21sed07:40:241
1345699211,17cyclictest2157-21diskmemload10:51:461
1345699211,15cyclictest0-21swapper/110:03:041
1345699211,15cyclictest0-21swapper/108:32:281
1345699211,15cyclictest0-21swapper/108:06:021
1345699211,15cyclictest0-21swapper/107:28:591
1345699211,14cyclictest0-21swapper/111:07:531
1345699211,12cyclictest12172-21latency_hist12:00:011
1345699211,11cyclictest8344-21df11:50:131
1345699211,11cyclictest827-21runrttasks07:15:141
1345699211,11cyclictest11544-21grep09:30:141
1345699210,4cyclictest6341-21kworker/u8:209:41:221
1345099216,6cyclictest0-21swapper/011:17:350
1345099216,6cyclictest0-21swapper/010:53:240
1345099213,8cyclictest13437-21cyclictest09:23:270
1345099213,8cyclictest13437-21cyclictest09:23:270
148702200,12sleep341-21ksoftirqd/308:25:013
1346799201,9cyclictest495-21jbd2/dm-0-807:51:173
1346799201,11cyclictest6167-21/usr/sbin/munin08:05:273
1346799201,11cyclictest18357-21sshd07:18:113
1346799201,10cyclictest0-21swapper/311:24:363
1346799201,10cyclictest0-21swapper/308:02:173
1346799201,10cyclictest0-21swapper/307:32:123
1346799200,14cyclictest0-21swapper/311:08:353
1346299202,7cyclictest481-21munin-run10:20:012
1346299201,6cyclictest0-21swapper/211:48:542
1346299201,5cyclictest25040-21hddtemp_smartct10:00:152
1346299201,15cyclictest0-21swapper/210:40:212
1346299201,14cyclictest5997-21kworker/2:211:54:372
1345699209,3cyclictest32274-21diskstats07:50:141
1345699207,10cyclictest0-21swapper/111:00:191
1345699205,4cyclictest0-21swapper/111:16:271
13456992016,2cyclictest0-21swapper/110:25:131
13456992012,3cyclictest0-21swapper/108:11:591
13456992012,3cyclictest0-21swapper/108:11:591
13456992012,3cyclictest0-21swapper/107:30:211
1345699201,18cyclictest0-21swapper/111:28:151
1345699201,17cyclictest0-21swapper/111:44:211
1345699201,15cyclictest0-21swapper/109:39:321
1345699201,15cyclictest0-21swapper/108:17:011
1345699201,14cyclictest0-21swapper/110:17:451
1345699201,14cyclictest0-21swapper/109:29:171
1345699201,14cyclictest0-21swapper/109:17:031
1345699201,14cyclictest0-21swapper/107:20:451
1345699201,11cyclictest28350-21perl12:35:141
1345699201,11cyclictest28169-21vmstat10:05:271
1345699201,10cyclictest13774-21perf10:50:021
1345699200,14cyclictest0-21swapper/108:27:561
1345099209,3cyclictest5813-21/usr/sbin/munin11:45:190
1345099208,3cyclictest19354-21/usr/sbin/munin08:35:160
1345099203,2cyclictest0-21swapper/009:27:400
1345099202,10cyclictest20598-21/usr/sbin/munin11:05:150
1345099201,6cyclictest0-21swapper/012:19:230
1345099201,17cyclictest9993-21awk11:55:010
1345099201,11cyclictest3620-21/usr/sbin/munin11:40:240
1345099201,11cyclictest30343-21/usr/sbin/munin09:00:230
1345099201,11cyclictest2157-21diskmemload12:10:200
1345099201,11cyclictest1459-21hddtemp_smartct10:20:170
58532191,3sleep00-21swapper/008:04:090
202972190,3sleep00-21swapper/011:04:160
1346799198,3cyclictest6779-21/usr/sbin/munin09:20:213
1346799198,3cyclictest6779-21/usr/sbin/munin09:20:213
1346799197,3cyclictest9360-21latency_hist10:40:013
1346799197,3cyclictest2489-21ntp_states11:35:213
1346799193,7cyclictest25946-21/usr/sbin/munin08:50:163
1346799191,9cyclictest0-21swapper/310:53:353
1346799191,7cyclictest0-21swapper/309:50:163
13467991913,3cyclictest31515-21cut11:30:013
1346799191,12cyclictest0-21swapper/311:44:533
1346799191,12cyclictest0-21swapper/310:13:173
1346799191,11cyclictest21551-21/usr/sbin/munin08:40:263
1346799191,11cyclictest0-21swapper/312:15:223
1346799191,10cyclictest25603-21/usr/sbin/munin12:30:173
1346799191,10cyclictest25329-21/usr/sbin/munin07:35:143
1346799191,10cyclictest0-21swapper/312:01:183
1346299196,2cyclictest32654-21open_files11:30:222
1346299194,9cyclictest0-21swapper/207:11:392
1346299191,7cyclictest5578-21sh11:45:022
1346299191,7cyclictest0-21swapper/210:11:162
1346299191,6cyclictest0-21swapper/209:36:122
1346299191,5cyclictest0-21swapper/212:37:312
1346299191,5cyclictest0-21swapper/210:05:362
1346299191,3cyclictest0-21swapper/209:44:362
13462991912,4cyclictest21661-21ntp_states11:05:222
13462991911,5cyclictest0-21swapper/210:30:152
13462991911,5cyclictest0-21swapper/210:30:152
1346299191,14cyclictest0-21swapper/208:13:132
1346299191,14cyclictest0-21swapper/208:13:132
1346299191,11cyclictest0-21swapper/211:11:392
1345699198,8cyclictest31674-21cpuspeed_turbos11:30:121
1345699194,9cyclictest0-21swapper/111:14:081
1345699194,13cyclictest22489-21switchtime12:20:241
1345699191,17cyclictest0-21swapper/112:27:081
13456991911,3cyclictest0-21swapper/109:03:471
1345699191,10cyclictest7775-21latency_hist11:50:011
1345699191,10cyclictest2111-21diskstats07:55:151
1345699191,10cyclictest19020-21basename12:15:121
1345099198,3cyclictest27185-21/usr/sbin/munin11:20:260
1345099198,3cyclictest24364-21/usr/sbin/munin10:00:240
1345099196,4cyclictest0-21swapper/009:43:220
1345099195,3cyclictest0-21swapper/012:09:260
1345099194,6cyclictest0-21swapper/011:14:560
1345099194,6cyclictest0-21swapper/009:56:220
1345099191,11cyclictest20936-21/usr/sbin/munin07:25:140
1345099191,11cyclictest0-21swapper/010:58:390
1345099191,11cyclictest0-21swapper/010:10:280
1345099191,10cyclictest23137-21/usr/sbin/munin07:30:270
1345099191,10cyclictest13651-21/usr/sbin/munin07:10:140
1345099191,10cyclictest0-21swapper/010:35:040
13450991910,2cyclictest30123-21awk09:00:000
1345099190,6cyclictest0-21swapper/010:44:450
1345099190,6cyclictest0-21swapper/009:12:290
1345099190,11cyclictest0-21swapper/009:18:530
1346799181,9cyclictest8357-21/usr/sbin/munin08:10:133
1346799181,9cyclictest8357-21/usr/sbin/munin08:10:133
1346799181,7cyclictest0-21swapper/309:45:163
1346799181,6cyclictest1164-21sh11:35:013
1346799181,6cyclictest0-21swapper/310:40:123
1346799181,5cyclictest27639-21ntp_states10:05:213
13467991812,4cyclictest0-21swapper/311:49:183
13467991811,4cyclictest28760-21fgrep07:40:213
1346799181,13cyclictest0-21swapper/308:25:213
1346799181,10cyclictest13817-21kworker/3:208:48:173
1346299189,7cyclictest16645-21cpuspeed_turbos07:15:132
1346299189,2cyclictest87450irq/127-enp1s0-07:44:272
1346299185,4cyclictest2912-21tune2fs09:10:172
1346299183,8cyclictest0-21swapper/210:39:392
1346299181,7cyclictest27179-21/usr/sbin/munin08:50:232
1346299181,6cyclictest0-21swapper/210:56:122
13462991814,2cyclictest28245-21cut08:55:142
13462991812,3cyclictest3862-21sort10:25:182
1346299181,15cyclictest0-21swapper/207:45:512
1346299181,14cyclictest12379-21grep10:45:152
1346299181,14cyclictest0-21swapper/212:32:012
1346299181,14cyclictest0-21swapper/212:02:012
1346299181,13cyclictest0-21swapper/208:23:292
1345699181,16cyclictest0-21swapper/110:24:151
1345099186,3cyclictest0-21swapper/010:29:300
1345099182,13cyclictest26584-21head08:50:150
1345099181,5cyclictest0-21swapper/009:38:120
13450991813,3cyclictest27579-21awk12:35:000
1345099181,15cyclictest1527-21awk07:55:000
1345099181,14cyclictest21738-21cstates08:40:130
1345099181,11cyclictest0-21swapper/012:20:050
1345099181,11cyclictest0-21swapper/011:31:050
117922180,4sleep339-21rcuc/310:45:113
247072170,2sleep30-21swapper/310:00:133
1346799176,3cyclictest17549-21awk09:45:023
1346799176,3cyclictest16195-21/usr/sbin/munin10:55:253
1346799171,7cyclictest0-21swapper/311:15:233
1346799171,6cyclictest0-21swapper/311:55:133
1346799171,6cyclictest0-21swapper/309:25:203
1346799171,3cyclictest0-21swapper/307:59:003
13467991713,3cyclictest0-21swapper/308:58:363
1346799171,2cyclictest87650irq/129-enp1s0-11:10:163
13467991712,3cyclictest19354-21/usr/sbin/munin08:35:223
1346799171,15cyclictest0-21swapper/307:26:203
13467991711,3cyclictest2452-21cpu09:10:133
13467991710,4cyclictest14728-21tr12:05:133
1346299177,3cyclictest87550irq/128-enp1s0-07:50:122
1346299176,3cyclictest87350irq/126-enp1s0-08:15:162
1346299171,3cyclictest0-21swapper/211:20:512
13462991711,3cyclictest18197-21ntpq08:30:212
1345099177,8cyclictest31702-21awk07:50:000
1345099173,9cyclictest0-21swapper/009:05:230
1345099173,8cyclictest0-21swapper/011:35:410
1345099171,9cyclictest0-21swapper/012:03:490
1345099171,6cyclictest29037-21sed07:40:240
1345099171,3cyclictest6159-21kworker/0:208:15:180
1345099171,14cyclictest0-21swapper/008:20:070
1345099171,10cyclictest0-21swapper/012:27:100
1345099171,10cyclictest0-21swapper/010:05:280
13450991710,4cyclictest26400-21sh07:35:210
35782160,2sleep00-21swapper/007:58:160
1346799162,6cyclictest11523-21diskstats09:30:143
1346799161,6cyclictest0-21swapper/309:36:313
1346799161,5cyclictest5884-21/usr/sbin/munin10:30:173
1346799161,5cyclictest5884-21/usr/sbin/munin10:30:173
1346799161,14cyclictest0-21swapper/310:26:173
13467991611,3cyclictest1945-21awk10:20:233
13467991611,3cyclictest12601-21cat08:20:013
13467991610,3cyclictest31941-21cat10:15:203
1346299161,2cyclictest0-21swapper/208:42:092
1345099166,8cyclictest10179-21sshd08:13:130
1345099166,8cyclictest10179-21sshd08:13:130
1345099161,6cyclictest17807-21smart_sda07:15:250
13450991611,3cyclictest18745-21/usr/sbin/munin07:20:280
13450991610,3cyclictest18709-21awk08:30:270
1346799159,3cyclictest32599-21cpu09:05:123
13467991510,3cyclictest4854-21expr09:15:153
1346299152,2cyclictest87650irq/129-enp1s0-07:35:112
1345099151,5cyclictest0-21swapper/008:25:110
13450991510,3cyclictest11-21rcuc/008:07:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional