You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-17 - 23:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 17, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
65042239163,23sleep20-21swapper/207:09:372
62082202164,26sleep00-21swapper/007:05:450
45772202167,23sleep30-21swapper/307:05:083
61892201165,24sleep10-21swapper/107:05:301
2536721510,3sleep3663199cyclictest07:50:113
556121340,1sleep10-21swapper/109:30:221
662699104100,3cyclictest0-21swapper/210:53:082
236322760,3sleep19-21rcu_preempt09:00:121
6626995951,3cyclictest0-21swapper/210:48:452
6626995951,3cyclictest0-21swapper/210:48:442
155612550,2sleep121-21rcuc/107:25:251
196612540,4sleep123-21ksoftirqd/111:15:241
291382490,4sleep07-21ksoftirqd/010:25:150
244642480,2sleep20-21swapper/210:15:132
661099317,12cyclictest2555-21awk10:40:010
662199301,3cyclictest13231-21/usr/sbin/munin09:50:121
662699298,13cyclictest11136-21cstates09:45:122
663199275,7cyclictest29346-21/usr/sbin/munin11:40:133
176192270,5sleep011-21rcuc/011:10:260
663199263,9cyclictest87650irq/129-enp1s0-09:40:203
661099267,16cyclictest12926-21munin-run09:50:010
6610992616,2cyclictest0-21swapper/008:50:160
663199251,7cyclictest1128-21crond11:50:003
662699251,6cyclictest0-21swapper/210:02:382
662199251,3cyclictest12263-21diskstats07:20:141
6621992510,3cyclictest12018-21ntp_states09:45:221
661099259,13cyclictest5703-21munin-run08:20:010
661099250,12cyclictest20466-21idleruntime-cro07:40:010
254002251,20sleep125483-21timerandwakeup10:15:251
663199241,7cyclictest0-21swapper/310:19:033
663199241,7cyclictest0-21swapper/309:26:083
663199241,6cyclictest0-21swapper/311:28:383
662699247,4cyclictest21692-21strings07:40:172
662699241,5cyclictest27689-21diskmemload12:37:222
662699241,13cyclictest0-21swapper/210:43:362
662199241,11cyclictest6967-21ntp_kernel_err08:20:221
662199241,11cyclictest6967-21ntp_kernel_err08:20:211
662199241,11cyclictest10523-21df12:10:131
662199241,10cyclictest8338-21diskstats12:05:141
661099248,4cyclictest6925-21kworker/0:310:50:270
661099240,10cyclictest0-21swapper/007:45:130
78862231,18sleep27901-21open_files07:10:222
663199239,3cyclictest12050irq/123-ahci[0009:10:023
663199235,8cyclictest87350irq/126-enp1s0-09:45:163
663199231,7cyclictest0-21swapper/311:35:303
663199231,6cyclictest0-21swapper/311:18:273
663199231,6cyclictest0-21swapper/311:08:143
662699231,6cyclictest0-21swapper/211:08:002
662199235,4cyclictest19416-21sshd10:03:511
6621992317,3cyclictest21983-21hddtemp_smartct08:55:151
662199231,19cyclictest31365-21sed10:30:151
662199231,19cyclictest23961-21ntp_states07:45:211
662199230,16cyclictest0-21swapper/111:46:081
661099239,12cyclictest6318-21munin-run09:35:010
661099238,3cyclictest0-21swapper/007:13:500
661099236,3cyclictest87350irq/126-enp1s0-11:55:210
661099236,3cyclictest87350irq/126-enp1s0-11:55:210
661099231,6cyclictest23104-21diskstats11:25:130
661099231,6cyclictest0-21swapper/011:53:230
661099231,6cyclictest0-21swapper/011:16:350
292432231,2sleep10-21swapper/110:25:161
267662231,18sleep126790-21ntp_states09:05:221
663199227,3cyclictest87350irq/126-enp1s0-10:25:243
663199225,6cyclictest9595-21cpuspeed_turbos10:55:123
663199222,11cyclictest839-21lldpd07:40:193
663199221,7cyclictest13740-21latency_hist11:05:023
663199221,6cyclictest87350irq/126-enp1s0-09:20:013
6631992211,3cyclictest87350irq/126-enp1s0-09:50:193
662699226,4cyclictest14654-21latency_hist08:40:022
662699224,12cyclictest87650irq/129-enp1s0-11:20:122
662699224,10cyclictest27335-21wc11:35:122
662699223,5cyclictest8564-1kworker/u9:111:41:072
662699223,10cyclictest20631-21unixbench_multi08:50:262
662699222,6cyclictest8564-1kworker/u9:112:08:502
662699221,12cyclictest19619-21sensors_temp11:15:232
662699221,11cyclictest30727-21kworker/u8:111:35:012
662699221,11cyclictest0-21swapper/211:27:082
662199221,3cyclictest11048-21unixbench_singl10:55:261
662199221,10cyclictest21345-21if_err_enp2s011:20:171
662199220,4cyclictest0-21swapper/112:31:381
662199220,3cyclictest0-21swapper/111:02:121
661099229,2cyclictest0-21swapper/008:25:110
661099228,3cyclictest31831-21expr11:45:120
661099228,2cyclictest24700-21kworker/0:108:05:210
661099227,6cyclictest0-21swapper/009:55:020
661099226,5cyclictest15116-21grep11:05:230
661099225,3cyclictest30491-21grep11:40:220
661099221,6cyclictest0-21swapper/011:00:430
661099220,9cyclictest0-21swapper/012:20:170
663199215,7cyclictest401ktimersoftd/308:25:013
663199215,7cyclictest401ktimersoftd/308:25:013
663199214,9cyclictest0-21swapper/312:18:513
663199211,6cyclictest0-21swapper/310:52:283
663199211,6cyclictest0-21swapper/309:56:413
663199211,11cyclictest4542-21sshd10:42:413
663199210,5cyclictest0-21swapper/310:50:023
663199210,5cyclictest0-21swapper/310:50:023
662699214,9cyclictest0-21swapper/212:30:082
662699213,9cyclictest0-21swapper/210:56:042
662699213,5cyclictest8564-1kworker/u9:111:53:072
662699212,11cyclictest15697-21users07:25:262
662699212,10cyclictest0-21swapper/209:40:132
662699211,6cyclictest0-21swapper/212:04:202
662699211,11cyclictest9-21rcu_preempt11:55:172
662699211,11cyclictest9-21rcu_preempt11:55:172
662199217,3cyclictest0-21swapper/111:13:001
662199212,2cyclictest87650irq/129-enp1s0-11:55:091
662199212,2cyclictest87650irq/129-enp1s0-11:55:091
662199211,6cyclictest5595-21sshd08:18:511
6621992115,3cyclictest27076-21perf07:55:021
662199211,3cyclictest28106-21missed_timers11:35:191
662199211,3cyclictest0-21swapper/112:19:351
662199211,18cyclictest0-21swapper/112:00:371
662199211,18cyclictest0-21swapper/107:17:571
662199210,4cyclictest0-21swapper/111:06:251
662199210,2cyclictest0-21swapper/110:12:561
661099213,12cyclictest0-21swapper/011:36:070
661099212,8cyclictest839-21lldpd12:08:240
661099211,9cyclictest0-21swapper/009:18:100
661099211,6cyclictest0-21swapper/010:40:420
661099211,6cyclictest0-21swapper/009:50:220
6610992116,3cyclictest18881-21crond08:50:010
661099211,10cyclictest0-21swapper/010:20:250
661099210,8cyclictest0-21swapper/012:15:260
30892210,4sleep03088-21unixbench_singl08:10:260
279352211,16sleep228219-21expr09:10:122
663199206,3cyclictest87650irq/129-enp1s0-11:55:173
663199206,3cyclictest87650irq/129-enp1s0-11:55:163
663199204,8cyclictest12050irq/123-ahci[0008:15:013
663199204,4cyclictest41-21ksoftirqd/311:30:153
663199201,8cyclictest0-21swapper/311:14:093
6631992015,3cyclictest20408-21cut10:05:163
663199201,4cyclictest23038-21awk10:10:233
6631992013,4cyclictest10705-21needreboot07:15:193
663199201,11cyclictest14721-21/usr/sbin/munin08:40:243
663199201,10cyclictest6544-21/usr/sbin/munin09:35:213
663199201,10cyclictest6544-21/usr/sbin/munin09:35:213
6631992010,2cyclictest87650irq/129-enp1s0-12:20:203
662699205,4cyclictest8564-1kworker/u9:109:20:522
662699205,3cyclictest87650irq/129-enp1s0-09:15:202
662699205,3cyclictest16751-21gltestperf11:10:152
662699204,12cyclictest87650irq/129-enp1s0-12:26:032
662699204,12cyclictest87650irq/129-enp1s0-10:36:262
662699203,9cyclictest27689-21diskmemload10:32:132
662699202,5cyclictest11256-21ntp_states12:10:222
662699202,4cyclictest23501-21head07:45:152
662699202,3cyclictest0-21swapper/211:02:102
662699201,5cyclictest8564-1kworker/u9:110:26:052
662699201,3cyclictest0-21swapper/210:22:512
662699201,11cyclictest4345-21/usr/sbin/munin09:30:172
662699201,11cyclictest31894-21diskstats11:45:132
662699201,11cyclictest21311-21/usr/sbin/munin08:55:222
662699201,11cyclictest0-21swapper/209:55:232
662699201,10cyclictest3617-21cat08:15:012
662699201,10cyclictest25720-21/usr/sbin/munin09:05:172
662199208,3cyclictest0-21swapper/108:10:271
662199208,2cyclictest3785-21sshd09:25:511
662199206,6cyclictest0-21swapper/111:40:141
662199203,13cyclictest0-21swapper/107:35:141
662199202,5cyclictest10326-21/usr/sbin/munin08:30:131
662199201,7cyclictest0-21swapper/110:50:141
662199201,6cyclictest31122-21sshd08:00:511
6621992016,3cyclictest0-21swapper/109:35:031
6621992016,3cyclictest0-21swapper/109:35:031
6621992016,3cyclictest0-21swapper/108:08:341
662199201,2cyclictest0-21swapper/109:18:451
662199201,18cyclictest0-21swapper/112:28:441
662199201,18cyclictest0-21swapper/110:20:541
662199201,18cyclictest0-21swapper/108:50:021
662199201,17cyclictest0-21swapper/109:59:201
662199201,17cyclictest0-21swapper/109:12:381
662199201,16cyclictest9426-21hddtemp_smartct09:40:161
662199201,11cyclictest0-21swapper/112:24:181
662199200,3cyclictest0-21swapper/110:38:281
661099207,4cyclictest0-21swapper/012:00:080
661099207,11cyclictest11-21rcuc/007:25:010
661099203,2cyclictest0-21swapper/011:32:270
661099202,9cyclictest0-21swapper/012:36:160
661099201,9cyclictest0-21swapper/011:24:020
661099201,9cyclictest0-21swapper/010:31:490
661099201,8cyclictest0-21swapper/012:33:580
661099201,7cyclictest0-21swapper/009:10:070
661099201,5cyclictest0-21swapper/012:13:150
661099201,5cyclictest0-21swapper/010:04:190
661099201,4cyclictest0-21swapper/012:28:430
661099201,4cyclictest0-21swapper/010:06:110
661099201,14cyclictest0-21swapper/010:13:340
246072201,3sleep024814-21hddtemp_smartct10:15:160
142442201,2sleep10-21swapper/108:36:561
663199199,2cyclictest87350irq/126-enp1s0-10:20:243
663199198,3cyclictest827-21runrttasks12:07:033
663199198,3cyclictest15303-21processes07:25:213
663199196,3cyclictest12050irq/123-ahci[0007:15:023
663199195,3cyclictest39-21rcuc/308:00:133
663199194,8cyclictest87350irq/126-enp1s0-12:00:123
663199194,2cyclictest87650irq/129-enp1s0-10:00:143
663199193,8cyclictest401ktimersoftd/310:30:153
663199193,7cyclictest401ktimersoftd/308:50:163
663199191,7cyclictest5581-21sendmail_mailqu09:30:223
663199191,7cyclictest0-21swapper/312:10:253
663199191,7cyclictest0-21swapper/308:25:203
663199191,4cyclictest0-21swapper/311:52:553
663199191,10cyclictest27292-21/usr/sbin/munin07:55:123
663199191,10cyclictest11915-21/usr/sbin/munin07:20:173
663199190,5cyclictest87650irq/129-enp1s0-12:30:213
662699195,3cyclictest87650irq/129-enp1s0-08:40:132
662699194,3cyclictest87650irq/129-enp1s0-09:25:252
662699192,10cyclictest8277-21kworker/u8:009:37:542
662699192,10cyclictest8277-21kworker/u8:009:37:542
662699191,6cyclictest24359-21latency09:00:172
662699191,5cyclictest16932-21acpi08:45:112
662699191,5cyclictest14718-21cpuspeed_turbos12:20:122
662699191,5cyclictest13748-21grep09:50:152
662699191,4cyclictest13938-21grep12:15:272
662699191,10cyclictest827-21runrttasks08:00:232
662699191,10cyclictest16618-21expr07:30:132
662699191,10cyclictest0-21swapper/210:05:032
662199192,3cyclictest1001-21ntp_states09:20:211
662199191,7cyclictest0-21swapper/111:31:471
662199191,6cyclictest0-21swapper/110:07:381
662199191,6cyclictest0-21swapper/107:40:211
662199191,5cyclictest8130-21/usr/sbin/munin08:25:141
662199191,16cyclictest0-21swapper/112:35:431
662199191,16cyclictest0-21swapper/108:50:381
662199191,15cyclictest5633-21gltestperf10:45:161
662199191,15cyclictest5633-21gltestperf10:45:161
662199190,17cyclictest0-21swapper/111:25:251
661099197,2cyclictest0-21swapper/010:45:270
661099197,2cyclictest0-21swapper/010:45:270
661099196,3cyclictest0-21swapper/009:44:370
661099195,5cyclictest0-21swapper/009:21:450
661099191,6cyclictest0-21swapper/009:39:580
661099191,6cyclictest0-21swapper/009:39:580
661099191,16cyclictest0-21swapper/007:30:250
661099190,14cyclictest0-21swapper/009:26:320
663199189,2cyclictest87650irq/129-enp1s0-09:20:243
663199187,3cyclictest31694-21/usr/sbin/munin08:05:253
663199187,3cyclictest10326-21/usr/sbin/munin08:30:213
663199181,9cyclictest808-21snmpd07:37:443
663199181,8cyclictest0-21swapper/311:22:473
663199181,7cyclictest18066-21pmu-power08:45:223
663199181,7cyclictest0-21swapper/312:36:543
663199181,6cyclictest0-21swapper/307:30:223
663199180,2cyclictest165050irq/12-ttyS008:55:213
662699185,3cyclictest87350irq/126-enp1s0-08:15:222
662699181,9cyclictest31858-21cpuspeed_turbos08:05:122
662699181,5cyclictest10350-21cpu08:30:112
6621991814,3cyclictest0-21swapper/108:43:021
662199181,3cyclictest0-21swapper/107:30:371
662199181,16cyclictest0-21swapper/107:56:061
662199181,16cyclictest0-21swapper/107:10:501
662199181,15cyclictest0-21swapper/110:40:291
662199180,8cyclictest6608-21cyclictest11:50:261
661099182,3cyclictest26531-21irqstats09:05:180
661099181,7cyclictest27689-21diskmemload10:57:200
661099181,6cyclictest11072-21timerandwakeup07:15:240
661099181,5cyclictest0-21swapper/008:30:160
661099181,5cyclictest0-21swapper/008:21:230
661099181,5cyclictest0-21swapper/008:21:230
661099181,4cyclictest12525-21/usr/sbin/munin08:35:270
661099181,2cyclictest26232-21pmu-power07:50:210
6610991812,3cyclictest21878-21processes07:40:210
661099181,14cyclictest14721-21/usr/sbin/munin08:40:190
119022181,3sleep20-21swapper/207:20:052
663199173,3cyclictest87350irq/126-enp1s0-10:35:133
663199172,5cyclictest41-21ksoftirqd/309:10:193
663199171,9cyclictest0-21swapper/308:15:223
6631991713,3cyclictest0-21swapper/312:27:323
663199171,14cyclictest0-21swapper/309:00:563
662699174,5cyclictest8146-21cpu08:25:112
662699174,3cyclictest87650irq/129-enp1s0-07:35:152
662699171,9cyclictest0-21swapper/207:15:122
662699171,8cyclictest87350irq/126-enp1s0-07:55:192
662699171,4cyclictest7404-21sed08:20:272
662699171,4cyclictest7404-21sed08:20:272
6626991710,4cyclictest22387-21cut10:10:162
661099171,5cyclictest0-21swapper/008:55:230
661099171,5cyclictest0-21swapper/007:25:230
661099171,4cyclictest28848-21sed07:55:240
6631991612,3cyclictest0-21swapper/308:37:163
6631991610,3cyclictest23513-21head07:45:153
662699161,2cyclictest0-21swapper/207:50:102
661099161,4cyclictest29483-21/usr/sbin/munin08:00:120
661099161,3cyclictest0-21swapper/009:00:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional