You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-19 - 21:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Dec 19, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109172239163,25sleep20-21swapper/207:06:312
92292232165,56sleep00-21swapper/007:05:070
109732204169,23sleep30-21swapper/307:07:173
109092200164,24sleep10-21swapper/107:06:261
2410821650,3sleep01127099cyclictest08:50:150
2096821200,2sleep31128699cyclictest11:10:163
11280997471,2cyclictest0-21swapper/207:46:132
243982580,3sleep011-21rcuc/010:05:120
11280995552,2cyclictest0-21swapper/208:56:302
246762450,1sleep10-21swapper/110:05:131
1127699381,17cyclictest12541-21ntp_states07:10:221
1127699381,17cyclictest0-21swapper/110:20:201
225302330,4sleep339-21rcuc/310:00:133
1127699331,12cyclictest0-21swapper/109:25:401
1127699321,24cyclictest0-21swapper/112:19:141
1127699321,22cyclictest0-21swapper/107:16:221
1127699321,14cyclictest0-21swapper/110:30:321
1127699311,13cyclictest0-21swapper/110:52:401
1127699311,11cyclictest0-21swapper/108:47:441
1127699301,21cyclictest0-21swapper/110:25:461
1127699301,17cyclictest0-21swapper/110:42:261
1127699301,13cyclictest0-21swapper/110:48:141
1127699301,12cyclictest0-21swapper/112:27:591
1127099301,14cyclictest16944-21munin-run08:35:010
1127699291,21cyclictest0-21swapper/112:09:381
1127699291,14cyclictest0-21swapper/110:17:041
1127699291,14cyclictest0-21swapper/108:31:201
1127699291,13cyclictest0-21swapper/112:04:381
1127699291,13cyclictest0-21swapper/111:14:561
1127699291,13cyclictest0-21swapper/111:09:501
1127699291,13cyclictest0-21swapper/109:57:161
1127699291,13cyclictest0-21swapper/109:41:101
1127699291,12cyclictest0-21swapper/111:47:321
1127699291,12cyclictest0-21swapper/111:24:381
1127699291,11cyclictest28142-21sed11:25:221
1127699291,11cyclictest0-21swapper/110:36:161
1127699290,11cyclictest0-21swapper/109:16:081
1127699282,18cyclictest24441-40dnf11:15:561
1127699281,8cyclictest0-21swapper/111:50:181
1127699281,15cyclictest0-21swapper/108:25:141
1127699281,13cyclictest0-21swapper/111:04:501
1127699281,13cyclictest0-21swapper/111:04:501
1127699281,12cyclictest0-21swapper/112:33:581
1127699281,11cyclictest0-21swapper/109:31:121
1127699281,11cyclictest0-21swapper/108:00:201
1127699281,11cyclictest0-21swapper/107:20:261
1127699281,10cyclictest0-21swapper/107:51:281
1127099289,16cyclictest4999-21basename10:35:020
1127099286,10cyclictest8778-21perf09:30:010
1127699271,9cyclictest0-21swapper/111:56:321
1127699271,12cyclictest0-21swapper/111:00:001
1127699271,12cyclictest0-21swapper/109:39:421
1127699271,12cyclictest0-21swapper/109:11:141
1127699271,11cyclictest0-21swapper/110:11:181
1127099271,12cyclictest28561-21awk10:15:010
1127699263,12cyclictest8094-40dnf08:14:281
11276992622,2cyclictest86950irq/126-enp1s0-08:43:421
11276992622,2cyclictest86950irq/126-enp1s0-08:43:421
1127699261,9cyclictest0-21swapper/109:51:281
1127699261,9cyclictest0-21swapper/108:22:441
1127699261,5cyclictest0-21swapper/109:45:421
1127699261,16cyclictest0-21swapper/107:35:201
1127699261,13cyclictest0-21swapper/107:42:001
1127699261,12cyclictest0-21swapper/112:21:211
1127699261,12cyclictest0-21swapper/111:30:341
1127699261,11cyclictest0-21swapper/112:35:041
1127699261,11cyclictest0-21swapper/112:10:341
1127699261,11cyclictest0-21swapper/112:10:341
11280992512,4cyclictest0-21swapper/210:35:112
1127699251,9cyclictest0-21swapper/108:18:181
1127699251,13cyclictest0-21swapper/107:34:481
1127699251,13cyclictest0-21swapper/107:34:481
1127699251,12cyclictest0-21swapper/110:02:591
1127699251,11cyclictest0-21swapper/111:44:521
1127699251,10cyclictest0-21swapper/111:36:101
1127699251,10cyclictest0-21swapper/107:47:441
1127699251,10cyclictest0-21swapper/107:29:041
1127099259,14cyclictest0-21swapper/008:10:010
1127099254,10cyclictest26988-21perf11:25:010
86302241,19sleep38720-21df_inode08:15:123
1128699231,7cyclictest0-21swapper/312:39:273
1128099231,6cyclictest0-21swapper/209:43:482
1127699231,14cyclictest0-21swapper/107:55:191
1127699231,10cyclictest0-21swapper/108:53:381
1127099231,13cyclictest0-21swapper/012:33:200
1128699227,5cyclictest0-21swapper/309:29:273
1128699225,8cyclictest0-21swapper/312:15:273
1128699223,11cyclictest24446-21chrt11:15:583
1128699222,10cyclictest0-21swapper/312:02:273
1128699221,9cyclictest15595-21users10:55:263
1128699221,9cyclictest0-21swapper/310:22:253
1128699221,10cyclictest904-21sssd_nss12:20:153
1128699220,6cyclictest15768-21kworker/u8:211:50:003
1128699220,10cyclictest0-21swapper/310:28:543
1128099229,3cyclictest12658-21cat12:05:132
1127699221,5cyclictest0-21swapper/108:37:461
1127699221,11cyclictest0-21swapper/109:02:121
1127099225,4cyclictest904-21sssd_nss12:25:180
11270992215,4cyclictest23744-21grep10:00:240
1127099221,11cyclictest10223-21/usr/sbin/munin12:00:210
1128699215,6cyclictest0-21swapper/309:36:303
1128699211,11cyclictest4082-21ntp_states10:30:223
1128099211,7cyclictest27340-21kworker/2:107:58:322
1127699215,3cyclictest0-21swapper/108:05:241
1127099211,6cyclictest0-21swapper/011:33:560
1127099211,14cyclictest0-21swapper/010:50:200
322372200,3sleep31128699cyclictest07:55:123
250382201,3sleep30-21swapper/307:39:123
1128699205,4cyclictest0-21swapper/312:31:153
1128699204,7cyclictest0-21swapper/311:00:103
1128699204,7cyclictest0-21swapper/311:00:103
1128699203,7cyclictest41-21ksoftirqd/307:40:213
1128699201,7cyclictest68722awk11:50:203
11286992015,3cyclictest28557-21timerandwakeup11:25:273
11286992014,3cyclictest10486-21grep09:30:233
11286992013,4cyclictest10914-21df_inode08:20:143
1128699201,11cyclictest0-21swapper/309:17:033
1128699200,9cyclictest0-21swapper/310:43:443
1128699200,10cyclictest0-21swapper/310:10:483
1128099207,5cyclictest835-21runrttasks12:14:462
1128099207,5cyclictest835-21runrttasks12:14:462
11280992015,3cyclictest18801-21sh12:20:012
1127699202,2cyclictest0-21swapper/108:59:041
1127699201,12cyclictest0-21swapper/109:23:521
1127099209,3cyclictest8026-21/usr/sbin/munin11:55:120
1127099208,11cyclictest0-21swapper/009:55:010
1127099203,4cyclictest7-21ksoftirqd/010:25:020
1127099202,8cyclictest0-21swapper/011:50:170
1127099202,15cyclictest21437-21grep09:55:240
1127099202,15cyclictest19638-21grep12:20:160
1127099201,6cyclictest32320-21idleruntime-cro09:10:010
1127099201,5cyclictest0-21swapper/011:11:530
1127099201,13cyclictest0-21swapper/009:13:120
1127099201,12cyclictest0-21swapper/010:40:180
1127099201,10cyclictest0-21swapper/010:59:150
1128699196,4cyclictest15210-21grep07:15:183
1128699195,5cyclictest0-21swapper/311:32:023
1128699194,5cyclictest0-21swapper/310:18:143
1128699194,4cyclictest0-21swapper/309:53:293
1128699194,12cyclictest41-21ksoftirqd/309:10:203
1128699194,12cyclictest41-21ksoftirqd/308:45:183
1128699191,9cyclictest9026-21needreboot11:55:203
1128699191,9cyclictest17109-21unixbench_multi09:45:263
1128699191,11cyclictest0-21swapper/312:25:113
1128699191,10cyclictest4596-21/usr/sbin/munin09:20:143
1128699191,10cyclictest0-21swapper/312:11:143
1128699191,10cyclictest0-21swapper/312:11:143
1128699191,10cyclictest0-21swapper/311:20:483
1128699191,10cyclictest0-21swapper/310:53:523
1128699190,6cyclictest0-21swapper/311:35:123
1128099191,9cyclictest0-21swapper/211:50:262
1128099191,7cyclictest9605-21/usr/sbin/munin10:45:262
1128099191,3cyclictest0-21swapper/209:53:002
1127699191,11cyclictest0-21swapper/109:06:361
1127099197,3cyclictest0-21swapper/011:04:190
1127099197,3cyclictest0-21swapper/011:04:190
1127099194,8cyclictest14190-21taskset12:07:240
1127099194,3cyclictest0-21swapper/011:25:500
1127099191,8cyclictest0-21swapper/009:01:210
1127099191,16cyclictest0-21swapper/008:35:190
1127099191,12cyclictest0-21swapper/008:25:180
1127099191,12cyclictest0-21swapper/008:02:350
1127099191,10cyclictest9795-21irqstats09:30:160
1127099191,10cyclictest9605-21/usr/sbin/munin10:45:200
1127099191,10cyclictest32506-21memory11:35:170
1128699189,2cyclictest39-21rcuc/310:35:233
1128699188,3cyclictest801-21snmpd07:45:163
1128699188,2cyclictest41-21ksoftirqd/308:25:113
1128699187,2cyclictest20355-21needreboot08:40:203
1128699187,2cyclictest20355-21needreboot08:40:193
1128699183,12cyclictest39-21rcuc/307:20:183
1128699181,6cyclictest28752-21grep09:00:163
1128699181,6cyclictest0-21swapper/309:55:363
1128699181,6cyclictest0-21swapper/309:43:243
1128699181,11cyclictest0-21swapper/312:05:343
1128699181,10cyclictest0-21swapper/310:45:533
1128699180,10cyclictest0-21swapper/308:55:193
1128099186,4cyclictest9269-21expr09:30:112
1128099186,4cyclictest13170-21awk09:40:012
1128099184,7cyclictest86950irq/126-enp1s0-10:10:012
1128099181,7cyclictest22603-21ntp_kernel_pll_08:45:192
1128099181,7cyclictest0-21swapper/211:31:362
1128099181,7cyclictest0-21swapper/207:40:262
1128099181,5cyclictest0-21swapper/212:30:442
1128099181,5cyclictest0-21swapper/211:35:172
1128099181,4cyclictest986-1kworker/u9:111:29:362
11280991812,4cyclictest2930-21grep11:40:272
11280991811,4cyclictest24102-21/usr/sbin/munin08:50:152
11280991811,4cyclictest18473-21/usr/sbin/munin08:35:252
1127099185,3cyclictest0-21swapper/007:58:100
1127099184,2cyclictest0-21swapper/012:35:180
1127099181,9cyclictest10573-21/usr/sbin/munin08:20:120
1127099181,8cyclictest1607-21irqstats10:25:180
1127099181,5cyclictest0-21swapper/010:35:050
11270991813,3cyclictest12255-21ntp_states09:35:210
1127099181,11cyclictest0-21swapper/008:12:470
1127099181,10cyclictest0-21swapper/007:45:170
1127099181,10cyclictest0-21swapper/007:36:210
90472171,2sleep20-21swapper/208:15:142
257252171,3sleep20-21swapper/211:20:192
1128699178,3cyclictest0-21swapper/311:40:113
1128699177,2cyclictest18758-21/usr/sbin/munin07:25:123
1128699176,2cyclictest11538-21cpuspeed_turbos07:10:123
1128699174,3cyclictest0-21swapper/311:07:333
1128699171,8cyclictest0-21swapper/310:05:123
1128699171,13cyclictest16118-21pmu-power08:30:223
1128099175,7cyclictest24441-40dnf11:15:582
1128099175,3cyclictest86950irq/126-enp1s0-08:25:002
1128099174,4cyclictest21033-21ntp_offset09:55:202
1128099172,12cyclictest6768-21gltestperf08:10:162
1128099171,7cyclictest28150-21/usr/sbin/munin09:00:152
1128099171,7cyclictest23148-21/usr/sbin/munin07:35:262
1128099171,7cyclictest11297-21ntp_states12:00:212
1128099171,7cyclictest0-21swapper/211:49:282
1128099171,7cyclictest0-21swapper/209:21:112
1128099171,6cyclictest29836-21ntp_states10:15:212
1128099171,6cyclictest0-21swapper/211:56:252
1128099171,5cyclictest0-21swapper/211:10:262
1128099171,5cyclictest0-21swapper/209:18:032
1128099171,3cyclictest86950irq/126-enp1s0-10:50:212
1128099171,14cyclictest0-21swapper/207:30:102
1128099171,14cyclictest0-21swapper/207:30:102
1128099171,14cyclictest0-21swapper/207:10:032
11280991711,3cyclictest32367-21cut09:10:012
11280991711,3cyclictest19031-21/usr/sbin/munin12:20:232
11280991711,3cyclictest16559-21/usr/sbin/munin07:20:232
1127099177,2cyclictest23439-21sshd08:49:020
1127099172,6cyclictest0-21swapper/011:15:560
1127099171,9cyclictest3454-21ntpq09:15:200
11270991713,3cyclictest0-21swapper/011:49:550
11270991713,3cyclictest0-21swapper/011:07:470
11270991713,3cyclictest0-21swapper/008:43:260
11270991713,3cyclictest0-21swapper/008:43:260
11270991712,3cyclictest16251-21gltestperf09:45:170
1127099171,15cyclictest0-21swapper/007:30:230
1127099171,15cyclictest0-21swapper/007:30:220
1127099170,15cyclictest11-21rcuc/009:00:010
43862160,4sleep21128099cyclictest08:05:142
1128699162,4cyclictest31763-21cat07:55:013
1128699161,9cyclictest0-21swapper/308:10:243
1128699161,6cyclictest22779-21seq07:33:223
1128699161,6cyclictest22779-21seq07:33:223
1128699161,4cyclictest3467-21sshd08:01:263
11286991611,3cyclictest4316-21expr08:05:133
1128099167,2cyclictest87150irq/128-enp1s0-10:20:212
1128099165,3cyclictest13849-21ntpq07:15:022
1128099161,9cyclictest0-21swapper/207:50:242
1128099161,6cyclictest8555-21awk10:40:222
1128099161,6cyclictest1999-21expr08:00:122
1128099161,6cyclictest16040-21ntp_states08:30:212
1128099161,6cyclictest0-21swapper/212:27:222
1128099161,6cyclictest0-21swapper/211:01:122
1128099161,6cyclictest0-21swapper/211:01:122
1128099161,6cyclictest0-21swapper/210:31:582
1128099161,6cyclictest0-21swapper/209:29:282
1128099161,6cyclictest0-21swapper/209:11:292
1128099161,2cyclictest0-21swapper/212:39:512
1128099161,2cyclictest0-21swapper/210:14:552
1128099161,2cyclictest0-21swapper/208:29:542
11280991612,3cyclictest0-21swapper/209:45:022
11280991611,3cyclictest201392sleep207:25:252
11280991611,3cyclictest19365-21/usr/sbin/munin08:40:202
11280991611,3cyclictest19365-21/usr/sbin/munin08:40:202
11280991610,3cyclictest22172-21/usr/sbin/munin10:00:122
1128099160,14cyclictest0-21swapper/210:55:082
1127099165,3cyclictest12050irq/123-ahci[0011:40:240
1127099161,9cyclictest0-21swapper/012:13:330
1127099161,9cyclictest0-21swapper/012:13:330
1127099161,7cyclictest0-21swapper/009:40:200
1127099161,7cyclictest0-21swapper/008:18:160
1127099161,6cyclictest0-21swapper/007:20:160
1127099161,4cyclictest0-21swapper/009:22:140
1127099161,3cyclictest0-21swapper/012:15:250
1127099161,12cyclictest29736-21/usr/sbin/munin07:50:160
1127099161,12cyclictest26256-21memory07:40:200
1127099161,12cyclictest14366-21/usr/sbin/munin07:15:190
11270991611,2cyclictest13635-21latency_hist07:15:010
11270991610,3cyclictest29689-21missed_timers10:15:190
1128699156,3cyclictest86950irq/126-enp1s0-09:05:123
1128099154,3cyclictest1530-21awk10:25:162
1128099151,5cyclictest19206-21interrupts11:05:162
1127099150,9cyclictest0-21swapper/007:26:280
1128699148,3cyclictest17351-21cat08:35:133
1128699141,6cyclictest0-21swapper/308:50:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional