You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 12:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri Apr 03, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75462236165,58sleep20-21swapper/219:07:072
75242236162,24sleep10-21swapper/119:06:501
75962232163,56sleep30-21swapper/319:07:443
75922201164,25sleep00-21swapper/019:07:410
1732521720,4sleep1785999cyclictest20:40:241
1470821410,5sleep2786499cyclictest00:15:242
2701621300,4sleep2786499cyclictest22:16:042
7312700,2sleep1785999cyclictest20:05:131
61462620,4sleep1785999cyclictest23:55:241
249052610,1sleep10-21swapper/121:00:101
141972560,1sleep30-21swapper/300:15:173
277562480,1sleep20-21swapper/219:50:252
258582470,2sleep20-21swapper/223:30:012
7864994441,2cyclictest0-21swapper/220:08:512
306972440,1sleep30-21swapper/320:00:003
7864994340,2cyclictest0-21swapper/219:45:592
299882430,2sleep228920-21diskmemload21:10:192
256582420,1sleep30-21swapper/319:45:233
785999411,31cyclictest0-21swapper/122:51:211
785999395,4cyclictest16284-21runrttasks21:54:261
785999390,9cyclictest23-21ksoftirqd/100:33:031
785999385,24cyclictest0-21swapper/122:03:191
785999380,12cyclictest28292-21crond19:55:021
785999376,7cyclictest12569-21sshd19:15:391
785999376,4cyclictest23-21ksoftirqd/121:57:451
7859993716,18cyclictest28013-21ntp_states21:05:221
785999367,2cyclictest10656-21awk22:55:121
785999363,31cyclictest16778-21kworker/u8:023:12:121
785999359,2cyclictest0-21swapper/120:30:111
7859993513,18cyclictest827-21runrttasks23:16:451
785999351,12cyclictest0-21swapper/100:29:231
785999351,11cyclictest6030-21ntp_states20:15:211
282292350,1sleep20-21swapper/221:05:262
785999348,4cyclictest0-21swapper/122:30:171
785999341,9cyclictest0-21swapper/123:45:211
785999341,30cyclictest7544-21turbostat.cron21:35:011
785999341,30cyclictest31117-21turbostat21:15:001
7859993411,20cyclictest30378-21/usr/sbin/munin23:40:191
785999341,10cyclictest15773-21cat20:40:011
785999341,10cyclictest0-21swapper/100:12:081
785499342,10cyclictest21475-21kworker/0:319:41:390
785499342,10cyclictest13293-21kworker/0:220:32:340
785999338,3cyclictest16695-21ntp_states00:20:201
785999338,3cyclictest0-21swapper/122:48:381
785999338,3cyclictest0-21swapper/122:48:381
785999331,29cyclictest16047-21/usr/sbin/munin19:25:161
785999331,24cyclictest87550irq/128-enp1s0-00:35:101
785999331,11cyclictest22996-21ntp_states19:40:211
7859993311,19cyclictest26086-21cat22:15:171
7859993311,19cyclictest19734-21/usr/sbin/munin19:35:191
785999330,7cyclictest0-21swapper/119:46:251
785999328,3cyclictest0-21swapper/123:25:181
785999323,3cyclictest0-21swapper/100:20:021
785999321,6cyclictest32416-21ntp_states21:15:231
785999321,2cyclictest1327-21cpuspeed_turbos21:20:111
785999321,2cyclictest0-21swapper/122:35:031
785999321,12cyclictest22052-21fschecks_count23:20:151
785499329,2cyclictest15595-21kworker/0:020:41:420
785499329,12cyclictest2550-21awk20:10:010
785499329,11cyclictest30509-21perf20:00:010
785499321,11cyclictest3990-21kworker/0:222:58:350
785499321,10cyclictest28905-21kworker/0:221:19:230
785499321,10cyclictest14122-21kworker/0:321:53:150
785999319,3cyclictest2765-21/usr/sbin/munin20:10:161
785999314,3cyclictest18575-21kworker/u8:021:40:211
785999311,26cyclictest0-21swapper/120:20:151
785999311,26cyclictest0-21swapper/120:20:151
7859993112,16cyclictest30716-21/usr/sbin/munin20:00:181
785999311,13cyclictest7367-21fschecks_time00:00:151
785999311,11cyclictest0-21swapper/122:29:431
7859993111,17cyclictest10581-21awk00:05:261
7859993110,18cyclictest10149-21cat19:15:011
785499311,23cyclictest87350irq/126-enp1s0-21:25:180
785499311,12cyclictest29385-21kworker/0:022:45:030
785499311,12cyclictest29385-21kworker/0:022:45:030
785499311,11cyclictest32245-21kworker/0:123:44:230
7871993011,2cyclictest0-21swapper/323:35:153
7864993017,6cyclictest6564-21vmstat20:15:262
785999302,5cyclictest17535-21/usr/sbin/munin19:30:261
785999301,6cyclictest0-21swapper/122:05:211
785999301,26cyclictest11332-21latency_hist20:30:011
7859993011,16cyclictest3816-21cat23:50:241
7859993010,3cyclictest4534-21grep21:25:221
785499302,21cyclictest14013-21kworker/0:122:00:550
785499302,11cyclictest10042-21kworker/0:022:53:030
787199294,7cyclictest0-21swapper/323:59:113
787199293,7cyclictest0-21swapper/322:12:073
785999299,3cyclictest29208-21hwlatdetect19:55:161
785999295,3cyclictest14927-21cat23:05:021
785999291,7cyclictest0-21swapper/122:40:261
785999291,4cyclictest27565-21/usr/sbin/munin22:20:261
785999291,3cyclictest87550irq/128-enp1s0-19:20:101
785999291,1cyclictest0-21swapper/121:35:241
785499296,2cyclictest20628-21kworker/0:319:17:390
785499295,2cyclictest15595-21kworker/0:020:57:050
785499292,7cyclictest12733-21kworker/0:119:23:230
785499292,10cyclictest10679-21kworker/0:100:09:110
200962290,2sleep00-21swapper/000:30:150
175422290,2sleep20-21swapper/219:30:102
787199281,20cyclictest0-21swapper/322:18:373
787199281,19cyclictest0-21swapper/321:57:033
787199281,18cyclictest0-21swapper/321:27:313
785999289,3cyclictest16785-21sshd23:07:501
785999287,4cyclictest827-21runrttasks23:30:191
785999286,3cyclictest28179-21/usr/sbin/munin23:35:211
785999285,3cyclictest13087-21memory21:45:171
7859992812,13cyclictest18148-21/usr/sbin/munin20:45:121
785999280,2cyclictest0-21swapper/122:10:121
785499289,3cyclictest14122-21kworker/0:321:55:110
785499281,16cyclictest20628-21kworker/0:319:13:180
7854992810,15cyclictest17324-21awk19:30:010
787199271,19cyclictest0-21swapper/323:07:213
787199271,18cyclictest0-21swapper/321:42:053
787199271,17cyclictest0-21swapper/323:16:153
785999271,23cyclictest20609-21expr20:50:101
785499274,11cyclictest28920-21diskmemload22:25:010
785499272,11cyclictest5075-21kworker/0:221:38:470
785499271,2cyclictest19925-21kworker/0:120:53:330
785499270,8cyclictest15595-21kworker/0:020:45:550
787199264,6cyclictest0-21swapper/321:46:113
787199263,6cyclictest0-21swapper/323:21:153
787199261,16cyclictest0-21swapper/321:52:593
7871992611,3cyclictest5633-21hddtemp_smartct20:15:153
785999266,2cyclictest0-21swapper/120:55:191
785499268,3cyclictest14013-21kworker/0:121:48:590
785499262,11cyclictest15202-21kworker/0:200:18:470
785499260,9cyclictest506-21kworker/0:021:33:410
785499260,12cyclictest31748-21perf22:30:010
7871992521,3cyclictest0-21swapper/320:05:193
787199251,4cyclictest0-21swapper/322:26:413
787199251,18cyclictest0-21swapper/323:32:343
787199251,18cyclictest0-21swapper/322:44:283
787199251,18cyclictest0-21swapper/300:31:473
787199251,18cyclictest0-21swapper/300:29:213
787199251,17cyclictest0-21swapper/321:30:113
787199251,17cyclictest0-21swapper/321:12:413
787199251,17cyclictest0-21swapper/300:05:093
787199251,16cyclictest0-21swapper/322:48:583
787199251,16cyclictest0-21swapper/322:48:573
787199251,16cyclictest0-21swapper/322:20:083
787199251,16cyclictest0-21swapper/300:11:153
787199251,12cyclictest32261-21memory21:15:213
785499255,3cyclictest19480-21kworker/0:119:39:550
785499252,8cyclictest3990-21kworker/0:222:42:150
785499252,8cyclictest31606-21kworker/0:222:31:590
785499252,7cyclictest14568-21kworker/0:023:06:140
785499251,9cyclictest19019-21kworker/0:023:15:340
7854992512,2cyclictest30167-21awk23:40:010
785499251,10cyclictest16772-21kworker/0:123:10:260
787199243,6cyclictest0-21swapper/323:26:273
787199243,5cyclictest0-21swapper/322:07:193
787199242,7cyclictest0-21swapper/322:59:353
787199242,7cyclictest0-21swapper/300:23:073
7871992418,4cyclictest0-21swapper/321:35:453
787199241,17cyclictest0-21swapper/323:02:123
787199241,17cyclictest0-21swapper/322:35:493
787199241,17cyclictest0-21swapper/322:32:423
787199241,17cyclictest0-21swapper/322:00:033
787199241,17cyclictest0-21swapper/321:22:343
787199241,17cyclictest0-21swapper/319:12:103
787199241,17cyclictest0-21swapper/300:35:493
787199241,16cyclictest0-21swapper/323:13:493
787199241,16cyclictest0-21swapper/300:01:523
7871992411,1cyclictest0-21swapper/321:04:313
785499248,2cyclictest14013-21kworker/0:122:16:590
785499242,9cyclictest4702-21kworker/0:120:16:090
785499241,7cyclictest14013-21kworker/0:122:05:470
785499241,10cyclictest30014-21kworker/0:023:48:150
785499241,10cyclictest25697-21kworker/0:123:33:230
785499240,9cyclictest30284-21kworker/0:320:02:510
787199236,2cyclictest0-21swapper/319:40:063
7871992319,3cyclictest0-21swapper/323:50:153
787199231,16cyclictest0-21swapper/323:45:573
787199231,16cyclictest0-21swapper/320:26:223
785499233,2cyclictest87450irq/127-enp1s0-19:53:170
785499232,9cyclictest506-21kworker/0:021:23:550
785499232,9cyclictest23497-21kworker/0:023:25:590
785499231,11cyclictest9981-21/usr/sbin/munin21:40:180
785499230,21cyclictest0-21swapper/000:10:130
787199224,3cyclictest0-21swapper/320:31:213
7871992218,3cyclictest0-21swapper/320:42:353
787199221,19cyclictest87350irq/126-enp1s0-22:50:183
787199221,15cyclictest0-21swapper/323:42:583
787199221,10cyclictest7159-21/usr/sbin/munin20:20:253
787199221,10cyclictest7159-21/usr/sbin/munin20:20:253
786499221,6cyclictest0-21swapper/221:30:232
786499221,5cyclictest0-21swapper/200:14:212
785499228,3cyclictest22223-21/usr/sbin/munin00:35:210
785499225,2cyclictest26552-21kworker/0:321:07:090
785499221,12cyclictest2439-21kworker/0:020:27:540
785499221,10cyclictest808-21snmpd23:56:010
785499221,10cyclictest15595-21kworker/0:021:03:130
47982220,3sleep29-21rcu_preempt20:15:012
185972220,1sleep30-21swapper/319:30:183
787199212,11cyclictest15343-21/usr/sbin/munin19:25:233
787199211,4cyclictest0-21swapper/319:15:193
786499212,4cyclictest7156-21diskstats00:00:132
786499211,7cyclictest0-21swapper/222:49:022
786499211,7cyclictest0-21swapper/222:49:022
786499211,11cyclictest8454-21hddtemp_smartct21:35:152
785499217,3cyclictest12787-21/usr/sbin/munin23:00:240
785499212,10cyclictest15202-21kworker/0:200:26:350
121942210,4sleep20-21swapper/221:45:112
787199203,3cyclictest0-21swapper/320:37:513
787199201,8cyclictest28385-21timerwakeupswit21:05:263
787199201,5cyclictest579-21systemd-journal20:00:203
785499207,3cyclictest2439-21kworker/0:020:10:030
785499206,6cyclictest0-21swapper/000:00:130
785499202,2cyclictest6922-21kworker/0:220:21:010
785499202,2cyclictest6922-21kworker/0:220:21:010
785499201,16cyclictest21585-21/usr/sbin/munin23:20:240
785499201,11cyclictest30014-21kworker/0:023:51:590
785499200,18cyclictest0-21swapper/019:45:200
231202200,5sleep2786499cyclictest23:20:262
787199193,3cyclictest0-21swapper/319:35:143
787199192,3cyclictest0-21swapper/320:46:113
787199191,9cyclictest0-21swapper/319:20:233
787199191,6cyclictest20431-21/usr/sbin/munin20:50:113
787199191,10cyclictest87350irq/126-enp1s0-19:50:223
786499194,4cyclictest0-21swapper/223:07:282
786499191,8cyclictest12602-21awk20:30:192
7864991914,3cyclictest17920-21dump-pmu-power20:45:012
7864991912,4cyclictest4488-21/usr/sbin/munin21:25:232
786499191,10cyclictest0-21swapper/223:02:092
786499190,8cyclictest0-21swapper/221:16:472
785499195,3cyclictest87350irq/126-enp1s0-00:20:110
785499194,2cyclictest24844-21kworker/0:022:13:110
785499190,6cyclictest0-21swapper/020:35:170
787199181,6cyclictest3357-21head20:10:163
787199181,5cyclictest22542-21/usr/sbin/munin20:55:123
786499185,4cyclictest15112-21crond19:25:002
786499185,2cyclictest87350irq/126-enp1s0-22:55:132
786499181,8cyclictest29542-21munin-run22:25:002
786499181,8cyclictest20067-21smartctl23:15:162
786499181,7cyclictest0-21swapper/222:35:592
786499181,6cyclictest24052-21unixbench_multi20:55:242
786499181,5cyclictest0-21swapper/223:54:292
786499181,3cyclictest0-21swapper/223:44:142
786499181,3cyclictest0-21swapper/223:32:152
786499181,3cyclictest0-21swapper/221:50:102
786499181,3cyclictest0-21swapper/200:26:262
7864991813,3cyclictest21065-21grep22:05:112
7864991812,4cyclictest0-21swapper/223:45:132
786499181,15cyclictest0-21swapper/200:30:112
786499181,11cyclictest0-21swapper/200:35:162
7854991813,3cyclictest18359-21kernelversion19:30:170
786499176,3cyclictest827-21runrttasks19:25:222
786499175,5cyclictest0-21swapper/222:50:102
786499175,3cyclictest87350irq/126-enp1s0-21:55:242
786499175,3cyclictest12426-21unixbench-2d19:15:252
786499173,4cyclictest14451-21wc20:35:152
786499172,14cyclictest0-21swapper/222:44:262
786499172,12cyclictest22026-21grep19:40:122
786499172,11cyclictest10089-21cpuspeed_turbos21:40:132
786499171,7cyclictest20685-21df_inode20:50:122
786499171,7cyclictest0-21swapper/222:10:232
786499171,6cyclictest18183-21needreboot23:10:202
786499171,6cyclictest0-21swapper/223:35:292
786499171,4cyclictest0-21swapper/222:00:192
7864991713,3cyclictest0-21swapper/220:02:032
7864991712,3cyclictest10176-21irqstats20:25:182
786499171,15cyclictest0-21swapper/221:04:012
786499171,10cyclictest0-21swapper/222:31:112
786499171,10cyclictest0-21swapper/219:58:332
785499175,2cyclictest28764-21kworker/0:121:10:590
785499171,3cyclictest810-21sshd22:35:410
63022171,2sleep20-21swapper/223:56:342
317182170,5sleep2786499cyclictest22:29:572
172202170,2sleep20-21swapper/200:20:272
786499165,3cyclictest87650irq/129-enp1s0-21:20:152
786499164,3cyclictest20721-21strings19:35:182
786499161,6cyclictest7964-21interrupts20:20:162
786499161,6cyclictest7964-21interrupts20:20:162
7864991611,3cyclictest18148-21/usr/sbin/munin20:45:152
7864991610,4cyclictest0-21swapper/219:10:132
786499160,15cyclictest0-21swapper/200:06:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional