You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-04 - 23:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 04, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
284922240163,25sleep20-21swapper/207:09:582
282382239163,24sleep10-21swapper/107:06:441
284742204168,24sleep30-21swapper/307:09:473
284192204165,26sleep00-21swapper/007:09:050
1122821560,5sleep32859499cyclictest11:20:253
28587991070,1cyclictest0-21swapper/212:00:232
28587999184,2cyclictest0-21swapper/211:54:582
28587998683,2cyclictest0-21swapper/209:26:332
28587997773,3cyclictest0-21swapper/209:38:012
28587997773,3cyclictest0-21swapper/209:38:012
28587997068,1cyclictest0-21swapper/211:32:212
28587997067,2cyclictest0-21swapper/208:12:592
26232540,3sleep121-21rcuc/112:15:171
167802520,4sleep22858799cyclictest11:35:142
20622490,2sleep30-21swapper/307:20:153
2859499424,5cyclictest13249-40dnf10:11:463
28574993910,13cyclictest2521-21munin-run09:50:020
2857499389,13cyclictest18875-21awk08:00:010
2859499358,3cyclictest27888-21/usr/sbin/munin08:20:253
2859499356,3cyclictest0-21swapper/309:20:263
28594993512,3cyclictest25287-21processes09:25:223
28594993511,3cyclictest5904-21/usr/sbin/munin07:30:223
28594993411,3cyclictest3371-21/usr/sbin/munin11:05:223
28594993411,3cyclictest27273-21uptime08:15:263
28594993411,3cyclictest19716-21/usr/sbin/munin09:15:233
28594993410,3cyclictest10575-21/usr/sbin/munin12:35:163
2857499348,12cyclictest3770-21awk12:20:010
2859499335,3cyclictest17137-21gltestperf11:35:163
2859499332,3cyclictest12050irq/123-ahci[0011:30:013
2859499331,6cyclictest0-21swapper/310:35:553
2859499331,10cyclictest4318-21/usr/sbin/munin08:40:283
2859499330,9cyclictest0-21swapper/310:15:123
2859499328,3cyclictest808-21snmpd10:50:453
2859499328,3cyclictest808-21snmpd10:50:453
2859499324,3cyclictest12050irq/123-ahci[0007:40:013
2859499323,6cyclictest17288-21diskmemload09:49:063
2859499322,5cyclictest9443-21ntp_states12:30:213
2859499321,6cyclictest5618-21sh09:55:163
2859499321,6cyclictest17288-21diskmemload09:30:133
2859499321,3cyclictest31229-21grep09:40:143
28594993211,4cyclictest30346-21grep07:10:273
28594993211,4cyclictest30346-21grep07:10:263
28594993211,3cyclictest3484-21if_enp2s009:50:153
28594993211,3cyclictest16160-21expr10:20:113
28594993210,3cyclictest20948-21/usr/sbin/munin11:45:153
2859499312,12cyclictest839-21lldpd08:25:593
2859499312,12cyclictest839-21lldpd08:25:593
2859499311,6cyclictest15087-21latency_hist09:05:023
28594993110,3cyclictest23823-21sh11:50:163
28594993110,3cyclictest20103-21cat11:40:243
28594993110,2cyclictest14696-21/usr/sbin/munin07:50:223
2859499309,3cyclictest25343-21/usr/sbin/munin11:55:213
2859499305,8cyclictest23915-21fschecks_count08:10:133
2859499301,5cyclictest808-21snmpd10:33:063
2859499301,3cyclictest0-21swapper/310:49:193
28594993010,3cyclictest31721-21latency_hist12:10:013
2859499292,3cyclictest87650irq/129-enp1s0-12:24:383
2859499291,6cyclictest31242-21ntp_offset07:15:033
2859499291,6cyclictest29532-21perf12:05:003
2859499291,5cyclictest0-21swapper/310:07:523
2859499291,12cyclictest12500-21/usr/sbin/munin07:45:233
28594992910,3cyclictest6879-21forks08:45:163
28594992910,2cyclictest5675-21latency_hist07:30:023
2857499291,13cyclictest23889-21munin-run09:25:010
2859499281,7cyclictest808-21snmpd12:18:243
2859499281,6cyclictest17288-21diskmemload10:59:013
2859499281,6cyclictest0-21swapper/311:30:263
2859499281,3cyclictest10304-21/usr/sbin/munin07:40:163
2859499281,2cyclictest0-21swapper/312:29:473
28594992810,3cyclictest25638-21load10:40:203
28594992810,3cyclictest21385-21ls08:05:123
2857499281,11cyclictest0-21swapper/010:30:150
261612280,4sleep226166-21fschecks_time08:15:162
2859499279,4cyclictest0-21swapper/309:05:223
2859499279,3cyclictest2090-21latency_hist08:35:023
2859499275,3cyclictest17288-21diskmemload11:15:313
2859499271,7cyclictest0-21swapper/310:02:443
2859499271,5cyclictest17517-21/usr/sbin/munin09:10:243
28594992711,3cyclictest20685-21vmstat08:00:273
2858299271,23cyclictest28996-21cstates07:10:141
2858299271,23cyclictest28996-21cstates07:10:131
2857499272,12cyclictest32683-21munin-run09:45:020
2857499272,12cyclictest0-21swapper/009:40:010
2857499272,12cyclictest0-21swapper/009:40:010
2859499268,3cyclictest0-21swapper/311:01:303
2859499261,5cyclictest0-21swapper/311:14:483
2859499261,3cyclictest6820-21kworker/u8:010:27:393
2859499261,14cyclictest0-21swapper/307:59:473
2858299262,16cyclictest790-21in:imjournal10:46:421
190942261,3sleep119114-21forks11:40:151
2859499259,3cyclictest0-21swapper/308:54:063
2859499256,9cyclictest0-21swapper/308:35:263
2859499251,3cyclictest26054-21kworker/u8:208:55:153
2859499250,14cyclictest0-21swapper/309:38:173
2859499250,14cyclictest0-21swapper/309:38:173
2858299250,18cyclictest0-21swapper/112:05:371
2858799248,8cyclictest87350irq/126-enp1s0-10:40:142
2858799246,5cyclictest18245-1kworker/u9:212:20:192
2858799241,6cyclictest0-21swapper/210:31:452
2858299241,3cyclictest0-21swapper/107:34:441
2858299240,19cyclictest0-21swapper/109:29:381
2858299240,15cyclictest0-21swapper/110:35:091
2857499248,3cyclictest8379-21/usr/sbin/munin12:30:110
2857499248,3cyclictest24209-21ntp_states11:50:210
2857499241,5cyclictest0-21swapper/011:35:320
2857499240,12cyclictest10304-21/usr/sbin/munin07:40:150
2857499240,10cyclictest25100-21strings09:25:200
2859499231,6cyclictest17288-21diskmemload12:10:243
2858799231,5cyclictest8154-21kworker/u8:109:15:242
2858799230,9cyclictest0-21swapper/209:10:292
2858799230,10cyclictest0-21swapper/212:08:322
2858299231,19cyclictest18699-21perl10:25:181
2858299231,14cyclictest0-21swapper/112:26:001
2858299231,14cyclictest0-21swapper/111:15:111
2858299230,18cyclictest0-21swapper/110:24:391
2858299230,17cyclictest0-21swapper/109:33:241
2858299230,16cyclictest0-21swapper/111:55:021
2857499238,3cyclictest23252-21kworker/0:308:17:590
2857499237,3cyclictest19716-21/usr/sbin/munin09:15:240
2857499232,11cyclictest5904-21/usr/sbin/munin07:30:120
2857499231,5cyclictest0-21swapper/011:41:320
2857499231,5cyclictest0-21swapper/009:50:050
2857499231,12cyclictest0-21swapper/008:35:010
2857499231,10cyclictest10917-21/usr/sbin/munin08:55:210
2857499230,2cyclictest15315-21unixbench_singl10:15:260
188382230,5sleep12858299cyclictest09:10:241
8342220,4sleep230-21rcuc/209:45:122
2858799226,2cyclictest87550irq/128-enp1s0-07:45:212
2858799221,6cyclictest0-21swapper/211:28:572
2858799221,6cyclictest0-21swapper/209:32:012
2857499229,4cyclictest7510-21needreboot08:45:190
2857499227,3cyclictest13782-21sshd11:25:450
2857499222,3cyclictest0-21swapper/008:00:430
2857499221,9cyclictest0-21swapper/011:08:080
2857499221,4cyclictest13894-21kworker/u8:111:32:590
28574992213,5cyclictest29757-40dnf07:10:220
28574992213,5cyclictest29757-40dnf07:10:220
2857499221,10cyclictest7766-21/usr/sbin/munin11:15:180
2857499220,4cyclictest0-21swapper/010:24:440
2858799211,6cyclictest0-21swapper/212:19:362
2858799211,6cyclictest0-21swapper/209:53:402
2858799211,5cyclictest0-21swapper/211:41:412
2858799211,5cyclictest0-21swapper/211:05:042
2858799211,5cyclictest0-21swapper/209:42:182
2858799211,4cyclictest0-21swapper/211:14:192
2858299217,7cyclictest18245-1kworker/u9:211:49:231
28582992116,3cyclictest6013-21fschecks_count11:10:151
2858299211,4cyclictest11641-21sh12:35:201
2858299211,3cyclictest2145-21vmstat09:45:271
2858299211,18cyclictest0-21swapper/109:35:141
2858299211,18cyclictest0-21swapper/109:35:141
2858299211,18cyclictest0-21swapper/108:09:171
2858299211,12cyclictest0-21swapper/109:18:501
2858299210,19cyclictest0-21swapper/107:15:321
2858299210,15cyclictest0-21swapper/112:01:011
2858299210,14cyclictest0-21swapper/109:58:021
2857499219,3cyclictest31676-21/usr/sbin/munin07:15:140
2857499211,5cyclictest0-21swapper/012:04:550
2857499211,10cyclictest5599-21sed09:55:170
2857499211,10cyclictest0-21swapper/011:10:240
2858799208,4cyclictest17263-21fschecks_count07:55:152
2858799207,3cyclictest311ktimersoftd/208:30:022
2858799207,3cyclictest311ktimersoftd/208:30:022
2858799206,3cyclictest87650irq/129-enp1s0-10:45:152
2858799204,8cyclictest87350irq/126-enp1s0-10:10:262
2858799202,2cyclictest87650irq/129-enp1s0-09:20:162
2858799201,7cyclictest0-21swapper/210:00:342
28587992017,2cyclictest0-21swapper/207:18:342
2858799201,5cyclictest0-21swapper/212:30:262
2858799201,5cyclictest0-21swapper/211:59:212
2858799201,5cyclictest0-21swapper/211:19:442
2858799201,5cyclictest0-21swapper/210:53:182
2858799201,5cyclictest0-21swapper/210:53:182
2858799201,5cyclictest0-21swapper/209:57:002
28587992011,2cyclictest87350irq/126-enp1s0-10:55:192
2858799201,10cyclictest0-21swapper/212:36:012
2858299209,3cyclictest0-21swapper/112:32:271
2858299207,3cyclictest0-21swapper/111:07:281
2858299206,3cyclictest0-21swapper/112:25:011
2858299202,5cyclictest18245-1kworker/u9:211:34:011
2858299201,2cyclictest0-21swapper/108:35:111
2858299201,18cyclictest0-21swapper/108:49:561
2858299201,18cyclictest0-21swapper/108:10:111
2858299201,17cyclictest0-21swapper/107:56:351
2858299201,11cyclictest12159-21sshd07:43:441
2858299200,6cyclictest0-21swapper/110:06:261
2858299200,4cyclictest0-21swapper/109:44:281
2858299200,18cyclictest0-21swapper/107:20:161
2858299200,14cyclictest0-21swapper/110:14:271
2858299200,13cyclictest0-21swapper/111:54:441
2857499202,3cyclictest0-21swapper/012:24:300
2857499201,9cyclictest0-21swapper/010:52:060
2857499201,9cyclictest0-21swapper/010:52:050
2857499201,4cyclictest17288-21diskmemload11:21:530
2857499201,11cyclictest0-21swapper/010:35:570
2857499200,10cyclictest0-21swapper/010:09:020
209992200,2sleep2311ktimersoftd/211:45:132
141342201,3sleep00-21swapper/007:45:530
49982191,2sleep05176-21unixbench-2d07:25:270
2858799197,2cyclictest87350irq/126-enp1s0-11:01:002
2858799191,8cyclictest579-21systemd-journal10:30:012
2858799191,6cyclictest4584-21chrt07:25:192
2858799191,6cyclictest0-21swapper/212:29:362
2858799191,5cyclictest0-21swapper/208:00:232
2858799191,10cyclictest0-21swapper/212:14:162
2858299196,4cyclictest18245-1kworker/u9:210:15:101
2858299196,4cyclictest0-21swapper/111:20:121
2858299194,8cyclictest755-21rngd10:52:571
2858299194,8cyclictest755-21rngd10:52:571
2858299193,15cyclictest0-21swapper/110:04:251
2858299192,14cyclictest9981-21smart_sda08:50:231
28582991915,3cyclictest0-21swapper/108:00:361
2858299191,16cyclictest0-21swapper/108:15:561
2858299191,10cyclictest3038-21expr09:50:121
2858299190,4cyclictest0-21swapper/110:33:281
2858299190,4cyclictest0-21swapper/107:25:121
2858299190,3cyclictest0-21swapper/107:36:001
2858299190,13cyclictest0-21swapper/111:00:111
2858299190,12cyclictest0-21swapper/111:28:241
2857499198,2cyclictest12416-21/usr/sbin/munin10:10:170
2857499194,9cyclictest0-21swapper/008:25:140
2857499194,9cyclictest0-21swapper/008:25:140
2857499191,6cyclictest0-21swapper/012:06:260
2857499191,3cyclictest0-21swapper/009:10:300
2857499191,10cyclictest0-21swapper/011:49:180
2857499191,10cyclictest0-21swapper/010:56:150
2857499190,9cyclictest0-21swapper/010:40:150
2857499190,9cyclictest0-21swapper/010:02:230
39712181,2sleep00-21swapper/008:38:380
2858799188,2cyclictest12050irq/123-ahci[0008:45:022
2858799187,3cyclictest87350irq/126-enp1s0-10:15:152
2858799185,2cyclictest17291-21munin-run09:10:012
2858799181,8cyclictest12050irq/123-ahci[0007:35:012
2858799181,7cyclictest9948-21gltestperf10:05:162
2858799181,5cyclictest30-21rcuc/208:30:182
28587991813,3cyclictest16357-21fschecks_count10:20:152
2858299185,8cyclictest0-21swapper/110:42:211
2858299185,4cyclictest31738-21fschecks_count10:55:151
28582991815,2cyclictest0-21swapper/109:04:081
2858299181,4cyclictest0-21swapper/111:36:211
28582991814,3cyclictest0-21swapper/108:21:481
2858299181,3cyclictest22257-21df_inode09:20:131
2858299181,3cyclictest0-21swapper/107:51:431
2858299181,16cyclictest0-21swapper/108:41:031
2858299181,16cyclictest0-21swapper/108:26:131
2858299181,16cyclictest0-21swapper/108:26:121
2858299181,15cyclictest0-21swapper/108:55:491
28582991811,3cyclictest0-21swapper/108:30:131
2858299181,10cyclictest0-21swapper/107:45:161
2858299180,11cyclictest0-21swapper/112:12:011
2857499188,2cyclictest0-21swapper/012:25:170
2857499187,3cyclictest0-21swapper/007:20:160
2857499181,4cyclictest19295-21tr10:25:250
2857499181,4cyclictest14696-21/usr/sbin/munin07:50:200
2857499181,15cyclictest0-21swapper/009:01:010
2857499181,14cyclictest27117-21irqstats09:30:180
2857499181,10cyclictest0-21swapper/011:00:270
2857499180,5cyclictest3847-21kworker/u8:212:35:400
2857499180,5cyclictest0-21swapper/010:45:090
2857499180,11cyclictest0-21swapper/012:11:020
162232180,5sleep121-21rcuc/109:05:181
2858799173,7cyclictest311ktimersoftd/207:40:122
2858799172,7cyclictest87550irq/128-enp1s0-08:05:252
2858799171,6cyclictest0-21swapper/210:36:112
2858799171,4cyclictest0-21swapper/207:35:162
2858799171,3cyclictest0-21swapper/211:21:072
28587991713,3cyclictest32-21ksoftirqd/207:20:192
2858799170,8cyclictest311ktimersoftd/208:45:122
2857499174,4cyclictest5037-21idleruntime08:40:160
2857499173,4cyclictest15320-21/usr/sbin/munin09:05:130
2857499171,5cyclictest10073-21latency_hist07:40:020
2857499171,3cyclictest0-21swapper/008:24:280
28574991711,3cyclictest9320-21gltestperf08:50:160
28574991710,4cyclictest24801-21switchtime08:10:230
2857499170,10cyclictest0-21swapper/011:55:240
2858799168,3cyclictest87350irq/126-enp1s0-08:50:442
2858799162,5cyclictest11975-21munin-node07:10:292
2858799162,5cyclictest11975-21munin-node07:10:292
2858799161,8cyclictest0-21swapper/207:55:012
2858799161,7cyclictest808-21snmpd08:24:452
2858799161,6cyclictest0-21swapper/209:00:122
2858799161,4cyclictest0-21swapper/208:35:152
2858799161,3cyclictest87350irq/126-enp1s0-08:55:122
28574991610,3cyclictest21411-21cpuspeed_turbos08:05:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional