You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 05:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu May 14, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13642238163,24sleep20-21swapper/219:08:472
12092226194,21sleep30-21swapper/319:06:473
11892202164,25sleep10-21swapper/119:06:331
11332199162,25sleep00-21swapper/019:05:470
1570998683,2cyclictest0-21swapper/223:33:282
1570997673,2cyclictest0-21swapper/219:38:542
1570997673,2cyclictest0-21swapper/219:38:542
1570997471,2cyclictest0-21swapper/200:13:402
59142640,4sleep341-21ksoftirqd/320:30:153
188222510,4sleep230-21rcuc/221:00:152
32402470,4sleep011-21rcuc/019:10:250
1570994643,2cyclictest0-21swapper/219:47:332
155099431,33cyclictest21897-21/usr/sbin/munin23:35:250
155099391,24cyclictest13445-21/usr/sbin/munin19:35:120
155099391,24cyclictest13445-21/usr/sbin/munin19:35:120
1570993835,2cyclictest0-21swapper/219:11:052
155099371,3cyclictest10745-21tune2fs21:55:130
155099371,23cyclictest16258-21/usr/sbin/munin20:55:210
1550993712,22cyclictest10611-21smart_sda00:20:230
155099363,7cyclictest25061-21/usr/sbin/munin21:15:200
155099361,23cyclictest20672-21needreboot23:30:180
1550993611,22cyclictest31647-21/usr/sbin/munin21:30:250
155099358,4cyclictest0-21swapper/000:35:230
155099356,3cyclictest0-21swapper/021:36:440
155099350,9cyclictest0-21swapper/022:00:220
155099341,8cyclictest15536-21sshd00:32:530
155099341,6cyclictest9162-21cut00:20:010
1550993414,17cyclictest20202-21tr19:50:100
155099341,29cyclictest6385-21pmu-power20:30:200
157799331,21cyclictest0-21swapper/319:18:193
157799330,12cyclictest0-21swapper/322:50:063
155099334,3cyclictest87550irq/128-enp1s0-22:40:170
155099328,21cyclictest4676-21grep21:40:180
155099328,1cyclictest0-21swapper/022:50:250
155099328,11cyclictest22653-21idleruntime-cro21:10:010
155099324,3cyclictest87550irq/128-enp1s0-22:45:180
155099323,6cyclictest9971-21smart_sda23:05:230
155099323,3cyclictest30801-21latency_hist20:15:010
155099323,19cyclictest0-21swapper/000:09:540
155099321,3cyclictest25358-21smartctl23:40:230
155099321,24cyclictest5637-21kworker/0:021:50:120
1550993211,18cyclictest26642-21/usr/sbin/munin20:05:240
1550993210,19cyclictest11545-21/usr/sbin/munin00:25:220
157799311,11cyclictest0-21swapper/319:48:263
157799311,11cyclictest0-21swapper/300:25:103
155099319,19cyclictest31060-21fschecks_count23:55:140
155099318,3cyclictest30480-21awk23:55:010
155099318,3cyclictest27174-21cat23:45:190
155099318,3cyclictest16894-21/usr/sbin/munin19:40:230
155099318,20cyclictest5999-21tr21:45:120
155099318,19cyclictest31561-21cat21:30:020
155099311,4cyclictest26423-21basename20:05:010
157799301,21cyclictest0-21swapper/323:51:423
157799300,10cyclictest0-21swapper/323:18:103
157799300,10cyclictest0-21swapper/323:18:103
155099309,18cyclictest827-21runrttasks19:58:000
155099308,3cyclictest1802-21smart_sda00:00:220
155099302,7cyclictest14102-21needreboot23:15:190
155099302,7cyclictest14102-21needreboot23:15:190
155099302,6cyclictest0-21swapper/020:35:220
155099302,2cyclictest0-21swapper/022:06:350
155099301,5cyclictest822-21/usr/sbin/munin20:20:170
155099301,5cyclictest5724-21ntp_states19:15:210
155099301,4cyclictest14382-21expr20:50:120
155099301,4cyclictest14382-21expr20:50:120
1550993012,15cyclictest5243-21cut20:30:010
1550993011,16cyclictest12275-21seq19:30:190
1550993010,17cyclictest21562-21ls22:20:120
155099300,9cyclictest0-21swapper/023:20:240
157799291,13cyclictest0-21swapper/320:17:113
157799291,11cyclictest0-21swapper/320:05:103
157799290,12cyclictest0-21swapper/321:28:303
157799290,11cyclictest0-21swapper/300:02:063
155099298,18cyclictest23010-21tr21:10:130
155099297,3cyclictest18548-21tr21:00:120
155099296,3cyclictest19494-21grep23:30:010
155099292,4cyclictest27652-21munin-run22:35:020
155099292,4cyclictest18932-40dnf19:45:230
1550992911,15cyclictest6774-21seq00:13:120
1550992911,15cyclictest19323-21expr22:15:120
1550992910,16cyclictest14027-21latency_hist20:50:020
157799281,12cyclictest0-21swapper/300:32:293
157799281,11cyclictest0-21swapper/319:20:103
157799281,10cyclictest0-21swapper/321:46:503
157799280,12cyclictest0-21swapper/322:10:513
157799280,11cyclictest0-21swapper/323:47:023
1565992810,8cyclictest0-21swapper/122:55:001
155099289,16cyclictest827-21runrttasks19:29:070
155099289,16cyclictest27397-21expr21:20:130
155099286,4cyclictest24271-21if_err_enp2s022:25:180
155099286,4cyclictest11954-21sadc23:10:200
155099285,3cyclictest4311-21/usr/sbin/munin22:55:170
155099281,5cyclictest635-21cat20:20:010
155099280,6cyclictest0-21swapper/023:00:240
157799271,9cyclictest0-21swapper/320:38:343
157799271,11cyclictest0-21swapper/319:10:483
157799270,12cyclictest0-21swapper/319:40:103
157799270,11cyclictest0-21swapper/322:22:343
157799270,11cyclictest0-21swapper/321:13:023
155099279,15cyclictest827-21runrttasks20:41:270
155099276,3cyclictest17012-21cpuspeed_turbos22:10:110
155099272,22cyclictest27875-21/usr/sbin/munin22:35:180
155099271,4cyclictest8834-21awk19:25:010
157799265,3cyclictest0-21swapper/321:07:583
157799260,2cyclictest0-21swapper/322:19:553
157799260,13cyclictest0-21swapper/323:26:383
157799260,11cyclictest0-21swapper/322:01:473
157799260,11cyclictest0-21swapper/320:20:033
157799260,11cyclictest0-21swapper/300:10:143
157799260,10cyclictest0-21swapper/323:36:103
157799260,10cyclictest0-21swapper/321:24:343
157799260,10cyclictest0-21swapper/300:16:023
1565992611,13cyclictest7425-21kworker/u8:221:16:181
157799251,9cyclictest0-21swapper/323:14:263
157799251,11cyclictest0-21swapper/319:28:463
157799251,10cyclictest20712-21hddtemp_smartct19:50:143
157799250,12cyclictest0-21swapper/320:54:503
157799250,12cyclictest0-21swapper/320:54:503
157799250,10cyclictest0-21swapper/321:52:383
157799250,10cyclictest0-21swapper/321:36:503
157799250,10cyclictest0-21swapper/320:58:163
156599254,18cyclictest6138-21latency20:30:171
1565992514,3cyclictest7929-21chrt19:20:211
157799246,3cyclictest0-21swapper/300:06:423
157799245,3cyclictest0-21swapper/300:22:103
157799241,9cyclictest0-21swapper/320:42:303
157799241,9cyclictest0-21swapper/319:58:503
157799241,9cyclictest0-21swapper/319:35:103
157799241,9cyclictest0-21swapper/319:35:103
157799241,12cyclictest0-21swapper/323:55:203
157799241,12cyclictest0-21swapper/321:17:143
157799241,11cyclictest0-21swapper/320:02:223
157799241,10cyclictest12158-21memory19:30:183
157799240,9cyclictest0-21swapper/322:34:143
157799240,16cyclictest0-21swapper/322:40:163
157799240,12cyclictest0-21swapper/322:26:443
157099241,7cyclictest0-21swapper/200:39:052
156599249,13cyclictest30059-21kworker/u8:122:42:521
1565992418,3cyclictest21860-21cut23:35:021
241612230,2sleep20-21swapper/223:40:112
157799238,3cyclictest0-21swapper/320:46:543
157799231,9cyclictest0-21swapper/300:39:053
157799231,6cyclictest0-21swapper/322:45:543
157799231,12cyclictest0-21swapper/321:43:103
1577992310,3cyclictest0-21swapper/321:00:503
157799230,9cyclictest0-21swapper/320:27:473
157799230,11cyclictest0-21swapper/322:37:043
156599232,15cyclictest0-21swapper/119:35:541
156599232,15cyclictest0-21swapper/119:35:541
156599231,19cyclictest16041-21perf20:55:011
156599231,19cyclictest16041-21perf20:55:011
39282221,2sleep20-21swapper/221:40:142
157799221,7cyclictest9295-21gltestperf23:05:143
157799220,9cyclictest0-21swapper/321:55:193
157799220,10cyclictest0-21swapper/323:22:283
157799220,10cyclictest0-21swapper/323:00:283
156599224,11cyclictest23-21ksoftirqd/123:10:151
156599222,2cyclictest23-21ksoftirqd/119:45:231
156599221,9cyclictest0-21swapper/100:16:211
156599221,19cyclictest0-21swapper/100:20:141
156599221,18cyclictest18804-21diskstats21:00:151
156599221,18cyclictest15643-21df23:20:151
156599221,16cyclictest0-21swapper/121:46:341
156599221,16cyclictest0-21swapper/120:45:241
156599221,15cyclictest0-21swapper/122:25:281
156599221,15cyclictest0-21swapper/122:01:331
156599221,12cyclictest16287-21diskstats00:35:131
156599221,10cyclictest21-21rcuc/123:15:161
156599221,10cyclictest21-21rcuc/123:15:161
157799216,3cyclictest0-21swapper/321:30:143
157799211,11cyclictest14690-21/usr/sbin/munin22:05:133
157099212,8cyclictest7558-40dnf21:46:242
157099211,7cyclictest27032-21munin-run21:20:012
156599213,10cyclictest3881-21munin-node22:55:281
156599212,14cyclictest0-21swapper/120:29:001
156599212,11cyclictest0-21swapper/100:10:001
156599211,3cyclictest25216-21cat20:00:171
156599211,2cyclictest0-21swapper/119:50:101
156599211,15cyclictest0-21swapper/121:23:051
156599211,15cyclictest0-21swapper/120:59:431
156599211,15cyclictest0-21swapper/119:27:341
156599211,14cyclictest0-21swapper/123:49:361
156599211,14cyclictest0-21swapper/121:08:061
156599211,14cyclictest0-21swapper/120:18:331
156599211,14cyclictest0-21swapper/119:14:101
156599211,11cyclictest24129-21smart_sda21:10:261
1565992110,5cyclictest16487-21kworker/u8:222:35:461
1565992110,3cyclictest9038-21unixbench_singl20:35:261
156599210,20cyclictest0-21swapper/123:42:151
157799201,10cyclictest25555-21unixbench-2d23:40:243
157099204,7cyclictest0-21swapper/221:35:242
157099203,14cyclictest6731-21grep20:30:232
157099201,8cyclictest579-21systemd-journal23:25:012
1570992015,3cyclictest11756-21grep21:55:232
157099201,16cyclictest8138-21hddtemp_smartct20:35:162
156599207,4cyclictest30589-21cat23:55:031
156599205,2cyclictest21-21rcuc/121:55:111
1565992017,2cyclictest0-21swapper/122:15:211
1565992016,3cyclictest0-21swapper/123:58:161
1565992013,2cyclictest0-21swapper/120:05:101
156599201,17cyclictest22638-21diskmemload22:45:481
156599201,14cyclictest0-21swapper/121:42:251
156599201,14cyclictest0-21swapper/121:38:081
156599201,14cyclictest0-21swapper/121:28:511
156599201,14cyclictest0-21swapper/120:43:261
156599201,14cyclictest0-21swapper/119:55:591
156599201,14cyclictest0-21swapper/119:16:291
156599201,11cyclictest6847-21df_inode23:00:131
156599201,11cyclictest30097-21smart_sda20:10:231
156599201,10cyclictest19055-21cat22:15:011
156599200,5cyclictest0-21swapper/100:02:501
157099197,4cyclictest30-21rcuc/220:25:252
157099191,7cyclictest0-21swapper/223:17:482
157099191,7cyclictest0-21swapper/223:17:482
157099191,6cyclictest0-21swapper/223:29:102
1570991913,3cyclictest31857-21seq23:55:222
157099191,12cyclictest0-21swapper/219:30:112
157099191,11cyclictest0-21swapper/221:21:562
156652190,2sleep20-21swapper/200:34:342
1565991916,1cyclictest0-21swapper/100:25:221
156599191,3cyclictest11661-21/usr/sbin/munin19:30:151
1565991913,4cyclictest22963-21ntp_states23:35:221
1565991913,2cyclictest0-21swapper/122:05:211
156599191,17cyclictest0-21swapper/100:33:571
156599191,16cyclictest0-21swapper/123:27:201
156599191,14cyclictest0-21swapper/121:54:421
156599190,14cyclictest0-21swapper/119:41:261
156599190,11cyclictest0-21swapper/120:20:191
15012191,3sleep20-21swapper/222:45:382
157799184,3cyclictest0-21swapper/323:32:503
157799180,9cyclictest0-21swapper/322:59:303
157099182,5cyclictest14711-21egrep22:05:112
157099181,9cyclictest0-21swapper/220:55:242
157099181,8cyclictest790-21in:imjournal20:43:442
157099181,8cyclictest0-21swapper/223:02:542
157099181,8cyclictest0-21swapper/222:10:162
157099181,7cyclictest10042-21needreboot19:25:192
157099181,3cyclictest87650irq/129-enp1s0-22:44:252
1570991813,3cyclictest2157-21grep20:20:242
1570991812,4cyclictest15793-21tr19:40:122
1570991812,3cyclictest2742-21gltestperf22:50:142
157099180,5cyclictest0-21swapper/200:05:012
156599186,5cyclictest31409-21kworker/u8:121:30:511
1565991814,3cyclictest0-21swapper/100:10:291
1565991812,2cyclictest0-21swapper/122:22:171
156599181,16cyclictest0-21swapper/123:05:521
156599181,16cyclictest0-21swapper/122:30:161
121902180,4sleep232-21ksoftirqd/220:45:142
46482171,3sleep20-21swapper/219:15:052
298862170,2sleep30-21swapper/320:10:213
157099174,5cyclictest0-21swapper/200:27:542
157099174,4cyclictest22231-21cat22:20:182
157099173,2cyclictest87550irq/128-enp1s0-20:05:262
157099172,5cyclictest0-21swapper/222:35:202
157099171,7cyclictest31845-21irqstats20:15:192
157099171,7cyclictest26909-21fgrep22:30:222
157099171,7cyclictest25005-21cat21:15:012
157099171,7cyclictest22638-21diskmemload22:55:202
157099171,6cyclictest0-21swapper/223:36:192
157099171,6cyclictest0-21swapper/221:31:462
157099171,6cyclictest0-21swapper/219:20:242
157099171,6cyclictest0-21swapper/200:15:432
157099171,5cyclictest0-21swapper/200:21:042
157099171,4cyclictest11351-21cut23:10:142
157099171,14cyclictest20774-21cut21:05:112
157099171,14cyclictest0-21swapper/219:55:032
157099170,6cyclictest10198-21latency_hist21:55:022
157099165,3cyclictest87350irq/126-enp1s0-22:15:112
157099164,5cyclictest0-21swapper/222:00:102
157099161,7cyclictest0-21swapper/221:29:392
157099161,6cyclictest30925-21date20:15:012
157099161,6cyclictest27363-21ntp_states23:45:222
157099161,6cyclictest14736-21hddtemp_smartct20:50:152
157099161,6cyclictest14736-21hddtemp_smartct20:50:152
157099161,6cyclictest0-21swapper/223:05:192
157099161,5cyclictest0-21swapper/219:50:162
157099161,2cyclictest0-21swapper/223:50:112
1570991611,3cyclictest3980-21find00:05:232
1570991611,3cyclictest25469-21ntp_kernel_err20:00:222
1570991611,3cyclictest24163-21hwlatdetect22:25:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional