You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 21:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Jan 10, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302712237164,25sleep20-21swapper/207:07:142
302712237164,25sleep20-21swapper/207:07:142
302072203165,25sleep10-21swapper/107:06:241
302072203165,25sleep10-21swapper/107:06:241
303862200164,24sleep00-21swapper/007:08:460
303862200164,24sleep00-21swapper/007:08:450
302722199163,25sleep30-21swapper/307:07:153
302722199163,25sleep30-21swapper/307:07:153
200921650,5sleep23058199cyclictest10:55:202
296152550,1sleep10-21swapper/110:45:161
168332530,1sleep00-21swapper/010:15:210
117352530,4sleep011-21rcuc/010:05:140
51062520,4sleep25108-21needreboot08:35:192
79792500,1sleep00-21swapper/009:55:180
67592440,1sleep30-21swapper/308:40:173
3056799378,13cyclictest2312-21munin-run09:45:020
3056799335,13cyclictest25273-21perf08:10:010
3056799333,14cyclictest6091-21munin-run08:40:010
3056799323,13cyclictest24296-21munin-run10:35:010
30567993210,10cyclictest9545-21perf11:15:010
3056799312,16cyclictest20516-21munin-run11:40:010
3058199302,11cyclictest31139-21kworker/2:208:45:032
30588992915,8cyclictest0-21swapper/312:19:193
30588992914,9cyclictest0-21swapper/310:28:563
30588992913,9cyclictest0-21swapper/311:55:493
3058899291,20cyclictest0-21swapper/310:04:013
3058899291,19cyclictest0-21swapper/310:05:543
3056799296,11cyclictest13937-21perf11:25:010
3058899281,19cyclictest0-21swapper/309:56:273
3056799289,16cyclictest19904-21munin-run10:25:000
30588992713,8cyclictest0-21swapper/310:17:263
30588992713,5cyclictest0-21swapper/311:14:313
30588992712,9cyclictest0-21swapper/311:40:073
3058899271,19cyclictest0-21swapper/309:25:303
3058899271,18cyclictest0-21swapper/311:06:423
3058899271,18cyclictest0-21swapper/311:04:093
327172260,1sleep10-21swapper/112:05:181
30588992614,2cyclictest24000-21ntp_states11:45:193
3058899261,19cyclictest0-21swapper/310:34:233
3058899261,18cyclictest0-21swapper/311:22:123
3058899261,17cyclictest0-21swapper/312:02:063
3058899261,17cyclictest0-21swapper/309:16:403
3058899260,19cyclictest0-21swapper/310:20:413
3058899260,17cyclictest0-21swapper/310:35:443
3056799263,11cyclictest11121-21awk10:05:010
3056799261,12cyclictest4523-21basename09:50:010
3056799261,12cyclictest27455-21munin-run08:15:010
83192250,4sleep011-21rcuc/007:30:150
30588992517,2cyclictest0-21swapper/311:15:313
30588992514,3cyclictest0-21swapper/312:10:123
3058899251,18cyclictest0-21swapper/311:38:413
3058899251,17cyclictest0-21swapper/312:34:533
3058899251,17cyclictest0-21swapper/310:41:103
3058899251,17cyclictest0-21swapper/309:52:373
3058899250,19cyclictest0-21swapper/309:11:293
3058899250,17cyclictest0-21swapper/309:41:383
30581992521,3cyclictest0-21swapper/211:44:172
3058199251,12cyclictest31139-21kworker/2:211:05:382
3057599252,9cyclictest19287-21diskmemload12:10:591
3056799251,21cyclictest16686-21/usr/sbin/munin07:50:220
3056799251,21cyclictest10097-21/usr/sbin/munin07:35:220
3056799251,20cyclictest11519-21pmu-power12:30:240
3058899244,13cyclictest31103-21kworker/u8:210:55:433
3058899244,12cyclictest14636-21kworker/u8:010:48:303
30588992412,3cyclictest0-21swapper/309:34:463
30588992412,3cyclictest0-21swapper/309:34:463
3058899241,10cyclictest8710-21grep12:25:173
3057599247,9cyclictest8075-21grep11:10:141
3056799241,20cyclictest0-21swapper/009:05:570
30567992411,3cyclictest87350irq/128-enp1s0-09:29:040
3056799241,11cyclictest30775-21/usr/sbin/munin07:10:250
3056799241,11cyclictest14144-21/usr/sbin/munin11:25:090
3056799240,21cyclictest6698-21basename12:20:150
71312231,3sleep37289-21users07:25:263
4342231,18sleep1540-21memory08:25:191
314092230,1sleep00-21swapper/010:50:120
3058899231,16cyclictest0-21swapper/311:31:593
3058899231,15cyclictest0-21swapper/311:53:403
3058899231,15cyclictest0-21swapper/310:50:303
3058899231,15cyclictest0-21swapper/309:39:183
3058899231,15cyclictest0-21swapper/309:24:263
3058899231,14cyclictest26994-21kworker/u8:112:22:333
3058899231,11cyclictest26994-21sed08:10:233
3058899230,14cyclictest0-21swapper/312:38:393
3058199235,2cyclictest31139-21kworker/2:212:35:022
3057599238,4cyclictest0-21swapper/110:07:381
3057599234,12cyclictest0-21swapper/110:25:221
3057599231,3cyclictest5756-21munin-run12:20:011
3057599231,17cyclictest0-21swapper/109:33:591
3057599231,17cyclictest0-21swapper/109:33:591
3057599231,15cyclictest0-21swapper/110:54:171
3057599231,14cyclictest0-21swapper/109:55:101
3056799231,2cyclictest12892-21grep07:40:150
3056799231,19cyclictest814-21snmpd10:13:120
3056799231,10cyclictest814-21snmpd12:08:340
30567992310,3cyclictest0-21swapper/009:30:560
30567992310,3cyclictest0-21swapper/009:30:560
114502231,18sleep311644-21grep07:35:253
3058899229,3cyclictest12903-21/usr/sbin/munin08:55:233
3058899222,15cyclictest87150irq/126-enp1s0-10:10:413
30588992218,3cyclictest0-21swapper/308:50:233
3058899221,6cyclictest3889-21sh08:35:013
3058899221,13cyclictest32418-21hddtemp_smartct12:05:153
3058199226,2cyclictest9988-21kworker/2:008:15:122
30581992216,4cyclictest29616-21basename10:45:162
30581992210,2cyclictest31139-21kworker/2:209:37:302
3057599228,9cyclictest0-21swapper/112:32:201
3057599222,15cyclictest0-21swapper/111:22:521
30575992218,2cyclictest13250-21diskstats08:55:121
30575992214,3cyclictest0-21swapper/111:28:231
3057599221,16cyclictest0-21swapper/109:44:321
3057599220,6cyclictest0-21swapper/112:00:001
3056799229,4cyclictest161462ntp_kernel_pll_09:00:200
3056799225,3cyclictest0-21swapper/012:19:220
3056799221,5cyclictest3231-40dnf12:11:000
3056799221,10cyclictest8512-21/usr/sbin/munin08:45:160
3056799221,10cyclictest814-21snmpd11:18:230
3056799221,10cyclictest0-21swapper/008:17:580
3056799220,20cyclictest0-21swapper/010:25:100
3058899214,9cyclictest0-21swapper/309:46:573
3058199219,4cyclictest0-21swapper/212:10:262
3058199219,2cyclictest31139-21kworker/2:212:05:342
3058199218,2cyclictest9988-21kworker/2:007:26:582
3058199218,2cyclictest31139-21kworker/2:210:38:432
3058199211,7cyclictest0-21swapper/211:10:162
30581992111,2cyclictest31139-21kworker/2:209:25:062
3057599216,10cyclictest87350irq/128-enp1s0-11:51:181
3057599212,17cyclictest0-21swapper/108:08:101
3057599212,17cyclictest0-21swapper/107:25:311
3057599212,17cyclictest0-21swapper/107:17:511
30575992117,3cyclictest0-21swapper/107:52:131
30575992116,3cyclictest32222-21sed07:10:251
3057599211,3cyclictest0-21swapper/111:41:021
3057599211,18cyclictest0-21swapper/110:16:261
3057599211,18cyclictest0-21swapper/110:11:591
3057599211,17cyclictest23760-21ls10:30:261
3057599211,15cyclictest0-21swapper/111:33:451
3057599211,15cyclictest0-21swapper/111:05:371
3057599211,13cyclictest0-21swapper/110:01:161
3057599211,12cyclictest11684-21needreboot08:50:181
3057599211,12cyclictest0-21swapper/109:15:151
3057599211,11cyclictest28020-21df_inode08:15:121
3057599211,11cyclictest21078-21/usr/sbin/munin08:00:191
3057599211,11cyclictest19457-21memory11:35:171
3057599211,11cyclictest14274-21awk07:45:011
3057599211,11cyclictest0-21swapper/112:24:571
3057599211,10cyclictest0-21swapper/109:21:551
3057599210,19cyclictest0-21swapper/109:45:231
3056799218,2cyclictest27332-21/usr/sbin/munin11:55:200
3056799217,3cyclictest32082-21/usr/sbin/munin08:25:200
3056799217,3cyclictest25133-21/usr/sbin/munin11:50:240
30567992115,3cyclictest9060-21cat12:25:200
3058899202,9cyclictest12638-21diskstats07:40:123
3058899201,13cyclictest0-21swapper/307:55:253
3058899201,13cyclictest0-21swapper/307:18:043
30588992011,2cyclictest8246-21taskset07:30:143
3058899201,12cyclictest0-21swapper/308:00:123
3058899201,10cyclictest5372-21mailstats08:35:223
3058899201,10cyclictest16645-21unixbench_singl09:00:273
3058199204,3cyclictest31139-21kworker/2:210:30:232
3058199202,9cyclictest31139-21kworker/2:211:15:342
3058199202,11cyclictest31139-21kworker/2:211:48:382
3058199201,9cyclictest31139-21kworker/2:211:57:302
3058199201,9cyclictest15519-21cut09:00:152
30581992013,4cyclictest24957-21systemd-journal09:20:182
3058199201,11cyclictest7737-21interrupts09:55:162
3057599207,4cyclictest0-21swapper/109:37:591
3057599205,12cyclictest29657-21cpuspeed_turbos12:00:101
30575992017,2cyclictest0-21swapper/108:24:401
30575992016,3cyclictest0-21swapper/110:41:371
30575992016,3cyclictest0-21swapper/109:27:251
30575992016,3cyclictest0-21swapper/107:33:461
30575992015,3cyclictest5720-21strings09:50:191
3057599201,17cyclictest0-21swapper/108:31:261
3057599201,17cyclictest0-21swapper/107:45:591
3057599201,15cyclictest0-21swapper/111:47:271
3057599201,15cyclictest0-21swapper/111:19:501
3057599201,14cyclictest0-21swapper/112:27:161
3057599201,10cyclictest0-21swapper/107:55:201
3057599200,18cyclictest0-21swapper/109:08:581
3057599200,13cyclictest0-21swapper/112:39:381
3056799207,3cyclictest87350irq/128-enp1s0-11:32:150
3056799201,9cyclictest814-21snmpd12:36:460
3056799201,5cyclictest0-21swapper/009:51:010
3056799201,3cyclictest0-21swapper/008:52:590
257042201,3sleep00-21swapper/009:23:010
324622191,2sleep30-21swapper/307:11:333
3058899195,3cyclictest87450irq/129-enp1s0-07:50:183
3058899195,3cyclictest30872-21strings08:20:193
3058899191,6cyclictest696-21ntpq08:25:213
3058899191,11cyclictest0-21swapper/311:25:073
3058199199,2cyclictest31139-21kworker/2:210:20:142
3058199194,8cyclictest31139-21kworker/2:208:30:582
3058199193,3cyclictest9988-21kworker/2:007:35:142
3058199192,9cyclictest9988-21kworker/2:007:20:382
3058199192,11cyclictest0-21swapper/211:30:162
30581991914,3cyclictest11936-21grep10:05:142
30581991913,3cyclictest21191-21ntpq10:25:202
3058199191,16cyclictest0-21swapper/212:20:172
30581991911,5cyclictest32-21ksoftirqd/208:04:572
3057599194,5cyclictest763-21in:imjournal10:25:011
3057599192,16cyclictest0-21swapper/109:10:501
30575991916,2cyclictest0-21swapper/109:02:071
30575991916,2cyclictest0-21swapper/108:43:591
30575991913,1cyclictest0-21swapper/111:03:351
3057599191,16cyclictest0-21swapper/108:45:091
3057599191,10cyclictest0-21swapper/110:36:161
3057599190,6cyclictest0-21swapper/110:56:381
3056799191,5cyclictest0-21swapper/010:41:430
3056799191,3cyclictest7391-21ntp_states08:40:240
3056799191,3cyclictest15968-21sed07:45:260
30567991913,3cyclictest2851-21grep07:15:250
264172190,5sleep13057599cyclictest08:10:181
3058899181,7cyclictest18820-21grep09:05:283
30588991812,3cyclictest4541-21ntp_kernel_pll_07:20:213
3058199185,5cyclictest12694-21awk08:55:012
3058199185,4cyclictest4717-21cat12:15:192
3058199185,2cyclictest31139-21kworker/2:210:53:462
3058199185,2cyclictest31139-21kworker/2:209:43:342
3058199181,7cyclictest31139-21kworker/2:212:25:342
3058199181,7cyclictest0-21swapper/209:10:252
3058199181,3cyclictest0-21swapper/211:20:142
30581991813,3cyclictest32749-21munin-plugin-st07:15:022
30581991812,3cyclictest28554-21ls09:30:132
30581991812,3cyclictest28554-21ls09:30:132
30581991812,3cyclictest13205-21memory07:40:172
3058199181,10cyclictest0-21swapper/211:00:332
3058199181,10cyclictest0-21swapper/210:41:052
3057599181,9cyclictest0-21swapper/107:20:191
3057599181,10cyclictest0-21swapper/107:35:161
3057599180,16cyclictest0-21swapper/108:35:181
3056799185,2cyclictest0-21swapper/008:20:220
3056799184,3cyclictest87150irq/126-enp1s0-07:55:210
3056799183,3cyclictest3583-21grep07:20:120
3056799181,9cyclictest0-21swapper/011:49:030
3056799181,9cyclictest0-21swapper/009:40:000
3056799181,7cyclictest30500-21users10:45:250
3056799181,6cyclictest0-21swapper/011:01:040
3056799181,5cyclictest933-21/usr/sbin/munin10:55:230
3056799181,5cyclictest24867-21diskstats10:35:150
3056799181,5cyclictest23669-21latency_hist09:20:010
3056799181,15cyclictest22222-21sed11:40:250
3056799181,14cyclictest7167-21grep07:25:250
161812181,2sleep30-21swapper/307:46:363
70292170,2sleep00-21swapper/011:06:150
3058899172,8cyclictest87450irq/129-enp1s0-08:45:113
3058899171,3cyclictest87450irq/129-enp1s0-08:15:173
3058199178,2cyclictest31139-21kworker/2:208:55:142
3058199174,3cyclictest31139-21kworker/2:212:01:022
3058199174,2cyclictest9988-21kworker/2:008:14:182
3058199171,7cyclictest4872-21cpuspeed_turbos09:50:122
3058199171,7cyclictest20579-21latency_hist11:40:012
3058199171,7cyclictest14490-21/usr/sbin/munin07:45:152
3058199171,3cyclictest0-21swapper/210:04:052
3058199171,3cyclictest0-21swapper/208:26:552
30581991712,3cyclictest9623-21sed08:45:232
30581991712,3cyclictest20451-21sed07:55:242
3058199171,15cyclictest0-21swapper/210:16:482
3058199171,15cyclictest0-21swapper/210:13:242
3058199171,14cyclictest0-21swapper/207:30:082
30581991711,3cyclictest23279-21awk09:15:262
30581991711,3cyclictest13195-21expr12:35:172
3058199171,10cyclictest0-21swapper/207:50:022
30581991710,4cyclictest23702-21fschecks_count08:05:142
3056799174,3cyclictest87150irq/126-enp1s0-12:01:260
3056799171,4cyclictest21078-21/usr/sbin/munin08:00:170
3056799171,4cyclictest19510-21/usr/sbin/munin09:10:220
3058899161,4cyclictest0-21swapper/308:05:153
3058199169,4cyclictest14584-21tune2fs11:25:122
3058199166,3cyclictest87150irq/126-enp1s0-11:50:112
3058199161,2cyclictest0-21swapper/207:16:492
30581991611,3cyclictest30152-21expr08:20:122
30581991611,3cyclictest17311-21/usr/sbin/munin09:05:152
3058199160,15cyclictest0-21swapper/209:48:072
30567991610,3cyclictest1995-21sort08:30:130
30567991610,3cyclictest13245-21sh08:55:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional