You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-18 - 06:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Sep 18, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75922230161,57sleep00-21swapper/019:05:550
70202216148,56sleep30-21swapper/319:05:223
78692202166,24sleep20-21swapper/219:09:302
78582202167,23sleep10-21swapper/119:09:211
161821840,5sleep1800099cyclictest20:05:201
8006998077,2cyclictest0-21swapper/220:24:212
8006996158,2cyclictest0-21swapper/220:43:372
308412590,4sleep011-21rcuc/023:40:130
12292550,1sleep30-21swapper/323:45:183
217412540,3sleep3401ktimersoftd/300:30:253
13772540,2sleep19-21rcu_preempt23:45:201
8006995350,2cyclictest0-21swapper/223:00:362
70122410,1sleep10-21swapper/100:00:121
800099321,28cyclictest13435-21grep19:20:121
800099311,26cyclictest19068-21seq23:12:121
799299310,29cyclictest0-21swapper/021:28:280
38612310,7sleep339-21rcuc/320:10:193
801299301,4cyclictest0-21swapper/321:28:293
801299300,12cyclictest29648-21kworker/u8:222:32:403
799299301,19cyclictest29793-21cut22:25:020
800099291,20cyclictest0-21swapper/121:53:271
799299293,18cyclictest0-21swapper/021:50:200
800099281,25cyclictest0-21swapper/119:55:261
800099280,11cyclictest0-21swapper/121:26:151
799299281,6cyclictest0-21swapper/023:09:020
8012992716,6cyclictest827-21runrttasks20:27:253
801299271,4cyclictest0-21swapper/323:12:113
8012992712,3cyclictest29900-21/usr/sbin/munin22:25:153
801299271,13cyclictest11861-21fschecks_time00:10:183
801299271,12cyclictest22436-21sshd20:54:503
800099278,16cyclictest20555-21ntp_kernel_err23:15:201
8000992722,2cyclictest17255-21unixbench_multi00:20:271
800099271,9cyclictest0-21swapper/122:44:301
8000992713,7cyclictest0-21swapper/119:10:151
800099271,23cyclictest827-21runrttasks23:55:241
800099271,23cyclictest14099-21cut00:15:151
799299276,8cyclictest0-21swapper/021:42:240
799299271,17cyclictest0-21swapper/000:06:430
801299261,6cyclictest29082-21fschecks_count19:55:143
801299261,6cyclictest0-21swapper/322:09:203
801299261,5cyclictest0-21swapper/323:41:353
801299261,3cyclictest0-21swapper/321:51:523
8012992612,3cyclictest13569-21/usr/sbin/munin00:15:183
801299261,17cyclictest87350irq/126-enp1s0-21:05:173
801299261,16cyclictest87350irq/126-enp1s0-23:05:503
800699263,14cyclictest8564-1kworker/u9:122:37:022
800699262,14cyclictest11252-40dnf20:27:252
800699261,8cyclictest0-21swapper/222:57:392
800699261,5cyclictest0-21swapper/221:54:562
800699261,10cyclictest0-21swapper/221:18:282
8000992622,3cyclictest0-21swapper/120:03:101
8000992622,3cyclictest0-21swapper/100:35:281
8000992622,3cyclictest0-21swapper/100:35:271
8000992620,3cyclictest29292-21users22:20:271
8000992613,6cyclictest0-21swapper/120:50:071
800099261,23cyclictest0-21swapper/123:40:131
800099261,20cyclictest0-21swapper/122:29:301
800099261,11cyclictest579-21systemd-journal21:35:011
800099261,11cyclictest17841-21tr19:30:111
800099260,10cyclictest0-21swapper/123:08:521
800099260,10cyclictest0-21swapper/122:30:131
799299264,3cyclictest0-21swapper/022:36:150
7992992615,4cyclictest0-21swapper/022:47:510
7992992615,2cyclictest29061-21diskmemload22:15:190
799299261,16cyclictest0-21swapper/023:29:410
799299260,6cyclictest0-21swapper/000:10:510
801299254,6cyclictest579-21systemd-journal23:55:023
801299251,7cyclictest0-21swapper/321:35:043
801299251,6cyclictest0-21swapper/322:20:143
8012992516,3cyclictest0-21swapper/322:18:063
801299251,5cyclictest0-21swapper/322:57:433
801299251,5cyclictest0-21swapper/321:24:503
801299251,4cyclictest0-21swapper/322:03:233
801299251,18cyclictest87350irq/126-enp1s0-21:34:463
8012992511,7cyclictest0-21swapper/300:35:263
8012992511,7cyclictest0-21swapper/300:35:263
801299251,15cyclictest0-21swapper/321:16:403
8012992510,3cyclictest19875-21/usr/sbin/munin19:35:203
800699257,9cyclictest8564-1kworker/u9:122:48:372
800699253,5cyclictest0-21swapper/223:43:172
800699252,12cyclictest9601-21ntp_states22:50:222
800699251,9cyclictest0-21swapper/223:33:342
800699251,9cyclictest0-21swapper/223:25:402
800699251,8cyclictest0-21swapper/222:15:042
800699251,5cyclictest0-21swapper/222:40:252
800699251,10cyclictest22277-21cut23:20:162
800099252,2cyclictest87650irq/129-enp1s0-21:10:181
8000992522,2cyclictest0-21swapper/123:36:341
8000992522,2cyclictest0-21swapper/122:00:551
8000992522,2cyclictest0-21swapper/121:10:011
8000992522,2cyclictest0-21swapper/120:59:171
8000992522,2cyclictest0-21swapper/120:17:191
800099251,9cyclictest9646-21kworker/u8:121:55:351
800099251,3cyclictest0-21swapper/122:48:421
800099251,21cyclictest14314-21latency_hist21:50:001
800099251,10cyclictest32553-21ntp_states21:15:211
800099250,10cyclictest0-21swapper/121:41:071
800099250,10cyclictest0-21swapper/100:32:531
799299258,9cyclictest0-21swapper/023:58:410
799299256,13cyclictest0-21swapper/023:49:200
799299255,8cyclictest0-21swapper/023:38:120
799299253,3cyclictest0-21swapper/021:10:210
799299251,15cyclictest0-21swapper/000:29:080
7992992510,8cyclictest0-21swapper/022:08:370
7992992510,7cyclictest0-21swapper/022:55:410
799299250,22cyclictest0-21swapper/022:32:220
310582250,4sleep011-21rcuc/022:25:210
801299244,3cyclictest9053-21latency_hist00:05:023
801299241,5cyclictest0-21swapper/322:39:323
801299241,4cyclictest0-21swapper/323:26:443
801299241,15cyclictest87350irq/126-enp1s0-19:12:503
801299241,11cyclictest26118-21/usr/sbin/munin23:30:163
801299241,11cyclictest11751-21sshd21:40:523
8012992410,8cyclictest0-21swapper/319:25:033
801299240,7cyclictest3039-21kworker/u8:122:50:193
800699247,4cyclictest13224-21cat19:20:022
800699247,10cyclictest3191-21sshd21:23:512
800699246,8cyclictest8564-1kworker/u9:123:48:462
800699245,11cyclictest12596-21rt-features00:10:262
800699241,4cyclictest0-21swapper/223:37:202
800699241,13cyclictest0-21swapper/221:45:342
8000992421,2cyclictest0-21swapper/123:27:381
8000992421,2cyclictest0-21swapper/121:04:371
8000992420,3cyclictest0-21swapper/100:26:291
800099241,9cyclictest0-21swapper/122:18:021
8000992419,3cyclictest0-21swapper/119:15:161
8000992412,6cyclictest0-21swapper/123:34:201
8000992412,6cyclictest0-21swapper/121:22:101
800099241,11cyclictest23080-21sed23:20:261
800099240,9cyclictest0-21swapper/121:37:221
800099240,4cyclictest0-21swapper/122:56:091
799299244,3cyclictest87450irq/127-enp1s0-21:00:460
799299241,7cyclictest6952-21/usr/sbin/munin00:00:140
7992992412,3cyclictest0-21swapper/022:11:140
799299241,16cyclictest0-21swapper/021:38:570
799299241,15cyclictest21399-21kworker/u8:223:31:480
799299241,12cyclictest13902-21/usr/sbin/munin20:35:200
799299241,11cyclictest23080-21if_enp1s000:35:170
799299241,11cyclictest23080-21if_enp1s000:35:170
7992992410,7cyclictest0-21swapper/022:53:290
7992992410,7cyclictest0-21swapper/000:31:010
7992992410,4cyclictest0-21swapper/022:40:030
799299240,5cyclictest0-21swapper/023:21:170
801299238,3cyclictest12929-21/usr/sbin/munin23:00:243
801299237,4cyclictest87350irq/126-enp1s0-20:50:003
801299235,9cyclictest87350irq/126-enp1s0-20:00:143
801299235,3cyclictest12050irq/123-ahci[0000:00:023
801299234,10cyclictest29061-21diskmemload22:48:453
801299233,3cyclictest87650irq/129-enp1s0-20:55:203
801299232,3cyclictest24037-21sh19:45:013
801299231,7cyclictest0-21swapper/320:30:113
801299231,6cyclictest7636-21df_inode20:20:123
801299231,5cyclictest0-21swapper/323:35:373
801299231,5cyclictest0-21swapper/300:28:263
801299231,4cyclictest0-21swapper/321:58:103
801299231,4cyclictest0-21swapper/321:49:253
801299231,11cyclictest26465-21/usr/sbin/munin19:50:153
801299231,11cyclictest26465-21/usr/sbin/munin19:50:153
800699238,3cyclictest87650irq/129-enp1s0-22:20:252
800699237,4cyclictest9501-21df00:05:152
800699236,4cyclictest25484-21grep21:00:162
800699235,9cyclictest29061-21diskmemload00:15:232
800699234,10cyclictest6892-21cat00:00:022
800699234,10cyclictest18789-21sed23:10:252
800699233,9cyclictest2980-21fschecks_count23:50:172
800699233,10cyclictest21619-21unixbench-2d00:30:242
800699232,10cyclictest32451-21diskstats22:30:172
800699232,10cyclictest21112-21users23:15:262
800699231,8cyclictest0-21swapper/223:05:042
800699231,8cyclictest0-21swapper/221:55:552
800699231,7cyclictest0-21swapper/222:27:332
800699231,13cyclictest30637-21munin-run20:00:002
800699231,13cyclictest0-21swapper/200:37:012
800699231,13cyclictest0-21swapper/200:37:012
800699231,12cyclictest0-21swapper/221:27:322
800699230,10cyclictest29648-21kworker/u8:222:00:582
800099234,3cyclictest87650irq/129-enp1s0-22:10:271
800099234,3cyclictest0-21swapper/123:00:371
800099231,20cyclictest0-21swapper/119:43:341
800099231,11cyclictest18061-21latency_hist20:45:011
800099230,21cyclictest0-21swapper/122:35:151
800099230,21cyclictest0-21swapper/120:10:121
800099230,10cyclictest2900-21diskstats23:50:171
799299239,7cyclictest0-21swapper/022:01:170
799299239,7cyclictest0-21swapper/000:21:280
799299237,7cyclictest0-21swapper/023:01:100
799299231,15cyclictest9646-21kworker/u8:121:45:160
799299231,15cyclictest0-21swapper/023:51:250
799299231,14cyclictest29283-21kworker/u8:021:21:430
799299231,14cyclictest0-21swapper/000:15:080
799299231,11cyclictest5719-21/usr/sbin/munin21:30:210
7992992310,6cyclictest0-21swapper/023:13:110
799299230,14cyclictest0-21swapper/021:15:410
801299223,3cyclictest87650irq/129-enp1s0-21:10:233
801299222,6cyclictest22637-21memory23:20:193
801299221,8cyclictest24269-21/usr/sbin/munin19:45:193
801299221,8cyclictest23306-21/usr/sbin/munin22:10:153
801299221,8cyclictest19525-21/usr/sbin/munin23:15:163
801299221,7cyclictest0-21swapper/320:43:223
801299221,6cyclictest0-21swapper/319:31:283
801299221,3cyclictest87350irq/126-enp1s0-00:20:053
800699226,9cyclictest22767-21kworker/u8:122:13:042
800699226,4cyclictest7430-21fschecks_time00:00:162
800699225,4cyclictest21138-21sh19:35:242
800699222,11cyclictest30207-21memory21:10:202
800699221,5cyclictest0-21swapper/200:28:292
800699221,12cyclictest4881-21munin-run20:15:002
800699221,11cyclictest9181-21smart_sda21:35:232
800699221,11cyclictest0-21swapper/221:41:132
800699221,10cyclictest808-21snmpd20:50:012
800699221,10cyclictest31197-21df20:00:122
800099224,3cyclictest0-21swapper/100:13:131
800099222,2cyclictest87650irq/129-enp1s0-00:09:121
8000992220,1cyclictest0-21swapper/120:20:251
800099221,8cyclictest0-21swapper/122:05:061
800099221,19cyclictest0-21swapper/120:45:311
800099221,19cyclictest0-21swapper/120:35:171
800099221,18cyclictest25419-21pmu-power19:45:221
800099221,18cyclictest0-21swapper/122:50:121
799299222,17cyclictest10171-21munin-run19:15:020
799299221,15cyclictest0-21swapper/021:58:160
801299211,6cyclictest0-21swapper/319:20:113
801299211,6cyclictest0-21swapper/319:17:273
801299211,6cyclictest0-21swapper/300:05:273
801299211,5cyclictest0-21swapper/322:44:123
800699216,8cyclictest21111-21/usr/sbin/munin22:05:132
800699213,10cyclictest26540-21cpuspeed_turbos19:50:112
800699213,10cyclictest26540-21cpuspeed_turbos19:50:112
800699212,6cyclictest21557-21ntp_states20:50:222
800699212,11cyclictest11701-21/usr/sbin/munin20:30:242
800699211,6cyclictest14040-21tr20:35:132
800699211,5cyclictest10306-21munin-node19:15:012
800699211,11cyclictest7078-21perf20:20:012
800699211,11cyclictest24039-21latency_hist19:45:012
800699211,10cyclictest8564-1kworker/u9:100:20:532
800099211,18cyclictest11252-40dnf20:27:231
800099211,10cyclictest0-21swapper/119:35:191
800099210,9cyclictest0-21swapper/120:30:141
800099210,19cyclictest0-21swapper/119:50:171
800099210,19cyclictest0-21swapper/119:50:171
800099210,19cyclictest0-21swapper/119:25:161
799299219,3cyclictest0-21swapper/023:15:400
799299214,9cyclictest0-21swapper/020:55:220
799299214,8cyclictest5681-21perl20:15:160
799299211,4cyclictest0-21swapper/020:04:510
799299211,18cyclictest0-21swapper/019:45:190
801299203,4cyclictest25798-21memory21:00:203
801299201,6cyclictest1619-21memory20:05:203
801299201,6cyclictest0-21swapper/320:15:503
801299200,10cyclictest0-21swapper/320:35:143
800699201,7cyclictest15481-21/usr/sbin/munin19:25:152
800699201,11cyclictest673-21cpu20:05:122
800699201,11cyclictest5719-21/usr/sbin/munin21:30:112
800699201,11cyclictest27664-21fschecks_time21:05:162
800699201,10cyclictest13284-21/usr/sbin/munin19:20:182
799299208,2cyclictest0-21swapper/019:25:210
799299204,3cyclictest10525-21grep19:15:020
799299201,4cyclictest0-21swapper/019:40:140
800699194,4cyclictest808-21snmpd19:30:042
800699192,9cyclictest22688-21/usr/sbin/munin20:55:122
800699191,6cyclictest24269-21/usr/sbin/munin19:45:132
799299195,2cyclictest0-21swapper/020:40:270
799299192,3cyclictest0-21swapper/020:50:180
7992991913,3cyclictest3569-21sed20:10:160
799299191,11cyclictest0-21swapper/019:30:250
7992991910,2cyclictest13284-21/usr/sbin/munin19:20:250
799299190,17cyclictest0-21swapper/020:20:130
113702191,2sleep00-21swapper/020:28:510
799299184,3cyclictest87350irq/126-enp1s0-20:05:200
7992991812,3cyclictest28664-21/usr/sbin/munin19:55:170
799299181,14cyclictest21100-21gunzip19:35:230
799299171,5cyclictest0-21swapper/021:07:130
799299171,5cyclictest0-21swapper/020:45:140
7992991712,3cyclictest26465-21/usr/sbin/munin19:50:180
7992991712,3cyclictest26465-21/usr/sbin/munin19:50:180
7992991710,4cyclictest0-21swapper/020:30:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional