You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 10:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Mar 26, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305522233165,57sleep00-21swapper/019:09:180
303792230162,55sleep20-21swapper/219:07:032
303022229154,24sleep30-21swapper/319:06:043
302652201165,24sleep10-21swapper/119:05:331
330021510,5sleep03069999cyclictest20:30:250
1450321400,5sleep13070699cyclictest20:55:261
908521370,4sleep33071699cyclictest22:00:013
3071699109105,3cyclictest0-21swapper/321:12:243
3071699109105,3cyclictest0-21swapper/321:12:243
30716999491,2cyclictest0-21swapper/319:38:483
30716998077,2cyclictest0-21swapper/321:45:113
30716996863,3cyclictest0-21swapper/320:05:143
30716995955,3cyclictest0-21swapper/321:15:563
285972580,1sleep20-21swapper/223:55:222
30716995350,2cyclictest0-21swapper/322:35:003
142792460,1sleep00-21swapper/000:35:240
30716993827,2cyclictest23380-21kworker/3:223:26:583
30699993811,13cyclictest11846-21munin-run23:20:010
3070699361,10cyclictest0-21swapper/119:35:061
3071099351,17cyclictest0-21swapper/222:50:202
3071099351,14cyclictest0-21swapper/200:10:132
30710993419,10cyclictest0-21swapper/200:09:172
3071099341,15cyclictest0-21swapper/200:23:432
3070699341,12cyclictest0-21swapper/121:06:031
3070699341,10cyclictest0-21swapper/120:36:231
3070699340,11cyclictest0-21swapper/123:54:571
3070699340,10cyclictest0-21swapper/100:12:541
131052340,5sleep011-21rcuc/020:55:120
3071099331,14cyclictest0-21swapper/222:11:102
3071099331,14cyclictest0-21swapper/221:12:232
3071099331,14cyclictest0-21swapper/221:12:232
3071099331,14cyclictest0-21swapper/219:43:112
3070699331,9cyclictest0-21swapper/120:18:421
3070699331,11cyclictest0-21swapper/119:56:381
3070699331,10cyclictest0-21swapper/119:24:271
3070699330,11cyclictest0-21swapper/122:39:361
3069999337,12cyclictest21610-21perf21:15:010
3069999337,12cyclictest21610-21perf21:15:010
30710993220,3cyclictest0-21swapper/220:23:002
3071099321,17cyclictest0-21swapper/223:00:502
3071099321,15cyclictest0-21swapper/220:44:342
3071099321,14cyclictest0-21swapper/221:22:192
3070699321,19cyclictest0-21swapper/123:05:351
3070699321,10cyclictest0-21swapper/100:34:581
289292320,2sleep00-21swapper/020:15:240
30716993122,2cyclictest23380-21kworker/3:219:57:453
3071699312,21cyclictest23380-21kworker/3:222:53:163
30716993120,5cyclictest23380-21kworker/3:220:00:113
30716993120,5cyclictest23380-21kworker/3:220:00:113
3071099311,13cyclictest0-21swapper/222:40:402
3071099311,13cyclictest0-21swapper/220:58:072
3071099311,13cyclictest0-21swapper/219:59:102
3071099311,13cyclictest0-21swapper/200:17:192
3071099310,15cyclictest0-21swapper/222:55:202
3070699311,10cyclictest0-21swapper/122:44:061
3070699310,10cyclictest0-21swapper/123:26:311
3069999318,11cyclictest28816-21perf22:45:020
3069999314,13cyclictest19409-21munin-run21:10:000
30716993019,2cyclictest23380-21kworker/3:221:43:273
3071099301,14cyclictest0-21swapper/200:00:272
3071099301,13cyclictest0-21swapper/223:49:302
3071099301,13cyclictest0-21swapper/222:26:222
3071099301,13cyclictest0-21swapper/222:26:222
3071099301,13cyclictest0-21swapper/221:28:352
3071099301,13cyclictest0-21swapper/200:31:072
3071099301,12cyclictest0-21swapper/223:11:502
3071099301,12cyclictest0-21swapper/221:50:232
3071099301,12cyclictest0-21swapper/220:04:462
3071099301,12cyclictest0-21swapper/220:04:462
3071099301,12cyclictest0-21swapper/200:36:192
3071099301,10cyclictest0-21swapper/223:53:302
3070699305,13cyclictest0-21swapper/121:28:591
3070699301,11cyclictest0-21swapper/120:49:411
3070699301,10cyclictest0-21swapper/121:12:491
3070699301,10cyclictest0-21swapper/121:12:491
3070699300,11cyclictest0-21swapper/122:04:141
3069999302,13cyclictest11227-21munin-run22:05:000
3069999301,13cyclictest15629-21awk22:15:000
3071099291,14cyclictest0-21swapper/220:11:382
3071099291,13cyclictest0-21swapper/222:49:182
3071099291,13cyclictest0-21swapper/222:05:022
3071099291,13cyclictest0-21swapper/221:19:272
3071099291,13cyclictest0-21swapper/220:26:142
3071099291,12cyclictest0-21swapper/222:16:542
3071099291,12cyclictest0-21swapper/221:59:112
3071099291,12cyclictest0-21swapper/221:34:512
3071099291,12cyclictest0-21swapper/220:51:522
3070699291,7cyclictest0-21swapper/119:54:261
3070699291,2cyclictest0-21swapper/120:00:101
3070699291,2cyclictest0-21swapper/120:00:101
3070699291,26cyclictest0-21swapper/122:17:081
3070699291,26cyclictest0-21swapper/121:40:241
3070699291,22cyclictest0-21swapper/123:33:071
3070699291,16cyclictest30978-21needreboot20:20:201
3070699290,22cyclictest0-21swapper/121:39:021
3069999292,13cyclictest20975-21munin-run20:00:000
3069999291,13cyclictest3702-21awk00:15:010
30699992911,15cyclictest31970-21munin-run20:25:020
30716992817,2cyclictest23380-21kworker/3:221:07:103
3071699281,20cyclictest23380-21kworker/3:221:00:183
3071699280,21cyclictest23380-21kworker/3:220:12:563
3071099287,8cyclictest0-21swapper/219:22:492
3071099281,13cyclictest0-21swapper/223:30:342
3071099281,12cyclictest0-21swapper/223:16:262
3071099281,12cyclictest0-21swapper/221:00:272
3071099281,12cyclictest0-21swapper/219:51:482
3071099281,11cyclictest0-21swapper/221:36:392
3071099281,10cyclictest0-21swapper/223:26:522
3071099281,10cyclictest0-21swapper/223:21:102
3071099281,10cyclictest0-21swapper/221:40:152
3071099281,10cyclictest0-21swapper/219:11:582
3070699281,3cyclictest0-21swapper/119:31:221
3069999281,13cyclictest16242-21munin-run23:30:010
30716992718,2cyclictest23380-21kworker/3:200:05:563
3071699271,18cyclictest0-21swapper/321:30:393
3071699270,20cyclictest23380-21kworker/3:220:31:523
3071099276,3cyclictest0-21swapper/222:31:142
30710992712,3cyclictest0-21swapper/223:39:362
30710992712,3cyclictest0-21swapper/223:39:362
3071099271,14cyclictest0-21swapper/223:40:422
3071099271,13cyclictest0-21swapper/219:18:142
3071099271,12cyclictest0-21swapper/222:38:462
3071099271,10cyclictest0-21swapper/220:34:262
3071099271,10cyclictest0-21swapper/219:46:072
3071099271,10cyclictest0-21swapper/219:37:152
3070699271,9cyclictest22149-21sshd22:29:241
3070699271,9cyclictest22149-21sshd22:29:241
3070699271,9cyclictest0-21swapper/122:57:071
3070699271,16cyclictest0-21swapper/119:19:031
3070699270,9cyclictest0-21swapper/100:29:261
3069999279,15cyclictest13437-21basename22:10:000
3071699262,8cyclictest23380-21kworker/3:200:29:153
30716992614,3cyclictest0-21swapper/300:21:373
30716992611,8cyclictest23380-21kworker/3:223:17:083
3071699261,16cyclictest23380-21kworker/3:220:43:243
30716992610,8cyclictest23380-21kworker/3:222:23:153
3071099261,13cyclictest0-21swapper/221:08:372
3071099261,11cyclictest0-21swapper/223:05:582
3071099261,11cyclictest0-21swapper/221:46:552
3070699261,9cyclictest0-21swapper/120:10:541
3070699261,9cyclictest0-21swapper/120:08:141
3070699261,3cyclictest32160-21missed_timers22:50:181
3070699261,16cyclictest19930-21chrt22:24:061
3070699261,10cyclictest0-21swapper/123:39:511
3070699261,10cyclictest0-21swapper/123:39:511
3070699260,3cyclictest0-21swapper/100:05:571
3070699260,11cyclictest0-21swapper/120:54:051
3071699254,2cyclictest23380-21kworker/3:223:49:503
3071699252,9cyclictest23380-21kworker/3:221:54:173
3071699251,9cyclictest23380-21kworker/3:223:44:003
30716992515,2cyclictest23380-21kworker/3:223:04:163
3071699251,18cyclictest0-21swapper/322:36:563
30716992511,6cyclictest23380-21kworker/3:222:44:543
30716992511,6cyclictest23380-21kworker/3:222:17:493
30716992511,5cyclictest23380-21kworker/3:200:10:383
3071699251,14cyclictest23380-21kworker/3:219:31:593
30716992511,2cyclictest23380-21kworker/3:220:18:183
3071099259,3cyclictest0-21swapper/200:27:152
3071099251,15cyclictest0-21swapper/220:15:382
3071099251,14cyclictest0-21swapper/222:20:182
3071099251,11cyclictest0-21swapper/220:49:502
3070699251,9cyclictest0-21swapper/120:34:551
3070699251,9cyclictest0-21swapper/100:16:181
3070699251,2cyclictest0-21swapper/123:10:221
3070699251,2cyclictest0-21swapper/122:05:291
3070699251,16cyclictest0-21swapper/119:48:071
3070699251,12cyclictest16481-21smart_sda21:00:241
3070699251,11cyclictest29837-21irqstats22:45:181
3070699251,11cyclictest0-21swapper/120:41:511
30706992511,12cyclictest28624-21kworker/u8:123:00:121
3070699250,10cyclictest0-21swapper/123:59:191
3070699250,10cyclictest0-21swapper/122:30:221
3069999259,14cyclictest26003-21perf21:25:010
3071699248,7cyclictest23380-21kworker/3:200:31:513
3071699247,4cyclictest0-21swapper/322:10:223
3071699247,2cyclictest23380-21kworker/3:222:25:193
3071699247,2cyclictest23380-21kworker/3:222:25:193
3071699242,9cyclictest23380-21kworker/3:220:35:043
3071699241,9cyclictest23380-21kworker/3:221:22:163
30716992413,2cyclictest23380-21kworker/3:220:48:103
30716992412,2cyclictest23380-21kworker/3:222:56:063
3071699241,15cyclictest23380-21kworker/3:220:28:543
3071699241,11cyclictest23380-21kworker/3:219:10:023
3071699240,14cyclictest23380-21kworker/3:221:26:503
3071099249,3cyclictest0-21swapper/219:32:552
3071099241,15cyclictest0-21swapper/220:38:582
3071099241,13cyclictest24864-21/usr/sbin/munin20:05:242
3071099241,10cyclictest0-21swapper/222:03:072
3070699241,8cyclictest0-21swapper/119:40:181
3070699241,2cyclictest0-21swapper/122:11:321
3070699241,10cyclictest0-21swapper/121:20:171
3070699240,9cyclictest0-21swapper/123:19:151
3070699240,10cyclictest0-21swapper/123:24:231
3071699237,2cyclictest23380-21kworker/3:220:58:383
3071699233,13cyclictest20591-21kworker/u8:123:56:093
3071699232,9cyclictest23380-21kworker/3:223:36:503
3071699232,9cyclictest23380-21kworker/3:223:36:503
3071699232,8cyclictest23380-21kworker/3:223:32:403
3071699231,3cyclictest0-21swapper/322:45:563
30716992311,2cyclictest23380-21kworker/3:222:07:093
30716992310,2cyclictest23380-21kworker/3:200:00:243
3071699230,16cyclictest23380-21kworker/3:219:49:193
3071099231,10cyclictest0-21swapper/219:27:432
30706992311,3cyclictest86750irq/128-enp1s0-00:21:401
3070699231,11cyclictest0-21swapper/119:26:381
3070699231,10cyclictest0-21swapper/121:45:501
3070699230,9cyclictest0-21swapper/121:53:501
3070699230,8cyclictest0-21swapper/123:40:551
3069999236,14cyclictest3060-21perf23:00:020
9822220,8sleep341-21ksoftirqd/319:15:023
3071699229,6cyclictest23380-21kworker/3:223:24:003
3071699227,6cyclictest0-21swapper/323:54:473
3071699226,6cyclictest0-21swapper/300:37:313
30716992212,2cyclictest86550irq/126-enp1s0-22:00:253
3071699221,15cyclictest0-21swapper/323:10:563
30716992211,3cyclictest23380-21kworker/3:219:26:453
3071699220,8cyclictest23380-21kworker/3:219:42:513
3071699220,11cyclictest23380-21kworker/3:220:52:083
30706992218,3cyclictest0-21swapper/121:56:181
3070699221,3cyclictest32544-21diskstats20:25:151
30706992213,3cyclictest0-21swapper/119:10:231
30706992211,2cyclictest0-21swapper/121:33:041
3070699220,20cyclictest0-21swapper/121:15:151
3069999221,18cyclictest8082-21munin-run00:25:000
3071699219,2cyclictest23380-21kworker/3:221:35:403
3071699211,8cyclictest23380-21kworker/3:220:22:203
3071699211,3cyclictest0-21swapper/319:21:023
3071699211,14cyclictest0-21swapper/300:16:203
3070699214,3cyclictest86750irq/128-enp1s0-23:49:511
3070699211,17cyclictest14429-21runrttasks00:37:051
3070699210,7cyclictest0-21swapper/100:02:031
3069999211,6cyclictest29592-21processes21:30:220
3069999211,4cyclictest27526-21switchtime21:25:250
3069999210,5cyclictest22104-21kworker/u8:022:52:060
3071699205,2cyclictest23380-21kworker/3:219:52:033
3071699201,14cyclictest0-21swapper/323:05:113
3069999204,14cyclictest9-21rcu_preempt20:50:000
3069999202,6cyclictest25771-1kworker/u9:023:12:140
3069999202,15cyclictest6210-21munin-run20:40:010
3069999202,15cyclictest12191-21munin-run19:40:000
3069999201,9cyclictest0-21swapper/000:00:220
3069999201,11cyclictest20858-21/usr/sbin/munin23:40:260
228902201,2sleep00-21swapper/020:01:290
228902201,2sleep00-21swapper/020:01:280
3069999192,8cyclictest30096-21ntp_states22:45:220
3069999192,14cyclictest1950-21grep21:40:240
3069999191,8cyclictest812-21snmpd20:40:170
3069999191,6cyclictest5892-21munin-run00:20:010
3069999191,6cyclictest0-21swapper/000:08:180
3069999191,10cyclictest580-21systemd-journal19:41:250
3069999191,10cyclictest25253-21/usr/sbin/munin23:50:150
3069999188,3cyclictest5818-21/usr/sbin/munin19:25:180
3069999187,3cyclictest16441-21sshd19:48:240
3069999184,3cyclictest812-21snmpd19:20:020
3069999183,8cyclictest833-21runrttasks23:49:500
3069999181,8cyclictest0-21swapper/023:35:180
3069999181,8cyclictest0-21swapper/023:35:180
3069999181,8cyclictest0-21swapper/022:20:220
3069999181,7cyclictest31426-21irqstats21:35:170
3069999181,7cyclictest0-21swapper/022:17:340
3069999181,6cyclictest0-21swapper/021:52:100
3069999181,5cyclictest1806-21latency_hist20:30:020
3069999181,5cyclictest0-21swapper/019:54:240
30699991813,3cyclictest29410-40updatedb00:00:010
30699991811,4cyclictest4166-21sed21:45:240
3069999181,11cyclictest0-21swapper/022:30:270
3069999176,3cyclictest86750irq/128-enp1s0-20:50:220
3069999173,7cyclictest407-21munin-run19:15:020
3069999171,7cyclictest0-21swapper/023:30:130
3069999171,6cyclictest0-21swapper/023:24:340
3069999171,6cyclictest0-21swapper/021:55:060
30699991711,4cyclictest25701-21ntp_states22:35:220
30699991711,3cyclictest12471-21turbostat.cron00:35:000
3069999162,7cyclictest25600-21/usr/sbin/munin20:10:250
3069999161,6cyclictest0-21swapper/021:18:560
3069999161,6cyclictest0-21swapper/000:26:250
3069999161,5cyclictest0-21swapper/023:00:530
30699991612,3cyclictest0-21swapper/021:01:480
30699991611,3cyclictest6536-21ntp_offset23:05:210
30699991611,2cyclictest545-21munin-node22:25:110
30699991611,2cyclictest545-21munin-node22:25:110
3069999160,10cyclictest0-21swapper/019:30:130
3069999151,11cyclictest3624-21/usr/sbin/munin19:20:120
3069999150,14cyclictest0-21swapper/020:08:520
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional