You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-15 - 23:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun Mar 15, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
144032233165,22sleep00-21swapper/007:07:040
1282722220,7sleep312832-21cpuspeed_turbos07:05:123
144192204168,24sleep10-21swapper/107:07:161
145462201165,24sleep20-21swapper/207:08:542
2759921710,2sleep20-21swapper/207:35:192
499721210,3sleep011-21rcuc/009:10:230
14734991100,108cyclictest0-21swapper/311:25:253
1473499106103,2cyclictest0-21swapper/312:39:313
147349910198,2cyclictest0-21swapper/309:27:043
147349910197,3cyclictest0-21swapper/310:35:433
14734996663,2cyclictest0-21swapper/309:09:193
267872650,4sleep31473499cyclictest07:35:123
14734996460,3cyclictest0-21swapper/311:15:523
14734995955,3cyclictest0-21swapper/308:55:443
212162570,4sleep114344-21kworker/u8:009:50:011
14734995652,3cyclictest0-21swapper/308:08:423
14734995652,3cyclictest0-21swapper/308:08:413
254652520,2sleep21472899cyclictest07:30:222
14734995148,2cyclictest0-21swapper/309:17:583
14734995148,2cyclictest0-21swapper/309:17:583
1471699340,27cyclictest0-21swapper/010:50:210
14734993127,3cyclictest0-21swapper/307:50:053
106972300,2sleep00-21swapper/010:40:010
1473499281,7cyclictest0-21swapper/310:02:053
1473499281,17cyclictest14637-21/usr/sbin/munin09:35:133
1471699284,16cyclictest30383-21kworker/u8:011:45:470
1471699284,16cyclictest17599-21kworker/u8:312:27:390
1471699282,18cyclictest0-21swapper/009:41:130
1471699281,17cyclictest3271-21perl11:35:150
1471699281,17cyclictest0-21swapper/012:30:190
125182280,4sleep121-21rcuc/109:30:111
1473499272,6cyclictest20006-21diskstats11:00:133
1473499271,16cyclictest0-21swapper/312:20:253
1471699275,11cyclictest0-21swapper/012:20:250
1471699271,17cyclictest0-21swapper/007:20:100
1473499261,7cyclictest0-21swapper/312:19:513
1473499261,11cyclictest0-21swapper/310:40:193
1473499261,11cyclictest0-21swapper/309:20:173
1471699269,2cyclictest16479-21taskset07:10:260
1471699267,6cyclictest6418-21grep11:40:270
1471699263,14cyclictest86550irq/126-enp1s0-09:20:130
14716992621,3cyclictest6219-21grep08:00:200
14716992619,4cyclictest0-21swapper/009:56:240
14716992619,4cyclictest0-21swapper/009:17:000
14716992619,4cyclictest0-21swapper/009:17:000
1471699261,6cyclictest0-21swapper/011:04:070
1471699261,18cyclictest0-21swapper/011:17:120
1471699261,18cyclictest0-21swapper/011:10:180
1471699261,18cyclictest0-21swapper/009:28:530
1471699261,16cyclictest0-21swapper/010:57:230
1471699261,11cyclictest812-21snmpd09:30:250
1473499252,7cyclictest11261-21munin-run11:55:013
14734992510,3cyclictest31409-21timerandwakeup10:10:233
1471699259,8cyclictest30383-21kworker/u8:011:25:240
1471699254,14cyclictest86550irq/126-enp1s0-10:46:200
1471699251,17cyclictest0-21swapper/011:51:100
1471699251,17cyclictest0-21swapper/011:22:530
1471699251,17cyclictest0-21swapper/010:28:030
1471699251,17cyclictest0-21swapper/010:28:020
1471699251,17cyclictest0-21swapper/010:01:110
1471699251,17cyclictest0-21swapper/009:49:000
1471699251,16cyclictest14344-21kworker/u8:010:30:080
1471699251,16cyclictest0-21swapper/012:38:530
1471699251,16cyclictest0-21swapper/012:12:170
1471699251,16cyclictest0-21swapper/009:51:090
1471699251,15cyclictest86550irq/126-enp1s0-07:45:260
1471699251,15cyclictest0-21swapper/010:07:470
1471699251,12cyclictest11613-21perf08:15:010
1471699251,11cyclictest3428-21diskmemload11:55:260
1471699250,5cyclictest0-21swapper/011:31:110
1471699250,17cyclictest30383-21kworker/u8:012:00:530
1473499241,7cyclictest0-21swapper/311:30:463
1473499241,7cyclictest0-21swapper/310:18:033
1473499241,16cyclictest0-21swapper/310:20:163
1473499241,16cyclictest0-21swapper/308:15:213
1473499241,14cyclictest17314-21fschecks_time09:40:153
1473499241,11cyclictest22203-21/usr/sbin/munin07:25:143
1473499241,11cyclictest15891-21/usr/sbin/munin12:05:143
1472899245,10cyclictest2151-21grep10:20:132
1472499246,10cyclictest6356-21grep08:00:221
1472499242,18cyclictest9411-21munin-run08:10:011
1472499242,18cyclictest9411-21munin-run08:10:011
14724992415,3cyclictest0-21swapper/110:25:231
14724992415,3cyclictest0-21swapper/110:25:231
1472499241,17cyclictest0-21swapper/110:24:531
1472499241,17cyclictest0-21swapper/109:24:561
1472499241,17cyclictest0-21swapper/107:13:061
1471699249,12cyclictest18211-21basename08:30:020
1471699242,15cyclictest0-21swapper/010:20:380
1471699241,17cyclictest0-21swapper/010:15:450
1471699241,16cyclictest0-21swapper/011:05:060
1471699241,16cyclictest0-21swapper/007:40:140
1471699241,15cyclictest86550irq/126-enp1s0-09:35:150
1471699241,15cyclictest0-21swapper/012:17:170
1471699241,15cyclictest0-21swapper/012:09:250
1471699240,6cyclictest0-21swapper/010:42:040
169462231,3sleep216958-21ntp_states12:05:212
1473499239,2cyclictest18106-21sshd08:29:033
1473499231,9cyclictest0-21swapper/310:05:153
14734992319,3cyclictest0-21swapper/309:10:223
1472899234,10cyclictest6120-21gunzip11:40:232
1472499231,17cyclictest0-21swapper/107:53:311
1472499231,16cyclictest0-21swapper/110:39:151
1472499231,14cyclictest0-21swapper/110:05:131
1472499230,6cyclictest0-21swapper/109:25:041
1471699232,2cyclictest86550irq/126-enp1s0-10:10:140
1471699231,16cyclictest0-21swapper/007:30:200
1471699231,11cyclictest0-21swapper/009:00:140
1473499226,4cyclictest32419-21sshd07:45:253
1473499223,3cyclictest0-21swapper/307:10:113
1473499221,7cyclictest8487-21sed10:35:013
1473499221,19cyclictest0-21swapper/310:45:113
1473499221,12cyclictest23432-21/usr/sbin/munin09:55:183
1473499220,15cyclictest0-21swapper/311:20:173
1472899226,8cyclictest86850irq/129-enp1s0-11:30:232
1472899224,8cyclictest86850irq/129-enp1s0-09:15:022
1472899223,10cyclictest28649-21irqstats10:05:162
1472899222,11cyclictest30664-21acpi11:25:122
1472899221,12cyclictest1419-21unixbench_singl10:15:262
1472499229,3cyclictest15309-21grep09:35:141
1472499229,11cyclictest23-21ksoftirqd/109:15:321
1472499229,11cyclictest23-21ksoftirqd/109:15:321
1472499226,4cyclictest0-21swapper/112:14:321
1472499225,9cyclictest28455-21/usr/sbin/munin11:20:141
1472499222,15cyclictest0-21swapper/111:19:041
14724992215,2cyclictest0-21swapper/107:46:331
14724992214,3cyclictest0-21swapper/112:31:201
1472499221,16cyclictest0-21swapper/111:28:331
1472499221,15cyclictest0-21swapper/112:17:591
1472499220,16cyclictest0-21swapper/112:36:331
14716992219,2cyclictest0-21swapper/007:15:210
14716992218,3cyclictest0-21swapper/007:55:240
14716992214,5cyclictest30742-40dnf08:55:250
1471699221,10cyclictest234562gltestperf08:40:140
1471699221,10cyclictest22381-21kworker/0:208:50:220
1471699220,10cyclictest580-21systemd-journal09:05:250
1473499218,2cyclictest13680-21/usr/sbin/munin12:00:213
1473499215,3cyclictest23064-21sshd09:53:243
1473499212,3cyclictest812-21snmpd08:12:153
1473499211,9cyclictest26875-21/usr/sbin/munin12:30:213
1473499211,8cyclictest289912expr07:40:123
14734992115,3cyclictest23832-21sh11:10:023
1473499211,10cyclictest0-21swapper/310:55:183
1473499210,9cyclictest0-21swapper/310:50:243
1472899219,6cyclictest86850irq/129-enp1s0-11:18:332
1472899216,8cyclictest11833-21users09:25:272
1472899216,3cyclictest20505-21latency11:00:172
1472899212,11cyclictest4457-21cstates10:25:142
1472899212,11cyclictest4457-21cstates10:25:142
1472899212,11cyclictest16180-21memory10:50:202
1472899212,11cyclictest13792-21sh08:20:002
1472899212,10cyclictest29409-21sort11:20:162
1472899212,10cyclictest17811-21/usr/sbin/munin07:15:232
1472899211,12cyclictest18903-21irqstats12:10:182
1472899211,10cyclictest26137-21latency_hist11:15:012
1472499216,12cyclictest4017-21needreboot07:55:191
1472499212,15cyclictest0-21swapper/108:45:271
1472499211,9cyclictest0-21swapper/110:31:581
1472499211,8cyclictest15891-21/usr/sbin/munin12:05:111
1472499211,3cyclictest0-21swapper/108:18:081
1472499211,17cyclictest6672-21taskset11:42:431
1472499211,15cyclictest0-21swapper/111:31:351
1472499211,15cyclictest0-21swapper/111:11:391
1472499211,15cyclictest0-21swapper/110:59:451
1472499211,15cyclictest0-21swapper/110:16:251
1472499211,15cyclictest0-21swapper/109:01:231
1472499211,15cyclictest0-21swapper/107:21:431
1472499211,10cyclictest12838-21munin-run10:45:021
14724992110,3cyclictest1802-21diskstats09:05:141
1472499210,15cyclictest0-21swapper/108:21:461
1471699210,15cyclictest0-21swapper/008:20:180
178272201,2sleep20-21swapper/208:25:292
1473499205,3cyclictest86850irq/129-enp1s0-07:55:273
1473499205,3cyclictest86550irq/126-enp1s0-09:00:233
1473499204,3cyclictest86550irq/126-enp1s0-09:45:133
1473499201,6cyclictest5329-21ntp_offset10:25:213
1473499201,6cyclictest5329-21ntp_offset10:25:213
1473499201,5cyclictest3428-21diskmemload11:56:453
1473499201,5cyclictest26978-21latency_hist08:50:013
1473499201,5cyclictest19594-21ps08:30:203
1473499201,3cyclictest7086-21/usr/sbin/munin11:45:223
1473499201,18cyclictest0-21swapper/307:24:233
1473499201,17cyclictest0-21swapper/308:35:133
1473499200,18cyclictest0-21swapper/311:40:113
1472899208,2cyclictest407-1kworker/u9:012:30:222
1472899205,8cyclictest14921-21/usr/sbin/munin07:10:192
1472899204,9cyclictest0-21swapper/209:18:242
1472899204,9cyclictest0-21swapper/209:18:242
1472899204,8cyclictest26961-21cat08:49:572
1472899203,9cyclictest0-21swapper/209:32:312
1472899203,8cyclictest14988-21df_inode09:35:122
1472899203,8cyclictest0-21swapper/212:15:132
1472899203,4cyclictest23835-21crond11:10:022
1472899202,9cyclictest851-21sssd_nss10:10:222
1472899201,6cyclictest19451-21awk11:00:022
1472899201,6cyclictest17650-21/usr/sbin/munin09:40:162
1472899201,6cyclictest10873-21users11:50:252
1472899201,5cyclictest2868-21kworker/u8:109:10:012
1472899201,11cyclictest3428-21diskmemload12:00:422
1472899201,11cyclictest29605-21irqstats07:40:162
1472899201,10cyclictest23556-21ntpq12:20:222
1472899201,10cyclictest11482-21/usr/sbin/munin11:55:152
1472899201,10cyclictest0-21swapper/210:30:252
1472499202,10cyclictest3427-21dump-pmu-power09:14:551
1472499201,8cyclictest0-21swapper/111:07:211
14724992013,3cyclictest0-21swapper/108:53:321
14724992013,3cyclictest0-21swapper/108:30:091
14724992013,3cyclictest0-21swapper/107:31:521
14724992013,2cyclictest0-21swapper/108:44:511
14724992013,2cyclictest0-21swapper/108:25:051
1472499201,15cyclictest0-21swapper/112:25:321
1472499201,15cyclictest0-21swapper/111:46:101
1472499201,15cyclictest0-21swapper/109:50:361
1472499201,14cyclictest0-21swapper/111:35:451
1472499201,14cyclictest0-21swapper/111:04:221
1472499201,14cyclictest0-21swapper/109:43:041
1472499201,14cyclictest0-21swapper/108:11:451
1472499201,14cyclictest0-21swapper/107:40:371
1472499201,11cyclictest30737-21sed08:55:251
1472499201,11cyclictest20681-21cpuspeed_turbos08:35:111
1472499201,11cyclictest15972-21idleruntime10:50:161
1472499201,10cyclictest24339-21load09:55:191
1472499201,10cyclictest22296-21cpuspeed_turbos07:25:111
1472499200,17cyclictest3428-21diskmemload10:01:311
1472499200,15cyclictest0-21swapper/111:52:161
1471699206,3cyclictest86750irq/128-enp1s0-07:25:260
1471699202,15cyclictest22186-21grep08:35:240
1473499194,3cyclictest86850irq/129-enp1s0-11:10:153
1473499194,3cyclictest86850irq/129-enp1s0-08:50:113
1473499194,3cyclictest86550irq/126-enp1s0-09:30:253
1473499191,5cyclictest17811-21/usr/sbin/munin07:15:243
1473499191,4cyclictest2692-21/usr/sbin/munin11:35:183
1473499191,4cyclictest0-21swapper/312:25:513
14734991911,4cyclictest6640-21sort08:00:263
1472899195,7cyclictest21235-21/usr/sbin/munin09:50:192
1472899193,8cyclictest29071-21/usr/sbin/munin12:35:212
1472899192,4cyclictest0-21swapper/211:35:112
1472899191,5cyclictest750-21/usr/sbin/munin07:50:132
1472899191,5cyclictest13408-21diskstats10:45:142
1472899191,5cyclictest0-21swapper/207:26:422
1472899191,4cyclictest0-21swapper/209:45:112
1472899191,11cyclictest0-21swapper/209:55:092
1472899191,10cyclictest0-21swapper/210:01:072
1472499198,2cyclictest15049-21awk10:50:011
1472499194,9cyclictest0-21swapper/112:22:551
14724991915,3cyclictest0-21swapper/112:02:011
1472499191,16cyclictest0-21swapper/111:58:111
1472499190,18cyclictest0-21swapper/107:39:231
1472499190,14cyclictest0-21swapper/110:12:061
1472499190,14cyclictest0-21swapper/107:15:541
1471699192,14cyclictest25669-21gltestperf08:45:140
1471699191,9cyclictest86550irq/126-enp1s0-08:32:230
1473499184,3cyclictest86850irq/129-enp1s0-07:30:203
1473499181,6cyclictest0-21swapper/312:11:313
1473499181,4cyclictest23164-21diskstats08:40:133
1473499181,3cyclictest14367-21diskstats08:20:133
1472899186,3cyclictest15610-21sshd08:20:262
1472899183,8cyclictest8667-21/usr/sbin/munin10:35:222
1472899183,8cyclictest7433-21/usr/sbin/munin08:05:202
1472899183,8cyclictest7433-21/usr/sbin/munin08:05:202
1472899181,9cyclictest0-21swapper/211:46:362
1472899181,9cyclictest0-21swapper/210:40:032
1472899181,7cyclictest0-21swapper/212:25:052
1472899181,5cyclictest399-21smart_sda09:00:242
1472899181,5cyclictest3033-21/usr/sbin/munin07:55:252
1472899181,5cyclictest29407-21/usr/sbin/munin08:55:162
1472899181,5cyclictest20007-21/usr/sbin/munin07:20:172
1472899181,3cyclictest0-21swapper/209:20:062
14728991812,3cyclictest18421-21/usr/sbin/munin08:30:242
1471699181,5cyclictest812-21snmpd07:37:100
14716991812,3cyclictest7685-21cat08:05:130
14716991812,3cyclictest7685-21cat08:05:130
1471699181,14cyclictest812-21snmpd08:17:170
1472899171,9cyclictest20617-21/usr/sbin/munin08:35:112
1472899171,7cyclictest0-21swapper/208:40:232
1472899171,5cyclictest5229-21/usr/sbin/munin08:00:212
1472899171,3cyclictest0-21swapper/207:45:172
1471699172,4cyclictest767-21acpi07:50:110
1472899161,3cyclictest0-21swapper/208:50:222
14728991510,3cyclictest10146-21cut08:10:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional