You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-03 - 05:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Mon Aug 03, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
241153221500,3sleep33056-21PK-Backend23:51:543
240939021340,4sleep33056-21PK-Backend23:45:213
231173321210,4sleep33056-21PK-Backend19:10:213
238882921150,2sleep10-21swapper/122:50:001
24161262970,3sleep10-21swapper/100:05:201
139099850,69rtkit-daemon0-21swapper/119:09:541
231028827151,14sleep20-21swapper/219:09:242
231022726444,14sleep30-21swapper/319:08:363
231027126241,14sleep00-21swapper/019:09:090
23894752480,3sleep00-21swapper/022:50:150
23719642480,2sleep20-21swapper/222:00:182
23415722410,2sleep20-21swapper/220:35:162
231053099389,14cyclictest0-21swapper/219:33:372
231053099388,15cyclictest0-21swapper/223:55:062
231053099363,18cyclictest0-21swapper/222:10:512
231053099362,19cyclictest0-21swapper/221:28:142
231053099355,17cyclictest0-21swapper/219:19:252
231053099352,19cyclictest0-21swapper/223:02:482
231053099345,17cyclictest0-21swapper/222:34:252
231053099341,19cyclictest2358588-21chrt21:23:132
231053099340,20cyclictest2337663-21taskset20:25:062
231053099332,18cyclictest0-21swapper/223:41:182
231053099320,18cyclictest2330505-21munin-run20:05:012
231053099312,15cyclictest0-21swapper/223:39:392
231053099312,15cyclictest0-21swapper/219:35:292
231053099312,15cyclictest0-21swapper/219:35:282
231053099311,17cyclictest2322073-21/usr/sbin/munin19:40:132
231053099310,19cyclictest2427573-21chrt00:35:402
231053099304,14cyclictest2395840-21runrttasks23:08:562
231053099302,14cyclictest0-21swapper/221:58:292
231053099301,16cyclictest978-21snmpd00:23:532
231053099301,15cyclictest0-21swapper/220:05:282
231053099300,17cyclictest2333771-21seq20:12:482
231053099300,17cyclictest2306-21gsd-housekeepin23:31:282
231053099300,17cyclictest2306-21gsd-housekeepin23:16:292
231051999309,10cyclictest1095-21runrttasks22:49:000
231053099292,14cyclictest0-21swapper/223:49:292
231053099292,14cyclictest0-21swapper/222:19:282
231053099292,14cyclictest0-21swapper/221:36:292
231051999291,15cyclictest0-21swapper/023:41:050
231053099282,13cyclictest0-21swapper/200:08:282
2310519992811,15cyclictest2360908-21tr21:30:130
231053099272,12cyclictest0-21swapper/219:10:052
231051999276,19cyclictest2357339-21cat21:20:120
231051999272,22cyclictest2396390-21cpuspeed_turbos23:10:130
2310519992712,13cyclictest2425356-21grep00:30:220
231051999270,3cyclictest2371398-21/usr/sbin/munin22:00:130
231051999267,17cyclictest2347685-21cat20:50:230
231051999265,19cyclictest2342882-21find20:40:110
2310519992611,14cyclictest2320225-21cat19:35:000
2310519992611,13cyclictest13-21ksoftirqd/020:34:590
231053099259,14cyclictest2374739-21grep22:10:012
231052499250,23cyclictest2322101-21apt19:40:101
231051999256,17cyclictest2365875-21cut21:45:010
231051999255,18cyclictest2378888-21cat22:20:170
231051999255,18cyclictest2345233-21cat20:45:140
2310519992512,11cyclictest2329833-21processes20:00:230
139099250,4rtkit-daemon1389-21rtkit-daemon20:18:341
231053099249,13cyclictest2361957-21gmain21:32:052
231053099248,14cyclictest2350673-21expr21:00:132
2310530992411,11cyclictest2384791-21timerandwakeup22:35:212
231051999248,14cyclictest2423676-21cat00:25:220
231051999246,16cyclictest2394169-21date23:05:010
231051999245,17cyclictest2392372-21wc23:00:000
231051999245,17cyclictest2385689-21expr22:40:120
231051999245,17cyclictest2323030-21cat19:40:190
231051999244,18cyclictest2325922-21cat19:50:130
231051999243,19cyclictest2384730-21cat22:35:220
231051999243,19cyclictest1095-21runrttasks00:06:110
2310519992419,4cyclictest0-21swapper/020:56:430
2310519992411,12cyclictest13-21ksoftirqd/023:22:510
231051999240,22cyclictest2316476-21mktemp19:25:010
231051999240,1cyclictest0-21swapper/019:36:090
231051999240,1cyclictest0-21swapper/019:36:090
139099240,4rtkit-daemon1389-21rtkit-daemon22:06:351
23911142230,4sleep20-21swapper/222:55:152
231053099238,13cyclictest2423403-21head00:25:172
231053099236,15cyclictest2337415-21cron20:25:002
231053099234,17cyclictest913-21gdbus20:55:012
2310530992315,6cyclictest2325284-21cron19:50:002
2310524992317,4cyclictest2405454-21expr23:35:131
231051999239,12cyclictest2420217-21timerandwakeup00:15:230
231051999239,12cyclictest2420217-21timerandwakeup00:15:230
231051999239,12cyclictest2403124-21latency_hist23:30:000
231051999238,14cyclictest2426669-21cat00:35:150
231051999237,14cyclictest2342779-21date20:40:010
231051999235,16cyclictest2376794-21cut22:15:120
231051999235,16cyclictest2363-21evolution-alarm23:59:580
231051999231,5cyclictest2336369-21basename20:20:150
231051999231,20cyclictest2382687-21taskset22:30:210
2310519992311,10cyclictest2418852-21latency_hist00:15:010
2310519992311,10cyclictest2418852-21latency_hist00:15:010
139099230,16rtkit-daemon3056-21PK-Backend00:09:293
23605062220,4chrt0-21swapper/021:29:590
231053399222,19cyclictest2303216-21kworker/3:022:35:013
2310533992220,1cyclictest3056-21PK-Backend20:36:133
2310533992219,2cyclictest3056-21PK-Backend20:51:543
231053399220,20cyclictest3056-21PK-Backend21:47:243
231053399220,20cyclictest3056-21PK-Backend21:39:253
231053399220,20cyclictest3056-21PK-Backend20:22:483
231053399220,20cyclictest3056-21PK-Backend00:20:133
231053399220,20cyclictest3056-21PK-Backend00:02:293
231053099223,17cyclictest2355731-21tr21:15:132
2310530992220,1cyclictest0-21swapper/200:16:342
2310530992220,1cyclictest0-21swapper/200:16:342
2310530992217,4cyclictest0-21swapper/200:30:062
231053099221,20cyclictest2343339-21awk20:40:152
231053099221,19cyclictest2156-21gnome-shell19:26:542
231053099221,17cyclictest2339432-21apt20:30:132
231053099220,20cyclictest2328761-21perf20:00:002
231053099220,20cyclictest0-21swapper/219:52:112
231053099220,19cyclictest2396943-21ntp_states23:10:202
2310524992220,1cyclictest0-21swapper/120:37:221
2310524992215,5cyclictest436-20systemd-journal00:30:001
231052499220,20cyclictest0-21swapper/121:13:591
231051999228,13cyclictest2414162-21cat00:00:120
231051999228,12cyclictest2331640-21switchtime20:05:220
231051999228,12cyclictest2318365-21latency_hist19:29:590
231051999225,15cyclictest2354770-21timerandwakeup21:10:220
231051999224,17cyclictest2394587-21cpuspeed_turbos23:05:130
231051999224,16cyclictest2352389-21expr21:05:140
231051999223,5cyclictest2325422-21cat19:50:010
231051999223,17cyclictest2367986-21cat21:50:130
2310519992220,1cyclictest0-21swapper/023:45:290
2310519992220,1cyclictest0-21swapper/023:37:590
231051999221,9cyclictest2398311-21diskstats23:15:130
231051999221,8cyclictest2311338-21hddtemp_smartct19:10:160
231051999221,18cyclictest2356288-21ntp_states21:15:200
231051999221,18cyclictest2332495-21apt20:10:120
231051999221,18cyclictest0-21swapper/023:50:120
231051999221,18cyclictest0-21swapper/023:30:110
231051999220,7cyclictest2421482-21irqstats00:20:150
231051999220,20cyclictest2327466-21cpuspeed_turbos19:55:130
139099220,4rtkit-daemon1389-21rtkit-daemon20:14:271
139099220,14rtkit-daemon0-21swapper/223:52:092
139099220,14rtkit-daemon0-21swapper/223:22:162
2310533992119,1cyclictest3056-21PK-Backend23:26:193
2310533992119,1cyclictest3056-21PK-Backend23:15:113
2310533992119,1cyclictest3056-21PK-Backend20:55:523
2310533992119,1cyclictest3056-21PK-Backend20:49:383
2310533992119,1cyclictest3056-21PK-Backend00:27:103
2310533992119,1cyclictest0-21swapper/320:32:343
2310533992118,2cyclictest3056-21PK-Backend21:03:163
2310533992118,2cyclictest3056-21PK-Backend00:10:483
2310533992118,2cyclictest3056-21PK-Backend00:10:483
231053099219,10cyclictest0-21swapper/221:40:172
231053099217,12cyclictest0-21swapper/200:10:162
231053099217,12cyclictest0-21swapper/200:10:152
231053099214,15cyclictest2367541-21latency_hist21:50:002
2310530992119,1cyclictest0-21swapper/222:48:332
2310530992119,1cyclictest0-21swapper/221:11:052
231053099211,18cyclictest2353557-21munin-run21:10:002
231053099210,20cyclictest0-21swapper/222:25:492
231053099210,20cyclictest0-21swapper/222:25:492
231053099210,19cyclictest2347322-21awk20:50:192
2310524992119,1cyclictest0-21swapper/123:49:591
2310524992119,1cyclictest0-21swapper/123:27:431
2310524992119,1cyclictest0-21swapper/122:56:061
2310524992119,1cyclictest0-21swapper/122:51:531
2310524992119,1cyclictest0-21swapper/121:17:571
2310524992119,1cyclictest0-21swapper/121:07:431
2310524992119,1cyclictest0-21swapper/120:40:121
2310524992119,1cyclictest0-21swapper/120:32:351
2310524992119,1cyclictest0-21swapper/100:18:461
2310524992119,1cyclictest0-21swapper/100:18:451
2310524992118,2cyclictest0-21swapper/122:15:151
2310524992118,2cyclictest0-21swapper/121:20:131
2310524992115,4cyclictest2330483-21cron20:05:001
2310524992113,6cyclictest918-21rs:main1
231052499210,20cyclictest0-21swapper/123:51:551
231052499210,20cyclictest0-21swapper/123:22:421
231052499210,20cyclictest0-21swapper/123:17:071
231052499210,20cyclictest0-21swapper/119:17:461
231052499210,19cyclictest0-21swapper/123:30:131
231051999215,15cyclictest2334544-21expr20:15:130
231051999215,14cyclictest2374889-21apt22:10:110
2310519992119,1cyclictest0-21swapper/021:47:000
231051999211,8cyclictest2380519-21/usr/sbin/munin22:25:170
231051999211,8cyclictest2380519-21/usr/sbin/munin22:25:170
231051999211,8cyclictest2370452-21/usr/sbin/munin21:55:210
231051999211,5cyclictest2350770-21cat21:00:150
231051999211,17cyclictest2312969-21python319:15:030
2310533992018,1cyclictest3056-21PK-Backend23:38:033
2310533992018,1cyclictest3056-21PK-Backend23:34:583
2310533992018,1cyclictest3056-21PK-Backend20:17:273
2310533992017,2cyclictest3056-21PK-Backend23:56:073
2310533992017,2cyclictest3056-21PK-Backend19:35:023
2310533992017,2cyclictest3056-21PK-Backend19:35:023
2310533992013,5cyclictest2426229-21wc00:35:013
2310533992013,5cyclictest2338335-21cat20:25:173
2310533992012,5cyclictest436-20systemd-journal21:25:543
2310533992011,7cyclictest978-21snmpd21:10:003
231053399200,1cyclictest3056-21PK-Backend21:42:513
231053399200,19cyclictest3056-21PK-Backend23:11:333
231053399200,19cyclictest3056-21PK-Backend22:11:313
231053399200,19cyclictest3056-21PK-Backend21:14:053
231053099208,10cyclictest2334811-21missed_timers20:15:172
231053099206,12cyclictest2385674-21tr22:40:122
231053099203,15cyclictest0-21swapper/200:01:102
2310530992018,1cyclictest0-21swapper/222:51:372
231053099201,17cyclictest2368551-21python321:50:212
231053099201,17cyclictest2345660-21/usr/sbin/munin20:45:212
231053099201,17cyclictest2315795-21taskset19:20:212
231053099201,16cyclictest2156-21gnome-shell22:20:092
2310524992018,1cyclictest0-21swapper/122:32:471
2310524992018,1cyclictest0-21swapper/120:54:581
2310524992018,1cyclictest0-21swapper/119:37:401
2310524992018,1cyclictest0-21swapper/119:37:401
2310524992018,1cyclictest0-21swapper/100:14:191
2310524992018,1cyclictest0-21swapper/100:14:191
2310524992014,5cyclictest26-21ksoftirqd/121:55:201
231052499200,1cyclictest3056-21PK-Backend00:39:001
231052499200,1cyclictest0-21swapper/122:25:101
231052499200,1cyclictest0-21swapper/122:25:101
231052499200,19cyclictest0-21swapper/119:50:581
231052499200,19cyclictest0-21swapper/119:47:051
231052499200,19cyclictest0-21swapper/119:14:551
231052499200,19cyclictest0-21swapper/100:32:031
231051999203,15cyclictest2373617-21cat22:05:160
231051999200,7cyclictest2362696-21cat21:35:130
2310533991917,1cyclictest3056-21PK-Backend23:40:333
2310533991917,1cyclictest3056-21PK-Backend20:11:323
2310533991917,1cyclictest3056-21PK-Backend00:17:443
2310533991917,1cyclictest3056-21PK-Backend00:17:433
2310533991916,2cyclictest3056-21PK-Backend19:45:033
2310533991914,3cyclictest2427210-21ntp_states00:35:223
231053399190,1cyclictest3056-21PK-Backend23:00:173
231053099190,18cyclictest0-21swapper/223:27:202
231052499191,8cyclictest2415273-21perf00:05:011
2310524991917,1cyclictest0-21swapper/120:22:131
2310524991913,4cyclictest26-21ksoftirqd/123:10:011
2310524991912,6cyclictest2393062-21memory23:00:181
2310524991912,5cyclictest2379062-21ntpq22:20:181
231052499190,1cyclictest0-21swapper/119:33:141
231052499190,18cyclictest3056-21PK-Backend20:25:531
231051999190,1cyclictest0-21swapper/020:26:280
2310533991816,1cyclictest3056-21PK-Backend22:24:493
2310533991816,1cyclictest3056-21PK-Backend19:55:503
2310533991816,1cyclictest0-21swapper/321:15:473
231052499182,12cyclictest2371426-21apt22:00:121
2310524991816,1cyclictest0-21swapper/119:56:461
2310524991816,1cyclictest0-21swapper/100:20:421
2310524991813,3cyclictest2397664-21idleruntime-cro23:14:591
2310524991812,5cyclictest2174-21JS10:27:481
2310524991812,4cyclictest436-20systemd-journal21:49:551
231052499181,12cyclictest2385552-21apt22:40:101
231052499181,12cyclictest2383282-21apt-get22:35:031
2310524991810,6cyclictest26-21ksoftirqd/122:10:201
2310524991810,6cyclictest2317608-21ntpq19:25:201
2310533991715,1cyclictest3056-21PK-Backend19:43:073
2310533991715,1cyclictest3056-21PK-Backend19:32:313
2310533991714,2cyclictest3056-21PK-Backend19:26:243
2310533991713,3cyclictest42-21ksoftirqd/323:08:353
2310533991713,3cyclictest3056-21PK-Backend22:01:153
2310533991711,4cyclictest42-21ksoftirqd/319:15:033
2310533991710,5cyclictest2361079-21expr21:30:143
231052499179,6cyclictest2350293-21date21:00:011
231052499172,12cyclictest2406838-21munin-node23:40:091
2310524991715,1cyclictest0-21swapper/123:57:441
139099170,15rtkit-daemon0-21swapper/121:38:001
139099170,14rtkit-daemon0-21swapper/121:43:011
231053399163,11cyclictest2316399-21unin-run19:25:013
2310533991614,1cyclictest3056-21PK-Backend22:40:373
2310533991614,1cyclictest3056-21PK-Backend19:53:553
2310533991613,2cyclictest3056-21PK-Backend20:04:283
2310533991612,3cyclictest2389474-21if_err_enp2s022:50:153
2310533991610,4cyclictest2369168-21munin-plugin-st21:55:013
2310533991610,4cyclictest2358145-21smartctl21:20:213
231053399160,15cyclictest23884262sleep322:46:063
231052499169,5cyclictest2331497-21ntpq20:05:191
231052499163,11cyclictest2362211-21cron21:35:001
2310524991613,2cyclictest3056-21PK-Backend21:25:201
2310533991513,1cyclictest3056-21PK-Backend22:09:133
231053399150,3cyclictest23850742chrt22:37:323
231052499158,5cyclictest2315487-21expr19:20:161
2310524991510,3cyclictest1095-21runrttasks20:49:181
2310533991412,1cyclictest0-21swapper/323:23:153
2310533991411,2cyclictest3056-21PK-Backend22:17:513
2310533991410,3cyclictest3056-21PK-Backend22:55:443
231053399129,2cyclictest3056-21PK-Backend22:25:233
231053399129,2cyclictest3056-21PK-Backend22:25:233
231053399129,2cyclictest3056-21PK-Backend21:55:093
231053399120,11cyclictest3056-21PK-Backend20:05:183
231053399119,1cyclictest3056-21PK-Backend20:41:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional