You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 16:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Wed May 06, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
258866021030,3sleep13056-21PK-Backend12:00:121
24977842990,4sleep33056-21PK-Backend07:40:193
139099760,60rtkit-daemon0-21swapper/307:08:563
248532626546,13sleep20-21swapper/207:07:542
248521026540,19sleep00-21swapper/007:06:230
248529826344,14sleep10-21swapper/107:07:331
25689952510,15sleep20-21swapper/211:05:122
25432512500,3sleep00-21swapper/009:50:160
139099380,3rtkit-daemon3056-21PK-Backend09:26:093
139099380,3rtkit-daemon3056-21PK-Backend09:10:273
139099370,3rtkit-daemon3056-21PK-Backend09:32:063
139099370,3rtkit-daemon3056-21PK-Backend08:58:493
139099370,3rtkit-daemon3056-21PK-Backend07:32:323
139099370,3rtkit-daemon0-21swapper/311:02:453
139099360,3rtkit-daemon3056-21PK-Backend09:48:343
139099350,3rtkit-daemon3056-21PK-Backend10:00:343
25028312340,4sleep13056-21PK-Backend07:55:131
25994952270,4chrt3056-21PK-Backend12:30:161
248564499278,10cyclictest13-21ksoftirqd/011:24:590
24951182260,3sleep2331rcuc/207:35:012
248565499250,23cyclictest2579798-21cstates11:35:143
248564499256,10cyclictest13-21ksoftirqd/011:35:000
248564499256,10cyclictest13-21ksoftirqd/011:35:000
24989132240,3sleep30-21swapper/307:45:123
2485650992411,11cyclictest2600410-21unin-run12:35:012
2485644992412,7cyclictest2583754-21cat11:45:150
248564499240,6cyclictest3056-21PK-Backend11:26:150
248565499230,21cyclictest2600387-21/usr/sbin/munin12:35:003
2485650992311,10cyclictest2561469-21unin-run10:45:002
2485650992311,10cyclictest2520207-21diskstats08:45:132
248564499239,13cyclictest2568402-21unin-run11:05:000
248564499232,9cyclictest13-21ksoftirqd/010:50:190
2485644992321,1cyclictest0-21swapper/010:03:370
2485644992318,4cyclictest13-21ksoftirqd/011:15:010
2485654992220,1cyclictest0-21swapper/308:37:233
2485654992220,1cyclictest0-21swapper/308:37:233
2485654992220,1cyclictest0-21swapper/307:50:583
248565499220,21cyclictest0-21swapper/311:23:373
2485650992214,6cyclictest2489020-21ntp_kernel_err07:15:192
2485648992219,2cyclictest3056-21PK-Backend10:10:011
2485648992213,7cyclictest2591711-21grep12:10:001
248564899220,20cyclictest3056-21PK-Backend11:36:391
2485644992220,1cyclictest0-21swapper/012:26:480
2485644992220,1cyclictest0-21swapper/012:26:480
2485644992220,1cyclictest0-21swapper/010:48:530
2485644992220,1cyclictest0-21swapper/008:34:360
2485644992220,1cyclictest0-21swapper/008:13:060
2485644992220,1cyclictest0-21swapper/007:16:350
2485644992218,3cyclictest161rcuc/009:44:160
2485644992218,3cyclictest161rcuc/009:13:010
2485644992214,6cyclictest2502632-21cpuspeed_turbos07:55:110
2485654992119,1cyclictest3056-21PK-Backend09:54:423
2485654992119,1cyclictest0-21swapper/311:59:333
2485654992119,1cyclictest0-21swapper/310:57:213
2485654992119,1cyclictest0-21swapper/310:50:593
2485654992119,1cyclictest0-21swapper/310:10:023
2485654992118,2cyclictest0-21swapper/312:25:163
2485654992118,2cyclictest0-21swapper/312:25:163
2485654992118,2cyclictest0-21swapper/311:50:543
248565499211,7cyclictest2490793-21ntpq07:20:193
248565499211,5cyclictest2595170-21perf12:19:593
248565499210,20cyclictest4176944-21snapd12:09:143
248565499210,20cyclictest0-21swapper/310:20:583
248565499210,20cyclictest0-21swapper/310:05:503
248565499210,20cyclictest0-21swapper/308:09:583
248565499210,20cyclictest0-21swapper/307:14:073
2485650992114,6cyclictest1-21systemd11:40:002
2485648992119,1cyclictest3056-21PK-Backend10:43:051
2485648992119,1cyclictest3056-21PK-Backend10:37:321
2485648992119,1cyclictest3056-21PK-Backend10:37:321
2485648992119,1cyclictest3056-21PK-Backend08:34:291
2485648992119,1cyclictest3056-21PK-Backend07:24:321
2485648992119,1cyclictest0-21swapper/109:59:461
2485648992119,1cyclictest0-21swapper/109:32:081
2485648992119,1cyclictest0-21swapper/109:19:521
2485648992119,1cyclictest0-21swapper/107:45:001
2485648992119,1cyclictest0-21swapper/107:30:421
248564899210,19cyclictest3056-21PK-Backend11:10:361
248564499214,7cyclictest13-21ksoftirqd/009:00:190
248564499213,6cyclictest13-21ksoftirqd/012:05:150
248564499213,10cyclictest13-21ksoftirqd/011:50:190
2485644992119,1cyclictest0-21swapper/012:03:230
2485644992119,1cyclictest0-21swapper/010:35:290
2485644992119,1cyclictest0-21swapper/010:35:280
2485644992119,1cyclictest0-21swapper/009:31:540
2485644992119,1cyclictest0-21swapper/009:28:570
2485644992119,1cyclictest0-21swapper/008:59:160
2485644992119,1cyclictest0-21swapper/008:42:130
2485644992119,1cyclictest0-21swapper/008:25:020
2485644992119,1cyclictest0-21swapper/007:49:200
2485644992116,4cyclictest13-21ksoftirqd/011:16:030
2485644992115,4cyclictest13-21ksoftirqd/010:00:010
2485644992110,3cyclictest13-21ksoftirqd/011:35:120
248564499210,20cyclictest0-21swapper/010:57:180
248564499210,20cyclictest0-21swapper/008:00:150
25976052200,4chrt3056-21PK-Backend12:25:141
25976052200,4chrt3056-21PK-Backend12:25:141
25620992200,4chrt0-21swapper/210:45:162
2485654992018,1cyclictest0-21swapper/312:12:313
2485654992018,1cyclictest0-21swapper/311:32:573
2485654992018,1cyclictest0-21swapper/311:32:563
2485654992018,1cyclictest0-21swapper/310:37:073
2485654992018,1cyclictest0-21swapper/310:37:063
2485654992018,1cyclictest0-21swapper/310:29:033
2485654992018,1cyclictest0-21swapper/307:56:193
2485654992015,3cyclictest2581686-21diskstats11:40:143
248565499201,14cyclictest2512032-21chrt08:20:203
248565499200,1cyclictest0-21swapper/311:14:463
248565499200,19cyclictest3056-21PK-Backend10:30:513
248565499200,19cyclictest3056-21PK-Backend08:53:513
248565499200,19cyclictest0-21swapper/312:21:573
248565499200,19cyclictest0-21swapper/311:48:383
248565499200,19cyclictest0-21swapper/311:29:253
248565499200,19cyclictest0-21swapper/310:45:393
248565499200,19cyclictest0-21swapper/310:15:313
248565499200,19cyclictest0-21swapper/308:28:233
2485650992018,1cyclictest0-21swapper/212:19:572
2485650992018,1cyclictest0-21swapper/210:13:022
2485650992018,1cyclictest0-21swapper/209:26:592
2485650992015,3cyclictest2156-21gnome-shell08:41:092
2485650992013,5cyclictest2553471-21cat10:20:152
2485650992012,6cyclictest2590299-21cat12:05:112
2485650992010,8cyclictest883-21dbus-daemon09:41:032
248565099200,19cyclictest2566647-21munin-run11:00:012
248565099200,19cyclictest0-21swapper/212:38:022
248565099200,19cyclictest0-21swapper/212:10:172
248565099200,19cyclictest0-21swapper/211:04:052
248565099200,19cyclictest0-21swapper/208:37:112
248565099200,19cyclictest0-21swapper/208:37:102
248565099200,19cyclictest0-21swapper/208:26:372
248565099200,19cyclictest0-21swapper/208:13:042
248565099200,19cyclictest0-21swapper/207:38:582
2485648992018,1cyclictest0-21swapper/112:35:211
2485648992018,1cyclictest0-21swapper/109:03:571
248564899200,1cyclictest0-21swapper/108:59:511
248564899200,19cyclictest3056-21PK-Backend07:11:491
248564899200,19cyclictest0-21swapper/107:39:411
2485644992018,1cyclictest0-21swapper/012:34:110
2485644992018,1cyclictest0-21swapper/010:43:330
2485644992018,1cyclictest0-21swapper/010:27:230
2485644992018,1cyclictest0-21swapper/010:11:020
2485644992018,1cyclictest0-21swapper/007:22:570
2485644992017,2cyclictest161rcuc/012:18:070
2485644992015,4cyclictest13-21ksoftirqd/012:25:000
248564499200,6cyclictest2570093-21cron11:10:010
248564499200,1cyclictest0-21swapper/010:21:570
248564499200,1cyclictest0-21swapper/008:50:030
248564499200,19cyclictest0-21swapper/011:40:300
248564499200,19cyclictest0-21swapper/010:06:340
248564499200,19cyclictest0-21swapper/008:22:380
248564499200,19cyclictest0-21swapper/008:15:360
248564499200,19cyclictest0-21swapper/007:11:280
248565499192,13cyclictest2488319-21apt07:15:143
2485654991917,1cyclictest0-21swapper/312:02:523
2485654991917,1cyclictest0-21swapper/308:46:483
2485654991917,1cyclictest0-21swapper/308:01:543
2485654991913,4cyclictest2539879-21awk09:40:163
2485654991912,5cyclictest42-21ksoftirqd/308:10:213
248565499190,6cyclictest2519569-21turbostat.cron08:45:003
248565499190,1cyclictest0-21swapper/308:18:303
248565499190,18cyclictest0-21swapper/310:41:333
2485650991917,1cyclictest0-21swapper/212:01:382
2485650991917,1cyclictest0-21swapper/211:58:202
2485650991917,1cyclictest0-21swapper/211:45:092
2485650991917,1cyclictest0-21swapper/211:19:412
2485650991917,1cyclictest0-21swapper/210:53:392
2485650991917,1cyclictest0-21swapper/210:06:482
2485650991917,1cyclictest0-21swapper/209:56:472
2485650991917,1cyclictest0-21swapper/209:35:522
2485650991917,1cyclictest0-21swapper/209:23:272
2485650991917,1cyclictest0-21swapper/207:57:572
248565099190,1cyclictest3056-21PK-Backend08:51:032
248565099190,1cyclictest0-21swapper/211:32:312
248565099190,1cyclictest0-21swapper/211:32:312
248565099190,1cyclictest0-21swapper/209:31:182
248565099190,1cyclictest0-21swapper/208:15:352
248565099190,1cyclictest0-21swapper/208:07:022
248565099190,1cyclictest0-21swapper/208:03:302
248565099190,1cyclictest0-21swapper/207:46:012
248565099190,18cyclictest3056-21PK-Backend07:25:462
248565099190,18cyclictest0-21swapper/211:43:542
248565099190,18cyclictest0-21swapper/211:23:132
248565099190,18cyclictest0-21swapper/210:25:472
248565099190,18cyclictest0-21swapper/209:13:342
248565099190,18cyclictest0-21swapper/208:58:052
248565099190,18cyclictest0-21swapper/207:23:292
2485648991917,1cyclictest3056-21PK-Backend11:59:381
2485648991917,1cyclictest3056-21PK-Backend10:57:531
2485648991917,1cyclictest3056-21PK-Backend10:54:151
2485648991917,1cyclictest0-21swapper/109:46:421
2485648991916,2cyclictest3056-21PK-Backend08:04:551
2485648991914,3cyclictest2498516-21idleruntime-cro07:45:001
2485648991913,4cyclictest1095-21runrttasks09:50:211
248564899190,1cyclictest3056-21PK-Backend11:30:451
248564899190,1cyclictest3056-21PK-Backend11:30:441
248564899190,1cyclictest3056-21PK-Backend11:24:361
248564899190,1cyclictest0-21swapper/109:28:371
248564899190,1cyclictest0-21swapper/108:42:471
248564899190,18cyclictest0-21swapper/110:01:551
2485644991917,1cyclictest0-21swapper/011:58:020
2485644991917,1cyclictest0-21swapper/008:47:260
2485644991917,1cyclictest0-21swapper/008:08:200
2485644991917,1cyclictest0-21swapper/007:29:330
248564499191,6cyclictest13-21ksoftirqd/009:15:100
2485644991916,2cyclictest161rcuc/009:48:090
2485644991914,3cyclictest2502054-21cat07:55:000
2485644991912,6cyclictest2497348-21ls07:40:130
248564499190,1cyclictest3056-21PK-Backend10:31:490
248564499190,1cyclictest0-21swapper/010:16:290
248564499190,1cyclictest0-21swapper/009:39:240
248564499190,1cyclictest0-21swapper/007:39:120
2485654991816,1cyclictest3056-21PK-Backend09:58:003
2485654991816,1cyclictest3056-21PK-Backend09:21:493
2485654991816,1cyclictest3056-21PK-Backend09:05:423
248565499181,3cyclictest2600649-21apt12:35:113
2485654991810,6cyclictest436-20systemd-journal11:16:153
2485650991816,1cyclictest0-21swapper/210:03:502
2485650991816,1cyclictest0-21swapper/209:50:582
2485650991813,3cyclictest2558969-21ntp_offset10:35:172
2485650991813,3cyclictest2558969-21ntp_offset10:35:172
2485650991812,4cyclictest1095-21runrttasks07:50:092
2485650991810,6cyclictest436-20systemd-journal09:05:002
2485650991810,6cyclictest1-21systemd07:15:002
248565099180,1cyclictest0-21swapper/211:51:442
2485648991816,1cyclictest3056-21PK-Backend11:06:111
2485648991816,1cyclictest3056-21PK-Backend08:06:581
2485648991816,1cyclictest0-21swapper/109:38:371
248564899181,13cyclictest82332-21update-manager09:41:041
2485648991810,6cyclictest1788610-21gdbus11:26:061
248564499184,3cyclictest13-21ksoftirqd/012:37:510
2485644991816,1cyclictest0-21swapper/007:34:530
2485644991814,3cyclictest161rcuc/009:09:040
2485644991814,3cyclictest13-21ksoftirqd/008:40:010
2485644991814,3cyclictest13-21ksoftirqd/008:40:010
2485644991812,4cyclictest2592136-21cpuspeed_turbos12:10:110
2485644991810,6cyclictest436-20systemd-journal09:25:010
248565499174,9cyclictest2514636-21apt08:30:113
2485654991715,1cyclictest3056-21PK-Backend09:16:363
2485654991715,1cyclictest3056-21PK-Backend07:38:483
2485654991714,2cyclictest3056-21PK-Backend07:28:073
2485654991711,4cyclictest2569707-21awk11:05:213
248565099176,2cyclictest141rcu_preempt11:25:152
2485650991715,1cyclictest0-21swapper/207:42:282
2485650991711,4cyclictest2511340-21grep08:20:132
248564899179,6cyclictest883-21dbus-daemon08:55:011
248564899179,6cyclictest2532673-21awk09:20:151
248564899172,13cyclictest2487762-21munin-node07:15:021
2485648991715,1cyclictest3056-21PK-Backend11:53:571
2485648991715,1cyclictest3056-21PK-Backend11:16:521
2485648991715,1cyclictest3056-21PK-Backend08:24:291
2485648991715,1cyclictest3056-21PK-Backend08:13:061
2485648991715,1cyclictest0-21swapper/110:31:101
2485648991715,1cyclictest0-21swapper/109:10:041
2485648991712,3cyclictest2567065-21tr11:00:121
2485648991710,5cyclictest436-20systemd-journal12:19:451
2485648991710,5cyclictest2492381-21sed07:25:161
248565099169,6cyclictest2514872-21expr08:30:122
248565099169,5cyclictest2596118-21netstat12:20:172
248565099169,5cyclictest2556855-21expr10:30:132
248565099166,2cyclictest141rcu_preempt09:05:132
2485650991614,1cyclictest3056-21PK-Backend09:17:262
2485650991614,1cyclictest0-21swapper/210:15:522
2485650991614,1cyclictest0-21swapper/209:48:282
2485650991611,3cyclictest2571702-21uptime11:10:212
2485650991610,5cyclictest2597393-21expr12:25:112
2485650991610,5cyclictest2597393-21expr12:25:112
2485648991614,1cyclictest3056-21PK-Backend11:45:241
2485648991614,1cyclictest3056-21PK-Backend11:44:231
2485648991614,1cyclictest3056-21PK-Backend08:26:391
248564899161,11cyclictest2526990-21apt-get09:05:031
248564899160,1cyclictest3056-21PK-Backend10:12:241
248564899160,15cyclictest3056-21PK-Backend10:47:451
248564899160,15cyclictest3056-21PK-Backend10:20:341
248564899160,15cyclictest3056-21PK-Backend07:54:201
2485654991511,3cyclictest3056-21PK-Backend09:38:483
248565499150,5cyclictest886-21NetworkManager09:05:013
2485648991513,1cyclictest3056-21PK-Backend12:11:471
2485648991513,1cyclictest0-21swapper/112:20:341
2485648991512,2cyclictest3056-21PK-Backend08:35:171
2485648991512,2cyclictest3056-21PK-Backend08:35:161
248564899141,5cyclictest26-21ksoftirqd/108:15:211
2485648991412,1cyclictest3056-21PK-Backend08:48:561
248564899139,3cyclictest3056-21PK-Backend10:27:331
2485648991310,2cyclictest3056-21PK-Backend10:18:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional