You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 21:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Jun 03, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
164120521180,4sleep00-21swapper/010:45:190
165852621140,3sleep30-21swapper/311:35:183
156327427341,26sleep30-21swapper/307:09:063
139099730,59rtkit-daemon0-21swapper/107:05:161
156314326646,14sleep20-21swapper/207:07:242
156326126344,14sleep00-21swapper/007:08:550
16465532520,1sleep30-21swapper/311:00:223
16409902500,3sleep13056-21PK-Backend10:45:161
16233202460,3sleep20-21swapper/209:55:172
16512742410,4sleep13056-21PK-Backend11:15:161
139099380,3rtkit-daemon0-21swapper/007:29:010
139099380,3rtkit-daemon0-21swapper/007:29:000
139099370,3rtkit-daemon0-21swapper/011:55:000
139099370,3rtkit-daemon0-21swapper/010:22:130
139099370,3rtkit-daemon0-21swapper/008:44:280
139099360,3rtkit-daemon3056-21PK-Backend10:36:370
139099360,3rtkit-daemon0-21swapper/012:07:070
139099360,3rtkit-daemon0-21swapper/009:15:520
139099340,3rtkit-daemon0-21swapper/009:26:310
139099340,3rtkit-daemon0-21swapper/008:48:210
156355399336,16cyclictest1663358-21irqstats11:50:162
1563553993110,19cyclictest1607883-21wc09:15:012
156355399308,8cyclictest0-21swapper/209:45:032
156355399307,9cyclictest1634691-21cut10:30:002
1563553993023,6cyclictest0-21swapper/209:40:182
156355399297,6cyclictest1593996-21apt08:35:122
156354699299,4cyclictest13-21ksoftirqd/008:05:020
15838412280,2sleep13056-21PK-Backend08:05:131
15662512280,3chrt0-21swapper/007:15:110
1563546992810,5cyclictest13-21ksoftirqd/012:13:150
1563553992711,14cyclictest1657844-21cpuspeed_turbos11:35:112
156355399269,15cyclictest1583358-21unin-run08:05:002
156355399264,8cyclictest0-21swapper/208:20:132
156355399260,12cyclictest1650498-21awk11:15:002
1563546992610,14cyclictest1655662-21cron11:30:000
16130302250,8chrt42-21ksoftirqd/309:30:013
1563553992515,8cyclictest1636008-21/usr/sbin/munin10:30:202
1563553992510,5cyclictest1675882-21python312:25:232
156355399250,23cyclictest1601447-21latency08:55:162
1563555992419,4cyclictest42-21ksoftirqd/310:20:003
156355599241,21cyclictest1565338-21sshd07:14:173
156355399243,8cyclictest0-21swapper/207:40:152
1563553992416,6cyclictest1567037-21python307:15:212
1563553992415,5cyclictest1638622-21apt10:40:112
1563553992414,8cyclictest1626806-21runrttasks10:05:172
1563553992414,6cyclictest1608115-21apt09:15:142
1563553992414,6cyclictest1568783-21python307:20:222
1563553992413,9cyclictest1671292-21idleruntime-cro12:15:002
156354699249,4cyclictest13-21ksoftirqd/011:09:340
1563546992412,4cyclictest13-21ksoftirqd/007:37:270
156355599230,21cyclictest1593908-21latency_hist08:35:013
156355399239,12cyclictest1578809-21idleruntime07:50:162
156355399233,2cyclictest0-21swapper/211:33:002
156355399233,2cyclictest0-21swapper/208:50:442
1563553992319,3cyclictest0-21swapper/210:17:362
1563553992315,6cyclictest1625434-21find10:00:232
1563553992314,7cyclictest436-20systemd-journal12:35:002
1563553992314,7cyclictest1642948-21python310:50:202
1563553992314,6cyclictest1580991-21python307:55:212
1563553992314,5cyclictest1661150-21apt11:45:112
1563553992313,8cyclictest1654220-21taskset11:25:132
1563553992312,9cyclictest883-21dbus-daemon07:50:012
1563553992312,9cyclictest1571115-21perf07:29:592
1563553992312,9cyclictest1571115-21perf07:29:592
1563553992311,10cyclictest1680365-21cat12:40:012
156354699239,5cyclictest13-21ksoftirqd/008:34:370
1563546992316,6cyclictest1624215-21cron10:00:000
1563546992311,4cyclictest13-21ksoftirqd/010:32:570
1563546992310,5cyclictest13-21ksoftirqd/010:54:340
1563546992310,4cyclictest13-21ksoftirqd/007:45:440
1563546992310,3cyclictest13-21ksoftirqd/011:36:300
139099230,6rtkit-daemon0-21swapper/109:55:181
139099230,15rtkit-daemon3056-21PK-Backend09:45:171
156355599222,5cyclictest1587920-21python308:15:203
156355399227,13cyclictest0-21swapper/207:35:002
1563553992216,3cyclictest1597460-21apt08:45:122
1563553992215,5cyclictest34-21ksoftirqd/212:00:202
1563553992215,3cyclictest1611584-21apt09:25:122
1563553992213,7cyclictest1604795-21grep09:05:122
1563553992213,7cyclictest1590921-21smartctl08:25:162
1563553992213,7cyclictest1565416-21cron07:15:002
1563553992212,8cyclictest1614041-21/usr/sbin/munin09:30:172
1563553992210,10cyclictest1629493-21cat10:15:002
1563550992220,1cyclictest3056-21PK-Backend10:26:441
1563550992220,1cyclictest3056-21PK-Backend09:12:211
1563550992220,1cyclictest3056-21PK-Backend08:55:181
156355099220,20cyclictest3056-21PK-Backend12:13:241
156355099220,20cyclictest3056-21PK-Backend09:37:431
156355099220,20cyclictest3056-21PK-Backend07:39:541
156355099220,20cyclictest3056-21PK-Backend07:17:301
156354699229,5cyclictest13-21ksoftirqd/011:44:400
1563546992220,1cyclictest0-21swapper/008:28:330
1563546992220,1cyclictest0-21swapper/007:43:090
1563546992211,4cyclictest13-21ksoftirqd/010:55:110
1563546992211,3cyclictest13-21ksoftirqd/009:33:420
1563546992210,4cyclictest13-21ksoftirqd/010:03:100
1563546992210,4cyclictest13-21ksoftirqd/009:49:580
156354699220,20cyclictest0-21swapper/009:22:270
156354699220,20cyclictest0-21swapper/009:22:260
139099220,14rtkit-daemon3056-21PK-Backend11:43:081
139099220,13rtkit-daemon0-21swapper/012:31:360
156355599214,15cyclictest1579158-21open_inodes07:50:193
156355599212,17cyclictest1586188-21python308:10:213
1563555992119,1cyclictest0-21swapper/307:22:413
156355599211,7cyclictest1632003-21memory10:20:173
1563555992113,6cyclictest1639200-21memory10:40:163
156355399219,10cyclictest1653935-21latency_hist11:25:022
156355399219,10cyclictest1621419-21fw_packets09:50:132
1563553992119,1cyclictest0-21swapper/210:47:512
1563553992119,1cyclictest0-21swapper/210:36:152
1563553992119,1cyclictest0-21swapper/207:39:052
1563553992115,3cyclictest1650747-21apt11:15:102
1563553992115,3cyclictest1602918-21apt09:00:112
1563553992113,6cyclictest82332-21update-manager12:24:372
1563553992113,6cyclictest1597176-21cron08:45:012
1563553992113,6cyclictest1586798-21munin-plugin-st08:15:002
1563553992113,6cyclictest1583771-21grep08:05:122
1563553992112,7cyclictest1665356-21taskset11:55:192
1563553992112,7cyclictest1610694-21/usr/sbin/munin09:20:182
1563553992112,7cyclictest1610694-21/usr/sbin/munin09:20:182
1563553992112,7cyclictest1587191-21tr08:15:142
1563553992111,8cyclictest436-20systemd-journal08:34:332
1563553992111,8cyclictest1671615-21cat12:15:112
1563553992111,8cyclictest1668488-21/usr/sbin/munin12:05:142
1563553992111,8cyclictest1647287-21/usr/sbin/munin11:05:122
1563553992111,7cyclictest0-21swapper/210:20:112
1563553992110,8cyclictest0-21swapper/209:35:112
1563553992110,10cyclictest1647010-21latency_hist11:05:002
1563550992119,1cyclictest3056-21PK-Backend11:31:131
1563550992119,1cyclictest3056-21PK-Backend11:23:491
1563550992119,1cyclictest3056-21PK-Backend10:50:301
1563550992119,1cyclictest3056-21PK-Backend09:41:151
1563550992119,1cyclictest3056-21PK-Backend07:27:451
1563550992119,1cyclictest3056-21PK-Backend07:27:451
1563550992119,1cyclictest0-21swapper/110:56:041
1563550992119,1cyclictest0-21swapper/110:16:361
1563550992118,2cyclictest3056-21PK-Backend08:24:031
156355099210,20cyclictest3056-21PK-Backend12:31:281
156355099210,20cyclictest3056-21PK-Backend08:02:461
156355099210,19cyclictest3056-21PK-Backend11:10:031
156355099210,19cyclictest3056-21PK-Backend11:01:261
156355099210,19cyclictest3056-21PK-Backend10:22:111
156355099210,19cyclictest3056-21PK-Backend10:00:411
156354699219,4cyclictest13-21ksoftirqd/010:42:120
156354699219,4cyclictest13-21ksoftirqd/009:42:410
156354699219,4cyclictest13-21ksoftirqd/009:35:110
156354699219,4cyclictest13-21ksoftirqd/007:53:460
156354699218,4cyclictest13-21ksoftirqd/008:19:280
156354699217,5cyclictest13-21ksoftirqd/010:09:590
156354699217,4cyclictest13-21ksoftirqd/007:32:300
156354699212,10cyclictest13-21ksoftirqd/008:05:220
1563546992119,1cyclictest0-21swapper/012:38:220
1563546992119,1cyclictest0-21swapper/010:19:050
1563546992119,1cyclictest0-21swapper/008:38:120
1563546992119,1cyclictest0-21swapper/008:20:070
1563546992119,1cyclictest0-21swapper/007:23:290
1563546992119,1cyclictest0-21swapper/007:11:200
1563546992112,2cyclictest141rcu_preempt11:58:220
1563546992111,4cyclictest13-21ksoftirqd/012:16:080
1563546992110,4cyclictest13-21ksoftirqd/011:34:560
1563546992110,4cyclictest13-21ksoftirqd/011:19:000
1563546992110,4cyclictest13-21ksoftirqd/011:11:280
1563546992110,4cyclictest13-21ksoftirqd/010:29:400
1563546992110,4cyclictest13-21ksoftirqd/009:14:260
1563546992110,4cyclictest13-21ksoftirqd/007:57:320
1563546992110,3cyclictest13-21ksoftirqd/012:28:460
139099210,15rtkit-daemon3056-21PK-Backend09:01:101
16050862200,9chrt26-21ksoftirqd/109:05:151
1563555992018,1cyclictest0-21swapper/312:24:363
1563555992018,1cyclictest0-21swapper/311:44:423
1563555992018,1cyclictest0-21swapper/310:53:033
1563555992015,4cyclictest1644558-21ntp_states10:55:183
1563555992014,5cyclictest1575491-21load07:40:173
1563555992013,5cyclictest436-20systemd-journal08:05:013
1563555992012,6cyclictest1651326-21memory11:15:173
156355599201,17cyclictest1625355-21python310:00:213
156355599200,1cyclictest3056-21PK-Backend11:50:023
156355599200,1cyclictest0-21swapper/309:36:523
156355599200,19cyclictest0-21swapper/312:10:573
156355599200,19cyclictest0-21swapper/311:26:503
156355599200,19cyclictest0-21swapper/310:05:523
156355599200,19cyclictest0-21swapper/309:58:503
156355599200,19cyclictest0-21swapper/309:42:403
156355599200,19cyclictest0-21swapper/309:18:403
156355599200,19cyclictest0-21swapper/309:08:483
156355599200,19cyclictest0-21swapper/307:30:463
1563553992018,1cyclictest0-21swapper/211:44:432
1563553992011,7cyclictest1644183-21df_inode10:55:132
1563550992018,1cyclictest3056-21PK-Backend12:17:161
1563550992018,1cyclictest3056-21PK-Backend11:59:181
1563550992018,1cyclictest3056-21PK-Backend10:12:051
1563550992018,1cyclictest3056-21PK-Backend09:33:101
1563550992018,1cyclictest3056-21PK-Backend09:27:361
1563550992018,1cyclictest3056-21PK-Backend09:24:191
1563550992018,1cyclictest3056-21PK-Backend09:24:181
1563550992018,1cyclictest3056-21PK-Backend08:48:421
1563550992018,1cyclictest3056-21PK-Backend07:31:231
1563550992018,1cyclictest3056-21PK-Backend07:21:431
1563550992018,1cyclictest0-21swapper/109:52:371
1563550992017,2cyclictest3056-21PK-Backend12:07:431
156355099200,2cyclictest3056-21PK-Backend07:45:011
156355099200,1cyclictest3056-21PK-Backend12:21:081
156355099200,1cyclictest3056-21PK-Backend07:54:561
156355099200,19cyclictest3056-21PK-Backend10:44:421
156355099200,19cyclictest3056-21PK-Backend07:57:121
156355099200,19cyclictest3056-21PK-Backend07:12:441
156355099200,19cyclictest0-21swapper/110:36:521
156355099200,18cyclictest3056-21PK-Backend07:42:591
156354699209,4cyclictest13-21ksoftirqd/011:21:450
156354699209,4cyclictest13-21ksoftirqd/010:11:120
156354699209,4cyclictest13-21ksoftirqd/009:06:480
156354699209,4cyclictest13-21ksoftirqd/008:54:530
156354699209,4cyclictest13-21ksoftirqd/008:10:340
156354699208,4cyclictest13-21ksoftirqd/011:49:520
1563546992018,1cyclictest0-21swapper/012:20:300
1563546992018,1cyclictest0-21swapper/008:57:460
1563546992010,4cyclictest13-21ksoftirqd/009:51:520
156354699200,1cyclictest3056-21PK-Backend12:02:460
156354699200,19cyclictest0-21swapper/011:00:340
15963162190,4chrt3056-21PK-Backend08:40:171
156355599197,4cyclictest141rcu_preempt12:35:043
156355599197,10cyclictest1576632-21/usr/sbin/munin07:45:153
156355599193,14cyclictest1661697-21ls11:45:163
1563555991917,1cyclictest0-21swapper/310:38:093
1563555991917,1cyclictest0-21swapper/310:31:203
1563555991917,1cyclictest0-21swapper/309:12:043
1563555991913,5cyclictest42-21ksoftirqd/311:55:133
1563555991913,4cyclictest42-21ksoftirqd/310:15:003
1563555991912,5cyclictest42-21ksoftirqd/307:39:373
1563555991912,4cyclictest1641216-21python310:45:213
156355599191,16cyclictest1600053-21python308:50:213
1563555991911,6cyclictest1567032-21python307:15:203
156355599190,1cyclictest0-21swapper/312:18:423
156355599190,1cyclictest0-21swapper/310:27:503
156355599190,1cyclictest0-21swapper/308:49:133
156355599190,1cyclictest0-21swapper/308:26:223
156355599190,1cyclictest0-21swapper/307:27:073
156355599190,1cyclictest0-21swapper/307:27:073
156355599190,18cyclictest0-21swapper/312:33:083
156355599190,18cyclictest0-21swapper/312:25:373
156355599190,18cyclictest0-21swapper/312:01:443
156355599190,18cyclictest0-21swapper/311:32:103
156355599190,18cyclictest0-21swapper/311:23:323
156355599190,18cyclictest0-21swapper/311:12:513
156355599190,18cyclictest0-21swapper/309:31:043
156355599190,18cyclictest0-21swapper/309:24:153
156355599190,18cyclictest0-21swapper/309:24:153
156355599190,18cyclictest0-21swapper/308:39:203
156355599190,18cyclictest0-21swapper/308:22:023
1563550991917,1cyclictest3056-21PK-Backend12:39:001
1563550991917,1cyclictest3056-21PK-Backend10:34:341
1563550991917,1cyclictest3056-21PK-Backend08:18:421
1563550991916,2cyclictest3056-21PK-Backend10:07:321
1563550991916,2cyclictest3056-21PK-Backend08:51:001
1563550991916,2cyclictest3056-21PK-Backend08:14:081
1563550991913,4cyclictest1663369-21kernelversion11:50:151
156355099190,1cyclictest3056-21PK-Backend12:26:491
156355099190,1cyclictest3056-21PK-Backend11:06:451
156355099190,1cyclictest3056-21PK-Backend09:17:431
156354699198,4cyclictest13-21ksoftirqd/009:01:270
156355599184,12cyclictest42-21ksoftirqd/308:43:243
1563555991816,1cyclictest0-21swapper/311:09:333
1563555991816,1cyclictest0-21swapper/309:04:283
1563555991811,5cyclictest42-21ksoftirqd/308:55:163
156355599181,15cyclictest1619239-21cat09:45:113
156355599181,15cyclictest1584456-21python308:05:223
1563550991811,5cyclictest1661260-21cat11:45:121
139099180,13rtkit-daemon3056-21PK-Backend12:04:151
1563555991715,1cyclictest0-21swapper/312:05:023
156355599170,15cyclictest1580132-21apt07:55:123
156355599169,5cyclictest1621581-21cat09:50:153
1563550991614,1cyclictest3056-21PK-Backend08:34:431
139099160,13rtkit-daemon3056-21PK-Backend11:28:061
156355099148,4cyclictest1659184-21grep11:40:011
1563550991412,1cyclictest3056-21PK-Backend08:29:231
156355099139,3cyclictest1594131-21tr08:35:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional