You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-11 - 09:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jun 11, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
161090922480,15sleep2341ksoftirqd/219:05:112
161204728040,14sleep10-21swapper/119:07:011
161202427839,13sleep30-21swapper/319:06:433
161070126543,14sleep00-21swapper/019:05:030
16868402490,3sleep33056-21PK-Backend22:40:113
16129832440,5sleep33056-21PK-Backend19:10:123
16200742360,3sleep126-21ksoftirqd/119:30:111
1612422993610,18cyclictest3056-21PK-Backend19:43:331
161242299357,15cyclictest0-21swapper/123:10:011
161242299330,18cyclictest1627872-21sed19:50:201
161242299321,15cyclictest1895007-21snapd22:59:141
161242299320,18cyclictest1674199-21taskset22:03:431
161242299320,18cyclictest1651235-21runrttasks20:56:431
161242299320,17cyclictest0-21swapper/100:25:021
161242299310,19cyclictest1626664-21seq19:49:381
161242299310,18cyclictest1677739-21taskset22:14:421
161242299310,18cyclictest1672471-21taskset21:58:451
161242299310,17cyclictest1639334-21taskset20:25:031
161242299310,16cyclictest1666880-21sed21:40:221
161242299310,16cyclictest0-21swapper/121:16:041
161242299310,16cyclictest0-21swapper/121:16:041
161242299301,18cyclictest1638949-21taskset20:23:161
161242299291,16cyclictest1682769-21taskset22:27:211
161242299290,15cyclictest1654872-21seq21:08:571
161242299281,13cyclictest0-21swapper/123:46:581
161242299280,16cyclictest0-21swapper/121:52:481
161242299280,15cyclictest1658956-21seq21:20:081
139099280,20rtkit-daemon1707529-21cron23:40:003
161242299271,14cyclictest1702159-21runrttasks23:22:511
161242299271,14cyclictest1619478-21runrttasks19:26:251
1612422992711,14cyclictest436-20systemd-journal19:40:001
161242299270,1cyclictest0-21swapper/121:13:251
161242299270,1cyclictest0-21swapper/121:13:251
161242299270,16cyclictest1668985-21runrttasks21:48:321
161242299270,14cyclictest0-21swapper/121:30:141
161242299270,14cyclictest0-21swapper/119:11:091
16827862260,13sleep342-21ksoftirqd/322:27:363
1612433992612,12cyclictest1616337-21unin-run19:20:003
161242299260,14cyclictest1684772-21sed22:35:011
161242299260,14cyclictest1633194-21seq20:05:171
16171522250,4sleep03056-21PK-Backend19:20:150
161243399250,23cyclictest436-20systemd-journal21:34:143
1612425992521,3cyclictest978-21snmpd19:11:202
16781292240,3sleep2331rcuc/222:15:112
16415902240,2sleep2331rcuc/220:30:142
1612433992420,2cyclictest1720259-21expr00:15:133
161242299249,13cyclictest1636030-21cpuspeed_turbos20:15:121
1612422992412,11cyclictest1648282-21/usr/sbin/munin20:50:161
161242299240,23cyclictest0-21swapper/123:19:101
161242299240,22cyclictest1684967-21/usr/sbin/munin22:35:161
161242299240,14cyclictest1696518-21taskset23:05:361
17046022230,4sleep30-21swapper/323:30:113
1612433992321,1cyclictest3056-21PK-Backend21:05:593
1612433992321,1cyclictest0-21swapper/319:27:023
1612433992320,2cyclictest0-21swapper/323:06:243
1612433992315,6cyclictest883-21dbus-daemon20:26:103
1612433992311,11cyclictest1637364-21perf20:19:593
161242299238,13cyclictest1723572-21wc00:25:011
161242299232,17cyclictest1628710-21apt19:55:121
161242299230,21cyclictest1687522-21grep22:40:191
139099230,16rtkit-daemon1211-21kerneloops21:15:082
139099230,16rtkit-daemon1211-21kerneloops21:15:082
16558602220,4sleep33056-21PK-Backend21:10:173
16558602220,4sleep33056-21PK-Backend21:10:163
1612433992220,1cyclictest3056-21PK-Backend21:15:263
1612433992220,1cyclictest3056-21PK-Backend21:15:263
1612433992220,1cyclictest0-21swapper/323:47:213
1612433992220,1cyclictest0-21swapper/322:49:013
1612433992220,1cyclictest0-21swapper/319:50:213
1612433992220,1cyclictest0-21swapper/319:42:183
1612433992214,6cyclictest1714089-21users23:55:223
1612425992220,1cyclictest0-21swapper/222:41:002
1612425992220,1cyclictest0-21swapper/222:21:482
1612425992220,1cyclictest0-21swapper/221:58:572
1612425992220,1cyclictest0-21swapper/221:25:312
1612425992219,2cyclictest1666586-21strings21:40:172
161242299229,11cyclictest1617606-21timerwakeupswit19:20:201
161242299227,13cyclictest1714475-21cron00:04:551
161242299221,19cyclictest1712762-21munin-run23:55:011
1612422992210,11cyclictest1665655-21ls21:40:001
161242299220,21cyclictest1-21systemd20:05:001
161242299220,20cyclictest1694383-21/usr/sbin/munin23:00:181
161241999229,11cyclictest1656574-21fwupdmgr21:13:250
161241999229,11cyclictest1656574-21fwupdmgr21:13:240
1612419992220,1cyclictest3056-21PK-Backend20:26:060
1612419992220,1cyclictest0-21swapper/000:31:170
161241999221,18cyclictest1686466-21sh22:40:000
161241999220,10cyclictest1704070-21idleruntime-cro23:29:590
139099220,14rtkit-daemon0-21swapper/322:55:093
139099220,14rtkit-daemon0-21swapper/221:46:332
1612433992119,1cyclictest3056-21PK-Backend21:24:043
1612433992119,1cyclictest0-21swapper/323:54:313
1612433992119,1cyclictest0-21swapper/323:19:433
1612433992119,1cyclictest0-21swapper/321:36:473
1612433992119,1cyclictest0-21swapper/300:39:193
1612433992118,2cyclictest3056-21PK-Backend23:02:183
1612433992114,5cyclictest1662172-21cron21:30:003
161243399210,20cyclictest3056-21PK-Backend22:36:093
161243399210,20cyclictest3056-21PK-Backend21:59:403
161243399210,20cyclictest0-21swapper/320:12:123
161243399210,20cyclictest0-21swapper/319:59:013
161243399210,1cyclictest0-21swapper/323:12:333
161243399210,19cyclictest3056-21PK-Backend00:25:123
1612425992119,1cyclictest3056-21PK-Backend23:50:032
1612425992119,1cyclictest0-21swapper/223:11:422
1612425992119,1cyclictest0-21swapper/222:38:582
1612425992119,1cyclictest0-21swapper/221:21:522
1612425992119,1cyclictest0-21swapper/221:00:302
1612425992119,1cyclictest0-21swapper/220:44:212
1612425992119,1cyclictest0-21swapper/220:08:382
1612425992119,1cyclictest0-21swapper/219:39:522
1612425992114,5cyclictest883-21dbus-daemon00:08:022
1612425992114,5cyclictest1710822-21sshd23:47:412
161242599210,20cyclictest0-21swapper/222:05:532
161242599210,20cyclictest0-21swapper/222:03:502
161242599210,20cyclictest0-21swapper/221:10:232
161242599210,20cyclictest0-21swapper/221:10:232
161242599210,20cyclictest0-21swapper/220:12:302
161242599210,20cyclictest0-21swapper/220:04:462
161242299217,12cyclictest1707581-21latency_hist23:40:011
161242299217,12cyclictest1645509-21grep20:40:171
161242299214,15cyclictest1714518-40updatedb.mlocat00:00:001
161242299213,16cyclictest918-21rs:main1
1612422992119,1cyclictest0-21swapper/122:45:181
1612422992119,1cyclictest0-21swapper/122:23:411
1612422992111,9cyclictest1651745-21/usr/sbin/munin21:00:191
161242299211,18cyclictest17195722sleep100:13:361
161242299211,18cyclictest1675053-21interrupts22:05:151
161242299211,18cyclictest1647063-21/usr/sbin/munin20:45:171
161242299211,18cyclictest1643234-21awk20:35:121
1612422992110,10cyclictest1615317-21irqstats19:15:151
1612419992119,1cyclictest3056-21PK-Backend23:46:220
1612419992119,1cyclictest3056-21PK-Backend23:44:190
1612419992119,1cyclictest3056-21PK-Backend22:59:090
1612419992119,1cyclictest3056-21PK-Backend22:34:420
1612419992119,1cyclictest3056-21PK-Backend21:43:330
1612419992119,1cyclictest3056-21PK-Backend19:19:140
1612419992119,1cyclictest0-21swapper/023:20:250
1612419992119,1cyclictest0-21swapper/022:07:390
1612419992119,1cyclictest0-21swapper/022:00:560
1612419992119,1cyclictest0-21swapper/021:00:410
1612419992119,1cyclictest0-21swapper/020:58:510
1612419992119,1cyclictest0-21swapper/019:11:300
1612419992119,1cyclictest0-21swapper/000:21:150
1612419992118,2cyclictest3056-21PK-Backend23:06:200
161241999210,20cyclictest3056-21PK-Backend21:25:480
161241999210,19cyclictest3056-21PK-Backend23:31:280
161241999210,19cyclictest3056-21PK-Backend21:37:180
139099210,15rtkit-daemon3056-21PK-Backend20:03:143
139099210,14rtkit-daemon3056-21PK-Backend19:35:073
139099210,14rtkit-daemon0-21swapper/219:45:002
139099210,13rtkit-daemon0-21swapper/219:53:232
1612433992018,1cyclictest3056-21PK-Backend22:03:323
1612433992018,1cyclictest3056-21PK-Backend20:06:313
1612433992018,1cyclictest0-21swapper/320:55:183
1612433992018,1cyclictest0-21swapper/320:50:523
1612433992018,1cyclictest0-21swapper/320:46:253
1612433992013,4cyclictest1684124-21python322:30:203
161243399201,17cyclictest1643708-21python320:35:193
161243399200,8cyclictest1645681-21/usr/sbin/munin20:40:213
161243399200,1cyclictest0-21swapper/321:40:273
161243399200,1cyclictest0-21swapper/300:02:413
161243399200,19cyclictest0-21swapper/323:42:143
161243399200,19cyclictest0-21swapper/322:19:003
161243399200,19cyclictest0-21swapper/319:46:513
161243399200,18cyclictest3056-21PK-Backend21:00:323
161243399200,18cyclictest0-21swapper/321:45:073
1612425992018,1cyclictest0-21swapper/222:25:542
1612425992018,1cyclictest0-21swapper/221:51:262
1612425992018,1cyclictest0-21swapper/221:32:472
1612425992018,1cyclictest0-21swapper/220:27:232
1612425992018,1cyclictest0-21swapper/219:28:442
1612425992018,1cyclictest0-21swapper/219:20:192
1612425992012,5cyclictest28650irq/123-ahci[0000:00:202
161242599200,1cyclictest0-21swapper/222:34:532
161242599200,1cyclictest0-21swapper/200:10:452
161242599200,19cyclictest0-21swapper/223:39:342
161242599200,19cyclictest0-21swapper/223:25:152
161242599200,19cyclictest0-21swapper/223:16:512
161242599200,19cyclictest0-21swapper/222:55:072
161242599200,19cyclictest0-21swapper/222:11:472
161242599200,19cyclictest0-21swapper/221:08:342
161242599200,19cyclictest0-21swapper/220:46:582
161242599200,19cyclictest0-21swapper/220:23:192
161242599200,19cyclictest0-21swapper/219:45:402
161242599200,19cyclictest0-21swapper/219:16:132
161242299208,10cyclictest1660924-21cstates21:25:121
161242299207,11cyclictest0-21swapper/123:40:181
161242299206,12cyclictest1725698-21cat00:30:111
161242299206,12cyclictest1634933-21open_inodes20:10:201
161242299201,17cyclictest1721959-21latency_hist00:20:001
161242299201,17cyclictest1641882-21taskset20:30:171
161242299201,16cyclictest0-21swapper/122:15:121
1612422992010,9cyclictest1727204-21/usr/sbin/munin00:35:171
161241999203,15cyclictest1660434-21cron21:25:000
161241999202,13cyclictest1626980-21taskset19:50:110
1612419992018,1cyclictest3056-21PK-Backend23:13:160
1612419992018,1cyclictest3056-21PK-Backend22:41:390
1612419992018,1cyclictest3056-21PK-Backend19:46:590
1612419992018,1cyclictest3056-21PK-Backend00:00:280
1612419992018,1cyclictest0-21swapper/021:56:170
1612419992018,1cyclictest0-21swapper/019:40:240
1612419992018,1cyclictest0-21swapper/019:33:480
161241999200,1cyclictest3056-21PK-Backend23:57:220
161241999200,19cyclictest3056-21PK-Backend20:53:380
161241999200,19cyclictest3056-21PK-Backend20:42:350
161241999200,19cyclictest3056-21PK-Backend20:36:000
161241999200,19cyclictest3056-21PK-Backend00:29:130
161241999200,19cyclictest0-21swapper/023:02:130
161241999200,19cyclictest0-21swapper/022:11:310
161241999200,19cyclictest0-21swapper/019:26:240
161241999200,18cyclictest3056-21PK-Backend00:18:120
139099200,14rtkit-daemon3056-21PK-Backend22:10:583
139099200,13rtkit-daemon0-21swapper/220:18:372
139099200,12rtkit-daemon3056-21PK-Backend00:20:533
1612433991917,1cyclictest3056-21PK-Backend22:07:233
161243399190,1cyclictest3056-21PK-Backend23:21:463
161243399190,1cyclictest3056-21PK-Backend21:50:343
161243399190,1cyclictest3056-21PK-Backend19:31:233
161243399190,1cyclictest0-21swapper/323:26:523
161243399190,1cyclictest0-21swapper/322:22:043
161243399190,18cyclictest3056-21PK-Backend20:24:503
1612425991917,1cyclictest0-21swapper/222:51:022
1612425991917,1cyclictest0-21swapper/219:31:002
1612425991917,1cyclictest0-21swapper/200:35:312
1612425991913,4cyclictest28650irq/123-ahci[0023:30:192
161242599190,1cyclictest0-21swapper/223:59:152
161242599190,18cyclictest0-21swapper/200:18:562
161242299191,16cyclictest1690177-21/usr/sbin/munin22:50:101
161242299190,17cyclictest1704027-21sshd23:29:381
161241999192,12cyclictest1711287-21apt23:50:100
161241999192,12cyclictest1621777-21apt19:35:110
1612419991917,1cyclictest3056-21PK-Backend22:48:220
1612419991917,1cyclictest3056-21PK-Backend22:16:250
1612419991917,1cyclictest3056-21PK-Backend21:05:070
1612419991917,1cyclictest3056-21PK-Backend20:18:220
1612419991917,1cyclictest0-21swapper/023:16:200
1612419991917,1cyclictest0-21swapper/022:20:170
1612419991917,1cyclictest0-21swapper/021:50:490
1612419991917,1cyclictest0-21swapper/020:31:000
1612419991917,1cyclictest0-21swapper/020:02:340
1612419991912,5cyclictest1633290-21sensors_temp20:05:190
161241999190,1cyclictest3056-21PK-Backend22:26:590
161241999190,1cyclictest0-21swapper/021:45:220
161241999190,18cyclictest3056-21PK-Backend20:24:040
161241999190,18cyclictest3056-21PK-Backend19:57:060
161241999190,18cyclictest0-21swapper/021:17:580
161241999190,18cyclictest0-21swapper/021:17:580
139099190,13rtkit-daemon0-21swapper/221:38:422
161243399185,2cyclictest141rcu_preempt00:06:193
1612433991810,6cyclictest1616941-21df_inode19:20:123
161242599184,9cyclictest34-21ksoftirqd/223:20:032
1612425991816,1cyclictest0-21swapper/222:48:582
1612425991815,2cyclictest1095-21runrttasks19:56:322
1612425991811,5cyclictest1095-21runrttasks20:50:112
161242299180,16cyclictest1717830-21chrt00:08:311
1612419991816,1cyclictest3056-21PK-Backend00:07:240
1612419991815,2cyclictest1646666-21cpuspeed_turbos20:45:130
1612419991815,2cyclictest0-21swapper/021:30:140
1612433991711,5cyclictest1718644-21ls00:10:153
1612433991710,5cyclictest1690555-21df22:50:133
161243399170,5cyclictest1641104-21awk20:30:093
161243399170,3cyclictest1726073-21memory00:30:163
1612425991715,1cyclictest0-21swapper/220:56:502
161242599171,12cyclictest1707809-21apt23:40:102
1612419991715,1cyclictest3056-21PK-Backend23:38:240
161242599169,5cyclictest1726085-21expr00:30:152
161242599167,7cyclictest1643029-21cpuspeed_turbos20:35:112
161242599163,8cyclictest1722027-21apt00:20:102
1612425991614,1cyclictest0-21swapper/200:25:182
1612425991611,3cyclictest1695387-21acpi23:05:092
1612419991613,2cyclictest3056-21PK-Backend20:13:430
1612419991611,3cyclictest1719320-21chrt00:10:230
1612425991511,3cyclictest1095-21runrttasks23:04:342
161241999159,4cyclictest1691660-21turbostat22:55:010
1612419991412,1cyclictest3056-21PK-Backend00:38:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional