You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-06 - 13:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Mar 06, 2026 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
158261099463456,4cyclictest1690175-21turbostat00:19:553
158261099462455,4cyclictest1589984-21turbostat19:30:003
158259599462455,4cyclictest1623173-21turbostat21:09:550
158261099461454,4cyclictest1598652-21turbostat19:55:003
158261099460433,24cyclictest1614220-21turbostat20:44:553
158260799459452,4cyclictest1658702-21turbostat22:49:552
158261099458451,4cyclictest1681499-21turbostat23:54:553
158261099457450,4cyclictest1662164-21turbostat22:55:003
158261099457450,4cyclictest1646583-21turbostat22:10:013
158261099457450,4cyclictest1646583-21turbostat22:10:013
158261099456447,6cyclictest1650053-21turbostat22:24:553
158259599456449,4cyclictest1628368-21turbostat21:24:550
158259599456449,4cyclictest1628368-21turbostat21:24:550
158261099455448,4cyclictest1694010-21turbostat00:25:003
158261099455448,4cyclictest1607301-21turbostat20:20:003
158261099454447,4cyclictest1679774-21turbostat23:49:563
158261099454447,4cyclictest1588217-21turbostat19:25:003
158261099454447,4cyclictest1588217-21turbostat19:25:003
158261099454446,5cyclictest1686717-21turbostat00:09:553
158261099453446,4cyclictest1664190-21turbostat23:04:553
158261099453446,4cyclictest1617979-21turbostat20:50:003
158261099452444,5cyclictest1584258-21turbostat19:15:003
158260799451444,4cyclictest1635298-21turbostat21:44:552
158261099450442,5cyclictest1662164-21turbostat22:59:563
158261099450442,5cyclictest1662164-21turbostat22:59:563
158261099450441,6cyclictest1598652-21turbostat19:59:553
158261099449442,4cyclictest1641085-21turbostat21:55:003
158261099449442,4cyclictest1586479-21turbostat19:20:003
158260799447440,4cyclictest1655245-21turbostat22:39:562
158261099444416,25cyclictest1595181-21turbostat19:45:003
158261099443432,8cyclictest1676319-21turbostat23:39:553
158261099437430,4cyclictest1630099-21turbostat21:25:003
158261099437430,4cyclictest1630099-21turbostat21:25:003
158261099437413,21cyclictest1688444-21turbostat00:14:563
158261099437408,26cyclictest1644860-21turbostat22:05:003
158261099435428,4cyclictest1674580-21turbostat23:30:003
158261099435427,5cyclictest1609028-21turbostat20:29:553
158260799433423,6cyclictest1602107-21turbostat20:09:552
158261099430405,22cyclictest1600376-21turbostat20:04:553
158261099428421,4cyclictest1653505-21turbostat22:30:003
158259599427420,4cyclictest1582485-21turbostat19:14:550
158261099424416,5cyclictest1595181-21turbostat19:49:553
158261099422414,5cyclictest1686717-21turbostat00:05:003
158261099421414,4cyclictest1669376-21turbostat23:19:553
158261099420412,5cyclictest1603834-21turbostat20:10:003
158261099420412,5cyclictest1591727-21turbostat19:39:553
158261099418411,4cyclictest1671120-21turbostat23:24:563
158261099418411,4cyclictest1671120-21turbostat23:24:563
158261099413406,4cyclictest1695745-21turbostat00:34:553
158261099410403,4cyclictest1695745-21turbostat00:30:003
158261099409401,5cyclictest1667651-21turbostat23:10:003
158261099408401,4cyclictest1612498-21turbostat20:35:013
158261099408401,4cyclictest1612498-21turbostat20:35:013
158261099408401,4cyclictest1607301-21turbostat20:24:553
158260799406399,4cyclictest1596915-21turbostat19:54:552
158261099404397,4cyclictest1650053-21turbostat22:20:003
158261099403395,5cyclictest1621450-21turbostat21:00:003
158261099403395,5cyclictest1621450-21turbostat21:00:003
158261099402395,4cyclictest1667651-21turbostat23:14:553
158261099402395,4cyclictest1589984-21turbostat19:34:553
158261099400393,4cyclictest1684969-21turbostat00:00:003
158261099400393,4cyclictest1684969-21turbostat00:00:003
158261099400393,4cyclictest1655245-21turbostat22:35:013
158261099400393,4cyclictest1655245-21turbostat22:35:013
158261099400393,4cyclictest1605567-21turbostat20:15:003
158261099400389,8cyclictest1643129-21turbostat22:00:003
158261099395387,5cyclictest1656976-21turbostat22:40:003
158260799395388,4cyclictest1674580-21turbostat23:34:552
158259599390383,4cyclictest1631826-21turbostat21:34:550
158261099387381,3cyclictest1628368-21turbostat21:20:003
158260799382373,6cyclictest1684969-21turbostat00:04:542
158261099361353,5cyclictest1646583-21turbostat22:14:563
158260799358351,4cyclictest1641085-21turbostat21:59:552
158261099356348,5cyclictest1656976-21turbostat22:44:543
158261099355348,4cyclictest1679774-21turbostat23:45:013
158261099354348,3cyclictest1676319-21turbostat23:35:003
158261099352345,4cyclictest1635298-21turbostat21:40:003
158261099348320,7cyclictest1660432-21turbostat22:50:003
158260799347340,4cyclictest1697477-21turbostat00:39:562
158260799344337,4cyclictest1648316-21turbostat22:19:552
158261099340334,4cyclictest1633572-21turbostat21:35:003
158261099340333,4cyclictest1626638-21turbostat21:15:003
158260799336329,4cyclictest1593444-21turbostat19:44:552
158261099330325,3cyclictest1639056-21turbostat21:50:003
158261099330323,4cyclictest1621450-21turbostat21:04:553
158260799330323,4cyclictest1617979-21turbostat20:54:552
158261099327321,3cyclictest1612498-21turbostat20:39:563
158260799324317,4cyclictest1678047-21turbostat23:44:552
158261099320313,4cyclictest1630099-21turbostat21:29:553
158261099317291,23cyclictest1624909-21turbostat21:10:003
158259599316308,5cyclictest1683234-21turbostat23:59:550
158259599316308,5cyclictest1683234-21turbostat23:59:550
158261099263249,10cyclictest1619716-21turbostat20:55:003
158261099251236,11cyclictest1699203-21turbostat00:40:003
158134522170,21sleep234-21ksoftirqd/219:05:172
1973991200,100rtkit-daemon0-21swapper/019:07:060
1582286210976,23sleep10-21swapper/119:07:501
1582222210875,23sleep30-21swapper/319:07:043
15826109910694,9cyclictest1637025-21turbostat21:45:013
15826079910493,8cyclictest1665927-21turbostat23:09:552
1973991000,9rtkit-daemon1972-21rtkit-daemon23:58:041
1973991000,9rtkit-daemon1972-21rtkit-daemon23:58:041
16592842650,16sleep126-21ksoftirqd/122:45:141
15912272620,6chrt0-21swapper/219:30:282
197399590,9rtkit-daemon1972-21rtkit-daemon23:11:052
197399570,10rtkit-daemon1656250-21ntp_states22:35:260
197399560,8rtkit-daemon1972-21rtkit-daemon23:48:392
16873732480,6chrt4497-21PK-Backend00:05:191
15849352470,16chrt34-21ksoftirqd/219:15:062
1582595994714,18cyclictest13-21ksoftirqd/023:45:230
1582599994629,13cyclictest976-21rs:main1
197399450,37rtkit-daemon4497-21PK-Backend23:26:022
1582595994433,8cyclictest13-21ksoftirqd/019:25:520
1582599994325,13cyclictest923-21dbus-daemon00:23:321
197399420,34rtkit-daemon4497-21PK-Backend23:37:042
16097262420,7sleep14497-21PK-Backend20:25:211
16085842420,5sleep10-21swapper/120:20:291
1582595994210,16cyclictest141rcu_preempt20:20:280
1582607994125,11cyclictest438-20systemd-journal22:01:392
158259599419,19cyclictest13-21ksoftirqd/000:30:100
197399400,5rtkit-daemon0-21swapper/121:45:101
16078632400,4sleep2331rcuc/220:20:142
1582607994015,5cyclictest141rcu_preempt20:45:132
1582607994010,6cyclictest141rcu_preempt19:25:522
1582599994022,13cyclictest1698446-21ntpq00:35:211
158259599402,8cyclictest1611429-21hddtemp_smartct20:30:210
158259599402,8cyclictest1611429-21hddtemp_smartct20:30:210
1582595994023,13cyclictest1617263-21ntp_states20:45:260
1582595994013,8cyclictest13-21ksoftirqd/022:05:290
1582595994013,8cyclictest13-21ksoftirqd/022:05:290
197399390,27rtkit-daemon4497-21PK-Backend21:04:051
197399390,26rtkit-daemon0-21swapper/019:39:070
197399390,25rtkit-daemon0-21swapper/022:46:010
197399390,25rtkit-daemon0-21swapper/019:55:580
197399390,25rtkit-daemon0-21swapper/000:21:090
158260799392,32cyclictest1637714-21cpuspeed_turbos21:45:132
158260799391,33cyclictest1054-21snmpd19:20:252
158260799391,33cyclictest1054-21snmpd19:20:252
1582599993921,14cyclictest1632460-21df21:30:171
1582595993910,9cyclictest13-21ksoftirqd/021:10:200
197399380,7rtkit-daemon4497-21PK-Backend19:55:291
197399380,25rtkit-daemon0-21swapper/022:22:040
197399380,25rtkit-daemon0-21swapper/020:07:410
197399380,24rtkit-daemon0-21swapper/020:50:050
16638812380,3chrt0-21swapper/122:56:291
16638812380,3chrt0-21swapper/122:56:291
15940952380,13sleep013-21ksoftirqd/019:40:160
1582607993823,11cyclictest1695234-21cat00:25:302
1582607993823,10cyclictest1632291-21cut21:30:162
1582607993821,13cyclictest1621641-21latency_hist21:00:012
1582607993821,13cyclictest1621641-21latency_hist21:00:012
1582607993821,12cyclictest1661068-21diskstats22:50:152
1582607993813,6cyclictest141rcu_preempt23:55:232
1582607993813,6cyclictest141rcu_preempt23:55:232
1582607993813,5cyclictest141rcu_preempt19:10:302
158259599389,5cyclictest141rcu_preempt23:40:020
158259599388,9cyclictest13-21ksoftirqd/020:00:090
158259599388,14cyclictest141rcu_preempt20:45:000
1582595993816,8cyclictest13-21ksoftirqd/019:45:290
1582595993812,8cyclictest13-21ksoftirqd/000:06:130
1582595993811,6cyclictest141rcu_preempt22:30:000
1582595993810,12cyclictest141rcu_preempt23:10:220
197399370,25rtkit-daemon4497-21PK-Backend21:22:131
197399370,25rtkit-daemon4497-21PK-Backend21:22:131
158260799372,10cyclictest0-21swapper/221:12:112
1582607993719,13cyclictest2345-21gdbus20:40:212
1582607993719,13cyclictest1647425-21load22:10:212
1582607993713,6cyclictest141rcu_preempt23:55:022
1582607993713,6cyclictest141rcu_preempt23:00:292
1582607993713,6cyclictest141rcu_preempt21:25:012
1582607993713,6cyclictest141rcu_preempt21:25:012
1582607993713,5cyclictest141rcu_preempt00:30:122
158260799371,32cyclictest1653456-21chrt22:29:372
1582607993712,6cyclictest141rcu_preempt19:55:132
1582607993712,5cyclictest141rcu_preempt22:40:212
1582607993710,6cyclictest141rcu_preempt23:15:202
158259999378,13cyclictest141rcu_preempt21:55:001
1582599993721,12cyclictest2193-21gdbus23:00:211
1582599993715,11cyclictest1633919-21cpuspeed_turbos21:35:131
158259999371,11cyclictest0-21swapper/120:59:481
158259999371,11cyclictest0-21swapper/120:59:481
158259599379,6cyclictest141rcu_preempt20:40:000
158259599378,6cyclictest141rcu_preempt23:40:230
158259599377,15cyclictest141rcu_preempt19:23:410
158259599377,15cyclictest141rcu_preempt19:23:410
1582595993720,12cyclictest438-20systemd-journal19:17:000
1582595993718,6cyclictest141rcu_preempt22:44:250
1582595993711,6cyclictest141rcu_preempt22:05:000
197399360,25rtkit-daemon4497-21PK-Backend00:13:461
197399360,25rtkit-daemon0-21swapper/022:50:210
197399360,25rtkit-daemon0-21swapper/021:26:060
197399360,24rtkit-daemon0-21swapper/120:45:061
197399360,24rtkit-daemon0-21swapper/000:03:200
197399360,23rtkit-daemon0-21swapper/122:44:441
16411012360,6chrt1641098-21awk21:55:002
1582607993627,6cyclictest1596920-21munin-plugin-st19:50:002
1582607993622,10cyclictest1693962-21sshd00:24:402
1582607993620,11cyclictest438-20systemd-journal20:19:392
1582607993617,14cyclictest1624163-21ntpq21:05:242
158260799361,30cyclictest1689449-21ntp_states00:10:242
1582607993612,6cyclictest141rcu_preempt21:30:012
1582599993621,11cyclictest1648313-21ls22:15:001
158259999361,9cyclictest0-21swapper/119:21:281
158259999361,9cyclictest0-21swapper/119:21:281
158259999361,9cyclictest0-21swapper/119:11:181
158259599367,7cyclictest13-21ksoftirqd/023:15:220
1582595993621,12cyclictest13-21ksoftirqd/022:18:060
1582595993620,5cyclictest13-21ksoftirqd/021:50:160
1582595993618,13cyclictest13-21ksoftirqd/023:50:140
197399350,24rtkit-daemon0-21swapper/021:19:270
158260799352,30cyclictest0-21swapper/200:20:012
1582607993519,12cyclictest1634274-21awk21:35:202
1582607993519,11cyclictest1609913-21grep20:25:232
1582607993512,5cyclictest141rcu_preempt21:15:192
158259999357,23cyclictest1697465-21perf00:35:011
1582599993525,6cyclictest1598642-21sh19:55:001
1582599993520,11cyclictest1627618-21ntpq21:15:231
158259999351,13cyclictest4497-21PK-Backend23:06:101
158259599357,8cyclictest13-21ksoftirqd/000:35:290
158259599357,7cyclictest141rcu_preempt20:55:130
158259599357,7cyclictest141rcu_preempt20:55:130
158259599355,8cyclictest13-21ksoftirqd/000:10:300
1582595993520,12cyclictest13-21ksoftirqd/019:30:130
1582595993519,12cyclictest13-21ksoftirqd/023:30:130
1582595993519,11cyclictest438-20systemd-journal20:11:390
1582595993517,13cyclictest1638781-40apt-check21:47:230
1582595993514,6cyclictest141rcu_preempt00:15:280
1582595993512,12cyclictest141rcu_preempt23:00:000
1582595993512,12cyclictest141rcu_preempt23:00:000
16806522340,8chrt4497-21PK-Backend23:45:221
16363612340,7chrt4497-21PK-Backend21:40:261
1582607993423,7cyclictest1604777-21needreboot20:10:212
1582607993421,9cyclictest34-21ksoftirqd/220:35:172
1582607993418,11cyclictest1651262-21threads22:20:322
1582607993417,12cyclictest1645758-21cat22:05:222
1582607993417,12cyclictest1645758-21cat22:05:222
1582599993428,4cyclictest0-21swapper/123:11:041
1582599993427,4cyclictest4497-21PK-Backend20:30:571
1582599993427,4cyclictest4497-21PK-Backend20:30:571
1582599993426,5cyclictest1586471-21cron19:20:011
158259999341,9cyclictest4497-21PK-Backend20:00:121
158259999341,9cyclictest4497-21PK-Backend19:47:001
158259999341,9cyclictest4497-21PK-Backend19:34:041
1582599993419,11cyclictest1675466-21grep23:30:221
158259999341,30cyclictest4497-21PK-Backend21:27:341
158259999341,14cyclictest4497-21PK-Backend00:18:381
158259999341,13cyclictest4497-21PK-Backend22:06:091
158259999341,13cyclictest4497-21PK-Backend22:06:091
158259599346,14cyclictest141rcu_preempt23:25:170
1582595993420,10cyclictest13-21ksoftirqd/023:00:160
1582595993419,11cyclictest13-21ksoftirqd/022:30:170
1582595993419,11cyclictest13-21ksoftirqd/022:30:170
1582595993418,6cyclictest13-21ksoftirqd/020:30:010
1582595993411,6cyclictest141rcu_preempt23:20:150
1582595993411,6cyclictest141rcu_preempt23:20:150
1582595993410,5cyclictest141rcu_preempt19:50:260
1582607993314,5cyclictest141rcu_preempt20:30:242
1582607993314,5cyclictest141rcu_preempt20:30:242
1582607993313,5cyclictest141rcu_preempt22:55:122
1582607993313,5cyclictest141rcu_preempt22:55:122
158260799331,29cyclictest0-21swapper/223:23:242
158260799331,29cyclictest0-21swapper/223:23:242
158260799331,29cyclictest0-21swapper/221:03:192
1582607993311,5cyclictest141rcu_preempt00:05:202
1582599993326,4cyclictest4497-21PK-Backend23:44:131
158259999331,9cyclictest4497-21PK-Backend22:16:551
1582599993319,10cyclictest1651356-21/usr/sbin/munin22:20:321
158259999331,8cyclictest4497-21PK-Backend20:17:431
158259999331,8cyclictest4497-21PK-Backend19:39:121
158259999331,8cyclictest4497-21PK-Backend00:01:031
1582599993318,11cyclictest1603356-21grep20:05:291
1582599993317,12cyclictest1673680-21cat23:25:241
1582599993317,11cyclictest1676577-21apt23:35:121
1582599993316,13cyclictest1588845-21df_inode19:25:151
158259999331,29cyclictest4497-21PK-Backend23:21:231
158259999331,29cyclictest4497-21PK-Backend23:21:231
158259999331,29cyclictest4497-21PK-Backend22:50:071
158259999331,29cyclictest4497-21PK-Backend20:11:531
158259999331,13cyclictest4497-21PK-Backend22:31:361
158259999331,13cyclictest4497-21PK-Backend22:31:361
158259999331,13cyclictest4497-21PK-Backend22:04:321
158259999331,13cyclictest4497-21PK-Backend21:08:131
158259599338,8cyclictest13-21ksoftirqd/000:28:470
158259599336,5cyclictest141rcu_preempt21:45:020
1582595993320,9cyclictest2069-21gnome-shell23:05:090
1582595993318,11cyclictest1647440-21grep22:10:220
1582595993318,11cyclictest13-21ksoftirqd/021:55:180
1582595993317,12cyclictest1633950-21grep21:35:130
1582595993313,5cyclictest141rcu_preempt20:15:210
16422302320,7chrt4497-21PK-Backend21:55:231
1582607993226,4cyclictest0-21swapper/222:31:302
1582607993226,4cyclictest0-21swapper/222:31:302
158260799321,28cyclictest4497-21PK-Backend20:04:502
158260799321,27cyclictest1592959-21sed19:35:292
1582599993225,4cyclictest4497-21PK-Backend19:44:031
1582599993219,9cyclictest1613743-21grep20:35:301
158259999321,8cyclictest4497-21PK-Backend22:36:031
1582599993217,11cyclictest1669644-21awk23:15:101
158259999321,4cyclictest4497-21PK-Backend20:40:581
1582595993216,5cyclictest13-21ksoftirqd/021:00:300
1582599993119,8cyclictest1695226-21xargs00:25:301
1582599993116,11cyclictest1653048-21sed22:25:301
1582599993016,10cyclictest1683220-21cron23:55:001
1582599992912,14cyclictest0-21swapper/120:55:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional