You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-09 - 18:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Jun 09, 2026 12:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1637578995837,6cyclictest171rcu_preempt09:40:282
1637577995547,5cyclictest1705455-21grep10:00:011
1637576995442,8cyclictest12650irq/32-eth010:05:310
1637576995228,19cyclictest12750irq/33-eth010:30:250
1637578995123,6cyclictest171rcu_preempt10:20:282
1637576994940,6cyclictest439-21systemd-logind08:20:010
1637576994938,8cyclictest151ktimers/008:40:240
1637576994938,8cyclictest151ktimers/008:40:240
1637578994843,2cyclictest0-21swapper/212:10:312
1637577994840,5cyclictest1697542-21unixbench_multi09:45:351
1637576994832,13cyclictest1730827-21munin-run10:40:010
1637578994636,6cyclictest1676759-21debian-sa109:15:012
1637578994631,7cyclictest0-21swapper/212:05:252
163757799465,32cyclictest1731934-21ssh10:40:291
163757699463,35cyclictest1789024-21ssh12:10:300
163757899451,39cyclictest1697040-21ssh09:45:272
1637578994433,8cyclictest1655029-21sshd08:05:292
1637578994433,8cyclictest1649556-21/usr/sbin/munin07:50:022
163757799443,32cyclictest1782634-21ssh12:00:271
1637576994429,12cyclictest0-21swapper/010:30:010
1637576994323,17cyclictest0-21swapper/011:20:010
1637576994322,15cyclictest12650irq/32-eth009:20:260
163757699432,14cyclictest1674626-21ntpq09:10:260
1637576994316,24cyclictest1686493-21cat09:30:020
1637576994315,25cyclictest1695952-21cron09:45:010
163757699421,8cyclictest1668532-21sshd08:50:290
1637576994216,24cyclictest1740471-21latency_hist10:55:020
1637576994212,6cyclictest1757787-21grep11:20:380
1637576994210,5cyclictest12750irq/33-eth011:05:230
1637578994118,6cyclictest171rcu_preempt12:25:292
163757799414,31cyclictest1712838-21ssh10:10:241
1637577994133,5cyclictest1695998-21latency_hist09:45:021
163757699414,8cyclictest1664013-21ntpq08:35:260
163757699414,8cyclictest1664013-21ntpq08:35:260
1637576994136,3cyclictest151ktimers/009:30:240
1637576994117,22cyclictest439-21systemd-logind09:10:010
1637576994116,22cyclictest0-21swapper/008:45:350
163757699411,15cyclictest1795706-21expr12:20:370
163757699411,13cyclictest1776391-21ps11:50:300
163757899407,22cyclictest1673554-21diskmemload10:32:292
163757899403,31cyclictest1677847-21ssh09:15:272
1637578994031,6cyclictest265-21systemd-journal10:00:132
1637578994021,14cyclictest351ktimers/212:30:222
1637577994030,8cyclictest1660875-21needreboot08:25:221
163757699409,12cyclictest1768671-21kworker/u8:12+rpciod12:00:220
163757699403,8cyclictest1706499-21ssh10:00:250
1637576994034,4cyclictest12650irq/32-eth008:00:200
1637576994034,4cyclictest12650irq/32-eth008:00:200
163757699402,11cyclictest1753041-21munin-run11:15:020
1637576994018,19cyclictest1746673-21ssh11:05:010
1637576994015,2cyclictest0-21swapper/009:55:270
163757699401,4cyclictest1773086-21ntpq11:45:250
1637576994013,24cyclictest1659380-21needreboot08:20:200
1637576994012,2cyclictest0-21swapper/008:25:230
163757699401,16cyclictest1712886-21sh10:10:250
163757699401,15cyclictest1740959-21sort10:55:160
163757699401,10cyclictest1763095-21ssh11:30:150
1637578993932,4cyclictest0-21swapper/212:40:012
163757799394,28cyclictest1674566-21ssh09:10:231
163757699395,30cyclictest1766700-21ssh11:35:220
163757699395,10cyclictest1690671-21ssh09:35:270
1637576993929,5cyclictest0-21swapper/008:55:220
163757699391,9cyclictest665-21snmpd07:20:230
163757699391,13cyclictest1645904-21ntpq07:35:200
163757699390,15cyclictest1643524-21users07:25:330
1637578993830,5cyclictest1748239-21grep11:05:352
1637578993828,6cyclictest1-21systemd10:30:012
1637578993817,17cyclictest351ktimers/209:00:272
163757799382,32cyclictest1804719-21ssh12:35:271
163757799382,29cyclictest1677979-21awk09:15:301
163757799380,15cyclictest1763582-21ssh11:30:251
163757699384,10cyclictest1735108-21ssh10:45:270
1637576993812,2cyclictest0-21swapper/012:15:120
163757699381,14cyclictest1785686-21ssh12:05:200
163757699381,13cyclictest1671464-21ntpq09:00:230
163757699380,20cyclictest1779117-21/usr/sbin/munin11:55:210
163757699380,14cyclictest1647538-21ntp_states07:40:350
163757899379,5cyclictest171rcu_preempt11:45:292
1637578993730,3cyclictest0-21swapper/210:20:012
1637578993727,6cyclictest1743538-21kworker/u8:111:00:012
163757799374,25cyclictest265-21systemd-journal10:30:291
1637577993729,5cyclictest1757772-21sed11:20:381
1637577993729,5cyclictest1647991-21unixbench_singl07:40:421
1637577993729,4cyclictest0-21swapper/111:35:211
1637577993720,3cyclictest171rcu_preempt10:45:311
163757799371,5cyclictest171rcu_preempt12:30:221
163757699374,13cyclictest627-21lldpd12:35:270
163757699373,8cyclictest1648932-21ntpq07:45:230
1637576993712,2cyclictest0-21swapper/007:10:140
163757699371,15cyclictest1651544-21diskstats07:55:160
163757699371,11cyclictest1700125-21ntpq09:50:230
163757699371,11cyclictest1698868-21ssh09:49:170
163757699370,14cyclictest1769889-21sh11:40:220
163757899363,25cyclictest1769985-21ssh11:40:262
1637578993629,4cyclictest1702367-21latency_hist09:55:022
1637578993626,6cyclictest1730862-21systemd-run10:40:012
1637578993612,18cyclictest351ktimers/209:25:232
163757799366,20cyclictest1760379-21ssh11:25:231
1637577993628,5cyclictest1740346-21systemd-run10:55:011
1637577993628,5cyclictest1721312-21cron10:25:011
163757799362,24cyclictest1776345-21ssh11:50:291
163757699361,12cyclictest1639906-21ntpq07:15:210
163757699361,11cyclictest1678803-21ssh09:16:410
1637576993610,4cyclictest1802394-21sh12:31:100
163757899355,23cyclictest1731881-21ssh10:40:272
1637578993527,5cyclictest421-21dbus-daemon09:40:012
1637577993529,3cyclictest0-21swapper/109:30:171
1637577993527,5cyclictest1756212-21tr11:20:011
1637577993525,7cyclictest1779235-21missed_timers11:55:221
1637577993525,6cyclictest0-21swapper/109:20:351
1637577993520,5cyclictest1683772-21ssh09:25:171
163757699352,8cyclictest1650461-21ntpq07:50:240
163757699351,17cyclictest1718228-21ssh10:20:030
1637576993511,6cyclictest1655575-21kworker/u8:14+xprtiod08:15:010
1637576993511,4cyclictest12650irq/32-eth011:25:200
163757699351,12cyclictest1644097-21tr07:30:160
163757699351,11cyclictest1719235-21ntpq10:20:270
1637578993428,3cyclictest0-21swapper/212:05:012
1637578993426,5cyclictest1709712-21ntp_kernel_pll_10:05:282
1637578993424,5cyclictest0-21swapper/211:30:222
163757799343,20cyclictest1747790-21ssh11:05:271
1637577993428,3cyclictest0-21swapper/107:20:001
1637577993411,11cyclictest28-21ksoftirqd/108:10:221
1637577993410,5cyclictest28-21ksoftirqd/108:30:301
1637576993427,4cyclictest1798775-21threads12:25:340
163757699341,6cyclictest1731647-21grep10:40:210
1637578993327,3cyclictest0-21swapper/208:40:012
1637578993327,3cyclictest0-21swapper/208:40:012
1637578993325,3cyclictest0-21swapper/211:50:252
163757799339,13cyclictest0-21swapper/108:45:231
1637577993326,4cyclictest1767018-21kworker/u8:10-rpciod11:50:011
1637577993326,3cyclictest0-21swapper/111:40:261
1637577993325,4cyclictest0-21swapper/112:15:271
1637577993323,7cyclictest1653366-21needreboot08:00:191
1637577993323,7cyclictest1653366-21needreboot08:00:191
163757799331,24cyclictest1700247-21ssh09:50:281
1637578993224,5cyclictest1660086-21systemd-run08:25:012
1637578993224,5cyclictest1652578-21systemd-run08:00:012
1637578993224,3cyclictest0-21swapper/211:15:272
1637578993222,4cyclictest1002-21mta-sts-daemon08:55:282
163757899321,7cyclictest171rcu_preempt11:00:212
163757799329,4cyclictest28-21ksoftirqd/112:27:341
1637577993225,4cyclictest0-21swapper/111:10:331
1637577993219,10cyclictest1728724-21collect210:35:281
163757899316,6cyclictest171rcu_preempt07:50:212
163757899315,13cyclictest1757231-21ssh11:20:272
163757899313,4cyclictest0-21swapper/207:40:352
1637578993121,6cyclictest1766420-21kernelversion11:35:172
163757899311,25cyclictest1735016-21ssh10:45:242
163757799319,5cyclictest28-21ksoftirqd/109:00:211
163757799319,5cyclictest28-21ksoftirqd/107:10:241
163757799319,12cyclictest28-21ksoftirqd/110:25:371
163757799316,4cyclictest28-21ksoftirqd/107:25:221
163757799313,21cyclictest1744604-21ssh11:00:261
1637577993124,4cyclictest1672978-21ntp_kernel_pll_09:05:231
1637577993123,5cyclictest420-21avahi-daemon07:35:281
1637577993122,7cyclictest1641818-21timerandwakeup07:20:331
1637577993116,3cyclictest171rcu_preempt12:23:501
1637577993111,4cyclictest28-21ksoftirqd/107:45:251
1637576993121,4cyclictest151ktimers/008:05:250
1637578993023,4cyclictest1711820-21kworker/u8:16-xprtiod10:10:152
1637578993022,5cyclictest1794603-21aten_r2power_en12:20:142
1637578993022,5cyclictest1794396-21latency_hist12:20:022
1637578993022,5cyclictest1668191-21kernelversion08:50:192
1637578993022,5cyclictest1642718-21kernelversion07:25:172
1637578993021,6cyclictest1640603-21unin-run07:20:012
1637578993017,6cyclictest351ktimers/209:55:242
163757799309,5cyclictest28-21ksoftirqd/108:50:241
163757799308,5cyclictest28-21ksoftirqd/108:15:241
163757799307,5cyclictest28-21ksoftirqd/107:50:231
1637577993023,4cyclictest0-21swapper/112:15:011
1637577993022,5cyclictest1741093-21latency10:55:181
1637577993022,5cyclictest1001-21mta-sts-daemon08:35:261
1637577993022,5cyclictest1001-21mta-sts-daemon08:35:261
1637577993010,5cyclictest28-21ksoftirqd/110:19:241
1637577993010,11cyclictest28-21ksoftirqd/109:40:021
163757899295,7cyclictest171rcu_preempt07:35:202
163757899293,22cyclictest1779389-21ssh11:55:262
163757899293,21cyclictest1681123-21ssh09:20:302
1637578992923,4cyclictest1003-21mta-sts-daemon10:50:242
1637578992912,10cyclictest351ktimers/208:45:262
163757799299,4cyclictest28-21ksoftirqd/108:40:211
163757799299,4cyclictest28-21ksoftirqd/108:40:211
163757799299,3cyclictest261rcuc/110:08:011
163757799296,5cyclictest261rcuc/110:03:331
1637577992922,4cyclictest1785511-21latency12:05:171
1637577992921,5cyclictest1645046-21cron07:35:011
1637577992920,7cyclictest1655578-21sh08:10:011
1637577992910,5cyclictest28-21ksoftirqd/107:55:251
1637578992821,3cyclictest0-21swapper/209:35:022
1637578992819,4cyclictest0-21swapper/211:10:262
1637578992816,9cyclictest1001-21mta-sts-daemon08:25:222
163757899281,4cyclictest171rcu_preempt11:25:252
163757799286,14cyclictest428-21mta-sts-daemon08:20:211
163757899273,6cyclictest171rcu_preempt07:30:232
163757899272,6cyclictest0-21swapper/208:30:292
1637578992720,5cyclictest0-21swapper/209:10:022
1637578992718,5cyclictest0-21swapper/208:20:002
163757799278,4cyclictest28-21ksoftirqd/108:55:221
1637578992617,7cyclictest1665256-21latency08:40:182
1637578992617,7cyclictest1665256-21latency08:40:182
1637578992518,4cyclictest1652783-21/usr/sbin/munin08:00:232
1637578992518,4cyclictest1652783-21/usr/sbin/munin08:00:232
163757899250,6cyclictest171rcu_preempt08:10:212
1637578992417,5cyclictest0-21swapper/207:25:012
1637578992313,7cyclictest1637589-21systemd07:10:022
1637578992313,7cyclictest1637589-21systemd07:10:022
163757799233,5cyclictest28-21ksoftirqd/107:10:021
163757799233,5cyclictest28-21ksoftirqd/107:10:021
1637576992318,3cyclictest1654037-21kworker/u8:0-xprtiod08:30:170
163757899212,14cyclictest1638403-21ntpq07:10:202
163757999201,6cyclictest0-21swapper/309:40:253
163757999183,7cyclictest0-21swapper/308:45:273
1637576991811,4cyclictest1001-21mta-sts-daemon07:10:020
1637576991811,4cyclictest1001-21mta-sts-daemon07:10:020
163757999172,6cyclictest0-21swapper/311:40:243
163757999172,6cyclictest0-21swapper/311:30:213
163757999172,10cyclictest0-21swapper/311:05:283
163757999171,7cyclictest0-21swapper/311:45:253
163757999171,7cyclictest0-21swapper/309:55:223
163757999171,7cyclictest0-21swapper/307:55:233
163757999171,6cyclictest0-21swapper/307:35:263
163757999163,7cyclictest0-21swapper/312:20:013
163757999163,5cyclictest0-21swapper/309:15:223
163757999162,8cyclictest0-21swapper/311:10:223
163757999162,7cyclictest0-21swapper/311:00:283
163757999162,7cyclictest0-21swapper/310:40:283
163757999162,6cyclictest0-21swapper/310:00:283
163757999162,6cyclictest0-21swapper/309:50:223
163757999161,7cyclictest0-21swapper/309:35:223
163757999161,6cyclictest0-21swapper/307:20:233
163757999161,5cyclictest0-21swapper/307:25:193
163757999155,5cyclictest0-21swapper/311:35:223
163757999154,5cyclictest0-21swapper/309:30:213
163757999153,5cyclictest0-21swapper/312:20:253
163757999152,7cyclictest0-21swapper/310:55:213
163757999152,5cyclictest0-21swapper/308:20:213
163757999151,8cyclictest0-21swapper/311:25:243
163757999151,7cyclictest0-21swapper/312:30:203
163757999151,7cyclictest0-21swapper/310:45:223
163757999151,7cyclictest0-21swapper/310:15:203
163757999151,7cyclictest0-21swapper/309:25:253
163757999151,6cyclictest0-21swapper/310:25:223
163757999151,6cyclictest0-21swapper/310:10:223
163757999144,4cyclictest0-21swapper/307:15:233
163757999143,7cyclictest0-21swapper/312:00:233
163757999143,6cyclictest0-21swapper/310:30:283
163757999143,6cyclictest0-21swapper/308:00:253
163757999143,6cyclictest0-21swapper/308:00:253
163757999143,5cyclictest0-21swapper/311:15:303
163757999142,7cyclictest0-21swapper/311:20:283
163757999142,7cyclictest0-21swapper/310:35:283
163757999142,6cyclictest0-21swapper/312:25:233
163757999142,5cyclictest0-21swapper/309:00:213
163757999142,5cyclictest0-21swapper/307:45:223
163757999141,7cyclictest0-21swapper/308:30:273
163757999141,6cyclictest0-21swapper/309:45:263
163757999141,6cyclictest0-21swapper/308:50:243
163757999141,5cyclictest0-21swapper/307:50:213
163757999133,6cyclictest0-21swapper/312:05:163
163757999133,6cyclictest0-21swapper/310:05:283
163757999132,6cyclictest0-21swapper/311:55:173
163757999132,6cyclictest0-21swapper/310:50:253
163757999131,5cyclictest0-21swapper/307:10:203
163757999123,4cyclictest0-21swapper/309:20:233
163757999121,7cyclictest0-21swapper/311:50:283
163757999121,6cyclictest0-21swapper/309:00:013
163757999121,6cyclictest0-21swapper/308:30:013
163757999114,5cyclictest0-21swapper/308:10:213
163757999113,4cyclictest0-21swapper/308:40:003
163757999113,4cyclictest0-21swapper/308:40:003
163757999112,6cyclictest0-21swapper/312:35:283
163757999112,5cyclictest0-21swapper/307:41:293
163757999112,5cyclictest0-21swapper/307:30:123
163757999111,6cyclictest0-21swapper/309:10:213
163757999104,4cyclictest0-21swapper/312:10:263
163757999104,4cyclictest0-21swapper/308:40:243
163757999104,4cyclictest0-21swapper/308:40:243
163757999102,5cyclictest0-21swapper/310:20:213
163757999101,5cyclictest0-21swapper/308:15:143
16375799992,4cyclictest0-21swapper/307:10:013
16375799992,4cyclictest0-21swapper/307:10:013
16375799991,6cyclictest0-21swapper/309:05:243
16375799991,5cyclictest0-21swapper/308:05:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional