You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 01:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Tue Jan 06, 2026 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
167935823040,8sleep3411rcuc/307:05:273
179088921030,7sleep3411rcuc/312:20:213
173216621030,16sleep126-21ksoftirqd/109:35:151
17939942760,5sleep20-21swapper/212:30:142
197399710,9rtkit-daemon1972-21rtkit-daemon11:35:262
167966226443,14sleep00-21swapper/007:07:380
197399630,9rtkit-daemon1972-21rtkit-daemon12:28:272
197399630,9rtkit-daemon1972-21rtkit-daemon12:19:132
167952626342,14sleep10-21swapper/107:05:491
197399610,9rtkit-daemon1972-21rtkit-daemon11:25:173
167954726141,14sleep20-21swapper/207:06:072
197399600,9rtkit-daemon1972-21rtkit-daemon08:39:492
197399600,8rtkit-daemon1972-21rtkit-daemon08:20:472
197399590,9rtkit-daemon1972-21rtkit-daemon09:04:312
197399590,8rtkit-daemon1972-21rtkit-daemon11:16:002
197399590,8rtkit-daemon1972-21rtkit-daemon09:27:022
197399590,8rtkit-daemon1972-21rtkit-daemon09:27:022
197399590,10rtkit-daemon1972-21rtkit-daemon08:44:132
197399580,9rtkit-daemon1972-21rtkit-daemon12:09:592
197399580,9rtkit-daemon1972-21rtkit-daemon10:08:043
197399580,8rtkit-daemon1972-21rtkit-daemon11:48:392
197399580,8rtkit-daemon1972-21rtkit-daemon09:53:292
197399580,8rtkit-daemon1972-21rtkit-daemon09:53:292
197399570,9rtkit-daemon1972-21rtkit-daemon11:11:203
197399560,9rtkit-daemon1972-21rtkit-daemon10:44:372
197399560,9rtkit-daemon1972-21rtkit-daemon10:44:362
197399560,8rtkit-daemon1972-21rtkit-daemon07:47:502
197399550,9rtkit-daemon1972-21rtkit-daemon12:01:232
197399550,9rtkit-daemon1972-21rtkit-daemon07:19:093
197399550,9rtkit-daemon1972-21rtkit-daemon07:12:582
197399550,8rtkit-daemon1972-21rtkit-daemon08:54:572
197399550,8rtkit-daemon1972-21rtkit-daemon08:00:222
197399550,5rtkit-daemon0-21swapper/009:22:090
197399540,9rtkit-daemon1972-21rtkit-daemon10:48:113
197399540,9rtkit-daemon1972-21rtkit-daemon08:16:192
197399540,9rtkit-daemon1972-21rtkit-daemon07:58:163
197399540,8rtkit-daemon1972-21rtkit-daemon09:56:332
197399540,4rtkit-daemon1054-21snmpd09:12:132
197399540,26rtkit-daemon1972-21rtkit-daemon07:23:092
168001399549,8cyclictest1704793-21apt-key08:20:003
168001099540,50cyclictest1972-21rtkit-daemon10:48:562
1680003995419,8cyclictest13-21ksoftirqd/012:10:020
1680003995411,6cyclictest141rcu_preempt11:31:290
1680003995411,6cyclictest141rcu_preempt11:31:290
197399530,9rtkit-daemon1972-21rtkit-daemon07:29:402
197399530,8rtkit-daemon1972-21rtkit-daemon11:30:392
197399530,8rtkit-daemon1972-21rtkit-daemon11:30:382
197399530,8rtkit-daemon1972-21rtkit-daemon10:36:392
1680013995324,7cyclictest1761445-21package-data-do11:00:013
1680013995310,22cyclictest1718357-21switchtime08:55:283
197399520,8rtkit-daemon1972-21rtkit-daemon11:14:062
197399520,8rtkit-daemon1972-21rtkit-daemon10:10:342
197399510,8rtkit-daemon1972-21rtkit-daemon12:23:252
168000399511,23cyclictest1720935-21chrt09:04:470
168000399501,24cyclictest1722515-21chrt09:07:540
1680013994914,18cyclictest1710272-21unin-run08:35:013
168001399491,30cyclictest1788962-21expr12:15:143
168000399492,20cyclictest13-21ksoftirqd/009:50:290
168000399492,20cyclictest13-21ksoftirqd/009:50:290
168000399488,17cyclictest1680940-21ntp_states07:10:250
168000399475,19cyclictest13-21ksoftirqd/011:00:110
168000399472,20cyclictest13-21ksoftirqd/012:25:260
1680003994717,26cyclictest1724717-21cron09:15:000
168000399471,22cyclictest1734823-21taskset09:41:280
168000399471,21cyclictest1733325-21chrt09:39:280
197399460,38rtkit-daemon1722315-21users09:05:303
168001399469,8cyclictest0-21swapper/311:40:013
1680013994622,20cyclictest1752757-21unin-run10:35:013
1680007994626,16cyclictest1705209-21gdbus08:20:031
1680007994610,9cyclictest26-21ksoftirqd/108:05:021
1680003994624,9cyclictest13-21ksoftirqd/011:55:580
1680003994624,9cyclictest13-21ksoftirqd/011:55:580
168000399462,13cyclictest13-21ksoftirqd/011:10:230
197399450,24rtkit-daemon1972-21rtkit-daemon07:39:552
197399450,22rtkit-daemon1074-21lldpd09:52:363
197399450,22rtkit-daemon1074-21lldpd09:52:353
168000399459,16cyclictest13-21ksoftirqd/011:35:180
168000399452,15cyclictest13-21ksoftirqd/011:05:180
168000399452,15cyclictest13-21ksoftirqd/011:05:180
168000399451,13cyclictest13-21ksoftirqd/011:50:260
197399440,36rtkit-daemon2212-21gvfs-afc-volume08:30:272
17486742440,6sleep20-21swapper/210:20:282
168000399442,15cyclictest0-21swapper/010:05:380
168000399441,15cyclictest1795553-21sort12:35:120
168000399441,13cyclictest13-21ksoftirqd/010:00:310
1680003994410,12cyclictest13-21ksoftirqd/012:15:170
197399430,34rtkit-daemon42-21ksoftirqd/309:35:133
197399430,34rtkit-daemon42-21ksoftirqd/309:30:273
17209742430,11sleep11720972-21latency_hist09:05:011
1680013994312,26cyclictest1753641-21needreboot10:35:233
168000399439,14cyclictest13-21ksoftirqd/007:17:560
1680003994326,12cyclictest438-20systemd-journal07:58:370
168000399432,10cyclictest0-21swapper/010:35:160
197399420,34rtkit-daemon4497-21PK-Backend11:41:492
1680010994213,10cyclictest34-21ksoftirqd/209:10:012
168000399422,21cyclictest13-21ksoftirqd/010:45:230
1680003994217,8cyclictest13-21ksoftirqd/011:20:240
168000399421,22cyclictest1692376-21chrt07:42:290
1680003994212,15cyclictest13-21ksoftirqd/010:55:060
168000399421,14cyclictest1690341-21runrttasks07:35:300
168000399421,13cyclictest13-21ksoftirqd/010:25:290
168000399421,10cyclictest1787553-21strings12:10:240
168000399421,10cyclictest1726093-21awk09:15:300
197399410,34rtkit-daemon4497-21PK-Backend07:10:283
197399410,32rtkit-daemon42-21ksoftirqd/310:03:123
17843002410,8sleep30-21swapper/312:01:373
1680010994124,5cyclictest34-21ksoftirqd/211:00:302
1680010994124,13cyclictest34-21ksoftirqd/208:10:012
168000399417,12cyclictest13-21ksoftirqd/010:15:040
168000399417,12cyclictest13-21ksoftirqd/010:15:030
1680003994131,6cyclictest1718342-21python308:55:270
1680003994125,12cyclictest1748847-21timerwakeupswit10:20:280
168000399411,22cyclictest1695741-21chrt07:51:150
168000399411,21cyclictest1745690-21chrt10:13:500
168000399411,13cyclictest1702186-21taskset08:10:220
1680013994013,6cyclictest141rcu_preempt10:15:213
1680013994013,6cyclictest141rcu_preempt10:15:203
1680010994023,6cyclictest34-21ksoftirqd/208:55:132
1680010994023,13cyclictest34-21ksoftirqd/212:35:282
168001099401,35cyclictest1751994-21awk10:30:232
1680010994011,13cyclictest141rcu_preempt09:15:272
168000799408,13cyclictest26-21ksoftirqd/110:25:141
1680007994013,5cyclictest141rcu_preempt11:05:091
1680007994013,5cyclictest141rcu_preempt11:05:091
168000399408,11cyclictest13-21ksoftirqd/011:26:140
168000399402,22cyclictest13-21ksoftirqd/012:21:090
1680003994022,13cyclictest438-20systemd-journal09:34:590
1680003994020,16cyclictest1784612-21unin-run12:05:000
1680003994016,20cyclictest1759674-21cron10:55:000
168000399401,20cyclictest13-21ksoftirqd/009:55:280
168000399401,12cyclictest13-21ksoftirqd/012:35:020
168000399401,12cyclictest0-21swapper/008:44:130
16882912390,6sleep30-21swapper/307:30:223
1680013993925,10cyclictest1720997-21systemd-run09:05:013
1680013993923,11cyclictest956-21systemd-logind08:38:493
1680010993928,6cyclictest34-21ksoftirqd/211:05:202
1680010993928,6cyclictest34-21ksoftirqd/211:05:192
1680010993923,6cyclictest34-21ksoftirqd/207:40:112
1680010993922,12cyclictest34-21ksoftirqd/209:40:152
1680010993921,13cyclictest34-21ksoftirqd/208:10:192
1680010993913,6cyclictest141rcu_preempt10:07:032
1680010993912,13cyclictest141rcu_preempt07:55:282
168000799395,16cyclictest26-21ksoftirqd/107:50:151
168000799395,16cyclictest26-21ksoftirqd/107:35:191
1680007993923,11cyclictest438-20systemd-journal11:17:361
1680007993912,7cyclictest1704385-21sshd08:15:371
168000399398,5cyclictest141rcu_preempt07:20:140
168000399396,12cyclictest13-21ksoftirqd/007:49:470
168000399396,10cyclictest13-21ksoftirqd/008:30:180
1680003993928,8cyclictest181irq_work/008:38:490
168000399391,21cyclictest1777220-21chrt11:43:400
168000399391,20cyclictest1752539-21chrt10:32:480
168000399391,11cyclictest4497-21PK-Backend08:27:270
168000399391,10cyclictest4497-21PK-Backend11:19:360
197399380,26rtkit-daemon4497-21PK-Backend10:40:260
197399380,26rtkit-daemon4497-21PK-Backend10:40:260
17609402380,5sleep14497-21PK-Backend10:55:311
168001099389,13cyclictest141rcu_preempt11:20:222
1680010993821,6cyclictest34-21ksoftirqd/208:45:212
1680010993821,13cyclictest34-21ksoftirqd/212:10:212
1680010993820,14cyclictest34-21ksoftirqd/211:25:152
168000799389,14cyclictest26-21ksoftirqd/111:40:281
1680007993823,11cyclictest1736853-21awk09:50:011
1680007993822,5cyclictest26-21ksoftirqd/108:38:501
1680007993821,8cyclictest26-21ksoftirqd/111:31:301
1680007993821,8cyclictest26-21ksoftirqd/111:31:291
1680007993821,12cyclictest26-21ksoftirqd/109:55:121
1680007993812,9cyclictest26-21ksoftirqd/109:35:011
168000399387,5cyclictest141rcu_preempt11:45:120
168000399384,8cyclictest0-21swapper/008:00:230
1680003993817,8cyclictest13-21ksoftirqd/008:16:410
1680003993812,9cyclictest13-21ksoftirqd/008:05:260
1680003993811,23cyclictest1728714-21cstates09:25:170
1680003993811,23cyclictest1728714-21cstates09:25:160
197399370,4rtkit-daemon0-21swapper/312:35:153
17927022370,6sleep14497-21PK-Backend12:25:221
17571042370,7sleep14497-21PK-Backend10:45:211
17547692371,9sleep342-21ksoftirqd/310:40:143
17547692371,9sleep342-21ksoftirqd/310:40:133
168001399378,25cyclictest1725337-21expr09:15:143
1680013993727,7cyclictest1137-21runrttasks07:40:173
1680013993723,10cyclictest42-21ksoftirqd/312:05:123
1680010993720,6cyclictest34-21ksoftirqd/209:30:282
1680010993720,5cyclictest34-21ksoftirqd/211:00:022
1680010993719,6cyclictest34-21ksoftirqd/207:50:162
1680010993716,8cyclictest141rcu_preempt07:15:272
168000799378,7cyclictest1692000-21/usr/sbin/munin07:40:261
1680007993720,6cyclictest26-21ksoftirqd/108:42:011
1680007993720,12cyclictest1714757-21ntpq08:45:241
168000799371,17cyclictest1702862-21systemd-udevd08:13:471
168000399371,19cyclictest13-21ksoftirqd/007:25:170
168000399371,15cyclictest1736559-21chrt09:46:320
197399360,28rtkit-daemon1941-21systemd11:56:393
197399360,28rtkit-daemon1941-21systemd11:56:393
168001399367,24cyclictest1773552-21systemd-udevd11:31:273
168001399367,24cyclictest1773552-21systemd-udevd11:31:273
1680013993620,11cyclictest438-20systemd-journal12:35:003
1680013993619,12cyclictest1690024-21netstat07:35:243
1680010993621,11cyclictest34-21ksoftirqd/210:25:222
1680010993618,14cyclictest34-21ksoftirqd/209:25:022
1680010993614,9cyclictest141rcu_preempt09:45:252
1680010993614,9cyclictest141rcu_preempt09:35:232
1680010993614,8cyclictest141rcu_preempt10:00:252
168000799369,7cyclictest1709153-21diskstats08:30:161
168000799368,13cyclictest1786191-21/usr/sbin/munin12:05:301
168000799367,13cyclictest1780410-21/usr/sbin/munin11:50:261
1680007993619,13cyclictest1781360-21sshd11:55:121
1680007993619,13cyclictest1781360-21sshd11:55:111
1680007993619,12cyclictest1716451-21ntp_kernel_err08:50:231
1680007993616,15cyclictest1727314-21memory09:20:221
1680007993613,5cyclictest141rcu_preempt12:23:461
168000399363,9cyclictest0-21swapper/008:45:160
1680003993620,11cyclictest1705407-21cut08:20:130
168000399361,17cyclictest1716319-21taskset08:50:210
168000399361,12cyclictest13-21ksoftirqd/007:31:500
197399350,28rtkit-daemon0-21swapper/311:09:113
197399350,28rtkit-daemon0-21swapper/311:09:113
168001399351,9cyclictest0-21swapper/309:40:203
1680013993518,13cyclictest1739501-21cat09:55:143
168001399351,14cyclictest1762430-21ntp_states11:00:253
168001399351,10cyclictest0-21swapper/311:22:423
1680010993518,13cyclictest1746515-21cat10:15:202
1680010993518,13cyclictest1746515-21cat10:15:192
1680010993513,13cyclictest34-21ksoftirqd/211:50:202
168000799357,4cyclictest141rcu_preempt10:35:581
1680007993520,11cyclictest26-21ksoftirqd/112:30:281
1680007993520,11cyclictest26-21ksoftirqd/110:14:541
1680007993520,11cyclictest26-21ksoftirqd/109:11:471
1680007993520,11cyclictest26-21ksoftirqd/109:05:441
1680007993520,11cyclictest1707761-21ntp_kernel_pll_08:25:241
1680007993519,6cyclictest26-21ksoftirqd/107:32:341
1680007993519,12cyclictest1783866-21ntpq12:00:251
1680007993519,11cyclictest26-21ksoftirqd/112:37:281
168000799351,30cyclictest4497-21PK-Backend11:00:141
1680007993512,11cyclictest141rcu_preempt09:52:471
1680007993512,11cyclictest141rcu_preempt09:52:471
168001399349,21cyclictest1792815-21netstat12:25:233
168001399347,23cyclictest1729911-21cron09:30:013
168001399347,23cyclictest1729911-21cron09:30:003
168001399346,24cyclictest438-20systemd-journal07:50:003
1680013993420,10cyclictest1-21systemd10:15:013
1680013993419,11cyclictest1758864-21cat10:50:213
1680013993419,11cyclictest1735757-21diskstats09:45:153
1680013993414,16cyclictest1747993-21cstates10:20:133
168001399341,29cyclictest1713731-21cron08:45:003
168001399341,28cyclictest438-20systemd-journal08:07:393
1680010993413,8cyclictest141rcu_preempt08:27:242
168000799347,10cyclictest1700790-21/usr/sbin/munin08:05:291
168000799346,7cyclictest26-21ksoftirqd/107:55:291
1680007993420,10cyclictest26-21ksoftirqd/111:25:191
1680007993419,11cyclictest26-21ksoftirqd/111:48:061
1680007993419,11cyclictest26-21ksoftirqd/110:00:411
1680007993419,11cyclictest26-21ksoftirqd/109:43:141
1680007993419,11cyclictest26-21ksoftirqd/108:58:401
1680007993419,11cyclictest26-21ksoftirqd/107:49:051
1680007993419,11cyclictest1758933-21ntp_kernel_err10:50:241
1680007993418,5cyclictest26-21ksoftirqd/111:36:201
1680007993418,12cyclictest26-21ksoftirqd/109:15:051
1680007993418,11cyclictest26-21ksoftirqd/111:20:501
1680007993418,11cyclictest1686809-21cat07:25:301
1680007993416,13cyclictest1081-21ntpd11:10:251
1680013993322,7cyclictest1728201-21sh09:25:003
1680013993321,8cyclictest1705894-21grep08:20:223
1680013993321,7cyclictest1724306-21users09:10:313
168001399331,9cyclictest4497-21PK-Backend08:01:393
1680013993319,10cyclictest1767196-21sort11:15:113
168001399331,8cyclictest0-21swapper/308:51:493
1680013993318,11cyclictest1776282-21hddtemp_smartct11:40:193
1680013993318,11cyclictest1684598-21seq07:20:203
1680013993317,12cyclictest1694938-21expr07:50:153
168001399331,28cyclictest1777971-21df11:45:143
168001399331,12cyclictest1787250-21seq12:10:153
168001399331,12cyclictest1701618-21cat08:10:143
1680013993310,7cyclictest141rcu_preempt08:50:003
1680010993313,8cyclictest141rcu_preempt07:30:172
1680010993312,8cyclictest141rcu_preempt11:56:162
1680010993312,8cyclictest141rcu_preempt11:56:152
1680007993318,5cyclictest26-21ksoftirqd/112:14:551
1680007993318,11cyclictest1751226-21date10:30:031
1680007993318,11cyclictest1682636-21expr07:15:141
1680007993316,13cyclictest1680312-21sort07:10:131
168000799331,5cyclictest4497-21PK-Backend12:16:321
168000799331,12cyclictest0-21swapper/110:40:271
168000799331,12cyclictest0-21swapper/110:40:271
1680013993216,12cyclictest1707626-21load08:25:203
1680013993216,12cyclictest1686027-21grep07:25:143
1680013993216,11cyclictest438-20systemd-journal11:53:073
168001099327,7cyclictest141rcu_preempt10:50:132
168000799321,7cyclictest1685618-21munin-run07:25:011
1680007993121,6cyclictest26-21ksoftirqd/110:24:331
1680013993021,5cyclictest42-21ksoftirqd/310:28:253
1680007993022,5cyclictest4497-21PK-Backend10:09:351
1680007993021,5cyclictest26-21ksoftirqd/110:15:251
1680007993021,5cyclictest26-21ksoftirqd/110:15:251
168000799301,4cyclictest4497-21PK-Backend09:28:001
168000799301,4cyclictest4497-21PK-Backend09:27:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional