You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-15 - 16:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Mon Dec 15, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1973991070,5rtkit-daemon0-21swapper/111:59:161
16556672990,5sleep14497-21PK-Backend10:35:141
197399950,6rtkit-daemon4497-21PK-Backend12:32:251
158220199802,73cyclictest2069-21gnome-shell07:32:583
158220199782,71cyclictest2069-21gnome-shell08:32:583
16061692770,5sleep24497-21PK-Backend08:15:182
158220199752,68cyclictest2069-21gnome-shell08:36:583
158220199751,69cyclictest2069-21gnome-shell11:45:593
158220199751,69cyclictest2069-21gnome-shell08:19:583
197399730,63rtkit-daemon0-21swapper/207:09:322
158220199732,31cyclictest0-21swapper/307:45:583
158220199722,65cyclictest2069-21gnome-shell11:44:593
158219999723,47cyclictest2069-21gnome-shell11:48:592
158220199711,65cyclictest2069-21gnome-shell08:20:583
158219999712,10cyclictest2069-21gnome-shell07:44:582
158219699712,65cyclictest231irq_work/112:36:591
158219699712,65cyclictest231irq_work/112:36:591
158220199702,64cyclictest2069-21gnome-shell11:31:593
158219999702,10cyclictest2069-21gnome-shell10:44:592
158220199692,63cyclictest0-21swapper/312:20:593
1582201996836,27cyclictest2069-21gnome-shell11:37:593
158220199682,63cyclictest0-21swapper/307:36:583
158220199682,26cyclictest0-21swapper/309:05:583
1582199996825,17cyclictest2069-21gnome-shell09:40:582
158219999682,46cyclictest2069-21gnome-shell07:28:582
158220199672,61cyclictest0-21swapper/312:27:593
158220199662,61cyclictest0-21swapper/310:56:593
158220199662,61cyclictest0-21swapper/307:52:583
158220199662,61cyclictest0-21swapper/307:52:583
158220199661,36cyclictest1642363-21taskset09:55:583
158220199652,60cyclictest0-21swapper/310:48:593
158168326545,14sleep10-21swapper/107:05:291
16272322640,6sleep11627233-21cat09:15:011
16005012640,6sleep30-21swapper/308:00:143
1582201996434,26cyclictest2069-21gnome-shell08:27:583
158185626445,13sleep00-21swapper/007:07:450
197399630,9rtkit-daemon1972-21rtkit-daemon08:42:532
158220199632,58cyclictest0-21swapper/309:29:593
158055026340,14sleep30-21swapper/307:05:033
158220199622,57cyclictest0-21swapper/309:52:583
158220199612,56cyclictest0-21swapper/309:12:583
158219199612,54cyclictest2069-21gnome-shell10:13:590
158219199612,54cyclictest2069-21gnome-shell10:13:590
197399600,9rtkit-daemon1972-21rtkit-daemon08:41:233
158220199601,33cyclictest1636012-21/usr/sbin/munin09:40:163
1582201996012,43cyclictest1661998-21sshd10:54:563
158220199601,11cyclictest1649680-21taskset10:17:133
197399580,9rtkit-daemon1972-21rtkit-daemon09:48:052
197399580,9rtkit-daemon1972-21rtkit-daemon09:48:052
158219999582,16cyclictest2069-21gnome-shell10:46:592
1582201995731,23cyclictest0-21swapper/310:43:593
1582201995716,35cyclictest438-20systemd-journal09:05:003
1582201995716,35cyclictest438-20systemd-journal09:05:003
158220199571,11cyclictest1621619-21taskset08:56:143
1582196995740,13cyclictest2069-21gnome-shell10:34:591
197399560,9rtkit-daemon1972-21rtkit-daemon09:37:072
1582201995638,14cyclictest2069-21gnome-shell11:59:593
158220199562,50cyclictest2069-21gnome-shell11:50:593
1582201995615,35cyclictest438-20systemd-journal10:25:463
158220199561,50cyclictest1655699-21expr10:35:143
158220199561,10cyclictest1674120-21taskset11:26:093
1582199995637,14cyclictest2069-21gnome-shell12:05:592
1582196995635,16cyclictest2069-21gnome-shell07:15:581
158219699563,10cyclictest1644436-40lsb_release10:01:591
197399550,9rtkit-daemon1972-21rtkit-daemon07:11:453
16695592550,7sleep0161rcuc/011:15:130
1582201995520,31cyclictest1646481-21pluginstate10:10:023
158219699555,47cyclictest4497-21PK-Backend12:13:591
197399540,8rtkit-daemon1972-21rtkit-daemon07:19:252
158220199542,48cyclictest2069-21gnome-shell08:45:583
1582196995437,13cyclictest2069-21gnome-shell07:54:581
1582196995437,13cyclictest2069-21gnome-shell07:54:581
158219199542,10cyclictest2069-21gnome-shell09:57:590
15896072530,14sleep013-21ksoftirqd/007:30:010
1582201995319,30cyclictest1692353-21latency_hist12:20:013
158220199531,15cyclictest1602365-21sort08:05:203
158220199531,12cyclictest1633869-21taskset09:32:583
158220199531,11cyclictest438-20systemd-journal07:43:573
1582196995343,6cyclictest2069-21gnome-shell12:27:591
1582196995332,16cyclictest2069-21gnome-shell08:04:581
158219699532,48cyclictest231irq_work/107:29:581
158219699532,47cyclictest2069-21gnome-shell07:22:581
158219199532,46cyclictest2069-21gnome-shell11:03:590
158219199531,10cyclictest2069-21gnome-shell12:17:590
158220199522,46cyclictest1664160-21tr11:00:133
1582201995220,28cyclictest1588029-21/usr/sbin/munin07:25:273
1582201995215,33cyclictest1670759-21seq11:17:263
1582201995215,33cyclictest1-21systemd10:35:013
158220199521,46cyclictest1650573-21fw_packets10:20:163
158220199521,12cyclictest1666171-21cpuspeed_turbos11:05:133
158219999522,45cyclictest2069-21gnome-shell11:11:592
158219699524,7cyclictest231irq_work/110:59:591
1582196995242,6cyclictest2069-21gnome-shell12:20:591
1582201995118,28cyclictest1635223-21switchtime09:35:283
158220199511,47cyclictest0-21swapper/312:11:593
158220199511,46cyclictest1672487-21runrttasks11:22:253
158219999512,45cyclictest2069-21gnome-shell09:59:592
158219999512,10cyclictest1612717-21kworker/2:209:22:582
1582196995144,4cyclictest251rcuc/108:19:581
158219699513,7cyclictest231irq_work/110:44:591
158219699513,7cyclictest231irq_work/110:23:591
158219699513,45cyclictest231irq_work/111:10:591
158219699513,39cyclictest231irq_work/112:15:591
1582196995131,15cyclictest2069-21gnome-shell08:09:581
158219699512,8cyclictest4497-21PK-Backend08:22:581
158219699512,8cyclictest231irq_work/112:01:581
158219699512,45cyclictest2069-21gnome-shell09:18:581
158219199512,6cyclictest4497-21PK-Backend08:30:580
1582201995041,6cyclictest0-21swapper/308:14:583
1582201995029,18cyclictest0-21swapper/312:06:593
158220199502,44cyclictest2069-21gnome-shell08:53:583
158220199501,6cyclictest438-20systemd-journal12:36:563
158220199501,6cyclictest438-20systemd-journal12:36:563
158220199501,45cyclictest1628831-21taskset09:19:583
1582201995013,32cyclictest438-20systemd-journal07:20:013
1582201995011,34cyclictest1629576-21if_err_enp2s009:20:183
1582196995040,6cyclictest2069-21gnome-shell09:08:581
158219699503,7cyclictest231irq_work/110:17:591
158219699503,7cyclictest231irq_work/109:45:591
158219699503,7cyclictest231irq_work/109:45:591
158219699503,7cyclictest231irq_work/109:28:591
158219699503,44cyclictest231irq_work/108:57:581
158219699503,38cyclictest231irq_work/111:02:581
158219699502,46cyclictest231irq_work/111:33:581
158219699502,45cyclictest231irq_work/109:53:591
158219699502,45cyclictest2069-21gnome-shell07:58:581
1582191995035,11cyclictest2069-21gnome-shell09:35:590
158220199492,7cyclictest0-21swapper/307:56:583
158220199492,7cyclictest0-21swapper/307:23:583
158220199491,44cyclictest1643623-21hddtemp_smartct10:00:183
158220199491,43cyclictest1646693-21/usr/sbin/munin10:10:193
158220199491,43cyclictest1646693-21/usr/sbin/munin10:10:193
158219999492,7cyclictest0-21swapper/211:54:592
158219699493,44cyclictest231irq_work/107:42:581
158219699493,43cyclictest231irq_work/111:51:591
158219699493,43cyclictest231irq_work/109:34:591
158219699492,7cyclictest231irq_work/107:12:581
158219699492,6cyclictest0-21swapper/109:41:591
158219699492,45cyclictest231irq_work/110:06:591
158219699492,44cyclictest231irq_work/109:36:591
158219699492,44cyclictest231irq_work/108:14:581
158219699491,7cyclictest231irq_work/112:08:591
158219699491,44cyclictest0-21swapper/107:45:581
158219199492,44cyclictest0-21swapper/011:33:590
158220199483,40cyclictest1686585-21switchtime12:00:283
158219999482,6cyclictest0-21swapper/209:25:582
158219699483,43cyclictest231irq_work/108:44:581
158219699482,44cyclictest231irq_work/111:48:591
158219699482,44cyclictest231irq_work/111:40:591
158219699482,44cyclictest231irq_work/107:36:581
158219699482,43cyclictest231irq_work/109:22:581
158219699481,45cyclictest231irq_work/110:25:591
1582191994838,7cyclictest2069-21gnome-shell07:17:580
1582191994837,7cyclictest2069-21gnome-shell11:06:590
158219199482,5cyclictest0-21swapper/008:21:590
158220199472,7cyclictest1696529-21taskset12:30:193
1582199994736,7cyclictest2069-21gnome-shell10:29:592
158219999472,6cyclictest4497-21PK-Backend12:22:582
158219999472,6cyclictest4497-21PK-Backend09:09:582
158219999472,5cyclictest0-21swapper/212:33:592
158219699472,43cyclictest231irq_work/111:25:591
158219699472,43cyclictest231irq_work/108:51:581
158219699472,43cyclictest231irq_work/108:28:581
158219699472,42cyclictest231irq_work/109:56:591
158219699471,44cyclictest231irq_work/110:45:591
158219699471,44cyclictest231irq_work/109:02:581
158219699471,44cyclictest231irq_work/109:02:581
158219699471,43cyclictest4497-21PK-Backend11:24:591
1582191994739,5cyclictest4497-21PK-Backend09:11:580
158219199472,42cyclictest4497-21PK-Backend12:23:590
158219199471,5cyclictest4497-21PK-Backend08:29:580
158219999461,5cyclictest4497-21PK-Backend08:54:582
158219999461,41cyclictest4497-21PK-Backend11:57:592
158219699462,42cyclictest231irq_work/108:38:581
158219699461,42cyclictest4497-21PK-Backend10:50:591
1582191994639,4cyclictest4497-21PK-Backend10:03:590
158219199461,42cyclictest4497-21PK-Backend10:22:590
1582199994539,4cyclictest0-21swapper/210:56:592
1582199994538,4cyclictest0-21swapper/210:13:592
1582199994538,4cyclictest0-21swapper/210:13:592
1582199994534,7cyclictest2069-21gnome-shell08:30:582
1582196994538,5cyclictest0-21swapper/107:32:581
1582196994524,16cyclictest2069-21gnome-shell11:05:591
1582191994534,7cyclictest1582186-21cyclictest07:13:580
1582191994528,12cyclictest1688877-21cron12:10:000
158220199444,35cyclictest1668361-21cut11:10:203
1582199994437,4cyclictest0-21swapper/212:17:592
1582199994431,9cyclictest2069-21gnome-shell08:29:582
158219199445,11cyclictest2069-21gnome-shell11:23:590
1582191994437,4cyclictest4497-21PK-Backend11:51:590
16214282430,7sleep04497-21PK-Backend08:55:290
158220199431,14cyclictest1638628-21grep09:45:253
158220199431,14cyclictest1638628-21grep09:45:253
158219999436,11cyclictest2069-21gnome-shell11:36:592
158219999431,36cyclictest2069-21gnome-shell12:10:592
1582196994336,5cyclictest231irq_work/111:15:591
1582191994336,4cyclictest4497-21PK-Backend09:17:580
1582191994314,6cyclictest1697541-21turbostat12:35:000
158219999423,34cyclictest2069-21gnome-shell07:33:582
158219999421,35cyclictest2069-21gnome-shell12:03:592
1582191994231,7cyclictest2069-21gnome-shell09:26:590
1582199994130,7cyclictest2069-21gnome-shell08:21:582
1582199994124,13cyclictest2069-21gnome-shell10:22:592
1582191994134,4cyclictest0-21swapper/010:42:590
158219199411,7cyclictest1638638-21python309:45:270
158219199411,7cyclictest1638638-21python309:45:270
158219199411,34cyclictest2069-21gnome-shell08:05:580
1582196994030,8cyclictest0-21swapper/108:32:581
1582191994029,7cyclictest2069-21gnome-shell10:15:590
1582191994025,11cyclictest1699258-21cron12:40:000
1582191994025,11cyclictest1699258-21cron12:40:000
197399390,26rtkit-daemon4497-21PK-Backend12:28:100
197399390,26rtkit-daemon4497-21PK-Backend07:37:020
1582199993933,4cyclictest0-21swapper/211:03:592
1582199993928,7cyclictest2069-21gnome-shell11:33:592
1582199993924,10cyclictest2069-21gnome-shell08:55:582
1582199993924,10cyclictest2069-21gnome-shell07:38:582
158219699396,10cyclictest1677040-21cat11:35:141
1582191993924,10cyclictest2069-21gnome-shell07:55:580
197399380,30rtkit-daemon0-21swapper/209:11:472
1582199993822,11cyclictest1674437-21apt-get11:30:012
158219999382,19cyclictest1605099-40lsb_release08:11:032
1582199993821,13cyclictest1617079-21/usr/sbin/munin08:45:172
158219699381,32cyclictest2069-21gnome-shell10:10:591
158219699381,32cyclictest2069-21gnome-shell10:10:591
158219199384,31cyclictest1598212-21turbostat07:55:000
158219199384,31cyclictest1598212-21turbostat07:55:000
1582199993721,12cyclictest1694930-21memory12:25:212
1582199993721,12cyclictest1583681-21seq07:13:152
1582191993726,7cyclictest0-21swapper/012:04:590
1582191993725,7cyclictest4497-21PK-Backend11:25:590
158219199371,30cyclictest4497-21PK-Backend08:10:580
1582199993624,8cyclictest2069-21gnome-shell11:09:592
1582199993620,11cyclictest1628092-21ntpq09:15:222
158219699362,11cyclictest1617965-21taskset08:47:391
1582191993626,6cyclictest0-21swapper/011:43:580
1582191993624,8cyclictest2069-21gnome-shell08:47:580
1582191993620,12cyclictest1661061-21tr10:50:220
1582191993620,12cyclictest1633547-21cat09:30:280
1582191993618,12cyclictest2069-21gnome-shell09:22:580
197399350,29rtkit-daemon0-21swapper/208:39:382
197399350,25rtkit-daemon0-21swapper/011:48:480
1582199993524,6cyclictest4497-21PK-Backend09:50:592
1582199993522,9cyclictest2093-21JS10:49:092
1582199993518,13cyclictest1595613-21memory07:45:192
158219999351,7cyclictest2069-21gnome-shell10:54:592
158219199353,28cyclictest1623644-21cron09:05:000
158219199353,28cyclictest1623644-21cron09:05:000
1582191993520,11cyclictest1689610-21if_err_enp2s012:10:180
1582191993520,10cyclictest2069-21gnome-shell10:31:590
1582191993518,12cyclictest1668246-21cat11:10:190
1582191993518,12cyclictest1606441-21ntpq08:15:220
158219199351,29cyclictest4497-21PK-Backend07:45:580
1582191993510,7cyclictest15898892grep07:30:120
1582199993425,6cyclictest0-21swapper/211:23:592
1582199993422,8cyclictest924-21NetworkManager10:16:432
1582199993420,9cyclictest34-21ksoftirqd/210:35:302
1582199993417,12cyclictest438-20systemd-journal09:35:002
1582191993427,4cyclictest4497-21PK-Backend09:53:590
1582191993425,5cyclictest4497-21PK-Backend09:05:240
1582191993425,5cyclictest4497-21PK-Backend07:22:580
1582191993419,10cyclictest2069-21gnome-shell10:35:590
158219199341,30cyclictest0-21swapper/008:39:580
158219199341,29cyclictest4497-21PK-Backend10:08:590
158219199341,29cyclictest4497-21PK-Backend08:01:580
1582199993324,5cyclictest4497-21PK-Backend09:01:582
1582199993324,5cyclictest4497-21PK-Backend09:01:582
1582199993318,11cyclictest1698321-21cut12:35:202
1582199993318,11cyclictest1698321-21cut12:35:202
1582199993317,11cyclictest1679341-21swap11:40:282
1582199993315,13cyclictest1643802-21memory10:00:212
158219999331,29cyclictest0-21swapper/207:21:172
158219999331,22cyclictest4497-21PK-Backend10:05:592
1582191993318,11cyclictest1684673-21/usr/sbin/munin11:55:230
158219199331,30cyclictest0-21swapper/007:42:380
158219199331,29cyclictest0-21swapper/010:26:120
158219199331,10cyclictest4497-21PK-Backend09:42:140
158219999326,22cyclictest1137-21runrttasks08:01:002
158219999321,5cyclictest0-21swapper/210:34:592
1582191993225,4cyclictest0-21swapper/010:57:590
1582191993221,7cyclictest1614924-21apt08:40:110
1582191993217,11cyclictest1659774-21threads10:45:290
1582191993216,11cyclictest1676910-21tr11:35:140
1582199993124,4cyclictest4497-21PK-Backend07:56:382
1582199993122,6cyclictest1598207-21cron07:54:592
1582199993122,6cyclictest1598207-21cron07:54:592
1582199993115,11cyclictest956-21systemd-logind11:18:402
1582191992923,4cyclictest0-21swapper/008:53:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional