You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-18 - 00:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Aug 17, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
170786199110102,6cyclictest37-21ksoftirqd/312:40:013
1707849999613,54cyclictest1738309-21sh10:00:380
170784999844,21cyclictest131rcu_preempt12:20:030
1707849998422,31cyclictest150-21pr/ttyS212:36:420
1707849998312,37cyclictest1741535-21sh10:07:420
170784999826,56cyclictest0-21swapper/009:50:310
1707849998123,22cyclictest1743638-21ssh10:12:000
1707849997837,36cyclictest1784813-21sshd11:38:390
1707849997819,24cyclictest1714110-21diskmemload11:12:200
170784999771,73cyclictest0-21swapper/011:05:020
1707849997536,35cyclictest1758009-21ssh10:42:060
1707849997516,18cyclictest1709407-21modprobe07:40:010
1707849997238,30cyclictest1762160-21ssh10:50:520
1707849997222,9cyclictest1755759-21ssh10:37:260
1707849997136,32cyclictest1799478-21ssh12:09:260
170784999707,33cyclictest0-21swapper/009:56:460
1707849997038,29cyclictest1809535-21ssh12:30:150
1707849997034,14cyclictest1-21systemd12:00:000
1707849997024,41cyclictest1708325-21cron07:20:000
1707859996842,9cyclictest1725924-21kworker/2:111:44:092
1707849996833,31cyclictest1747037-21ssh10:19:200
1707849996822,14cyclictest1709670-21/usr/sbin/munin07:45:010
1707849996819,15cyclictest1766321-21munin-run11:00:000
1707849996716,8cyclictest1723349-21ssh09:29:340
1707849996639,4cyclictest1806840-21munin-run12:25:010
1707849996639,4cyclictest1806840-21munin-run12:25:010
1707849996634,29cyclictest1749690-21ssh10:24:540
1707849996633,30cyclictest1718197-21ssh09:18:430
1707849996631,32cyclictest1760403-21ssh10:47:210
1707849996515,16cyclictest1726226-21ssh09:35:150
1707849996513,15cyclictest1776629-21sh11:21:130
1707849996426,12cyclictest0-21swapper/011:30:010
1707849996417,13cyclictest1789157-21ssh11:47:370
170784999633,32cyclictest0-21swapper/009:49:230
1707849996314,15cyclictest1708158-21latency_hist07:15:020
1707859996242,15cyclictest3467602-21systemd-journal12:19:592
170784999625,23cyclictest1750496-21ssh10:26:090
170784999623,52cyclictest1787811-21ssh11:45:000
1707849996225,4cyclictest1711736-21munin-run08:25:000
1707849996223,5cyclictest1-21systemd08:20:000
1707849996216,13cyclictest1782909-21sshd11:34:400
1707849996215,14cyclictest1753635-21ssh10:33:080
1707849996120,10cyclictest1791591-21sshd11:52:400
1707849996120,10cyclictest1791591-21sshd11:52:390
1707849996039,17cyclictest1796817-21sshd12:03:400
1707849996018,14cyclictest1715290-21ssh09:12:250
1707849996018,14cyclictest1715290-21ssh09:12:250
1707849996016,11cyclictest1793008-21kworker/u8:212:10:130
1707849995916,39cyclictest0-21swapper/007:31:390
1707849995913,39cyclictest0-21swapper/012:30:020
170786199583,37cyclictest37-21ksoftirqd/307:55:013
170785999581,25cyclictest1766313-21cron11:00:002
1707849995817,14cyclictest1721022-21ssh09:24:480
1707849995815,12cyclictest0-21swapper/011:10:020
1707849995815,10cyclictest1713527-21latency_hist09:00:020
1707849995815,10cyclictest1713527-21latency_hist09:00:020
1707859995748,6cyclictest1699406-21kworker/2:208:10:022
1707849995749,5cyclictest1712771-21munin-run08:45:010
1707859995547,5cyclictest1752130-21kworker/2:011:57:212
1707859995531,13cyclictest1714110-21diskmemload11:13:402
170784999555,22cyclictest0-21swapper/009:41:060
170784999555,22cyclictest0-21swapper/009:41:060
1707849995540,11cyclictest1709874-21sshd07:45:080
1707849995540,11cyclictest1709874-21sshd07:45:080
1707849995514,11cyclictest1714110-21diskmemload09:32:350
1707854995437,13cyclictest3467602-21systemd-journal07:20:001
1707849995449,3cyclictest19350irq/34-mmc108:05:020
1707849995449,3cyclictest19350irq/34-mmc108:05:020
1707849995437,12cyclictest1710922-21sshd08:08:390
1707849995435,14cyclictest1712503-21systemctl08:40:000
1707849995420,29cyclictest16850irq/134-800000011:16:210
1707859995345,5cyclictest1725924-21kworker/2:111:03:012
1707859995345,5cyclictest1725924-21kworker/2:110:00:472
1707859995345,5cyclictest1709683-21kworker/2:009:23:172
1707859995344,6cyclictest1802117-21kworker/2:212:20:512
1707859995344,6cyclictest1802117-21kworker/2:212:20:512
1707859995344,6cyclictest1752130-21kworker/2:010:46:112
1707859995344,5cyclictest1742597-21kworker/2:310:24:412
1707859995343,5cyclictest1699406-21kworker/2:209:40:112
1707859995343,5cyclictest1699406-21kworker/2:209:40:112
1707854995340,9cyclictest1-21systemd08:50:001
1707854995321,6cyclictest131rcu_preempt11:52:121
1707854995321,6cyclictest131rcu_preempt11:52:121
1707849995337,13cyclictest1710391-21sshd07:58:390
1707859995244,5cyclictest1795041-21kworker/2:312:30:512
1707859995243,6cyclictest1725924-21kworker/2:109:58:262
1707859995243,5cyclictest1725924-21kworker/2:109:50:292
1707859995243,5cyclictest1699406-21kworker/2:209:38:022
170785499529,7cyclictest131rcu_preempt09:20:001
1707854995210,5cyclictest131rcu_preempt07:15:011
1707849995220,27cyclictest364-21snmpd08:30:520
170784999521,2cyclictest0-21swapper/008:30:020
1707859995144,4cyclictest1699406-21kworker/2:209:06:162
1707859995143,5cyclictest1795041-21kworker/2:312:27:122
1707859995143,5cyclictest1795041-21kworker/2:312:14:472
1707859995143,5cyclictest1752130-21kworker/2:011:36:092
1707859995143,5cyclictest1752130-21kworker/2:011:21:102
1707859995143,5cyclictest1752130-21kworker/2:010:40:322
1707859995143,5cyclictest1752130-21kworker/2:010:36:092
1707859995143,5cyclictest1742597-21kworker/2:310:27:482
1707859995143,5cyclictest1742597-21kworker/2:310:17:222
1707859995143,5cyclictest1699406-21kworker/2:207:50:012
1707859995143,5cyclictest1699406-21kworker/2:207:50:012
1707859995142,6cyclictest1725924-21kworker/2:111:48:172
1707859995142,5cyclictest1752130-21kworker/2:011:29:322
1707859995129,15cyclictest275-21dbus-daemon07:40:012
170785499515,5cyclictest131rcu_preempt10:20:001
1707849995139,9cyclictest1708887-21munin-run07:30:010
1707849995133,14cyclictest1713275-21munin-run08:55:010
1707861995031,12cyclictest275-21dbus-daemon12:30:003
1707859995043,4cyclictest1725924-21kworker/2:110:10:442
1707859995042,5cyclictest1802117-21kworker/2:212:38:272
1707859995042,5cyclictest1752130-21kworker/2:012:00:512
1707859995042,5cyclictest1752130-21kworker/2:011:07:592
1707859995042,5cyclictest1725924-21kworker/2:111:53:302
1707859995042,5cyclictest1725924-21kworker/2:111:53:292
1707859995042,5cyclictest1709683-21kworker/2:009:26:102
1707859995041,6cyclictest1699406-21kworker/2:209:47:192
1707859995041,6cyclictest1699406-21kworker/2:209:32:572
1707859995041,5cyclictest1699406-21kworker/2:209:16:162
1707859995036,10cyclictest30-21ksoftirqd/211:20:002
1707859995035,10cyclictest30-21ksoftirqd/208:30:012
1707859995034,11cyclictest3467602-21systemd-journal08:47:402
1707854995037,8cyclictest23-21ksoftirqd/109:30:021
1707854995017,7cyclictest23-21ksoftirqd/110:40:011
170786199491,3cyclictest131rcu_preempt10:15:023
1707859994941,5cyclictest1752130-21kworker/2:011:31:342
1707859994941,5cyclictest1752130-21kworker/2:010:53:122
1707859994941,5cyclictest1725924-21kworker/2:110:30:372
1707859994941,5cyclictest1699406-21kworker/2:209:12:092
1707859994941,5cyclictest1699406-21kworker/2:209:12:092
1707859994941,5cyclictest1699406-21kworker/2:207:25:012
1707859994940,6cyclictest1725924-21kworker/2:110:08:272
1707859994920,10cyclictest30-21ksoftirqd/207:10:022
170785499498,6cyclictest131rcu_preempt09:10:451
170785499498,6cyclictest131rcu_preempt09:10:451
170785499497,7cyclictest131rcu_preempt12:40:001
170785499496,8cyclictest23-21ksoftirqd/111:25:021
1707854994937,10cyclictest1764855-21sshd10:56:401
1707849994936,9cyclictest1714074-21apt-config09:06:170
1707861994832,12cyclictest3467602-21systemd-journal09:30:023
1707859994840,5cyclictest1752130-21kworker/2:012:07:502
1707859994839,6cyclictest1699406-21kworker/2:208:00:022
1707859994811,5cyclictest131rcu_preempt09:05:022
170785499488,6cyclictest131rcu_preempt11:40:011
170785499486,6cyclictest131rcu_preempt09:56:501
1707854994835,9cyclictest1-21systemd08:05:011
1707854994835,9cyclictest1-21systemd08:05:011
1707854994831,13cyclictest1-21systemd11:15:001
1707854994822,22cyclictest23-21ksoftirqd/112:06:281
1707854994810,5cyclictest131rcu_preempt10:25:021
1707849994841,4cyclictest1708600-21sshd07:23:390
1707859994739,5cyclictest1699406-21kworker/2:208:41:042
1707859994739,5cyclictest1699406-21kworker/2:208:16:432
1707859994739,5cyclictest1699406-21kworker/2:207:14:252
170785499478,5cyclictest131rcu_preempt11:45:021
170785499478,5cyclictest131rcu_preempt10:10:021
170785499477,6cyclictest131rcu_preempt09:50:001
170785499474,6cyclictest131rcu_preempt07:45:011
170785499473,5cyclictest131rcu_preempt11:33:181
1707854994735,9cyclictest1713250-21sshd08:52:391
1707861994629,12cyclictest1-21systemd11:40:013
1707859994639,4cyclictest1699406-21kworker/2:208:33:532
1707859994638,5cyclictest1699406-21kworker/2:208:58:222
1707859994638,5cyclictest1699406-21kworker/2:208:58:222
170785499467,5cyclictest131rcu_preempt12:00:011
170785499467,5cyclictest131rcu_preempt07:34:391
1707854994630,12cyclictest1-21systemd08:15:011
1707854994630,12cyclictest1-21systemd08:15:011
1707854994630,11cyclictest275-21dbus-daemon09:00:011
1707854994630,11cyclictest275-21dbus-daemon09:00:011
1707854994629,13cyclictest275-21dbus-daemon10:55:011
1707854994623,19cyclictest23-21ksoftirqd/109:32:181
1707854994622,20cyclictest23-21ksoftirqd/111:30:011
170785499461,5cyclictest131rcu_preempt09:54:431
1707854994610,5cyclictest131rcu_preempt10:40:391
1707854994610,17cyclictest0-21swapper/110:50:031
1707849994638,4cyclictest1713024-21munin-run08:50:000
1707849994630,10cyclictest1707839-21cyclictest07:10:010
170786199453,3cyclictest131rcu_preempt10:51:543
170786199453,3cyclictest131rcu_preempt09:35:493
1707861994531,11cyclictest1754454-21systemd-run10:35:003
170785999455,10cyclictest30-21ksoftirqd/207:19:402
170785499459,7cyclictest131rcu_preempt11:45:241
170785499459,6cyclictest131rcu_preempt10:04:041
170785499456,8cyclictest23-21ksoftirqd/111:08:421
170785499454,5cyclictest131rcu_preempt12:13:311
1707854994521,19cyclictest23-21ksoftirqd/110:32:431
1707854994512,5cyclictest131rcu_preempt12:20:241
1707854994512,5cyclictest131rcu_preempt12:20:241
1707849994533,8cyclictest364-21snmpd09:03:230
1707849994522,19cyclictest1711255-21latency_hist08:15:020
1707849994522,19cyclictest1711255-21latency_hist08:15:020
1707861994433,8cyclictest1713497-21sshd08:56:393
1707861994433,8cyclictest1713497-21sshd08:56:393
170786199441,3cyclictest131rcu_preempt10:16:153
1707859994437,4cyclictest1699406-21kworker/2:208:36:102
1707859994436,5cyclictest1699406-21kworker/2:207:34:562
1707859994435,5cyclictest1699406-21kworker/2:208:00:062
1707859994435,5cyclictest1699406-21kworker/2:208:00:062
1707859994435,5cyclictest1699406-21kworker/2:207:27:112
170785999441,5cyclictest131rcu_preempt08:55:002
170785499449,6cyclictest131rcu_preempt09:10:001
170785499448,5cyclictest131rcu_preempt09:39:261
170785499447,5cyclictest131rcu_preempt10:25:101
170785499444,5cyclictest131rcu_preempt12:26:551
170785499443,5cyclictest131rcu_preempt10:10:221
1707854994430,11cyclictest1712776-21systemd-run08:45:011
170785499442,7cyclictest23-21ksoftirqd/112:00:391
170785499442,5cyclictest131rcu_preempt08:00:001
1707849994420,20cyclictest1710164-21cut07:55:000
170786199433,3cyclictest131rcu_preempt10:23:213
170786199433,3cyclictest131rcu_preempt09:54:393
1707861994331,8cyclictest1-21systemd11:45:023
1707861994330,9cyclictest1-21systemd09:15:013
1707861994330,9cyclictest1-21systemd09:15:013
1707861994324,7cyclictest37-21ksoftirqd/312:05:163
1707859994336,4cyclictest1699406-21kworker/2:207:43:412
170785499439,6cyclictest23-21ksoftirqd/111:18:161
170785499438,5cyclictest131rcu_preempt11:04:571
170785499435,5cyclictest131rcu_preempt12:15:391
170785499433,35cyclictest1711461-21munin-plugin-st08:20:001
1707854994327,12cyclictest3467602-21systemd-journal07:46:391
1707854994327,12cyclictest3467602-21systemd-journal07:46:391
170785499431,5cyclictest131rcu_preempt09:22:411
170785499431,5cyclictest131rcu_preempt08:25:021
170786199425,3cyclictest131rcu_preempt09:43:503
170786199425,3cyclictest131rcu_preempt09:43:503
1707861994229,10cyclictest37-21ksoftirqd/311:11:433
170786199421,3cyclictest131rcu_preempt11:48:163
170786199421,3cyclictest131rcu_preempt11:03:473
170786199421,3cyclictest131rcu_preempt10:55:173
170786199421,3cyclictest131rcu_preempt10:08:193
1707859994234,5cyclictest1699406-21kworker/2:207:50:052
170785499426,6cyclictest131rcu_preempt09:41:171
170785499426,6cyclictest131rcu_preempt09:41:171
170786199418,15cyclictest1737989-21ssh10:00:033
170786199414,3cyclictest131rcu_preempt11:51:353
170786199414,3cyclictest131rcu_preempt11:51:343
170786199414,3cyclictest131rcu_preempt09:18:383
1707861994130,6cyclictest309-21in:imuxsock12:35:013
1707861994129,9cyclictest275-21dbus-daemon09:50:003
1707861994121,13cyclictest1751075-21ssh10:27:363
170785999411,4cyclictest131rcu_preempt08:25:022
170785499419,6cyclictest131rcu_preempt08:10:001
170785499416,5cyclictest131rcu_preempt07:10:011
170785499414,5cyclictest131rcu_preempt08:40:021
170785499413,6cyclictest131rcu_preempt08:30:021
1707854994128,10cyclictest3467602-21systemd-journal09:05:011
170785499412,6cyclictest131rcu_preempt07:25:021
1707854994125,11cyclictest1-21systemd07:40:011
1707854994114,5cyclictest131rcu_preempt12:31:401
170786199403,3cyclictest131rcu_preempt12:20:433
170786199403,3cyclictest131rcu_preempt12:20:433
1707861994030,7cyclictest1776825-21sshd11:21:393
1707861994030,5cyclictest1721374-21sh09:25:033
1707861994028,9cyclictest37-21ksoftirqd/312:19:393
170786199402,3cyclictest131rcu_preempt11:55:233
170786199401,3cyclictest131rcu_preempt11:27:293
1707854994025,12cyclictest3467602-21systemd-journal07:53:021
170785499401,4cyclictest131rcu_preempt08:30:281
1707854994010,4cyclictest131rcu_preempt07:25:031
1707861993928,7cyclictest1761602-21cat10:50:023
1707861993925,10cyclictest37-21ksoftirqd/309:10:003
1707861993924,11cyclictest1757042-21cut10:40:023
170786199391,3cyclictest131rcu_preempt11:17:503
1707859993923,12cyclictest30-21ksoftirqd/208:10:022
1707859993923,12cyclictest30-21ksoftirqd/208:10:022
1707861993828,7cyclictest37-21ksoftirqd/311:10:033
170786199381,6cyclictest1712500-21munin-run08:40:003
1707861993729,4cyclictest1713028-21sh08:50:003
1707861993726,9cyclictest37-21ksoftirqd/312:00:153
1707861993725,10cyclictest37-21ksoftirqd/312:14:103
170786199372,29cyclictest1712466-21kworker/u8:008:55:023
170786199371,3cyclictest131rcu_preempt11:31:343
170786199371,33cyclictest37-21ksoftirqd/310:41:123
1707861993626,8cyclictest37-21ksoftirqd/308:10:013
1707861993625,8cyclictest1710449-21systemd-run08:00:013
1707861993620,11cyclictest275-21dbus-daemon09:05:013
1707861993619,12cyclictest309-21in:imuxsock07:34:593
170786199361,3cyclictest131rcu_preempt09:20:033
1707861993527,6cyclictest37-21ksoftirqd/308:02:463
1707861993527,6cyclictest37-21ksoftirqd/308:02:463
1707861993524,9cyclictest37-21ksoftirqd/310:03:383
1707861993524,6cyclictest37-21ksoftirqd/308:25:013
1707861993522,10cyclictest37-21ksoftirqd/308:30:013
1707861993425,6cyclictest1712870-21cat08:45:023
1707861993425,6cyclictest1708240-21latency_hist07:15:033
1707861993424,5cyclictest309-21in:imuxsock08:20:003
1707861993421,7cyclictest37-21ksoftirqd/307:40:013
1707861993418,5cyclictest131rcu_preempt07:42:433
1707861993418,5cyclictest131rcu_preempt07:30:013
170786199341,3cyclictest131rcu_preempt07:50:023
170786199341,3cyclictest131rcu_preempt07:50:023
1707861993221,7cyclictest37-21ksoftirqd/307:25:013
170786199311,5cyclictest131rcu_preempt07:10:023
170786199303,4cyclictest131rcu_preempt08:30:283
170786199295,3cyclictest131rcu_preempt08:14:393
170786199295,3cyclictest131rcu_preempt08:14:393
170786199293,4cyclictest131rcu_preempt07:15:393
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional