You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 05:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Mar 12, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1763920991185,90cyclictest1832888-21ssh23:30:022
176392399698,4cyclictest131rcu_preempt20:15:033
176391499625,33cyclictest131rcu_preempt22:43:320
176391499625,33cyclictest131rcu_preempt22:43:320
1763914996124,33cyclictest1832883-21cut23:30:030
1763914996118,22cyclictest1819568-21ssh23:00:030
1763914996017,25cyclictest1841479-21ssh23:49:450
1763914995928,14cyclictest1822322-21ssh23:06:240
1763914995819,30cyclictest1845123-21ssh23:57:380
176391499578,15cyclictest131rcu_preempt21:43:170
176391499576,6cyclictest131rcu_preempt00:20:000
1763914995729,12cyclictest1808901-21ssh22:36:220
1763920995646,5cyclictest1752567-21kworker/2:020:05:152
1763914995627,13cyclictest1862455-21ssh00:36:110
1763914995623,22cyclictest1799824-21ssh22:16:000
1763914995620,32cyclictest16850irq/134-800000021:20:400
1763918995527,18cyclictest23-21ksoftirqd/121:29:521
1763914995527,13cyclictest1855563-21ssh00:20:400
1763914995527,13cyclictest1846854-21ssh00:01:210
1763914995519,6cyclictest16850irq/134-800000022:52:170
1763914995519,19cyclictest16850irq/134-800000022:46:290
1763914995518,19cyclictest16850irq/134-800000022:07:550
1763914995518,19cyclictest16850irq/134-800000022:07:540
1763914995514,21cyclictest16850irq/134-800000000:28:000
1763923995425,7cyclictest37-21ksoftirqd/321:50:153
176392399541,22cyclictest17723731unin-run21:15:013
1763920995446,5cyclictest1767230-21kworker/2:223:37:132
1763918995437,12cyclictest1-21systemd22:05:011
1763914995425,13cyclictest1777519-21ssh21:26:180
176391499542,3cyclictest131rcu_preempt21:15:520
1763914995419,18cyclictest16850irq/134-800000023:32:110
1763914995419,18cyclictest16850irq/134-800000023:11:110
176391499541,37cyclictest1782559-21ssh21:37:450
1763920995345,5cyclictest1767230-21kworker/2:223:53:322
1763920995344,5cyclictest1767230-21kworker/2:200:26:342
1763918995336,12cyclictest275-21dbus-daemon21:00:021
1763914995334,5cyclictest16850irq/134-800000023:17:400
1763914995324,14cyclictest1807072-21ssh22:32:250
1763914995324,14cyclictest1807072-21ssh22:32:250
1763914995324,13cyclictest1860188-21ssh00:31:010
1763914995322,21cyclictest1786052-21ssh21:45:120
1763914995318,5cyclictest16850irq/134-800000022:26:550
1763914995318,19cyclictest1834020-21kworker/u8:200:10:440
1763914995317,6cyclictest16850irq/134-800000023:52:210
1763914995317,6cyclictest16850irq/134-800000021:33:470
1763914995316,7cyclictest16850irq/134-800000022:02:450
176391499531,3cyclictest131rcu_preempt00:05:590
1763920995244,5cyclictest1767230-21kworker/2:221:14:142
1763920995244,5cyclictest1752567-21kworker/2:020:00:032
1763920995244,5cyclictest1752567-21kworker/2:020:00:022
1763914995218,18cyclictest16850irq/134-800000023:41:540
1763914995218,17cyclictest16850irq/134-800000022:56:050
1763914995217,18cyclictest16850irq/134-800000023:22:260
1763914995216,6cyclictest16850irq/134-800000021:55:230
1763923995135,11cyclictest1-21systemd22:40:003
1763920995143,5cyclictest1767230-21kworker/2:200:30:502
1763920995142,5cyclictest1767230-21kworker/2:222:21:212
1763918995134,13cyclictest1-21systemd21:35:011
1763914995117,18cyclictest16850irq/134-800000022:21:560
1763923995035,12cyclictest1-21systemd23:25:003
1763920995043,5cyclictest1767230-21kworker/2:200:23:412
1763920995042,5cyclictest1767230-21kworker/2:222:47:182
1763920995042,5cyclictest1764651-21kworker/2:121:18:042
1763918995034,12cyclictest1-21systemd00:40:001
1763918995025,10cyclictest23-21ksoftirqd/123:29:581
1763918995012,6cyclictest131rcu_preempt21:44:591
1763914995037,9cyclictest16950irq/224-800000021:55:000
1763914995017,16cyclictest16850irq/134-800000022:11:380
1763914995017,16cyclictest16850irq/134-800000022:11:380
176392399499,8cyclictest131rcu_preempt22:05:033
176392399499,6cyclictest131rcu_preempt23:10:003
176392399494,3cyclictest131rcu_preempt21:32:033
1763923994933,12cyclictest3467602-21systemd-journal23:17:013
1763923994926,10cyclictest37-21ksoftirqd/300:34:473
1763920994941,5cyclictest1767230-21kworker/2:200:02:112
1763920994939,7cyclictest1764651-21kworker/2:121:45:012
176391899494,4cyclictest131rcu_preempt22:35:501
1763918994925,5cyclictest23-21ksoftirqd/122:30:001
1763914994923,12cyclictest0-21swapper/021:05:010
1763914994920,24cyclictest16850irq/134-800000021:10:000
176392399485,7cyclictest37-21ksoftirqd/321:50:023
176392399485,3cyclictest131rcu_preempt00:01:223
176392399481,8cyclictest0-21swapper/323:15:013
176392399481,6cyclictest131rcu_preempt23:00:073
1763920994841,4cyclictest1817144-21kworker/2:323:21:442
1763920994840,5cyclictest1767230-21kworker/2:223:58:342
1763920994840,5cyclictest1767230-21kworker/2:223:08:162
1763920994840,5cyclictest1767230-21kworker/2:222:34:532
1763920994840,5cyclictest1767230-21kworker/2:222:34:522
1763920994840,5cyclictest1764651-21kworker/2:122:39:192
1763920994840,5cyclictest1764651-21kworker/2:122:08:392
1763920994840,5cyclictest1764651-21kworker/2:122:08:392
1763920994840,5cyclictest1764651-21kworker/2:121:50:152
1763920994840,5cyclictest1764651-21kworker/2:121:27:072
1763920994840,4cyclictest1817144-21kworker/2:323:19:002
1763920994839,5cyclictest1767230-21kworker/2:200:36:472
1763920994833,6cyclictest1764651-21kworker/2:121:30:082
176391899484,8cyclictest131rcu_preempt00:00:001
176391899484,3cyclictest131rcu_preempt21:20:021
176391899484,3cyclictest131rcu_preempt00:28:521
1763918994837,7cyclictest23-21ksoftirqd/100:22:561
176391899483,3cyclictest131rcu_preempt22:35:011
176391899483,3cyclictest131rcu_preempt22:35:001
176392399476,4cyclictest131rcu_preempt00:22:113
176392399474,3cyclictest131rcu_preempt22:51:323
176392399474,3cyclictest131rcu_preempt22:27:353
176392399474,3cyclictest131rcu_preempt22:10:453
176392399474,3cyclictest131rcu_preempt22:10:453
176392399474,3cyclictest131rcu_preempt22:10:003
176392399474,3cyclictest131rcu_preempt22:10:003
176392399474,3cyclictest131rcu_preempt21:10:023
1763923994726,5cyclictest37-21ksoftirqd/321:00:003
176392399472,4cyclictest131rcu_preempt00:17:573
1763920994740,5cyclictest1767230-21kworker/2:222:54:002
1763920994740,4cyclictest1767230-21kworker/2:222:28:402
1763920994739,5cyclictest1767230-21kworker/2:223:12:122
1763920994739,5cyclictest1767230-21kworker/2:222:18:412
1763920994739,5cyclictest1764651-21kworker/2:121:10:022
1763920994738,6cyclictest1767230-21kworker/2:221:58:482
176391899474,4cyclictest131rcu_preempt23:51:051
176391899474,4cyclictest131rcu_preempt23:30:061
1763918994739,6cyclictest23-21ksoftirqd/121:50:021
1763914994732,10cyclictest3467602-21systemd-journal19:15:000
176392399467,4cyclictest131rcu_preempt23:37:143
176392399465,3cyclictest131rcu_preempt00:10:493
176392399464,3cyclictest131rcu_preempt23:41:093
176392399463,4cyclictest131rcu_preempt21:39:343
176392399463,3cyclictest131rcu_preempt23:48:433
176392399462,5cyclictest131rcu_preempt19:55:013
176392399462,3cyclictest131rcu_preempt00:37:093
1763923994619,6cyclictest131rcu_preempt00:09:063
1763923994617,4cyclictest37-21ksoftirqd/321:59:143
1763920994638,5cyclictest1767230-21kworker/2:222:40:192
1763920994638,5cyclictest1767230-21kworker/2:222:40:182
1763920994638,5cyclictest1767230-21kworker/2:200:11:522
1763920994637,6cyclictest1767230-21kworker/2:223:48:272
1763920994630,8cyclictest1764651-21kworker/2:121:49:212
176391899465,4cyclictest131rcu_preempt22:50:001
176391899465,3cyclictest131rcu_preempt21:36:091
176391899464,4cyclictest131rcu_preempt00:18:111
176391899464,3cyclictest131rcu_preempt23:20:011
176391899464,3cyclictest131rcu_preempt22:56:451
176391899464,3cyclictest131rcu_preempt22:05:161
176391899464,3cyclictest131rcu_preempt22:05:161
176391899464,3cyclictest131rcu_preempt21:12:391
1763918994637,6cyclictest23-21ksoftirqd/121:52:041
1763918994634,7cyclictest1812627-21cut22:45:011
1763918994634,7cyclictest1812627-21cut22:45:001
176391899462,4cyclictest131rcu_preempt23:40:161
176391899462,4cyclictest131rcu_preempt22:17:361
176391899462,3cyclictest131rcu_preempt00:01:571
176391899461,3cyclictest131rcu_preempt22:21:061
1763914994610,32cyclictest16850irq/134-800000021:11:520
176392399455,3cyclictest131rcu_preempt22:47:423
176392399455,3cyclictest131rcu_preempt21:21:013
176392399454,3cyclictest131rcu_preempt22:20:543
176392399454,3cyclictest131rcu_preempt21:41:163
176392399453,4cyclictest131rcu_preempt00:28:373
176392399453,3cyclictest131rcu_preempt23:32:113
176392399453,3cyclictest131rcu_preempt22:35:023
176392399453,3cyclictest131rcu_preempt22:35:013
1763923994528,11cyclictest37-21ksoftirqd/320:20:013
176392399452,3cyclictest131rcu_preempt19:20:033
176392399451,5cyclictest131rcu_preempt23:50:113
1763923994510,6cyclictest131rcu_preempt21:05:013
1763920994537,5cyclictest1764651-21kworker/2:122:02:572
1763920994536,6cyclictest1767230-21kworker/2:223:45:022
1763920994536,5cyclictest1764651-21kworker/2:120:52:232
1763920994525,15cyclictest3467602-21systemd-journal23:05:002
176392099452,3cyclictest131rcu_preempt20:45:012
1763920994517,18cyclictest30-21ksoftirqd/200:20:012
176391899454,3cyclictest131rcu_preempt23:38:581
176391899454,3cyclictest131rcu_preempt22:53:371
176391899454,3cyclictest131rcu_preempt20:40:001
176391899452,4cyclictest131rcu_preempt00:07:381
176391899452,3cyclictest131rcu_preempt23:01:581
1763918994523,6cyclictest23-21ksoftirqd/121:56:501
1763918994521,5cyclictest23-21ksoftirqd/100:14:361
176391499456,35cyclictest16850irq/134-800000023:39:460
1763914994533,9cyclictest1-21systemd20:30:000
1763914994528,13cyclictest3467602-21systemd-journal19:30:010
176392399446,3cyclictest131rcu_preempt19:30:033
176392399444,3cyclictest131rcu_preempt23:56:453
176392399444,3cyclictest131rcu_preempt19:40:013
176392399443,3cyclictest131rcu_preempt20:40:023
176392399442,8cyclictest37-21ksoftirqd/319:10:033
176392399442,5cyclictest131rcu_preempt21:16:423
176392399442,4cyclictest131rcu_preempt21:28:163
176392399442,3cyclictest131rcu_preempt22:40:253
176392399442,3cyclictest131rcu_preempt22:40:243
176392399442,3cyclictest131rcu_preempt22:16:193
1763923994420,5cyclictest37-21ksoftirqd/322:56:203
1763923994420,5cyclictest37-21ksoftirqd/319:35:013
1763923994420,5cyclictest37-21ksoftirqd/319:35:013
176392399441,3cyclictest131rcu_preempt20:00:033
176392399441,3cyclictest131rcu_preempt20:00:023
1763920994436,5cyclictest1764651-21kworker/2:121:35:282
1763920994435,6cyclictest1767230-21kworker/2:222:55:332
1763920994431,10cyclictest3467602-21systemd-journal20:38:182
176391899445,3cyclictest131rcu_preempt21:05:011
176391899444,3cyclictest131rcu_preempt23:45:591
176391899444,3cyclictest131rcu_preempt23:11:151
176391899444,3cyclictest131rcu_preempt21:21:021
176391899444,3cyclictest131rcu_preempt00:31:411
176391899443,3cyclictest131rcu_preempt22:15:011
176391899443,3cyclictest131rcu_preempt22:15:001
176391899443,3cyclictest131rcu_preempt20:41:191
1763918994429,12cyclictest1-21systemd20:10:001
1763918994429,11cyclictest275-21dbus-daemon20:50:001
176391899442,4cyclictest131rcu_preempt23:07:351
176391899441,3cyclictest131rcu_preempt23:20:171
1763914994437,4cyclictest16950irq/224-800000019:25:000
176392399435,3cyclictest131rcu_preempt20:30:003
176392399434,4cyclictest131rcu_preempt19:10:023
176392399433,3cyclictest131rcu_preempt20:45:013
176392399432,4cyclictest131rcu_preempt23:26:293
1763923994319,6cyclictest37-21ksoftirqd/320:55:013
176392399431,3cyclictest131rcu_preempt19:50:023
1763920994335,5cyclictest1767230-21kworker/2:200:09:192
1763920994334,5cyclictest1769857-21latency_hist21:05:022
176391899432,3cyclictest131rcu_preempt19:20:031
176391899432,10cyclictest1765870-21sed19:45:021
176391899431,3cyclictest131rcu_preempt20:30:021
176391899431,3cyclictest131rcu_preempt20:16:181
176392399423,3cyclictest131rcu_preempt20:05:023
176392399423,3cyclictest131rcu_preempt19:20:023
176392399421,3cyclictest131rcu_preempt20:05:023
176391899425,3cyclictest131rcu_preempt20:35:011
176391899423,3cyclictest131rcu_preempt20:05:011
176391899423,3cyclictest131rcu_preempt20:00:011
176391899423,3cyclictest131rcu_preempt20:00:011
176391899423,3cyclictest131rcu_preempt19:10:021
176391899422,3cyclictest131rcu_preempt21:10:001
176391899421,3cyclictest131rcu_preempt19:30:021
176391899421,3cyclictest131rcu_preempt19:30:021
1763914994235,4cyclictest16950irq/224-800000020:25:010
1763914994235,4cyclictest16950irq/224-800000019:50:020
1763914994228,9cyclictest1-21systemd20:05:000
176391499422,36cyclictest1766948-21sshd20:08:180
1763923994133,4cyclictest1765675-21munin-plugin-st19:45:003
176392399412,3cyclictest131rcu_preempt20:50:013
176392399411,3cyclictest131rcu_preempt20:30:023
1763920994132,6cyclictest1817144-21kworker/2:323:30:012
1763920994123,14cyclictest1-21systemd20:15:012
176391899413,3cyclictest131rcu_preempt19:38:191
176391899413,3cyclictest131rcu_preempt19:15:011
1763918994129,8cyclictest1767740-21sshd20:22:181
176391899411,4cyclictest131rcu_preempt20:15:031
176391899411,3cyclictest131rcu_preempt19:25:181
1763914994133,5cyclictest16950irq/224-800000021:00:000
1763914994128,9cyclictest1766426-21cron20:00:000
1763914994128,9cyclictest1766426-21cron20:00:000
176391499412,6cyclictest1766172-21modprobe19:55:000
1763914994125,12cyclictest12-21ksoftirqd/019:35:010
1763914994125,12cyclictest12-21ksoftirqd/019:35:000
1763920994032,5cyclictest1752567-21kworker/2:019:40:362
1763920994027,9cyclictest30-21ksoftirqd/222:13:302
1763920994027,9cyclictest30-21ksoftirqd/222:13:292
176391899403,3cyclictest131rcu_preempt19:47:181
1763914994028,9cyclictest12-21ksoftirqd/019:20:020
1763914994024,11cyclictest12-21ksoftirqd/020:50:010
1763914994012,6cyclictest12-21ksoftirqd/020:55:010
1763923993916,3cyclictest37-21ksoftirqd/320:21:183
1763920993928,9cyclictest30-21ksoftirqd/219:50:012
1763920993926,10cyclictest30-21ksoftirqd/221:23:382
176391899392,4cyclictest131rcu_preempt19:20:321
176391499391,3cyclictest131rcu_preempt19:10:030
176391499391,13cyclictest3467602-21systemd-journal20:30:180
1763920993830,5cyclictest1752567-21kworker/2:019:28:122
1763920993827,7cyclictest309-21in:imuxsock19:15:002
1763920993826,9cyclictest30-21ksoftirqd/220:35:012
1763920993826,9cyclictest1766280-21wc19:55:012
176391899383,3cyclictest131rcu_preempt20:55:011
176391899381,3cyclictest131rcu_preempt19:52:191
176391499382,4cyclictest1768562-21munin-run20:40:020
176391499381,33cyclictest1767466-21sshd20:18:180
1763920993718,14cyclictest275-21dbus-daemon20:20:002
176391499372,31cyclictest0-21swapper/020:15:000
176391499371,32cyclictest0-21swapper/020:44:290
176391499371,31cyclictest3467718-21lldpd19:37:400
1763920993620,12cyclictest1-21systemd19:25:012
176391499361,3cyclictest131rcu_preempt19:45:010
1763920993526,7cyclictest30-21ksoftirqd/219:40:012
1763920993525,7cyclictest1768117-21latency_hist20:30:012
1763920993525,7cyclictest1764341-21sshd19:18:182
1763920993520,12cyclictest1-21systemd20:05:012
176392099351,3cyclictest131rcu_preempt20:50:022
1763920993426,5cyclictest1764651-21kworker/2:120:24:252
1763920993422,9cyclictest309-21in:imuxsock21:00:002
1763920993221,7cyclictest30-21ksoftirqd/219:35:022
1763920993221,7cyclictest30-21ksoftirqd/219:35:022
176392099293,3cyclictest131rcu_preempt19:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional