You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 22:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Sep 01, 2026 12:45:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1606154991004,10cyclictest131rcu_preempt07:15:011
160616399831,65cyclictest1645701-21ssh10:20:033
1606154997974,3cyclictest231ksoftirqd/107:20:021
160615899744,57cyclictest1650369-21ssh10:30:012
160615499725,20cyclictest1664584-21modprobe10:59:591
160614999725,5cyclictest131rcu_preempt12:00:010
1606154997130,11cyclictest1611755-21sshd08:57:231
160615499695,16cyclictest1635911-21modprobe10:00:001
1606149996830,29cyclictest17060751ssh12:27:020
1606163996739,19cyclictest37-21ksoftirqd/310:05:013
1606149996738,12cyclictest1-21systemd10:50:010
1606163996421,11cyclictest37-21ksoftirqd/308:25:013
160616399637,6cyclictest131rcu_preempt11:35:003
1606163996311,10cyclictest37-21ksoftirqd/312:15:013
1606163996311,10cyclictest37-21ksoftirqd/312:15:013
1606158996315,13cyclictest30-21ksoftirqd/211:10:452
1606158996315,13cyclictest30-21ksoftirqd/211:10:452
1606158996312,12cyclictest30-21ksoftirqd/212:35:032
1606158996312,12cyclictest30-21ksoftirqd/212:35:022
1606158996115,8cyclictest30-21ksoftirqd/209:54:542
1606158996112,10cyclictest30-21ksoftirqd/211:34:432
1606149996111,9cyclictest12-21ksoftirqd/012:09:550
1606163996027,11cyclictest37-21ksoftirqd/311:20:003
1606163996020,10cyclictest37-21ksoftirqd/309:40:253
1606158996018,7cyclictest30-21ksoftirqd/210:53:592
1606158996015,9cyclictest30-21ksoftirqd/209:24:212
1606158996015,10cyclictest30-21ksoftirqd/209:42:412
1606158996012,10cyclictest30-21ksoftirqd/212:10:002
160614999606,6cyclictest131rcu_preempt09:40:000
1606149996017,34cyclictest16850irq/134-800000011:41:020
1606149996017,34cyclictest16850irq/134-800000011:41:020
1606158995917,8cyclictest30-21ksoftirqd/209:13:142
1606158995917,8cyclictest30-21ksoftirqd/209:13:142
1606158995916,8cyclictest30-21ksoftirqd/211:15:392
1606158995915,11cyclictest30-21ksoftirqd/210:49:012
1606158995914,8cyclictest30-21ksoftirqd/211:02:332
1606158995914,8cyclictest30-21ksoftirqd/211:02:332
1606158995914,10cyclictest30-21ksoftirqd/212:22:152
1606158995914,10cyclictest30-21ksoftirqd/212:22:142
160616399587,6cyclictest131rcu_preempt09:40:003
1606163995812,5cyclictest131rcu_preempt07:40:013
1606163995811,8cyclictest131rcu_preempt11:45:033
1606163995811,8cyclictest131rcu_preempt11:45:023
1606163995811,6cyclictest131rcu_preempt10:35:023
1606163995811,6cyclictest131rcu_preempt10:35:023
160614999589,8cyclictest12-21ksoftirqd/010:42:490
1606149995839,16cyclictest16850irq/134-800000009:15:020
1606149995839,16cyclictest16850irq/134-800000009:15:010
1606149995829,13cyclictest1673252-21ssh11:18:000
1606163995718,10cyclictest37-21ksoftirqd/312:30:013
1606163995712,6cyclictest131rcu_preempt11:02:273
1606163995712,6cyclictest131rcu_preempt11:02:273
1606158995717,36cyclictest1611016-21cron08:45:002
1606158995716,9cyclictest30-21ksoftirqd/211:55:002
1606158995716,9cyclictest30-21ksoftirqd/211:55:002
1606158995712,9cyclictest30-21ksoftirqd/211:20:112
1606154995735,13cyclictest1652802-21date10:35:021
1606154995735,13cyclictest1652802-21date10:35:021
1606149995717,35cyclictest16850irq/134-800000012:15:030
1606149995717,35cyclictest16850irq/134-800000012:15:030
160614999571,40cyclictest1702064-21ssh12:18:450
1606149995711,41cyclictest16850irq/134-800000012:31:270
1606149995711,41cyclictest16850irq/134-800000012:31:260
160616399569,6cyclictest131rcu_preempt09:50:003
160616399567,6cyclictest131rcu_preempt10:00:003
1606163995618,10cyclictest37-21ksoftirqd/310:43:303
1606163995610,5cyclictest131rcu_preempt09:23:423
160615899564,9cyclictest30-21ksoftirqd/212:38:232
1606158995635,14cyclictest3467602-21systemd-journal09:40:002
1606158995620,10cyclictest30-21ksoftirqd/210:25:032
1606158995620,10cyclictest30-21ksoftirqd/210:25:022
1606158995617,10cyclictest30-21ksoftirqd/210:17:012
160615899561,16cyclictest1655984-21ssh10:41:342
160614999566,36cyclictest1670239-21kworker/u8:211:30:020
1606149995627,13cyclictest1690637-21ssh11:54:450
1606149995627,13cyclictest1690637-21ssh11:54:450
160614999562,34cyclictest131rcu_preempt10:55:020
160616399558,6cyclictest131rcu_preempt10:10:493
1606163995514,5cyclictest131rcu_preempt11:30:023
1606163995510,6cyclictest131rcu_preempt10:55:413
1606158995513,8cyclictest30-21ksoftirqd/211:36:462
1606158995512,9cyclictest30-21ksoftirqd/210:00:092
160614999556,9cyclictest12-21ksoftirqd/010:16:320
160614999552,37cyclictest1634995-21ssh09:57:450
1606149995518,33cyclictest16850irq/134-800000010:39:130
1606149995515,35cyclictest1612329-21grep09:10:000
1606149995515,11cyclictest12-21ksoftirqd/010:07:250
160616399548,6cyclictest131rcu_preempt09:05:013
160616399548,6cyclictest131rcu_preempt09:05:003
1606163995436,14cyclictest1611773-21cron09:00:003
160616399543,11cyclictest37-21ksoftirqd/310:21:263
160616399543,11cyclictest37-21ksoftirqd/310:21:263
1606163995414,6cyclictest131rcu_preempt11:13:183
1606163995414,6cyclictest131rcu_preempt11:13:173
1606163995410,7cyclictest131rcu_preempt11:25:023
160615899549,9cyclictest30-21ksoftirqd/211:57:432
1606158995414,8cyclictest30-21ksoftirqd/211:30:012
1606149995429,10cyclictest1694659-21ssh12:03:080
1606149995429,10cyclictest1694659-21ssh12:03:080
1606149995427,12cyclictest1675233-21ssh11:22:040
1606149995427,12cyclictest1648612-21ssh10:26:010
1606149995422,16cyclictest1642663-21ssh10:13:550
1606149995413,7cyclictest16850irq/134-800000009:50:020
160614999541,37cyclictest1662525-21ssh10:55:190
160616399539,6cyclictest131rcu_preempt09:30:293
160616399539,6cyclictest131rcu_preempt09:30:293
160616399539,5cyclictest131rcu_preempt11:57:273
160616399538,6cyclictest131rcu_preempt08:00:013
160616399538,5cyclictest131rcu_preempt09:55:003
160616399536,13cyclictest0-21swapper/308:40:013
1606163995315,7cyclictest37-21ksoftirqd/311:10:013
1606163995313,6cyclictest37-21ksoftirqd/310:36:223
1606163995310,6cyclictest131rcu_preempt11:50:003
1606163995310,5cyclictest131rcu_preempt09:28:533
160615899539,8cyclictest30-21ksoftirqd/211:44:352
160615899539,8cyclictest30-21ksoftirqd/211:44:352
160615899538,10cyclictest30-21ksoftirqd/209:30:002
1606158995313,7cyclictest30-21ksoftirqd/212:28:412
1606158995310,9cyclictest30-21ksoftirqd/209:49:142
1606158995310,8cyclictest30-21ksoftirqd/209:20:012
160614999535,9cyclictest12-21ksoftirqd/011:30:430
1606149995334,14cyclictest1619419-21systemd-run09:25:010
1606149995322,16cyclictest1711871-21ssh12:39:340
160616399528,5cyclictest131rcu_preempt09:10:013
160616399522,32cyclictest361rcuc/307:10:023
160615899527,6cyclictest30-21ksoftirqd/209:32:052
160615899527,6cyclictest30-21ksoftirqd/209:32:042
1606158995213,7cyclictest30-21ksoftirqd/210:32:392
1606158995213,7cyclictest30-21ksoftirqd/210:32:392
1606154995234,13cyclictest275-21dbus-daemon09:05:011
1606154995234,13cyclictest275-21dbus-daemon09:05:011
1606149995233,14cyclictest3467602-21systemd-journal11:10:000
1606149995212,9cyclictest131rcu_preempt07:45:000
160616399519,6cyclictest131rcu_preempt12:38:543
160616399516,6cyclictest131rcu_preempt12:04:293
160616399516,6cyclictest131rcu_preempt12:04:283
160616399516,4cyclictest131rcu_preempt12:31:463
160616399516,4cyclictest131rcu_preempt12:31:463
1606163995114,6cyclictest131rcu_preempt10:54:333
1606163995114,6cyclictest131rcu_preempt08:55:013
1606163995112,6cyclictest131rcu_preempt08:10:003
160615899515,10cyclictest30-21ksoftirqd/210:57:172
1606158995142,6cyclictest1612341-21kworker/2:009:57:112
1606158995133,14cyclictest275-21dbus-daemon11:10:012
160615899513,16cyclictest1-21systemd07:55:012
1606149995127,9cyclictest1669930-21ssh11:10:510
1606149995127,9cyclictest1669930-21ssh11:10:500
1606149995125,12cyclictest1687999-21ssh11:49:080
1606149995123,24cyclictest1627318-21ssh09:41:220
1606149995116,31cyclictest16850irq/134-800000011:37:490
1606149995116,30cyclictest16850irq/134-800000012:20:500
1606149995116,30cyclictest16850irq/134-800000012:20:490
1606149995116,11cyclictest12-21ksoftirqd/009:16:560
1606149995115,31cyclictest16850irq/134-800000010:21:220
1606149995115,31cyclictest16850irq/134-800000010:21:220
1606149995115,31cyclictest16850irq/134-800000009:51:390
160616399508,6cyclictest131rcu_preempt09:20:013
160616399507,6cyclictest131rcu_preempt09:10:503
160616399507,6cyclictest131rcu_preempt09:10:503
160616399506,7cyclictest37-21ksoftirqd/311:37:163
160616399505,8cyclictest37-21ksoftirqd/311:53:073
160616399505,8cyclictest37-21ksoftirqd/311:53:063
160616399505,6cyclictest131rcu_preempt12:19:053
160616399505,6cyclictest131rcu_preempt08:20:003
1606163995019,5cyclictest37-21ksoftirqd/312:25:013
1606163995019,5cyclictest37-21ksoftirqd/312:25:013
1606163995012,5cyclictest131rcu_preempt10:25:333
160615899506,8cyclictest30-21ksoftirqd/210:10:012
160615499506,8cyclictest1643103-21modprobe10:15:001
160614999509,17cyclictest131rcu_preempt10:02:370
1606149995021,14cyclictest1666405-21ssh11:03:370
1606149995021,14cyclictest1666405-21ssh11:03:360
1606149995016,30cyclictest16850irq/134-800000009:25:060
1606149995014,32cyclictest16850irq/134-800000010:32:370
1606149995014,32cyclictest16850irq/134-800000010:32:370
160616399498,5cyclictest131rcu_preempt08:50:013
160616399496,11cyclictest37-21ksoftirqd/307:24:193
1606163994915,6cyclictest131rcu_preempt10:05:023
160616399491,10cyclictest37-21ksoftirqd/310:50:003
160615899498,9cyclictest30-21ksoftirqd/208:35:022
160615899495,10cyclictest30-21ksoftirqd/211:48:222
1606158994923,20cyclictest275-21dbus-daemon12:15:022
1606158994923,20cyclictest275-21dbus-daemon12:15:012
160615899492,14cyclictest1606667-21cat07:20:012
1606149994933,13cyclictest3467602-21systemd-journal09:00:030
1606149994933,13cyclictest3467602-21systemd-journal09:00:020
1606149994914,31cyclictest16850irq/134-800000009:32:300
1606149994914,31cyclictest16850irq/134-800000009:32:300
1606149994911,5cyclictest131rcu_preempt07:50:010
1606149994911,5cyclictest131rcu_preempt07:50:010
160616399489,5cyclictest131rcu_preempt07:45:013
160616399486,10cyclictest37-21ksoftirqd/312:05:273
160616399485,7cyclictest37-21ksoftirqd/308:12:203
160616399485,11cyclictest37-21ksoftirqd/307:30:193
160616399483,9cyclictest37-21ksoftirqd/308:35:003
1606163994817,14cyclictest0-21swapper/307:55:003
160615899486,8cyclictest30-21ksoftirqd/212:00:512
160615899486,8cyclictest30-21ksoftirqd/212:00:512
1606158994812,5cyclictest0-21swapper/210:12:322
1606154994832,12cyclictest23-21ksoftirqd/111:40:001
1606154994831,13cyclictest3467602-21systemd-journal08:30:001
1606154994830,13cyclictest275-21dbus-daemon11:05:021
1606154994830,13cyclictest275-21dbus-daemon11:05:011
1606149994840,5cyclictest16950irq/224-800000008:00:000
1606149994812,9cyclictest131rcu_preempt07:15:000
1606149994812,6cyclictest131rcu_preempt07:55:010
160616399477,7cyclictest131rcu_preempt07:50:023
160616399477,7cyclictest131rcu_preempt07:50:023
1606163994711,4cyclictest131rcu_preempt08:45:003
160615899472,13cyclictest1612059-21cron09:05:002
160615899472,13cyclictest1612059-21cron09:05:002
1606154994734,10cyclictest1606156-21ls07:10:021
1606154994730,13cyclictest1-21systemd12:05:021
1606154994730,13cyclictest1-21systemd12:05:011
1606154994727,16cyclictest3467602-21systemd-journal09:25:011
1606154994727,15cyclictest275-21dbus-daemon07:25:011
1606149994713,7cyclictest131rcu_preempt07:40:010
160616399464,7cyclictest131rcu_preempt07:15:003
160616399461,9cyclictest37-21ksoftirqd/307:19:123
160615899468,7cyclictest1611429-21cat08:50:022
1606158994638,5cyclictest1672136-21kworker/2:112:16:342
1606154994634,10cyclictest23-21ksoftirqd/108:05:021
1606154994627,14cyclictest23-21ksoftirqd/111:20:001
1606154994626,14cyclictest3467602-21systemd-journal12:30:001
160614999468,27cyclictest1606153-21latency_hist07:10:020
1606163994510,5cyclictest131rcu_preempt07:30:003
160615899455,10cyclictest1654308-21ssh10:38:152
1606154994531,10cyclictest1-21systemd09:55:021
1606154994529,13cyclictest1626520-21sh09:40:011
1606154994528,12cyclictest275-21dbus-daemon12:40:011
160616399445,10cyclictest37-21ksoftirqd/308:04:213
160616399442,7cyclictest37-21ksoftirqd/308:25:193
1606158994425,14cyclictest1609444-21cron08:15:002
1606158994423,15cyclictest0-21swapper/208:05:002
1606154994426,13cyclictest1-21systemd09:20:001
1606154994412,6cyclictest131rcu_preempt09:40:281
160614999442,6cyclictest1610762-21modprobe08:40:000
160614999442,5cyclictest1609705-21cron08:19:590
1606149994419,10cyclictest12-21ksoftirqd/007:25:010
160614999441,4cyclictest131rcu_preempt08:35:010
160614999441,3cyclictest16061431cyclictest08:25:000
1606149994412,5cyclictest131rcu_preempt08:44:190
1606158994327,12cyclictest275-21dbus-daemon09:00:002
1606154994330,10cyclictest1685756-21sshd11:44:271
1606154994330,10cyclictest1685756-21sshd11:44:261
1606154994328,12cyclictest0-21swapper/110:20:011
1606154994322,16cyclictest3467602-21systemd-journal11:09:591
160615499431,3cyclictest131rcu_preempt08:00:021
1606149994326,4cyclictest1606665-21kworker/0:008:03:370
160614999431,13cyclictest364-21snmpd08:48:240
160615899426,7cyclictest30-21ksoftirqd/207:39:162
1606158994232,6cyclictest309-21in:imuxsock07:15:002
160615499425,23cyclictest1607638-21modprobe07:40:011
160615499423,5cyclictest1630146-21ssh09:47:281
1606154994231,8cyclictest3467602-21systemd-journal11:48:271
1606154994210,6cyclictest131rcu_preempt10:08:411
1606149994233,6cyclictest1611772-21cron09:00:000
160614999421,36cyclictest1610254-21true08:30:010
160615899411,9cyclictest30-21ksoftirqd/208:20:022
160615499415,28cyclictest1624125-21kworker/u8:010:30:011
1606154994128,9cyclictest23-21ksoftirqd/112:35:001
1606154994128,9cyclictest23-21ksoftirqd/112:35:001
1606154994127,11cyclictest275-21dbus-daemon11:15:021
1606154994127,11cyclictest275-21dbus-daemon11:15:011
1606154994117,17cyclictest275-21dbus-daemon07:45:011
160614999413,6cyclictest131rcu_preempt08:55:010
160614999411,36cyclictest1607131-21latency_hist07:30:020
160615899403,9cyclictest30-21ksoftirqd/208:00:012
1606158994014,4cyclictest131rcu_preempt08:25:012
160615499405,4cyclictest131rcu_preempt08:25:001
1606154994025,11cyclictest23-21ksoftirqd/110:45:131
1606154994015,5cyclictest1621401-21sh09:29:031
160615499401,13cyclictest1691541-21ssh11:56:271
160615899396,7cyclictest131rcu_preempt09:10:012
1606154993928,9cyclictest23-21ksoftirqd/109:33:311
1606154993928,9cyclictest23-21ksoftirqd/109:33:311
1606154993927,9cyclictest23-21ksoftirqd/112:08:131
1606154993927,9cyclictest23-21ksoftirqd/111:30:511
160615499392,6cyclictest1698673-21ssh12:11:211
160615499392,6cyclictest1698673-21ssh12:11:201
1606154993926,10cyclictest23-21ksoftirqd/112:19:371
1606154993925,11cyclictest23-21ksoftirqd/111:23:201
1606158993827,8cyclictest30-21ksoftirqd/208:25:222
1606158993827,8cyclictest30-21ksoftirqd/207:25:012
1606158993825,10cyclictest30-21ksoftirqd/208:10:002
1606158993821,13cyclictest1608087-21sed07:45:022
1606154993829,7cyclictest23-21ksoftirqd/110:55:011
1606154993825,8cyclictest3467602-21systemd-journal07:47:201
1606154993825,8cyclictest3467602-21systemd-journal07:47:191
1606154993824,9cyclictest309-21in:imuxsock08:35:191
1606154993823,12cyclictest23-21ksoftirqd/108:20:011
1606154993822,10cyclictest309-21in:imuxsock12:25:021
1606154993822,10cyclictest309-21in:imuxsock12:25:011
160614999383,6cyclictest131rcu_preempt07:30:080
1606149993828,6cyclictest1606601-21modprobe07:20:010
1606149993822,13cyclictest12-21ksoftirqd/008:10:000
1606158993725,6cyclictest309-21in:imuxsock07:30:202
1606158993722,11cyclictest1610723-21sshd08:37:192
1606158993719,13cyclictest1-21systemd07:30:002
160615499374,3cyclictest131rcu_preempt10:39:131
160615499373,9cyclictest0-21swapper/110:44:411
160615499372,6cyclictest1614665-21true09:15:021
160615499372,6cyclictest1614665-21true09:15:011
1606154993725,9cyclictest23-21ksoftirqd/108:50:021
1606154993725,9cyclictest1637240-21sshd10:02:261
160615499371,4cyclictest131rcu_preempt11:55:011
160615499371,4cyclictest131rcu_preempt11:55:001
1606154993624,9cyclictest23-21ksoftirqd/111:27:151
160615499362,3cyclictest131rcu_preempt07:30:021
160615499361,3cyclictest131rcu_preempt08:10:001
160614999362,4cyclictest131rcu_preempt08:15:020
160615499353,5cyclictest131rcu_preempt08:45:001
1606154993525,8cyclictest23-21ksoftirqd/110:22:361
1606154993525,8cyclictest23-21ksoftirqd/110:22:351
1606154993522,9cyclictest1610468-21sshd08:31:191
1606158993420,10cyclictest48-21kcompactd008:54:512
1606158993325,6cyclictest30-21ksoftirqd/207:50:022
1606158993325,6cyclictest30-21ksoftirqd/207:50:022
1606154993322,8cyclictest23-21ksoftirqd/108:55:021
1606154993321,9cyclictest23-21ksoftirqd/107:33:261
1606154993321,8cyclictest1609215-21kworker/1:009:10:001
1606154993222,7cyclictest23-21ksoftirqd/107:55:021
1606158993123,5cyclictest1606168-21latency_hist07:10:022
160615499311,3cyclictest131rcu_preempt08:15:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional