You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 19:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Mar 11, 2026 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16080259911714,7cyclictest131rcu_preempt11:35:023
1608016998177,2cyclictest0-21swapper/011:45:020
1608025997624,10cyclictest275-21dbus-daemon11:50:013
1608022996730,31cyclictest291rcuc/212:00:022
1608020996721,5cyclictest1688963-21sh11:56:541
160802099663,19cyclictest1656673-21munin-run10:45:021
1608022996512,21cyclictest131rcu_preempt07:40:002
1608020996425,35cyclictest1647675-21sh10:24:491
1608020996422,5cyclictest1618399-21sh09:19:211
1608020996322,37cyclictest23-21ksoftirqd/108:00:031
1608020996228,30cyclictest23-21ksoftirqd/111:38:051
1608016996228,15cyclictest131rcu_preempt11:30:030
1608020996124,32cyclictest23-21ksoftirqd/112:33:011
1608020996118,24cyclictest1660891-21kworker/u8:010:55:031
1608016996114,42cyclictest12-21ksoftirqd/011:05:020
1608016996110,10cyclictest1608502-21modprobe07:20:010
1608020996025,32cyclictest23-21ksoftirqd/111:06:181
1608020996024,31cyclictest1619626-21sh09:21:521
1608020996019,11cyclictest23-21ksoftirqd/112:22:461
1608020996017,6cyclictest23-21ksoftirqd/109:39:361
1608025995938,19cyclictest37-21ksoftirqd/310:15:023
160802099592,20cyclictest1650982-21ssh10:31:511
1608020995921,9cyclictest23-21ksoftirqd/109:28:061
1608020995917,6cyclictest0-21swapper/112:15:281
1608020995913,9cyclictest1661348-21sh10:55:031
1608016995924,32cyclictest1645679-21latency_hist10:20:020
1608022995839,9cyclictest1618664-21cron09:20:002
1608020995824,7cyclictest23-21ksoftirqd/112:02:511
1608020995811,10cyclictest275-21dbus-daemon11:45:011
1608020995725,28cyclictest23-21ksoftirqd/111:46:201
1608020995722,10cyclictest23-21ksoftirqd/111:25:121
1608016995730,22cyclictest12-21ksoftirqd/012:20:010
160802299566,9cyclictest131rcu_preempt09:05:002
160802099563,17cyclictest1643342-21wc10:15:021
1608020995623,9cyclictest23-21ksoftirqd/109:44:241
1608020995618,4cyclictest3467562-21sshd10:27:191
1608020995617,7cyclictest23-21ksoftirqd/111:18:311
1608020995525,27cyclictest23-21ksoftirqd/111:33:561
1608020995523,29cyclictest1645548-21latency_hist10:20:021
1608020995523,28cyclictest1612934-21latency_hist08:45:011
1608020995514,9cyclictest1674371-21sh11:24:381
1608016995545,7cyclictest12-21ksoftirqd/009:37:590
1608016995542,3cyclictest0-21swapper/010:05:020
1608016995533,7cyclictest12-21ksoftirqd/009:26:470
1608016995522,29cyclictest12-21ksoftirqd/012:01:250
160801699551,38cyclictest1624402-21ssh09:32:300
160802099542,19cyclictest1636538-21modprobe10:00:001
160802099542,19cyclictest1636538-21modprobe10:00:001
160802099542,18cyclictest1608517-21cut07:20:001
1608020995420,5cyclictest23-21ksoftirqd/110:04:231
1608020995416,9cyclictest1625515-21wc09:35:011
1608020995416,11cyclictest1-21systemd09:00:011
1608020995416,11cyclictest1-21systemd09:00:011
1608016995439,11cyclictest275-21dbus-daemon09:50:010
1608016995439,11cyclictest275-21dbus-daemon09:50:000
1608016995433,15cyclictest1-21systemd12:25:010
1608016995428,11cyclictest1673682-21ssh11:22:480
1608016995427,12cyclictest1694344-21ssh12:09:070
1608016995427,12cyclictest1694344-21ssh12:09:070
1608016995427,12cyclictest1628374-21ssh09:41:130
1608016995426,12cyclictest1664903-21ssh11:03:180
1608016995425,13cyclictest1667371-21ssh11:08:470
160801699541,38cyclictest1695903-21ssh12:12:230
160801699541,37cyclictest1639158-21ssh10:05:150
1608025995335,13cyclictest3467602-21systemd-journal10:25:013
1608022995345,5cyclictest1652298-21kworker/2:210:37:362
1608022995343,6cyclictest1610334-21kworker/2:110:48:382
160802099533,13cyclictest1613433-21modprobe08:54:591
1608020995320,6cyclictest23-21ksoftirqd/112:35:201
1608020995320,5cyclictest1614230-21diskmemload11:15:021
160802099531,12cyclictest1693696-21ssh12:07:361
160802099531,12cyclictest1693696-21ssh12:07:351
1608016995328,9cyclictest1653440-21ssh10:37:300
1608016995327,11cyclictest1680139-21ssh11:37:160
1608016995326,24cyclictest1707570-21ssh12:38:360
1608016995321,27cyclictest1621122-21wc09:25:040
1608025995235,13cyclictest1-21systemd11:55:013
1608022995244,5cyclictest1663428-21kworker/2:012:38:032
1608022995244,5cyclictest1652298-21kworker/2:212:20:362
1608022995244,5cyclictest1610334-21kworker/2:110:20:572
1608022995244,5cyclictest1610334-21kworker/2:109:46:292
1608022995244,5cyclictest1610334-21kworker/2:109:46:282
1608022995243,5cyclictest1652298-21kworker/2:212:31:532
160802099523,17cyclictest1614253-21unin-run09:10:011
1608020995219,5cyclictest23-21ksoftirqd/107:40:021
1608016995224,13cyclictest1704144-21ssh12:30:460
1608025995138,9cyclictest3467602-21systemd-journal12:35:013
160802299519,6cyclictest131rcu_preempt12:00:422
1608022995144,4cyclictest1610334-21kworker/2:110:25:552
1608022995143,5cyclictest1663428-21kworker/2:012:25:212
1608022995143,5cyclictest1663428-21kworker/2:011:17:102
1608022995143,5cyclictest1652298-21kworker/2:210:53:312
1608022995143,5cyclictest1612180-21kworker/2:309:21:112
1608022995143,5cyclictest1610334-21kworker/2:109:40:482
1608022995142,6cyclictest1652298-21kworker/2:212:05:502
1608022995142,6cyclictest1652298-21kworker/2:212:05:502
1608022995142,6cyclictest1610334-21kworker/2:110:55:412
1608022995142,6cyclictest1610334-21kworker/2:109:39:222
1608022995141,7cyclictest1612180-21kworker/2:309:27:342
1608022995140,7cyclictest30-21ksoftirqd/207:45:012
1608020995126,4cyclictest23-21ksoftirqd/110:50:021
160802099512,17cyclictest1703757-21cat12:30:021
1608020995116,5cyclictest1615161-21rm09:12:001
160801699517,33cyclictest16950irq/224-800000009:55:020
1608016995133,3cyclictest12-21ksoftirqd/011:42:230
1608016995121,14cyclictest1642854-21ssh10:13:560
1608016995120,25cyclictest12-21ksoftirqd/010:26:390
1608022995043,4cyclictest1663428-21kworker/2:011:31:292
1608022995043,4cyclictest1652298-21kworker/2:211:10:172
1608022995042,5cyclictest1663428-21kworker/2:012:17:122
1608022995042,5cyclictest1663428-21kworker/2:011:45:072
1608022995042,5cyclictest1663428-21kworker/2:011:22:432
1608022995042,5cyclictest1663428-21kworker/2:011:02:342
1608022995042,5cyclictest1663428-21kworker/2:011:02:342
1608022995042,5cyclictest1652298-21kworker/2:212:11:512
1608022995042,5cyclictest1652298-21kworker/2:210:40:332
1608022995042,5cyclictest1612180-21kworker/2:309:12:552
1608022995042,5cyclictest1610334-21kworker/2:110:15:192
1608022995042,5cyclictest1610334-21kworker/2:110:12:462
1608022995042,5cyclictest1610334-21kworker/2:109:55:032
1608022995042,5cyclictest1610334-21kworker/2:109:55:022
1608022995041,6cyclictest1663428-21kworker/2:011:53:372
1608022995041,6cyclictest1610334-21kworker/2:108:14:262
1608022995016,9cyclictest30-21ksoftirqd/211:26:002
1608022995011,6cyclictest131rcu_preempt08:40:002
1608020995029,18cyclictest23-21ksoftirqd/112:12:061
1608020995019,4cyclictest1686443-21sh11:51:101
1608016995026,9cyclictest1688970-21ssh11:56:560
1608022994942,4cyclictest1610334-21kworker/2:110:09:252
1608022994941,5cyclictest1652298-21kworker/2:211:06:172
1608022994941,5cyclictest1612180-21kworker/2:309:05:452
1608022994941,5cyclictest1610334-21kworker/2:110:00:502
1608022994941,5cyclictest1610334-21kworker/2:109:50:242
1608022994941,5cyclictest1610334-21kworker/2:108:31:512
1608022994940,6cyclictest1663428-21kworker/2:011:40:182
1608022994940,6cyclictest1663428-21kworker/2:011:38:332
1608022994940,6cyclictest1612180-21kworker/2:309:30:332
160802099491,4cyclictest0-21swapper/110:08:371
1608016994936,8cyclictest12-21ksoftirqd/011:50:180
1608016994922,23cyclictest12-21ksoftirqd/008:00:010
1608016994920,14cyclictest1635103-21ssh09:56:190
1608016994920,14cyclictest1635103-21ssh09:56:180
1608025994838,6cyclictest1-21systemd11:10:003
1608025994832,10cyclictest309-21in:imuxsock08:25:033
1608025994832,10cyclictest309-21in:imuxsock08:25:033
1608025994831,13cyclictest275-21dbus-daemon12:00:013
160802599482,4cyclictest131rcu_preempt09:00:013
160802599482,4cyclictest131rcu_preempt09:00:013
160802599482,3cyclictest131rcu_preempt11:15:023
1608022994840,5cyclictest1610334-21kworker/2:110:30:112
1608022994840,5cyclictest1610334-21kworker/2:108:28:052
1608022994840,5cyclictest1610334-21kworker/2:108:28:052
1608022994840,5cyclictest1610334-21kworker/2:108:01:202
1608022994840,5cyclictest1599875-21kworker/2:207:21:012
1608022994840,5cyclictest1593895-21kworker/2:307:49:162
1608022994840,5cyclictest1593895-21kworker/2:307:49:152
1608022994840,5cyclictest1593895-21kworker/2:307:17:182
1608022994840,5cyclictest1593895-21kworker/2:307:10:212
160802099482,15cyclictest1665829-21latency_hist11:05:031
160802099482,15cyclictest1665829-21latency_hist11:05:031
1608016994826,18cyclictest131rcu_preempt11:35:020
160801699482,10cyclictest1613744-21latency_hist09:00:020
160801699482,10cyclictest1613744-21latency_hist09:00:010
1608016994818,27cyclictest12-21ksoftirqd/012:27:110
1608025994731,12cyclictest3467602-21systemd-journal12:15:003
1608025994730,13cyclictest37-21ksoftirqd/312:05:013
1608022994740,5cyclictest1610334-21kworker/2:108:24:192
1608022994739,5cyclictest1610334-21kworker/2:108:58:162
1608022994739,5cyclictest1610334-21kworker/2:108:58:152
1608022994739,5cyclictest1610334-21kworker/2:108:51:382
1608022994739,5cyclictest1610334-21kworker/2:108:48:252
1608022994739,5cyclictest1610334-21kworker/2:108:05:442
1608022994739,5cyclictest1593895-21kworker/2:307:58:242
1608022994739,5cyclictest1593895-21kworker/2:307:53:522
1608022994739,5cyclictest1593895-21kworker/2:307:30:112
1608022994738,6cyclictest1610334-21kworker/2:108:18:572
1608020994714,7cyclictest1614007-21cut09:05:001
160801699473,15cyclictest1616377-21sh09:15:000
1608016994722,20cyclictest12-21ksoftirqd/007:55:010
1608016994719,12cyclictest1660523-21ssh10:53:280
160801699471,41cyclictest1663113-21sshd10:59:200
1608025994631,11cyclictest3467602-21systemd-journal11:03:323
1608025994631,11cyclictest3467602-21systemd-journal11:03:323
1608022994639,5cyclictest1593895-21kworker/2:307:26:342
1608022994638,5cyclictest1610334-21kworker/2:108:44:212
160802099464,13cyclictest1654453-21munin-run10:40:001
160802099461,18cyclictest1632223-21grep09:50:031
160802099461,18cyclictest1632223-21grep09:50:031
1608016994631,11cyclictest16950irq/224-800000010:43:190
1608016994630,13cyclictest1650475-21ssh10:30:380
1608016994622,20cyclictest16850irq/134-800000010:25:010
1608016994618,23cyclictest12-21ksoftirqd/009:05:020
1608025994528,12cyclictest275-21dbus-daemon07:20:013
160802099452,16cyclictest1610083-21unin-run07:50:011
160802099452,16cyclictest1610083-21unin-run07:50:011
1608016994536,6cyclictest19350irq/34-mmc111:19:200
1608016994528,13cyclictest1618345-21ssh09:19:150
1608016994527,14cyclictest1611578-21sshd08:16:200
1608016994519,22cyclictest12-21ksoftirqd/007:40:020
160801699451,6cyclictest131rcu_preempt08:45:010
1608025994430,10cyclictest275-21dbus-daemon09:20:003
160802599441,3cyclictest131rcu_preempt11:25:023
1608025994413,7cyclictest131rcu_preempt09:10:003
160802099449,13cyclictest3467602-21systemd-journal08:38:201
1608020994417,7cyclictest23-21ksoftirqd/107:43:261
1608016994436,4cyclictest16850irq/134-800000009:10:010
1608016994428,12cyclictest12-21ksoftirqd/008:40:020
1608016994427,14cyclictest1669618-21ssh11:13:500
1608016994426,14cyclictest1657138-21ssh10:45:380
160802599438,5cyclictest131rcu_preempt09:13:273
1608020994312,9cyclictest1608312-21cut07:15:021
160801699431,38cyclictest0-21swapper/008:15:010
1608025994231,8cyclictest37-21ksoftirqd/308:05:013
1608025994228,10cyclictest37-21ksoftirqd/311:00:003
1608025994227,12cyclictest275-21dbus-daemon10:35:013
1608025994227,11cyclictest1-21systemd07:40:013
1608025994222,15cyclictest275-21dbus-daemon11:20:003
160802599421,4cyclictest131rcu_preempt10:45:013
160802599421,3cyclictest131rcu_preempt08:40:013
1608025994212,6cyclictest131rcu_preempt11:40:143
1608020994228,10cyclictest1611632-21run-parts08:20:001
160802099421,4cyclictest131rcu_preempt07:25:021
160802599413,3cyclictest131rcu_preempt08:25:023
1608025994127,11cyclictest309-21in:imuxsock09:35:033
160802599411,6cyclictest1613456-21munin-run08:55:003
160801699412,6cyclictest1608798-21munin-run07:25:010
1608016994125,9cyclictest12-21ksoftirqd/008:55:020
1608025994032,6cyclictest37-21ksoftirqd/309:04:593
1608025994027,10cyclictest275-21dbus-daemon08:15:013
1608025994026,11cyclictest37-21ksoftirqd/309:50:013
1608025994026,11cyclictest37-21ksoftirqd/309:50:013
160802599401,3cyclictest131rcu_preempt10:37:233
1608020994028,9cyclictest23-21ksoftirqd/108:35:001
1608020994020,14cyclictest275-21dbus-daemon08:25:011
160801699404,24cyclictest1609268-21kworker/u8:007:45:020
1608016994032,5cyclictest16950irq/224-800000008:05:000
1608025993925,11cyclictest37-21ksoftirqd/311:28:193
1608025993924,11cyclictest3467602-21systemd-journal09:30:023
1608025993913,6cyclictest131rcu_preempt10:01:213
160802099393,5cyclictest131rcu_preempt09:54:561
1608020993923,12cyclictest275-21dbus-daemon07:35:001
1608016993932,4cyclictest16950irq/224-800000007:15:000
1608016993930,5cyclictest1611888-21cron08:25:000
1608016993922,13cyclictest1613165-21sshd08:48:080
160802599388,5cyclictest131rcu_preempt07:10:033
160802599383,5cyclictest131rcu_preempt09:53:383
1608025993830,5cyclictest1611623-21sh08:19:593
1608025993828,8cyclictest37-21ksoftirqd/307:44:593
1608025993826,9cyclictest37-21ksoftirqd/312:18:423
1608025993826,8cyclictest37-21ksoftirqd/310:46:123
1608025993826,10cyclictest37-21ksoftirqd/308:47:193
1608025993824,10cyclictest275-21dbus-daemon08:10:013
1608022993812,6cyclictest131rcu_preempt07:10:022
1608020993830,5cyclictest1610578-21sh08:00:001
1608020993826,7cyclictest309-21in:imuxsock07:55:001
160802099381,27cyclictest1608462-21kworker/u8:107:30:021
1608016993830,4cyclictest1609269-21sshd07:34:190
1608016993825,8cyclictest12-21ksoftirqd/008:30:020
1608016993825,8cyclictest12-21ksoftirqd/008:30:010
1608016993819,13cyclictest3467602-21systemd-journal08:10:010
160802599376,4cyclictest150-21pr/ttyS208:00:003
160802599374,4cyclictest131rcu_preempt09:24:013
1608025993729,5cyclictest1610067-21latency_hist07:50:033
1608025993729,5cyclictest1610067-21latency_hist07:50:023
160802599372,4cyclictest131rcu_preempt10:52:403
1608025993724,10cyclictest37-21ksoftirqd/312:09:453
1608025993724,10cyclictest37-21ksoftirqd/312:09:453
1608025993722,11cyclictest275-21dbus-daemon12:30:013
160801699371,9cyclictest1612376-21sshd08:32:190
160802599367,5cyclictest131rcu_preempt12:38:053
160802599365,3cyclictest131rcu_preempt10:00:003
160802599365,3cyclictest131rcu_preempt09:59:593
160802599362,9cyclictest311-21rs:main3
1608025993624,9cyclictest37-21ksoftirqd/310:08:243
160802599362,3cyclictest131rcu_preempt11:36:273
1608025993620,12cyclictest275-21dbus-daemon07:30:003
1608020993620,10cyclictest23-21ksoftirqd/108:30:021
1608020993620,10cyclictest23-21ksoftirqd/108:30:011
1608016993626,7cyclictest1609241-21cat07:30:020
160802599355,4cyclictest131rcu_preempt09:33:213
1608025993527,6cyclictest37-21ksoftirqd/309:42:323
1608025993523,9cyclictest37-21ksoftirqd/312:21:023
1608020993524,5cyclictest309-21in:imuxsock08:10:001
1608016993526,6cyclictest1609807-21cron07:45:030
1608016993526,6cyclictest1609807-21cron07:45:020
160802599344,3cyclictest131rcu_preempt07:15:013
1608025993423,9cyclictest37-21ksoftirqd/310:17:503
1608025993421,9cyclictest3467602-21systemd-journal08:43:193
160802599333,4cyclictest131rcu_preempt10:26:463
1608025993324,7cyclictest37-21ksoftirqd/307:35:033
160802599332,3cyclictest131rcu_preempt08:35:013
1608025993323,7cyclictest37-21ksoftirqd/307:55:013
1608020993326,5cyclictest1607960-21latency_hist07:10:011
1608020993323,7cyclictest23-21ksoftirqd/108:15:001
1608020993221,9cyclictest23-21ksoftirqd/108:50:001
160801699327,6cyclictest131rcu_preempt07:10:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional