You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 14:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Apr 13, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32005379910816,13cyclictest131rcu_preempt21:10:032
320053299823,58cyclictest3295354-21ssh00:35:021
3200532996931,10cyclictest3467602-21systemd-journal22:35:001
3200532996733,11cyclictest23-21ksoftirqd/120:15:011
3200532996733,11cyclictest23-21ksoftirqd/120:15:001
3200527996425,29cyclictest12-21ksoftirqd/022:52:190
3200537996243,9cyclictest30-21ksoftirqd/222:35:032
3200532996240,17cyclictest3280392-21ssh00:00:151
3200527996121,3cyclictest131rcu_preempt00:32:420
3200537995935,17cyclictest3467602-21systemd-journal22:40:002
3200537995935,17cyclictest3467602-21systemd-journal22:39:592
320053799586,12cyclictest30-21ksoftirqd/200:00:012
320053799586,12cyclictest30-21ksoftirqd/200:00:002
3200527995824,24cyclictest12-21ksoftirqd/000:40:000
3200527995813,17cyclictest16850irq/134-800000021:42:190
3200541995719,10cyclictest37-21ksoftirqd/323:56:233
3200541995719,10cyclictest37-21ksoftirqd/323:56:223
3200537995748,6cyclictest3262923-21kworker/2:223:38:272
3200527995729,13cyclictest3291959-21ssh00:27:170
3200527995728,14cyclictest3256978-21ssh23:05:500
3200527995727,14cyclictest3246603-21ssh22:42:030
3200527995726,22cyclictest12-21ksoftirqd/022:16:360
3200527995721,12cyclictest12-21ksoftirqd/023:59:590
3200527995721,12cyclictest12-21ksoftirqd/000:00:000
3200527995710,32cyclictest3200521-21cyclictest23:35:010
3200537995648,5cyclictest3211787-21kworker/2:323:12:372
3200537995647,6cyclictest3206228-21kworker/2:121:12:592
320052799562,14cyclictest3275827-21cat23:50:010
3200527995617,18cyclictest16850irq/134-800000023:30:020
3200537995547,5cyclictest3262923-21kworker/2:223:41:122
3200537995547,5cyclictest3262923-21kworker/2:223:34:412
3200537995546,6cyclictest3280070-21kworker/2:000:26:312
3200537995546,6cyclictest3200511-21kworker/2:221:05:012
320052799552,27cyclictest131rcu_preempt23:53:000
3200527995514,20cyclictest16850irq/134-800000022:10:010
320052799551,27cyclictest131rcu_preempt21:54:520
3200537995447,4cyclictest3200511-21kworker/2:221:25:222
3200537995446,5cyclictest3211787-21kworker/2:323:04:182
3200537995446,5cyclictest3211787-21kworker/2:322:17:172
3200537995446,5cyclictest3211787-21kworker/2:322:08:452
3200537995446,5cyclictest3200511-21kworker/2:221:31:332
3200537995445,6cyclictest3280070-21kworker/2:000:34:122
3200537995444,6cyclictest3211787-21kworker/2:322:43:302
3200537995444,6cyclictest3200511-21kworker/2:222:01:162
3200527995446,4cyclictest3289066-21sh00:20:190
3200527995445,6cyclictest12-21ksoftirqd/023:25:030
3200527995420,18cyclictest16850irq/134-800000022:10:470
3200527995419,18cyclictest16850irq/134-800000021:34:210
3200527995418,6cyclictest16850irq/134-800000021:39:550
3200527995418,6cyclictest16850irq/134-800000021:39:550
3200527995418,18cyclictest16850irq/134-800000023:43:430
3200527995418,18cyclictest16850irq/134-800000022:31:100
3200527995418,18cyclictest16850irq/134-800000021:19:070
3200527995416,19cyclictest16850irq/134-800000022:35:080
3200527995416,19cyclictest16850irq/134-800000022:35:070
320052799541,37cyclictest3258786-21ssh23:10:050
3200537995345,5cyclictest3211787-21kworker/2:322:50:232
3200537995344,6cyclictest3280070-21kworker/2:000:21:022
3200537995344,6cyclictest3190569-21kworker/2:120:15:192
3200537995343,7cyclictest3262923-21kworker/2:223:49:492
3200527995327,23cyclictest3253205-21sshd22:57:210
3200527995327,10cyclictest3284127-21ssh00:09:190
3200527995325,13cyclictest3262039-21ssh23:17:570
3200527995319,16cyclictest16850irq/134-800000022:22:000
3200527995318,18cyclictest16850irq/134-800000021:10:200
3200527995317,6cyclictest16850irq/134-800000023:02:000
3200527995317,6cyclictest16850irq/134-800000022:02:430
3200527995317,6cyclictest16850irq/134-800000021:45:200
3200527995317,6cyclictest16850irq/134-800000021:45:200
3200527995317,18cyclictest16850irq/134-800000021:26:320
3200527995317,17cyclictest16850irq/134-800000022:45:300
3200527995316,19cyclictest16850irq/134-800000023:36:240
3200527995316,18cyclictest16850irq/134-800000000:16:440
320054199522,3cyclictest131rcu_preempt23:34:583
3200537995244,5cyclictest3280070-21kworker/2:000:09:092
3200537995244,5cyclictest3211787-21kworker/2:322:46:492
3200537995244,5cyclictest3211787-21kworker/2:322:14:042
3200537995244,5cyclictest3200511-21kworker/2:221:48:232
3200537995244,5cyclictest3200511-21kworker/2:221:48:222
3200537995243,6cyclictest3211787-21kworker/2:322:29:362
3200537995243,6cyclictest3200511-21kworker/2:221:53:192
3200537995243,5cyclictest3200511-21kworker/2:221:58:492
3200537995242,6cyclictest3280070-21kworker/2:000:35:582
3200537995235,12cyclictest3254245-21cron23:00:002
3200537995228,16cyclictest275-21dbus-daemon21:21:152
3200527995218,17cyclictest16850irq/134-800000021:59:000
3200527995218,17cyclictest16850irq/134-800000000:14:510
3200527995218,17cyclictest16850irq/134-800000000:03:090
3200527995217,18cyclictest16850irq/134-800000021:22:230
3200537995144,4cyclictest3200511-21kworker/2:221:43:142
3200537995143,5cyclictest3262923-21kworker/2:223:27:442
3200537995143,5cyclictest3243451-21kworker/2:123:19:562
3200537995143,5cyclictest3211787-21kworker/2:322:24:162
3200537995143,5cyclictest3200511-21kworker/2:221:18:172
3200537995142,6cyclictest3280070-21kworker/2:000:12:482
3200527995121,3cyclictest131rcu_preempt22:25:460
3200537995042,5cyclictest3262923-21kworker/2:223:51:102
3200537995042,5cyclictest3262923-21kworker/2:223:20:572
3200537995041,6cyclictest3211787-21kworker/2:323:08:412
3200537995041,5cyclictest3280070-21kworker/2:000:15:082
3200537995041,5cyclictest3243451-21kworker/2:100:01:072
3200537995035,7cyclictest30-21ksoftirqd/220:35:002
3200541994934,12cyclictest1-21systemd20:20:003
320054199492,4cyclictest131rcu_preempt23:41:003
3200537994941,5cyclictest3200511-21kworker/2:221:37:242
3200537994941,5cyclictest3200511-21kworker/2:221:37:242
3200537994940,5cyclictest3190569-21kworker/2:120:21:032
3200537994923,18cyclictest275-21dbus-daemon19:25:022
3200537994923,18cyclictest275-21dbus-daemon19:25:012
3200532994936,10cyclictest3202791-21cron19:55:011
3200532994936,10cyclictest3202791-21cron19:55:001
320054199489,10cyclictest0-21swapper/321:11:073
320054199484,3cyclictest131rcu_preempt22:30:013
320054199482,4cyclictest131rcu_preempt00:28:343
3200537994840,5cyclictest3190569-21kworker/2:119:51:202
3200537994840,5cyclictest3190569-21kworker/2:119:51:192
3200537994840,5cyclictest3190569-21kworker/2:119:35:012
3200537994839,5cyclictest3190569-21kworker/2:119:40:022
3200537994839,5cyclictest3190569-21kworker/2:119:40:012
3200532994832,12cyclictest3265034-21pluginstate23:25:011
320054199474,3cyclictest131rcu_preempt23:16:173
320054199474,3cyclictest131rcu_preempt22:02:293
3200541994734,9cyclictest3467602-21systemd-journal23:48:183
320054199471,42cyclictest3467602-21systemd-journal00:10:033
3200537994739,5cyclictest3190569-21kworker/2:120:00:012
3200537994739,5cyclictest3190569-21kworker/2:119:18:132
3200537994739,5cyclictest3190569-21kworker/2:119:11:452
320052799472,16cyclictest3200969-21cron19:20:000
3200541994635,7cyclictest3247804-21munin-plugin-st22:45:003
3200541994627,14cyclictest275-21dbus-daemon20:55:013
320054199462,3cyclictest131rcu_preempt23:22:313
320054199462,11cyclictest0-21swapper/321:29:323
320054199462,10cyclictest364-21snmpd23:51:003
3200537994639,4cyclictest3200511-21kworker/2:221:00:002
3200537994638,5cyclictest3200511-21kworker/2:220:36:082
3200537994638,5cyclictest3200511-21kworker/2:220:36:072
3200537994638,5cyclictest3190569-21kworker/2:119:45:172
3200537994628,13cyclictest309-21in:imuxsock20:50:032
3200527994638,4cyclictest16950irq/224-800000021:10:020
3200527994631,11cyclictest3467602-21systemd-journal20:42:190
3200527994617,24cyclictest1-21systemd20:20:020
320054199453,3cyclictest131rcu_preempt22:34:453
3200541994530,11cyclictest275-21dbus-daemon21:25:023
3200537994537,5cyclictest3190569-21kworker/2:120:14:422
3200537994537,5cyclictest3190569-21kworker/2:120:14:422
3200537994525,14cyclictest1-21systemd20:45:002
3200532994536,6cyclictest23-21ksoftirqd/123:32:311
3200532994532,10cyclictest3277613-21sshd23:54:171
320052799455,3cyclictest131rcu_preempt20:25:000
3200527994532,8cyclictest3203587-21munin-plugin-st20:10:000
320054199444,4cyclictest131rcu_preempt22:58:293
3200541994431,9cyclictest3200937-21sshd19:15:193
3200541994430,11cyclictest37-21ksoftirqd/300:00:153
3200541994426,14cyclictest3467602-21systemd-journal20:35:013
3200541994426,12cyclictest275-21dbus-daemon20:45:013
320054199441,3cyclictest131rcu_preempt21:59:033
320054199441,29cyclictest3260880-21latency_hist23:15:033
3200541994411,6cyclictest131rcu_preempt22:24:183
3200537994436,5cyclictest3200511-21kworker/2:220:47:122
3200532994429,11cyclictest275-21dbus-daemon21:10:011
3200532994418,18cyclictest0-21swapper/121:18:081
320053299441,3cyclictest131rcu_preempt22:10:031
320053299441,3cyclictest131rcu_preempt21:50:021
320053299441,3cyclictest131rcu_preempt21:50:021
3200527994433,6cyclictest3201520-21grep19:29:590
3200527994424,14cyclictest275-21dbus-daemon20:35:010
320054199431,3cyclictest131rcu_preempt20:09:593
320054199431,13cyclictest3293477-21ssh00:30:393
320054199431,12cyclictest3265267-21ssh23:25:063
3200537994335,5cyclictest3190569-21kworker/2:119:42:462
320053299434,3cyclictest131rcu_preempt21:24:591
3200532994331,8cyclictest3467602-21systemd-journal00:27:311
3200532994327,12cyclictest23-21ksoftirqd/121:40:031
3200532994326,13cyclictest3467602-21systemd-journal00:06:181
320053299431,38cyclictest23-21ksoftirqd/122:11:421
320052799434,4cyclictest131rcu_preempt19:43:180
320052799434,29cyclictest3200529-21cat19:10:020
320052799434,29cyclictest3200529-21cat19:10:010
3200527994325,14cyclictest275-21dbus-daemon20:30:000
320054199423,3cyclictest131rcu_preempt00:37:393
3200541994233,5cyclictest3201505-21cron19:29:593
3200541994230,9cyclictest3467602-21systemd-journal22:37:173
3200541994230,9cyclictest3467602-21systemd-journal22:37:163
3200541994230,9cyclictest275-21dbus-daemon23:40:013
3200541994229,10cyclictest37-21ksoftirqd/322:10:173
3200541994229,10cyclictest3206672-21sshd21:06:203
3200541994227,11cyclictest37-21ksoftirqd/322:50:013
3200541994226,11cyclictest3467602-21systemd-journal21:55:013
3200541994224,6cyclictest37-21ksoftirqd/321:18:293
320054199421,3cyclictest131rcu_preempt22:54:253
3200537994235,5cyclictest3190569-21kworker/2:120:28:282
3200537994234,5cyclictest3190569-21kworker/2:120:05:022
3200537994234,5cyclictest3190569-21kworker/2:120:05:022
320053299426,6cyclictest131rcu_preempt23:27:511
320053299424,3cyclictest131rcu_preempt22:40:021
320053299424,3cyclictest131rcu_preempt22:40:011
3200532994229,10cyclictest0-21swapper/122:02:571
3200532994228,10cyclictest23-21ksoftirqd/123:37:001
3200532994228,10cyclictest1-21systemd20:55:011
3200532994223,14cyclictest3467602-21systemd-journal23:19:591
320053299421,3cyclictest131rcu_preempt22:59:071
320053299421,3cyclictest131rcu_preempt21:55:401
320053299421,3cyclictest131rcu_preempt20:40:011
320053299421,3cyclictest131rcu_preempt20:40:001
320054199413,3cyclictest131rcu_preempt21:45:473
320054199413,3cyclictest131rcu_preempt21:45:463
3200541994128,10cyclictest275-21dbus-daemon21:05:003
3200541994127,11cyclictest37-21ksoftirqd/319:15:003
320054199412,3cyclictest131rcu_preempt22:16:063
320054199412,3cyclictest131rcu_preempt00:10:023
320053299414,4cyclictest131rcu_preempt19:50:011
3200532994131,7cyclictest23-21ksoftirqd/100:15:181
3200532994127,10cyclictest23-21ksoftirqd/122:30:001
3200532994126,11cyclictest3467602-21systemd-journal19:37:201
3200532994126,11cyclictest3467602-21systemd-journal19:37:191
320053299412,3cyclictest131rcu_preempt23:13:381
320053299411,3cyclictest131rcu_preempt21:44:531
320053299411,3cyclictest131rcu_preempt21:00:021
320054199408,6cyclictest131rcu_preempt00:23:393
320054199403,3cyclictest131rcu_preempt23:06:363
320054199402,4cyclictest131rcu_preempt22:06:293
320054199401,3cyclictest131rcu_preempt19:25:023
320054199401,3cyclictest131rcu_preempt19:25:013
320053299409,6cyclictest131rcu_preempt22:51:371
320053299404,3cyclictest131rcu_preempt19:20:011
320053299403,3cyclictest131rcu_preempt19:45:011
3200532994028,10cyclictest23-21ksoftirqd/122:20:011
3200532994027,9cyclictest23-21ksoftirqd/123:00:501
3200532994026,10cyclictest3467602-21systemd-journal20:26:561
320053299402,5cyclictest131rcu_preempt22:45:451
3200532994025,9cyclictest3467602-21systemd-journal21:05:011
3200532994017,8cyclictest23-21ksoftirqd/123:07:031
3200532994017,10cyclictest23-21ksoftirqd/123:45:171
320053299401,13cyclictest3278265-21ssh23:55:281
320053299401,13cyclictest3278265-21ssh23:55:271
3200527994028,8cyclictest12-21ksoftirqd/021:05:010
3200527994027,10cyclictest3201997-21sshd19:38:200
3200527994027,10cyclictest3201997-21sshd19:38:190
3200527994024,11cyclictest364-21snmpd20:11:020
3200527994024,11cyclictest364-21snmpd20:11:010
320052799401,35cyclictest3205105-21sshd20:36:200
320052799401,35cyclictest3205105-21sshd20:36:190
320054199393,6cyclictest3287546-21ssh00:17:033
3200541993931,5cyclictest1-21systemd19:40:013
3200541993931,5cyclictest1-21systemd19:40:013
320054199391,3cyclictest131rcu_preempt21:30:343
320054199391,3cyclictest131rcu_preempt19:55:023
320054199391,3cyclictest131rcu_preempt19:55:013
3200537993932,5cyclictest3190569-21kworker/2:120:08:552
3200537993923,7cyclictest0-21swapper/219:10:032
3200537993923,7cyclictest0-21swapper/219:10:022
3200537993915,6cyclictest131rcu_preempt19:30:012
320053299393,3cyclictest131rcu_preempt19:25:021
320053299393,3cyclictest131rcu_preempt19:25:011
3200532993929,7cyclictest23-21ksoftirqd/100:10:511
3200532993928,8cyclictest23-21ksoftirqd/119:13:191
320053299392,4cyclictest23-21ksoftirqd/121:50:161
320053299391,4cyclictest131rcu_preempt00:25:021
320053299391,3cyclictest131rcu_preempt23:40:271
3200527993928,7cyclictest12-21ksoftirqd/019:15:020
320052799392,6cyclictest3203324-21cron20:05:010
320052799392,6cyclictest3203324-21cron20:05:000
3200527993925,10cyclictest3467718-21lldpd19:46:200
3200527993920,15cyclictest3201743-21sshd19:32:190
3200541993830,5cyclictest3218668-21sh21:37:183
3200541993830,5cyclictest3218668-21sh21:37:173
320054199382,4cyclictest131rcu_preempt20:40:003
320054199382,4cyclictest131rcu_preempt20:40:003
320053299384,3cyclictest131rcu_preempt20:19:591
320053299384,3cyclictest131rcu_preempt00:40:011
3200532993825,9cyclictest3217820-21ssh21:35:071
3200532993825,8cyclictest23-21ksoftirqd/120:50:021
320053299381,4cyclictest131rcu_preempt19:59:591
320052799383,6cyclictest3467602-21systemd-journal20:55:220
320052799383,3cyclictest131rcu_preempt19:59:590
3200541993724,11cyclictest37-21ksoftirqd/323:01:523
320054199371,4cyclictest131rcu_preempt20:15:013
320054199371,4cyclictest131rcu_preempt20:15:003
320054199371,3cyclictest131rcu_preempt21:40:113
320054199371,3cyclictest131rcu_preempt20:00:013
320053299374,3cyclictest131rcu_preempt22:24:161
320053299373,3cyclictest131rcu_preempt20:10:001
320053299372,3cyclictest131rcu_preempt22:43:481
320053299372,3cyclictest131rcu_preempt21:31:211
320053299372,3cyclictest131rcu_preempt21:25:251
320053299372,3cyclictest131rcu_preempt21:10:421
3200532993723,11cyclictest3467602-21systemd-journal20:32:011
3200532993723,10cyclictest3467602-21systemd-journal20:25:011
320054199363,9cyclictest37-21ksoftirqd/319:35:023
3200541993623,10cyclictest37-21ksoftirqd/319:50:003
320053299363,3cyclictest131rcu_preempt19:30:011
320052799362,6cyclictest3205885-21sshd20:50:190
3200527993617,15cyclictest3202825-21latency_hist19:55:020
3200527993617,15cyclictest3202825-21latency_hist19:55:010
320054199358,3cyclictest131rcu_preempt20:50:003
320053299352,3cyclictest131rcu_preempt20:41:091
320052799351,30cyclictest0-21swapper/019:23:200
320052799351,30cyclictest0-21swapper/019:23:190
320054199342,8cyclictest37-21ksoftirqd/319:10:033
320054199342,8cyclictest37-21ksoftirqd/319:10:023
3200532993425,7cyclictest23-21ksoftirqd/119:10:031
3200532993425,7cyclictest23-21ksoftirqd/119:10:021
320052799341,29cyclictest0-21swapper/020:46:560
320054199335,3cyclictest131rcu_preempt19:45:013
320054199333,4cyclictest131rcu_preempt21:00:013
3200541993324,6cyclictest3203435-21latency_hist20:05:033
3200541993324,6cyclictest3203435-21latency_hist20:05:033
3200532993318,11cyclictest3467602-21systemd-journal20:05:011
3200532993318,11cyclictest3467602-21systemd-journal20:05:001
320053299331,3cyclictest131rcu_preempt19:35:011
320054199322,3cyclictest131rcu_preempt20:25:013
320054199314,3cyclictest131rcu_preempt20:26:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional