You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 22:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri May 22, 2026 12:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
319152699921,46cyclictest0-21swapper/011:38:190
319152699892,18cyclictest3292732-21ssh12:26:420
319152699892,18cyclictest3292732-21ssh12:26:410
3191526998328,51cyclictest3289324-21ssh12:19:580
3191526998328,51cyclictest3289324-21ssh12:19:570
319152699831,77cyclictest3200626-21ssh09:15:350
319152999824,4cyclictest131rcu_preempt09:40:031
3191526998210,45cyclictest19350irq/34-mmc112:35:020
319152999804,5cyclictest131rcu_preempt10:00:021
319152699793,45cyclictest364-21snmpd09:13:380
3191526997932,41cyclictest275-21dbus-daemon09:10:020
319152699791,74cyclictest3245759-21ssh10:49:300
319152699781,73cyclictest48-21kcompactd011:20:240
319152699781,5cyclictest3195632-21munin-run08:30:000
3191526997718,54cyclictest16850irq/134-800000011:14:230
319152699762,6cyclictest3224771-21ssh10:05:350
3191526997625,47cyclictest3249377-21ssh10:56:450
3191526997614,56cyclictest16850irq/134-800000011:30:130
3191526997541,30cyclictest16950irq/224-800000011:25:030
319152699753,67cyclictest3230402-21ssh10:17:270
3191526997524,29cyclictest3197756-21diskmemload12:35:050
319152999745,7cyclictest131rcu_preempt12:35:011
3191526997431,26cyclictest3282002-21ssh12:04:430
319152699742,55cyclictest3217460-21ssh09:50:300
319152699742,55cyclictest3217460-21ssh09:50:290
3191526997419,45cyclictest16850irq/134-800000009:55:310
3191526997352,18cyclictest3213873-21ssh09:43:250
3191526997327,42cyclictest3251492-21ssh11:01:120
319152699731,6cyclictest3203217-21ssh09:21:000
3191526997314,54cyclictest3207174-21ssh09:29:350
3191526997314,54cyclictest3207174-21ssh09:29:350
319152699731,17cyclictest3286111-21ssh12:13:090
319152699722,65cyclictest0-21swapper/011:50:010
3191526997223,44cyclictest3239160-21ssh10:35:270
319152699721,2cyclictest0-21swapper/010:53:100
319152699721,2cyclictest0-21swapper/010:53:090
3191526997134,34cyclictest16950irq/224-800000011:15:140
3191526997128,26cyclictest3242998-21ssh10:43:470
319152699712,16cyclictest3253834-21ssh11:05:580
319152699703,6cyclictest1-21systemd07:22:060
3191526997031,35cyclictest3270712-21ssh11:40:590
319152699702,10cyclictest48-21kcompactd010:25:170
3191526997017,22cyclictest3197756-21diskmemload10:12:320
3191529996923,40cyclictest3467602-21systemd-journal07:20:001
3191529996923,40cyclictest3467602-21systemd-journal07:20:001
319152699692,18cyclictest3210651-21ssh09:36:340
3191526996830,35cyclictest3207834-21ssh09:30:390
319152699681,4cyclictest3237470-21ssh10:32:010
319152699681,4cyclictest3237470-21ssh10:32:010
319152699681,4cyclictest3222562-21sshd10:01:070
3191537996756,7cyclictest3191540-21sort07:10:012
3191529996724,37cyclictest275-21dbus-daemon09:55:011
3191529996724,37cyclictest275-21dbus-daemon09:55:001
319152699671,2cyclictest0-21swapper/009:50:030
3191529996630,31cyclictest3194059-21cat08:00:001
3191529996623,5cyclictest3276457-21sh11:53:081
319152699661,19cyclictest3278639-21ssh11:57:340
319152699653,57cyclictest0-21swapper/012:08:050
319152699652,58cyclictest3193315-21munin-run07:45:010
319152699652,57cyclictest3195885-21munin-run08:35:010
319152699651,22cyclictest3233849-21ssh10:24:450
319152699651,22cyclictest3233849-21ssh10:24:450
3191529996426,35cyclictest3208559-21sh09:32:101
3191529996426,34cyclictest23-21ksoftirqd/111:24:421
3191529996425,36cyclictest3207564-21latency_hist09:30:031
3191529996425,36cyclictest3207564-21latency_hist09:30:021
3191529996425,35cyclictest3196664-21latency_hist08:50:011
319152699648,52cyclictest3276596-21ssh11:53:250
319152699642,14cyclictest3195140-21munin-run08:20:020
319152999633,27cyclictest3196145-21munin-run08:40:011
3191529996326,33cyclictest23-21ksoftirqd/111:15:061
3191529996325,34cyclictest275-21dbus-daemon10:39:591
3191529996320,7cyclictest3194726-21latency_hist08:10:021
3191529996319,8cyclictest3217098-21latency_hist09:50:021
319152699631,7cyclictest3291781-21latency_hist12:25:020
319152999623,13cyclictest3467602-21systemd-journal09:05:001
3191529996230,27cyclictest3238754-21sh10:35:021
3191529996230,27cyclictest3238754-21sh10:35:011
319152999622,17cyclictest3467602-21systemd-journal12:20:001
319152999622,17cyclictest3467602-21systemd-journal12:20:001
3191541996112,7cyclictest131rcu_preempt12:10:003
3191529996117,11cyclictest23-21ksoftirqd/112:00:001
319152699611,7cyclictest3191966-21sshd07:19:070
319152699611,7cyclictest3191966-21sshd07:19:060
319152699611,2cyclictest0-21swapper/008:56:060
319152699611,14cyclictest1-21systemd07:27:060
319152999602,21cyclictest3193555-21munin-run07:49:591
3191529996017,5cyclictest3255553-21cron11:10:001
3191529995941,13cyclictest1-21systemd11:00:001
3191541995825,11cyclictest37-21ksoftirqd/311:39:283
319152999586,5cyclictest131rcu_preempt12:39:011
319152999583,16cyclictest3257952-21unin-run11:15:001
319152699581,4cyclictest3196668-21munin-run08:50:010
3191541995711,10cyclictest131rcu_preempt12:35:023
3191537995713,6cyclictest131rcu_preempt07:40:012
3191529995738,14cyclictest3467602-21systemd-journal10:45:011
3191529995725,27cyclictest3253102-21sh11:04:581
3191529995723,5cyclictest3202896-21sh09:20:191
3191529995718,7cyclictest23-21ksoftirqd/109:44:031
319152699571,51cyclictest364-21snmpd08:11:090
319152699571,51cyclictest364-21snmpd08:11:090
319152699571,51cyclictest3196882-21sshd08:50:070
319152699571,51cyclictest3196882-21sshd08:50:060
3191537995610,5cyclictest131rcu_preempt12:05:012
319152999565,17cyclictest3191536-21latency_hist07:10:021
3191529995622,5cyclictest3224935-21sh10:06:021
3191529995622,4cyclictest3195882-21latency_hist08:35:011
319152999562,20cyclictest3193864-21grep07:55:011
3191529995619,5cyclictest3232473-21sh10:21:431
3191529995619,5cyclictest3232473-21sh10:21:421
319152699562,6cyclictest364-21snmpd08:37:070
319152699562,48cyclictest3192755-21sshd07:34:150
319152699562,47cyclictest3194681-21latency_hist08:10:020
319152699562,25cyclictest3191599-21latency_hist07:10:010
319152699561,2cyclictest0-21swapper/007:45:150
319153799554,7cyclictest30-21ksoftirqd/209:10:032
3191537995547,5cyclictest3193328-21kworker/2:008:34:072
3191537995541,7cyclictest3207447-21kworker/2:210:38:472
3191537995541,6cyclictest3272526-21kworker/2:311:57:212
3191537995513,5cyclictest131rcu_preempt09:54:432
3191537995513,5cyclictest131rcu_preempt09:54:432
3191537995510,5cyclictest131rcu_preempt10:09:022
319152999557,9cyclictest3270070-21modprobe11:40:001
319152999552,21cyclictest3193031-21munin-plugin-st07:40:001
319152999552,15cyclictest3229209-21cat10:15:021
319152999551,4cyclictest131rcu_preempt10:51:291
319152999551,4cyclictest131rcu_preempt10:51:281
319152699552,48cyclictest364-21snmpd09:03:470
319154199544,10cyclictest37-21ksoftirqd/309:57:313
3191541995413,6cyclictest131rcu_preempt10:39:593
3191541995410,6cyclictest131rcu_preempt11:41:453
3191541995410,5cyclictest131rcu_preempt10:48:053
319153799549,6cyclictest131rcu_preempt09:40:002
3191537995446,5cyclictest3207447-21kworker/2:209:40:512
3191537995446,5cyclictest3193328-21kworker/2:011:40:312
3191537995421,9cyclictest30-21ksoftirqd/210:54:212
3191537995421,9cyclictest30-21ksoftirqd/210:54:212
3191537995410,5cyclictest131rcu_preempt09:11:502
319152999543,4cyclictest131rcu_preempt11:26:061
3191529995419,5cyclictest3192873-21latency_hist07:35:031
319152699542,45cyclictest3195390-21unin-run08:25:010
319152699542,18cyclictest3193049-21sendmail-msp07:40:000
3191541995325,11cyclictest37-21ksoftirqd/311:47:023
319154199531,46cyclictest1-21systemd08:20:013
3191541995313,6cyclictest131rcu_preempt11:03:403
3191541995313,5cyclictest131rcu_preempt08:10:003
3191541995310,6cyclictest131rcu_preempt11:20:003
319153799538,5cyclictest131rcu_preempt10:13:072
319153799534,6cyclictest131rcu_preempt08:10:002
3191537995343,6cyclictest3193328-21kworker/2:010:20:022
3191537995343,6cyclictest3193328-21kworker/2:010:20:022
3191537995311,6cyclictest131rcu_preempt10:05:012
3191537995310,6cyclictest131rcu_preempt10:32:442
3191537995310,6cyclictest131rcu_preempt10:32:432
3191529995321,7cyclictest23-21ksoftirqd/109:57:331
3191529995320,5cyclictest3290037-21sh12:20:561
319152699531,47cyclictest364-21snmpd07:59:170
319154199524,7cyclictest37-21ksoftirqd/310:44:313
319154199523,7cyclictest37-21ksoftirqd/310:15:003
3191541995218,5cyclictest3467718-21lldpd11:51:543
3191541995212,6cyclictest131rcu_preempt10:05:003
319153799529,5cyclictest131rcu_preempt12:40:002
3191537995244,5cyclictest3246014-21kworker/2:311:12:422
3191537995244,5cyclictest3195405-21kworker/2:309:04:102
3191537995244,5cyclictest3193328-21kworker/2:011:33:522
3191537995211,6cyclictest131rcu_preempt12:06:112
3191537995210,5cyclictest131rcu_preempt10:18:172
319152999523,13cyclictest3231467-21cron10:19:591
3191529995217,4cyclictest3284464-21sh12:09:511
319152699522,46cyclictest364-21snmpd08:03:500
319152699522,44cyclictest3196414-21munin-plugin-st08:44:590
319152699521,6cyclictest364-21snmpd07:53:240
319154199517,10cyclictest37-21ksoftirqd/309:40:003
319154199515,6cyclictest131rcu_preempt10:55:013
319154199515,6cyclictest131rcu_preempt10:55:003
3191541995140,9cyclictest37-21ksoftirqd/310:10:033
3191541995131,14cyclictest3467602-21systemd-journal09:44:593
3191541995124,18cyclictest37-21ksoftirqd/312:19:263
3191541995124,18cyclictest37-21ksoftirqd/312:19:253
3191541995113,6cyclictest131rcu_preempt09:00:003
3191541995112,5cyclictest131rcu_preempt11:10:013
3191537995144,4cyclictest3193328-21kworker/2:010:47:142
3191537995143,5cyclictest3284789-21kworker/2:112:34:522
3191537995143,5cyclictest3193328-21kworker/2:010:55:232
3191537995143,5cyclictest3193328-21kworker/2:009:31:212
3191537995142,6cyclictest3246014-21kworker/2:311:17:032
3191537995135,12cyclictest275-21dbus-daemon09:25:002
3191529995120,7cyclictest3197172-21sh09:00:001
3191529995118,5cyclictest3271849-21sh11:43:351
3191529995114,5cyclictest0-21swapper/110:30:021
3191526995125,22cyclictest12-21ksoftirqd/007:15:020
3191541995032,14cyclictest275-21dbus-daemon08:30:003
319154199502,8cyclictest131rcu_preempt11:30:013
3191541995013,6cyclictest131rcu_preempt10:35:023
3191541995013,6cyclictest131rcu_preempt10:35:013
319153799507,8cyclictest30-21ksoftirqd/211:03:092
3191537995042,5cyclictest3284789-21kworker/2:112:17:482
3191537995042,5cyclictest3284789-21kworker/2:112:17:472
3191537995042,5cyclictest3257964-21kworker/2:111:21:562
3191537995042,5cyclictest3207447-21kworker/2:210:42:102
3191537995042,5cyclictest3193328-21kworker/2:012:10:372
3191537995042,5cyclictest3193328-21kworker/2:011:36:042
3191537995042,5cyclictest3193328-21kworker/2:011:05:302
3191537995042,5cyclictest3193328-21kworker/2:010:26:442
3191537995042,5cyclictest3193328-21kworker/2:009:58:452
3191537995042,5cyclictest3193328-21kworker/2:008:54:542
3191537995042,5cyclictest3193328-21kworker/2:008:54:532
3191537995041,6cyclictest3193328-21kworker/2:011:51:462
3191537995040,5cyclictest3180936-21kworker/2:107:10:522
3191537995012,7cyclictest30-21ksoftirqd/211:50:022
3191537995012,5cyclictest131rcu_preempt12:30:012
3191537995012,5cyclictest131rcu_preempt12:30:002
3191537995010,6cyclictest131rcu_preempt09:49:162
3191537995010,6cyclictest131rcu_preempt09:25:442
3191537995010,6cyclictest131rcu_preempt09:25:432
319152999503,12cyclictest3245990-21ssh10:50:001
3191529995018,7cyclictest23-21ksoftirqd/108:55:031
3191529995018,7cyclictest23-21ksoftirqd/108:55:021
319154199499,6cyclictest131rcu_preempt10:25:153
3191541994937,9cyclictest3194801-21sshd08:10:073
3191541994937,9cyclictest3194801-21sshd08:10:063
3191541994918,9cyclictest131rcu_preempt11:12:503
3191541994911,5cyclictest131rcu_preempt08:44:593
3191537994942,4cyclictest3193328-21kworker/2:012:21:022
3191537994941,5cyclictest3193328-21kworker/2:011:25:372
3191537994913,6cyclictest131rcu_preempt09:16:122
3191529994923,18cyclictest275-21dbus-daemon08:45:001
319152999492,20cyclictest3467602-21systemd-journal11:49:321
3191529994921,5cyclictest3197756-21diskmemload12:10:211
3191529994913,7cyclictest3195630-21munin-plugin-st08:29:591
319154199489,6cyclictest131rcu_preempt09:10:013
319154199488,6cyclictest131rcu_preempt10:59:593
319154199487,7cyclictest37-21ksoftirqd/312:30:023
319154199487,7cyclictest37-21ksoftirqd/312:30:013
319154199487,6cyclictest131rcu_preempt10:19:593
319154199483,7cyclictest37-21ksoftirqd/309:20:033
3191541994833,11cyclictest3467602-21systemd-journal09:27:063
3191541994833,11cyclictest3467602-21systemd-journal09:27:063
3191541994822,18cyclictest0-21swapper/308:00:023
3191541994812,6cyclictest131rcu_preempt09:19:213
319153799484,8cyclictest30-21ksoftirqd/207:25:002
3191537994840,5cyclictest3193328-21kworker/2:008:25:172
3191537994840,5cyclictest3193328-21kworker/2:008:11:222
3191537994840,5cyclictest3193328-21kworker/2:008:11:212
3191537994840,5cyclictest3181723-21kworker/2:307:42:502
3191537994840,5cyclictest3180936-21kworker/2:107:27:082
3191529994822,8cyclictest23-21ksoftirqd/107:42:581
319154199479,6cyclictest131rcu_preempt12:40:033
319154199479,5cyclictest131rcu_preempt12:25:023
319154199479,4cyclictest131rcu_preempt12:11:523
319154199478,7cyclictest131rcu_preempt10:22:473
319154199478,7cyclictest131rcu_preempt10:22:463
319154199474,6cyclictest131rcu_preempt09:15:033
3191541994712,8cyclictest131rcu_preempt12:04:483
3191541994710,6cyclictest131rcu_preempt09:50:033
319153799476,7cyclictest131rcu_preempt08:40:002
3191537994739,5cyclictest3193328-21kworker/2:008:46:462
3191537994739,5cyclictest3193328-21kworker/2:008:41:072
3191537994739,5cyclictest3193328-21kworker/2:008:23:002
3191537994739,5cyclictest3193328-21kworker/2:008:15:172
3191537994739,5cyclictest3193328-21kworker/2:008:04:362
3191537994739,5cyclictest3181723-21kworker/2:307:57:292
3191537994739,5cyclictest3181723-21kworker/2:307:48:462
3191537994739,5cyclictest3180936-21kworker/2:107:17:182
3191537994739,5cyclictest3180936-21kworker/2:107:17:172
3191537994738,5cyclictest3180936-21kworker/2:107:30:132
319154199466,6cyclictest131rcu_preempt07:15:063
319154199466,6cyclictest131rcu_preempt07:15:053
319154199465,6cyclictest131rcu_preempt12:00:003
319154199464,6cyclictest131rcu_preempt11:22:383
3191541994613,6cyclictest131rcu_preempt09:35:013
3191537994639,4cyclictest3181723-21kworker/2:307:51:072
3191537994638,5cyclictest3193328-21kworker/2:008:55:122
319152999468,8cyclictest0-21swapper/111:35:021
3191529994620,8cyclictest23-21ksoftirqd/112:25:391
3191529994620,8cyclictest23-21ksoftirqd/112:25:391
3191529994619,7cyclictest23-21ksoftirqd/109:10:561
319154199457,4cyclictest131rcu_preempt07:35:023
319154199455,5cyclictest131rcu_preempt08:40:003
3191529994533,7cyclictest3202495-21cron09:20:001
319154199449,6cyclictest131rcu_preempt07:35:023
3191541994436,5cyclictest37-21ksoftirqd/311:33:063
3191541994431,9cyclictest1-21systemd08:55:013
3191541994431,9cyclictest1-21systemd08:55:013
3191529994426,12cyclictest1-21systemd07:15:011
319154199426,5cyclictest131rcu_preempt07:25:013
319154199425,7cyclictest37-21ksoftirqd/307:46:063
319154199423,5cyclictest131rcu_preempt09:52:123
319154199423,5cyclictest131rcu_preempt09:52:123
3191541994231,8cyclictest37-21ksoftirqd/309:04:063
3191541994229,10cyclictest3191710-21sshd07:14:143
3191541994212,15cyclictest131rcu_preempt07:43:403
3191541994128,9cyclictest3467602-21systemd-journal07:55:003
3191541994021,13cyclictest275-21dbus-daemon08:25:023
319154199393,8cyclictest37-21ksoftirqd/308:00:433
3191541993921,15cyclictest37-21ksoftirqd/307:25:153
3191529993925,11cyclictest23-21ksoftirqd/112:03:001
3191529993921,13cyclictest1-21systemd07:25:001
3191529993822,10cyclictest309-21in:imuxsock08:25:021
3191529993811,6cyclictest131rcu_preempt09:10:021
319154199377,5cyclictest131rcu_preempt08:50:033
3191541993625,7cyclictest37-21ksoftirqd/308:35:013
3191529993625,7cyclictest3192518-21munin-plugin-st07:30:001
3191529993622,10cyclictest23-21ksoftirqd/108:04:591
3191529993523,7cyclictest0-21swapper/108:20:001
319154199333,5cyclictest131rcu_preempt07:10:023
3191529993319,9cyclictest0-21swapper/108:10:181
3191529993319,9cyclictest0-21swapper/108:10:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional