You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-15 - 20:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Mar 15, 2026 12:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28648009911527,18cyclictest2924551-21ssh11:10:010
2864800999335,52cyclictest16850irq/134-800000010:40:010
286480099871,82cyclictest2888613-21ssh09:49:310
286480099813,4cyclictest2926729-21cron11:15:000
2864800998015,56cyclictest16850irq/134-800000009:10:310
2864800997910,64cyclictest2877773-21grep09:25:020
2864800997819,53cyclictest16850irq/134-800000010:50:250
2864800997819,53cyclictest16850irq/134-800000010:50:250
2864800997732,28cyclictest2927102-21ssh11:15:210
286480099772,4cyclictest0-21swapper/010:25:010
2864800997624,47cyclictest16850irq/134-800000012:04:440
286480099752,9cyclictest2922254-21latency_hist11:05:030
2864800997529,30cyclictest2878393-21ssh09:26:200
2864800997529,30cyclictest2878393-21ssh09:26:190
286480099752,11cyclictest2955672-21modprobe12:20:000
286480099751,12cyclictest2880639-21sshd09:31:250
286480099741,56cyclictest2906169-21ssh10:28:530
286480099735,62cyclictest0-21swapper/010:00:010
2864800997324,30cyclictest16850irq/134-800000012:09:140
286480099728,7cyclictest12-21ksoftirqd/009:00:010
2864800997226,28cyclictest2930231-21ssh11:22:420
2864814997125,40cyclictest2949692-21sh12:06:103
286481099712,18cyclictest2929641-21ssh11:21:072
286480099712,64cyclictest2914321-21rm10:46:570
2864800997121,45cyclictest16850irq/134-800000010:17:470
286480099711,65cyclictest2924245-21kworker/u8:211:30:000
286480099706,6cyclictest16850irq/134-800000010:41:370
286480099698,51cyclictest16850irq/134-800000011:32:100
2864800996925,39cyclictest275-21dbus-daemon11:45:000
2864800996919,33cyclictest16850irq/134-800000010:00:510
286480099681,7cyclictest1-21systemd08:38:260
2864800996712,46cyclictest16850irq/134-800000011:39:280
2864800996712,46cyclictest16850irq/134-800000011:39:280
2864800996710,52cyclictest16850irq/134-800000010:13:300
2864800996710,52cyclictest16850irq/134-800000010:13:290
286480099667,52cyclictest2864805-21latency_hist07:10:010
286480099667,52cyclictest2864805-21latency_hist07:10:010
286480099662,7cyclictest2867312-21cron07:59:590
2864814996527,34cyclictest2900928-21cron10:16:593
286481499652,12cyclictest2876166-21rm09:21:243
2864810996526,35cyclictest2915939-21sh10:50:272
2864810996526,35cyclictest2915939-21sh10:50:262
2864810996517,43cyclictest30-21ksoftirqd/211:38:062
2864810996517,43cyclictest30-21ksoftirqd/211:38:052
286480099652,16cyclictest2942720-21ssh11:50:310
2864800996517,30cyclictest2960574-21ssh12:30:240
286480099651,11cyclictest2869171-21sshd08:31:240
286481099642,21cyclictest2910284-21ssh10:38:042
286480099647,52cyclictest16850irq/134-800000012:24:400
286480099642,13cyclictest2897252-21ssh10:08:530
286480099642,12cyclictest2867049-21sshd07:53:240
286480099641,59cyclictest48-21kcompactd011:58:150
286480099641,10cyclictest1-21systemd07:47:260
2864814996322,36cyclictest2937290-21sh11:38:433
2864814996322,36cyclictest2937290-21sh11:38:423
286481499632,18cyclictest2917258-21ssh10:53:453
286481499632,18cyclictest2917258-21ssh10:53:443
2864814996320,39cyclictest37-21ksoftirqd/309:11:263
2864810996323,36cyclictest2904936-21sh10:25:492
286481099632,17cyclictest2886623-21munin-run09:45:002
286480099633,7cyclictest2866275-21sshd07:39:260
286480099632,4cyclictest2907395-21sshd10:31:250
286480099632,4cyclictest2907395-21sshd10:31:240
2864800996313,31cyclictest16850irq/134-800000012:28:240
2864810996219,38cyclictest30-21ksoftirqd/210:34:332
2864810996219,38cyclictest30-21ksoftirqd/210:34:322
2864800996238,20cyclictest16850irq/134-800000011:50:010
2864800996238,20cyclictest16850irq/134-800000011:50:000
286480099622,5cyclictest2867921-21munin-run08:10:000
2864800996215,43cyclictest16850irq/134-800000012:10:470
2864814996125,31cyclictest2870474-21sshd08:59:183
286481499612,21cyclictest2884321-21cron09:40:003
286481499612,17cyclictest3467602-21systemd-journal11:05:003
2864814996121,33cyclictest275-21dbus-daemon10:40:003
2864814996119,38cyclictest2874518-21sh09:17:543
2864814996115,42cyclictest2953553-21sort12:15:023
2864810996132,25cyclictest2898235-21sh10:10:492
2864810996132,25cyclictest2898235-21sh10:10:492
286481099612,17cyclictest2920630-21ssh11:01:042
286481099612,17cyclictest2901674-21ssh10:18:562
286480799619,38cyclictest23-21ksoftirqd/112:20:031
2864807996141,14cyclictest1-21systemd08:50:011
2864800996128,29cyclictest2885402-21rm09:41:590
286480099611,5cyclictest2865800-21/usr/sbin/munin07:30:010
286480099611,56cyclictest0-21swapper/012:36:310
286480099611,56cyclictest0-21swapper/012:36:310
286480099611,15cyclictest2875345-21ssh09:19:530
286480099611,14cyclictest364-21snmpd09:36:240
286481499602,14cyclictest2931642-21ssh11:25:383
2864814996016,40cyclictest2887588-21sh09:46:573
286481099603,14cyclictest2951409-21latency_hist12:10:032
2864810996013,42cyclictest0-21swapper/209:10:022
286480099602,7cyclictest150-21pr/ttyS207:31:140
286480099602,12cyclictest2868142-21sshd08:14:260
286480099601,7cyclictest2870771-21munin-run09:05:000
286480099601,7cyclictest2869704-21sshd08:41:250
286481499593,15cyclictest2866547-21modprobe07:43:243
286481499592,14cyclictest2878463-21ssh09:26:293
286481499592,14cyclictest2878463-21ssh09:26:293
286481099592,15cyclictest2895023-21ssh10:03:592
286481099592,12cyclictest2875931-21rm09:20:492
286480099594,6cyclictest16850irq/134-800000009:54:150
286480099594,4cyclictest150-21pr/ttyS210:55:290
286480099592,15cyclictest2870002-21munin-run08:50:010
286480099591,13cyclictest2870240-21munin-run08:55:010
286481499583,12cyclictest2865523-21cron07:25:013
286481499583,12cyclictest2865523-21cron07:25:003
2864814995825,29cyclictest37-21ksoftirqd/312:29:583
286481499582,19cyclictest275-21dbus-daemon11:45:003
286481499582,15cyclictest2867367-21munin-run08:00:003
286481499582,14cyclictest2890405-21ssh09:53:253
286481499582,13cyclictest2942975-21ssh11:51:063
286481099582,15cyclictest2920118-21date11:00:022
286481099582,14cyclictest2954538-21ssh12:17:152
2864810995818,36cyclictest2865129-21latency_hist07:15:012
2864810995815,39cyclictest30-21ksoftirqd/209:52:332
2864810995813,40cyclictest2903033-21sh10:21:382
286481099581,15cyclictest2873994-21ssh09:16:312
286481099581,13cyclictest2964608-21ssh12:39:542
286481099581,13cyclictest2964608-21ssh12:39:532
286481099581,13cyclictest2881261-21ssh09:32:532
286480099582,17cyclictest2867605-21munin-run08:05:010
2864814995734,20cyclictest2884789-21sh09:40:313
286481499572,17cyclictest2913390-21cat10:45:023
286481499572,17cyclictest2896984-21ssh10:08:143
286481499571,15cyclictest2902891-21ssh10:21:173
2864810995732,21cyclictest2961337-21sshd12:32:242
286481099572,17cyclictest2896046-21ssh10:05:512
286481099571,16cyclictest2912460-21ssh10:42:542
286480099572,15cyclictest2868398-21sshd08:18:250
286480099572,13cyclictest2866549-21sshd07:44:040
286480099571,4cyclictest2864966-21sshd07:12:250
286480099571,4cyclictest131rcu_preempt07:17:250
286481499563,10cyclictest2868409-21cron08:20:013
286481499562,9cyclictest2864981-21cron07:15:003
2864814995624,28cyclictest2940683-21sh11:46:043
2864814995624,28cyclictest2940683-21sh11:46:043
286481499562,19cyclictest1-21systemd12:05:013
286481499562,15cyclictest2934188-21ssh11:31:293
286481499562,15cyclictest2928753-21ssh11:19:373
286481499562,13cyclictest3467602-21systemd-journal11:07:543
2864814995621,30cyclictest2892163-1kworker/u9:210:04:273
286481499562,12cyclictest2905192-21ssh10:26:243
2864814995619,33cyclictest2866438-21date07:40:033
2864814995616,36cyclictest2881677-21rm09:34:013
2864814995612,39cyclictest0-21swapper/308:05:013
286481499561,16cyclictest2956131-21ssh12:20:323
286481499561,16cyclictest2930666-21ssh11:23:453
286481099562,18cyclictest2943920-21ssh11:53:332
286481099562,16cyclictest2893249-21munin-run10:00:012
286481099562,14cyclictest2922901-21sh11:05:592
2864810995619,33cyclictest2871026-21diskmemload11:26:552
286481099561,50cyclictest2868401-21sshd08:19:112
2864810995614,37cyclictest275-21dbus-daemon09:00:002
286481099561,20cyclictest2957747-21ssh12:24:322
286481099561,15cyclictest2934977-21ssh11:33:222
286481099561,15cyclictest2888143-21ssh09:48:252
286481099561,14cyclictest2951970-21ssh12:11:222
286481099561,13cyclictest2879325-21sh09:28:442
286481099561,13cyclictest2879325-21sh09:28:442
2864807995636,14cyclictest1-21systemd11:30:001
286480099561,10cyclictest2865511-21sshd07:23:250
286480099561,10cyclictest2865511-21sshd07:23:240
286481499553,16cyclictest2919988-21cron10:59:593
2864814995524,27cyclictest2945222-21sh11:56:173
286481499552,11cyclictest2898188-21ssh10:10:443
286481499552,11cyclictest2898188-21ssh10:10:433
286481499551,15cyclictest2953772-21ssh12:15:163
286481499551,14cyclictest2871049-21pluginstate09:10:023
286481099553,7cyclictest2946719-21cron12:00:002
286481099552,17cyclictest2883322-21ssh09:37:292
286481099552,14cyclictest2914100-21sh10:46:252
286481099552,14cyclictest2869212-21systemd-run08:35:002
286481099552,12cyclictest2960060-21ssh12:29:442
286481099551,13cyclictest2938722-21rm11:41:412
2864800995511,39cyclictest16950irq/224-800000009:10:010
286481499543,16cyclictest2907408-21ssh10:31:273
286481499543,16cyclictest2907408-21ssh10:31:273
286481499542,15cyclictest2893051-21ssh09:59:323
2864814995412,38cyclictest0-21swapper/308:32:413
286481499541,18cyclictest2963547-21ssh12:37:223
286481499541,18cyclictest2963547-21ssh12:37:213
286481499541,13cyclictest2960874-21ssh12:31:193
286481499541,11cyclictest3467602-21systemd-journal10:48:063
286481099543,8cyclictest2942407-21grep11:50:022
286481099543,8cyclictest2942407-21grep11:50:012
286481099542,17cyclictest2927623-21ssh11:16:492
286481099542,17cyclictest2871481-21ssh09:10:322
286481099542,16cyclictest311-21rs:main2
286481099542,14cyclictest2925504-21rm11:11:572
286481099542,14cyclictest2868954-21munin-run08:30:012
286481099542,14cyclictest2868954-21munin-run08:30:002
286481099541,13cyclictest2867646-21cat08:05:012
2864800995419,31cyclictest12-21ksoftirqd/008:30:020
2864800995419,31cyclictest12-21ksoftirqd/008:30:010
286481499532,18cyclictest2868707-21true08:25:013
286481499531,12cyclictest2925481-21ssh11:11:523
286481099536,42cyclictest0-21swapper/208:45:022
2864810995317,32cyclictest30-21ksoftirqd/207:50:022
2864810995317,32cyclictest2866287-21modprobe07:40:002
286481099531,12cyclictest2868303-21sort08:15:022
2864807995312,36cyclictest2941191-21ssh11:47:171
2864807995312,36cyclictest2941191-21ssh11:47:161
2864814995239,8cyclictest2870757-21cron09:05:003
2864814995214,34cyclictest2866097-21latency_hist07:35:013
2864814995213,33cyclictest0-21swapper/308:35:373
286481099522,9cyclictest2870011-21wc08:50:022
286481099522,9cyclictest2867895-21cron08:09:592
286481099522,15cyclictest2869463-21munin-run08:40:012
286481099521,12cyclictest2865573-21cat07:25:022
286481099521,12cyclictest2865573-21cat07:25:012
286481099521,11cyclictest2870435-21date08:55:022
286481099521,11cyclictest2864933-21latency_hist07:10:032
286481099521,11cyclictest2864933-21latency_hist07:10:022
2864807995232,13cyclictest2902131-21cron10:19:591
286481499512,16cyclictest2868931-21sshd08:29:133
286481499512,16cyclictest2868931-21sshd08:29:123
286481499512,11cyclictest309-21in:imuxsock07:55:013
286481499512,10cyclictest2866826-21munin-plugin-st07:50:003
286481499511,11cyclictest364-21snmpd08:12:213
286481099512,14cyclictest1-21systemd07:55:002
2864810995120,28cyclictest30-21ksoftirqd/207:43:302
286481499503,6cyclictest3467602-21systemd-journal08:06:253
286481499501,14cyclictest2865798-21systemd-run07:30:003
286481099502,12cyclictest2865807-21cat07:30:012
2864810995018,27cyclictest0-21swapper/207:35:012
286480799502,8cyclictest0-21swapper/112:10:011
2864807995026,15cyclictest3467602-21systemd-journal07:50:011
286481499494,8cyclictest2865238-21sshd07:19:263
2864814994913,30cyclictest2869834-21cat08:45:033
286481099492,8cyclictest2867475-21latency_hist08:00:022
2864807994931,14cyclictest275-21dbus-daemon09:55:011
2864807994920,18cyclictest275-21dbus-daemon11:40:011
2864807994920,18cyclictest275-21dbus-daemon11:40:001
286481499482,7cyclictest2870010-21grep08:50:013
286481499481,11cyclictest364-21snmpd08:51:443
286481099481,13cyclictest364-21snmpd07:17:112
2864807994818,25cyclictest2940158-21ssh11:45:021
286481099472,11cyclictest2868754-21date08:25:012
2864807994729,10cyclictest309-21in:imuxsock11:35:011
2864807994727,15cyclictest275-21dbus-daemon11:20:001
286480799472,4cyclictest131rcu_preempt10:23:391
2864807994723,17cyclictest3467602-21systemd-journal07:35:001
286481099463,16cyclictest2870763-21modprobe09:05:012
2864807994628,9cyclictest2865265-21munin-plugin-st07:20:011
286480799462,4cyclictest131rcu_preempt10:27:101
2864807994622,15cyclictest275-21dbus-daemon10:15:011
2864807994622,15cyclictest275-21dbus-daemon10:15:001
286480799461,3cyclictest131rcu_preempt11:56:261
286480099462,12cyclictest2868702-21grep08:24:590
2864807994530,11cyclictest1-21systemd10:40:001
286480799452,4cyclictest131rcu_preempt09:29:181
286480799452,4cyclictest131rcu_preempt09:29:171
286481499441,14cyclictest2864909-21cat07:10:033
286481499441,14cyclictest2864909-21cat07:10:023
2864807994430,11cyclictest23-21ksoftirqd/110:04:091
2864807994429,12cyclictest23-21ksoftirqd/112:28:461
2864807994428,9cyclictest309-21in:imuxsock11:10:001
2864807994421,15cyclictest3467602-21systemd-journal09:20:001
286480799441,3cyclictest131rcu_preempt09:30:341
286480799435,26cyclictest2893388-21latency_hist10:00:021
286480799434,29cyclictest2877695-21latency_hist09:25:021
2864807994328,8cyclictest23-21ksoftirqd/110:59:591
2864807994327,11cyclictest23-21ksoftirqd/109:42:171
2864807994326,11cyclictest3467602-21systemd-journal07:40:001
2864807994324,12cyclictest309-21in:imuxsock09:50:011
2864807994322,17cyclictest1-21systemd08:35:011
286480799423,3cyclictest131rcu_preempt09:37:381
2864807994227,11cyclictest1-21systemd08:30:011
2864807994227,11cyclictest1-21systemd08:30:011
2864807994226,12cyclictest275-21dbus-daemon08:25:011
2864807994225,11cyclictest23-21ksoftirqd/112:33:181
286480799422,3cyclictest131rcu_preempt10:31:441
286480799422,3cyclictest131rcu_preempt10:31:431
286480799414,27cyclictest3467602-21systemd-journal08:55:001
2864807994129,9cyclictest23-21ksoftirqd/112:10:311
2864807994129,9cyclictest23-21ksoftirqd/111:51:111
2864807994128,8cyclictest2871165-21latency_hist09:10:021
2864807994127,11cyclictest23-21ksoftirqd/109:15:011
286480799403,31cyclictest2869466-21systemctl08:40:001
2864807994029,8cyclictest2868379-21sshd08:16:241
2864807994028,9cyclictest0-21swapper/107:30:011
2864807994027,6cyclictest23-21ksoftirqd/108:10:001
286480799402,3cyclictest131rcu_preempt12:37:461
286480799402,3cyclictest131rcu_preempt12:37:451
286480799402,3cyclictest131rcu_preempt12:23:121
2864807994022,12cyclictest2870491-21sh09:00:001
2864807994019,16cyclictest3467602-21systemd-journal11:15:001
286480799395,3cyclictest131rcu_preempt07:45:011
2864807993928,9cyclictest23-21ksoftirqd/108:00:011
2864807993925,10cyclictest23-21ksoftirqd/111:21:271
2864807993924,11cyclictest23-21ksoftirqd/110:55:011
2864807993924,11cyclictest23-21ksoftirqd/110:55:001
2864807993923,11cyclictest23-21ksoftirqd/110:40:471
2864807993921,14cyclictest2897405-21sshd10:09:241
2864807993918,11cyclictest275-21dbus-daemon07:55:001
286480799382,3cyclictest131rcu_preempt12:01:511
2864807993823,11cyclictest23-21ksoftirqd/108:05:001
2864807993725,8cyclictest0-21swapper/110:49:411
2864807993723,11cyclictest23-21ksoftirqd/111:04:501
2864807993525,5cyclictest309-21in:imuxsock07:14:591
2864807993523,9cyclictest23-21ksoftirqd/108:15:001
2864807993425,6cyclictest23-21ksoftirqd/107:25:021
2864807993425,6cyclictest23-21ksoftirqd/107:25:021
2864807993421,9cyclictest23-21ksoftirqd/109:04:171
2864807993225,4cyclictest2864740-21latency_hist07:10:031
2864807993225,4cyclictest2864740-21latency_hist07:10:021
2864807993223,6cyclictest23-21ksoftirqd/108:40:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional