You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-22 - 01:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 21, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1708999552,40cyclictest28407-21ssh10:00:121
1708599511,40cyclictest12174-21ssh09:15:140
1708999502,36cyclictest16588-21ssh12:00:121
1708599501,40cyclictest1339-21ssh09:45:120
17105994829,12cyclictest35966irq/58-eth012:15:133
17105994829,12cyclictest35966irq/58-eth012:15:123
1708599463,5cyclictest131rcu_preempt11:50:140
17098994516,24cyclictest3115-21diskmemload10:40:142
1708999452,6cyclictest131rcu_preempt08:10:121
17098994424,9cyclictest27-21ksoftirqd/211:00:112
1709899442,33cyclictest13318-21ssh11:40:112
1709899435,26cyclictest27-21ksoftirqd/209:05:132
1708999438,8cyclictest21-21ksoftirqd/112:20:141
1709899412,15cyclictest27-21ksoftirqd/207:50:132
17098994110,13cyclictest27-21ksoftirqd/211:05:142
1708999414,29cyclictest31210-21ssh11:50:141
1708999412,28cyclictest8515-21ssh10:25:141
17085994120,17cyclictest12-21ksoftirqd/008:20:110
17085994120,17cyclictest12-21ksoftirqd/008:20:110
17105994021,11cyclictest35966irq/58-eth011:10:143
1709899395,23cyclictest261rcuc/211:35:132
1708599392,25cyclictest6536-21sh11:00:120
1710599382,23cyclictest35966irq/58-eth011:30:133
1710599382,22cyclictest35966irq/58-eth009:45:133
1709899385,2cyclictest131rcu_preempt10:00:122
1708999382,28cyclictest12283-21fschecks_time10:45:151
1708999381,26cyclictest26613-21ssh10:35:161
1708999373,14cyclictest19383-21ssh09:55:141
1708999371,10cyclictest6148-21sendmail11:00:011
1710599363,14cyclictest35966irq/58-eth010:30:123
1710599361,22cyclictest35966irq/58-eth010:15:113
1709899362,28cyclictest6633-21ssh10:23:162
1709899362,28cyclictest18335-21ssh09:54:172
1708999362,9cyclictest21989-21kthreadcore10:50:211
1708999362,25cyclictest4410-21ssh11:35:141
1709899354,21cyclictest27-21ksoftirqd/212:15:132
1709899354,21cyclictest27-21ksoftirqd/212:15:122
1708999351,11cyclictest28910-21tr12:05:231
1708999345,5cyclictest29198-21latency_hist08:30:011
1708999341,14cyclictest15576-21ls09:35:021
1710599334,19cyclictest35966irq/58-eth008:00:133
1709899332,18cyclictest27346irq/40-dwc2_hso12:30:152
1709899332,18cyclictest27346irq/40-dwc2_hso12:30:142
17098993316,12cyclictest0-21swapper/210:10:122
1709899331,16cyclictest9777-21ssh09:49:492
1708999334,6cyclictest0-21swapper/111:20:131
1708999334,6cyclictest0-21swapper/111:20:121
1708999333,19cyclictest17500-21ssh10:30:141
1708999333,11cyclictest26773-21ssh11:45:311
17089993325,4cyclictest0-21swapper/111:55:171
1708999331,9cyclictest4628-21kthreadcore12:10:221
1708999331,9cyclictest4628-21kthreadcore12:10:211
1708999331,9cyclictest2642-21kthreadcore08:00:221
1708999331,11cyclictest25692-21cat09:00:011
1708999331,10cyclictest30874-21cut09:25:161
1708999331,10cyclictest27892-21ssh11:30:131
1708599332,23cyclictest3116-21ssh10:40:140
1708999321,9cyclictest22377-21usage_Anybus07:45:291
1708999321,21cyclictest15767-21ssh09:35:131
1708999321,10cyclictest15014-21ssh09:50:301
17085993216,8cyclictest10346-21ssh09:50:130
1710599312,16cyclictest35966irq/58-eth010:20:133
1709899313,17cyclictest27346irq/40-dwc2_hso08:25:142
1709899312,21cyclictest26083-21apt-get10:35:042
1708999317,6cyclictest24745-21tr10:15:191
1708999313,8cyclictest14129-21sh12:35:031
1708999313,8cyclictest14129-21sh12:35:021
1708999312,10cyclictest7612-21kthreadcore10:05:201
1708999312,10cyclictest17781-21kthreadcore08:15:221
1708999311,9cyclictest5620-21irqcore08:05:181
1708999311,9cyclictest5620-21irqcore08:05:171
1708999311,9cyclictest4064-21kthreadcore09:10:191
1708999311,9cyclictest25575-21kthreadcore09:40:181
1708999311,9cyclictest23128-21wc12:40:011
1708999311,11cyclictest12394-21fschecks_time09:15:161
1708999311,11cyclictest10134-21grep08:40:271
17089993110,4cyclictest0-21swapper/107:55:171
1708999310,10cyclictest17245-21wc07:40:291
1708599315,20cyclictest12-21ksoftirqd/010:25:140
1708599314,18cyclictest10987-21ssh12:15:130
1708599314,18cyclictest10987-21ssh12:15:120
1708599311,24cyclictest23015-21ssh10:15:120
1710599302,14cyclictest36066irq/59-eth011:15:123
1708999302,9cyclictest19691-21kthreadcore11:25:211
1708999301,9cyclictest3344-21find10:20:281
1708999301,9cyclictest27834-21cron07:55:021
1708999301,9cyclictest16208-21kthreadcore10:10:211
1708999301,9cyclictest1461-21diskstats09:45:151
1708999301,7cyclictest17458-21unixbench_singl11:40:301
1708999301,11cyclictest29332-21users11:10:321
1708999301,10cyclictest4681-21kthreadcore10:40:211
1708999301,10cyclictest20273-21ps08:20:171
1708999301,10cyclictest20273-21ps08:20:161
1708999300,11cyclictest28450-21kthreadcore09:00:211
1708599304,21cyclictest151rcuc/010:05:120
17085993012,13cyclictest12-21ksoftirqd/009:59:500
1709899292,20cyclictest0-21swapper/207:55:132
1709899292,16cyclictest27346irq/40-dwc2_hso09:20:122
17098992913,13cyclictest0-21swapper/210:48:162
1708999296,4cyclictest1667-21irqrtprio11:15:181
1708999295,7cyclictest17543-21kthreadcore11:05:221
1708599292,22cyclictest12284-21fschecks_time10:45:150
1708599292,21cyclictest14039-21ssh10:10:120
1709899285,14cyclictest27346irq/40-dwc2_hso08:45:132
1709899285,14cyclictest27346irq/40-dwc2_hso08:45:122
1709899282,21cyclictest19505-21ssh09:55:162
1709899282,10cyclictest24398-21ssh11:10:072
17098992810,9cyclictest0-21swapper/208:50:122
1708999281,9cyclictest32129-21irqcore07:25:171
1708999281,9cyclictest31532-21kthreadcore12:25:231
1708999281,9cyclictest18062-21kthreadcore08:50:201
1708999281,9cyclictest14767-21processes12:15:271
1708999281,9cyclictest14767-21processes12:15:271
1708999281,10cyclictest6623-21df11:00:141
1708599282,19cyclictest799-21ssh11:15:020
1708999271,9cyclictest24336-21kthreadcore09:20:211
1708599275,18cyclictest12-21ksoftirqd/011:30:120
1708599274,18cyclictest12-21ksoftirqd/009:25:100
1710599262,13cyclictest35966irq/58-eth011:20:143
1710599262,13cyclictest35966irq/58-eth011:20:133
1709899262,10cyclictest0-21swapper/209:15:152
17105992515,7cyclictest35966irq/58-eth009:50:243
1709899254,9cyclictest14449-21ssh12:35:162
1709899251,19cyclictest30204-21sh09:25:132
1709899250,15cyclictest8873-21find07:35:122
1708999251,9cyclictest29375-21/usr/sbin/munin08:30:241
1708599253,16cyclictest12-21ksoftirqd/011:05:140
1710599249,10cyclictest35966irq/58-eth010:10:133
17105992413,8cyclictest35966irq/58-eth009:10:153
1709899246,10cyclictest2102-21diskstats12:10:162
1709899246,10cyclictest2102-21diskstats12:10:152
1709899243,16cyclictest0-21swapper/210:59:172
1709899243,13cyclictest27-21ksoftirqd/208:00:132
17098992416,5cyclictest32323-21ssh11:50:202
17098992412,10cyclictest2370-21ssh12:26:532
1708599243,15cyclictest24786-21ssh09:40:120
1708599242,14cyclictest0-21swapper/009:20:130
1709899238,7cyclictest3005-1kworker/u9:209:10:142
1709899233,15cyclictest27346irq/40-dwc2_hso12:05:122
1709899232,6cyclictest0-21swapper/210:29:162
1709899232,18cyclictest0-21swapper/208:20:132
1709899232,18cyclictest0-21swapper/208:20:122
1709899232,16cyclictest0-21swapper/211:15:092
17098992312,4cyclictest0-21swapper/211:21:372
17098992312,4cyclictest0-21swapper/211:21:362
17098992311,9cyclictest15732-21/usr/sbin/munin09:35:232
1709899231,15cyclictest27346irq/40-dwc2_hso08:05:142
1709899231,15cyclictest27346irq/40-dwc2_hso08:05:132
1709899231,13cyclictest19886-21ssh12:20:132
1708999232,17cyclictest26892-21apt-get07:20:131
1708999232,17cyclictest26892-21apt-get07:20:121
1708599234,12cyclictest844-21munin-node09:10:150
1708599233,15cyclictest17489-21diskmemload10:30:130
1708599231,18cyclictest29057-21cut12:25:180
1710599223,13cyclictest0-21swapper/310:45:153
1710599221,16cyclictest35966irq/58-eth011:50:133
1710599221,15cyclictest0-21swapper/309:15:143
17105992210,8cyclictest0-21swapper/310:40:193
1709899222,6cyclictest0-21swapper/210:34:162
1709899222,6cyclictest0-21swapper/209:44:582
17098992213,7cyclictest0-21swapper/210:05:522
17098992211,9cyclictest15286-21ssh09:34:542
1709899221,14cyclictest0-21swapper/207:20:112
1709899221,14cyclictest0-21swapper/207:20:112
1708599228,4cyclictest131rcu_preempt11:55:250
1708599224,11cyclictest26592-21ssh10:35:160
1708599222,14cyclictest4080-21apt-get07:30:110
1708599221,18cyclictest30575-21ls07:20:270
1708599221,18cyclictest30575-21ls07:20:270
1708599221,10cyclictest2047-21tr08:00:200
17105992110,8cyclictest0-21swapper/312:03:003
1709899217,4cyclictest7465-21kthreadcore08:40:212
1709899215,7cyclictest10455-21kthreadcore11:55:282
1709899213,14cyclictest5586-21unixbench_multi08:35:292
1709899212,16cyclictest19220-21tr12:00:202
1709899211,17cyclictest23312-21hwlatdetect10:15:162
1709899211,11cyclictest30812-21munin-plugin-st11:50:022
1709899211,10cyclictest29871-21kthreadcore11:30:222
1709899211,10cyclictest26474-21kthreadcore09:00:192
1709899211,10cyclictest24993-21find10:50:292
1709899211,10cyclictest13973-21sed08:10:292
1708599213,14cyclictest844-21munin-node12:05:130
1708599211,5cyclictest13947-21apt-get12:35:010
1708599211,5cyclictest13947-21apt-get12:35:010
17085992114,4cyclictest19258-21latency09:35:230
1708599211,16cyclictest30621-21apt-get09:05:120
1708599211,14cyclictest28354-21timerwakeupswit08:25:290
1708599211,14cyclictest2751-21diskmemload11:25:130
1708599211,14cyclictest1993-21rm12:10:140
1708599211,14cyclictest1993-21rm12:10:130
1708599211,14cyclictest16966-21latency07:40:230
1708599211,10cyclictest23708-21cut12:20:270
17085992110,8cyclictest528-21cron11:15:000
1710599209,6cyclictest0-21swapper/311:48:273
1710599208,9cyclictest0-21swapper/312:09:013
1710599208,9cyclictest0-21swapper/311:30:003
1710599203,10cyclictest0-21swapper/310:50:143
17105992011,6cyclictest35966irq/58-eth012:10:133
17105992011,6cyclictest35966irq/58-eth012:10:123
1710599201,14cyclictest35966irq/58-eth007:45:143
1710599201,13cyclictest0-21swapper/308:15:173
1709899202,9cyclictest19324-21kthreadcore07:45:202
1709899201,9cyclictest1432-21kthreadcore07:25:212
1709899201,6cyclictest0-21swapper/207:30:122
1709899201,11cyclictest19123-21tune2fs11:25:162
1709899201,10cyclictest24778-21kthreadcore07:15:232
1709899201,10cyclictest13872-21fschecks_time07:40:172
1709899200,16cyclictest21129-21tune2fs08:55:172
1708999201,10cyclictest31119-21irqrtprio09:05:171
1708599204,3cyclictest131rcu_preempt11:20:140
1708599204,3cyclictest131rcu_preempt11:20:130
1708599203,9cyclictest21083-21ssh10:50:130
17085992014,4cyclictest6138-21sendmail11:00:010
1708599201,17cyclictest25726-21latency11:45:240
17085992011,6cyclictest7115-21irqcore09:30:170
1708599201,16cyclictest31958-21sh10:20:110
1708599200,6cyclictest30946-21tr08:30:220
1710599199,8cyclictest0-21swapper/310:56:233
1710599199,4cyclictest0-21swapper/311:35:153
1710599193,12cyclictest0-21swapper/310:05:133
1710599192,11cyclictest0-21swapper/311:00:073
17105991912,4cyclictest0-21swapper/312:25:193
1710599191,15cyclictest0-21swapper/311:05:203
1710599191,13cyclictest0-21swapper/308:50:283
1710599191,12cyclictest0-21swapper/308:45:213
1710599191,12cyclictest0-21swapper/308:45:203
17105991910,6cyclictest35966irq/58-eth012:35:133
17105991910,6cyclictest0-21swapper/310:39:453
1709899191,9cyclictest19448-21kthreadcore07:10:202
1709899191,9cyclictest17199-21cat07:10:022
1709899191,9cyclictest17155-21tr08:15:202
1709899190,10cyclictest29547-21fschecks_time08:30:172
17089991910,6cyclictest15919-21sshd08:46:171
17089991910,6cyclictest15919-21sshd08:46:161
17089991910,6cyclictest0-21swapper/107:10:271
1708599192,12cyclictest25784-21apt-get09:00:130
1708599192,12cyclictest11389-21apt-get08:45:120
1708599192,12cyclictest11389-21apt-get08:45:110
1708599191,9cyclictest10228-21fschecks_count08:10:160
17085991912,4cyclictest31983-21needreboot10:00:230
1708599191,15cyclictest17465-21kthreadcore08:15:210
1708599191,14cyclictest7624-21kthreadcore11:35:240
1708599191,10cyclictest13300-21/usr/sbin/munin11:40:110
1710599183,11cyclictest0-21swapper/309:44:163
1710599182,10cyclictest0-21swapper/308:20:133
1710599182,10cyclictest0-21swapper/308:20:123
1710599182,10cyclictest0-21swapper/307:20:143
1710599182,10cyclictest0-21swapper/307:20:133
1710599181,13cyclictest0-21swapper/309:55:153
1710599181,13cyclictest0-21swapper/307:15:223
17105991810,5cyclictest35966irq/58-eth012:30:303
17105991810,5cyclictest35966irq/58-eth012:30:293
17105991810,5cyclictest35966irq/58-eth010:00:033
1708599189,6cyclictest0-21swapper/008:50:240
1708599188,6cyclictest685-21in:imuxsock07:15:010
1708599185,10cyclictest18840-21irqcore07:45:190
1708599181,11cyclictest1913-21cut08:35:160
1710599179,6cyclictest0-21swapper/311:41:383
1710599178,6cyclictest0-21swapper/309:35:123
1710599178,6cyclictest0-21swapper/309:23:163
1710599178,4cyclictest0-21swapper/311:55:253
1710599172,11cyclictest0-21swapper/309:30:203
1710599172,10cyclictest0-21swapper/307:55:133
17105991711,4cyclictest0-21swapper/312:20:153
1710599171,12cyclictest0-21swapper/308:55:283
1710599171,11cyclictest0-21swapper/307:50:203
17105991710,5cyclictest35966irq/58-eth009:25:123
1708999178,6cyclictest2570-21kthreadcore08:35:201
1708999173,11cyclictest24701-21kthreadcore07:15:211
1708999172,11cyclictest0-21swapper/107:35:131
1708999172,11cyclictest0-21swapper/107:30:141
1708599179,6cyclictest0-21swapper/012:00:090
1708599175,10cyclictest9276-21kthreadcore08:40:230
1708599174,10cyclictest6717-21kthreadcore08:05:200
1708599174,10cyclictest6717-21kthreadcore08:05:190
1708599173,11cyclictest28748-21grep07:55:190
1708599173,11cyclictest1796-21kthreadcore07:25:220
1708599172,11cyclictest25695-21latency_hist09:00:010
1708599171,12cyclictest22192-21sshd07:15:150
1708599171,12cyclictest13947-21apt-get12:35:040
1708599171,11cyclictest11123-21kthreadcore07:35:210
1710599168,6cyclictest0-21swapper/309:10:013
1710599162,9cyclictest0-21swapper/308:25:143
1710599162,10cyclictest35966irq/58-eth010:25:143
1710599162,10cyclictest0-21swapper/308:35:143
1710599162,10cyclictest0-21swapper/307:25:133
1708599161,12cyclictest23283-21apt-get07:50:130
1710599151,12cyclictest0-21swapper/308:35:033
1710599151,10cyclictest0-21swapper/307:35:203
1708999151,11cyclictest17204-21sort07:10:021
1708599151,11cyclictest17130-21latency_hist07:10:010
1710599138,3cyclictest0-21swapper/308:40:243
1710599138,3cyclictest0-21swapper/308:06:193
1710599138,3cyclictest0-21swapper/308:06:183
1710599138,3cyclictest0-21swapper/307:44:213
1710599132,9cyclictest0-21swapper/307:10:133
1710599112,6cyclictest0-21swapper/307:30:173
1710599111,7cyclictest0-21swapper/309:00:133
1710599111,7cyclictest0-21swapper/308:10:173
1710599101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional