You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 23:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Jun 18, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
513899686,52cyclictest15316-21ssh11:10:121
513299667,14cyclictest12-21ksoftirqd/008:25:120
513299667,14cyclictest12-21ksoftirqd/008:25:120
513899654,47cyclictest31455-21ssh09:50:131
514899624,33cyclictest7950-21ssh12:35:102
514899624,33cyclictest7950-21ssh12:35:102
5161995935,15cyclictest35966irq/58-eth009:35:133
513899599,6cyclictest131rcu_preempt07:45:101
516199525,27cyclictest35966irq/58-eth009:40:123
513299511,40cyclictest7967-21ssh12:35:110
513299511,40cyclictest7967-21ssh12:35:110
513299502,42cyclictest4068-21ssh12:15:130
513899483,25cyclictest30528-21ssh10:25:151
513899482,40cyclictest25499-21ssh10:40:141
513299483,32cyclictest31290-21ssh12:30:120
516199455,25cyclictest35966irq/58-eth011:55:113
5161994536,5cyclictest35966irq/58-eth012:35:103
5161994536,5cyclictest35966irq/58-eth012:35:093
513899452,34cyclictest4255-21ssh09:35:121
513899443,34cyclictest18228-21ssh12:05:131
5161994325,11cyclictest35966irq/58-eth010:30:133
5161994325,11cyclictest35966irq/58-eth010:30:123
5148994316,19cyclictest0-21swapper/209:25:122
5161994110,19cyclictest35966irq/58-eth012:10:133
516199403,25cyclictest35966irq/58-eth010:40:143
513899404,29cyclictest20421-21ssh10:55:141
5138994024,9cyclictest0-21swapper/110:20:121
516199394,22cyclictest35966irq/58-eth009:05:113
513299392,28cyclictest28527-21ssh11:35:140
5161993817,10cyclictest35966irq/58-eth011:40:143
513299383,25cyclictest3074-21ssh10:10:110
513299382,30cyclictest10333-21ssh11:25:130
514899372,18cyclictest2530-21ssh11:01:102
514899357,20cyclictest0-21swapper/212:30:112
513899351,23cyclictest7284-21ssh10:30:131
513899351,23cyclictest7284-21ssh10:30:131
5161993316,12cyclictest35966irq/58-eth011:45:133
514899335,15cyclictest27346irq/40-dwc2_hso07:20:102
514899334,19cyclictest0-21swapper/211:35:142
5148993317,5cyclictest2106-21ssh10:45:142
5161993215,10cyclictest35966irq/58-eth012:30:123
514899328,15cyclictest0-21swapper/207:50:132
514899327,17cyclictest18222-21ssh12:05:142
513899326,15cyclictest27381-21ssh12:10:141
516199316,11cyclictest0-21swapper/311:10:113
516199315,16cyclictest35966irq/58-eth012:00:123
516199314,19cyclictest35966irq/58-eth010:55:133
513299311,22cyclictest18375-21sh09:25:120
514899302,22cyclictest14177-21ssh11:09:092
514899302,21cyclictest28521-21ssh12:27:102
5138993018,4cyclictest131rcu_preempt11:40:121
516199293,16cyclictest35966irq/58-eth011:20:103
514899292,21cyclictest0-21swapper/208:30:122
514899292,20cyclictest0-21swapper/208:00:122
513899292,19cyclictest1143-21ssh11:35:431
5138992920,3cyclictest0-21swapper/109:45:111
513299294,4cyclictest131rcu_preempt12:10:140
513299291,22cyclictest1233-21ssh11:20:110
516199282,20cyclictest35966irq/58-eth009:45:113
514899285,14cyclictest27346irq/40-dwc2_hso08:05:132
514899282,14cyclictest27346irq/40-dwc2_hso08:20:132
514899282,10cyclictest0-21swapper/208:25:132
514899282,10cyclictest0-21swapper/208:25:122
514899281,21cyclictest0-21swapper/211:25:132
514899281,10cyclictest684166ssh10:10:272
5148992810,8cyclictest27-21ksoftirqd/209:45:272
514899280,10cyclictest7160-21ssh10:30:052
514899280,10cyclictest7160-21ssh10:30:042
513899286,11cyclictest22282-21ssh12:25:131
513899282,19cyclictest6228-21ssh11:20:411
516199273,16cyclictest0-21swapper/311:53:103
516199273,16cyclictest0-21swapper/311:53:103
516199273,15cyclictest0-21swapper/311:05:393
513899278,4cyclictest21-21ksoftirqd/109:00:101
513899272,18cyclictest11180-21ssh11:05:391
513899271,17cyclictest17271-21ssh10:00:131
513899271,17cyclictest17271-21ssh10:00:131
5132992714,7cyclictest0-21swapper/010:27:480
5132992712,8cyclictest0-21swapper/010:44:530
5132992711,9cyclictest0-21swapper/010:58:550
516199262,14cyclictest35966irq/58-eth012:25:143
514899265,14cyclictest27346irq/40-dwc2_hso11:15:112
514899262,9cyclictest9124-21ssh12:00:112
514899262,15cyclictest27-21ksoftirqd/211:55:182
514899261,17cyclictest14272-21rm11:45:122
5148992610,5cyclictest0-21swapper/212:15:152
513899267,13cyclictest21-21ksoftirqd/110:50:121
513899263,4cyclictest131rcu_preempt10:35:111
513299263,17cyclictest29503-21ssh11:00:130
5132992612,7cyclictest12-21ksoftirqd/011:40:110
516199252,18cyclictest35966irq/58-eth010:35:113
516199252,11cyclictest35966irq/58-eth010:15:113
514899258,7cyclictest27-21ksoftirqd/210:44:522
514899258,11cyclictest27-21ksoftirqd/210:01:142
514899258,11cyclictest27-21ksoftirqd/210:01:142
514899257,11cyclictest27-21ksoftirqd/211:31:532
514899255,7cyclictest27-21ksoftirqd/207:30:172
514899255,7cyclictest27-21ksoftirqd/207:30:162
514899254,14cyclictest27346irq/40-dwc2_hso09:30:132
513899252,17cyclictest30493-21ssh11:53:101
513899252,17cyclictest30493-21ssh11:53:101
513299258,5cyclictest12-21ksoftirqd/012:00:100
513299252,19cyclictest19031-21ssh11:10:260
5132992511,7cyclictest0-21swapper/009:58:070
513299251,12cyclictest19375-21ssh11:30:120
513299250,19cyclictest29959-21ssh11:16:100
516199243,13cyclictest0-21swapper/311:15:043
516199243,12cyclictest35966irq/58-eth009:55:133
516199242,16cyclictest35966irq/58-eth009:50:133
514899249,7cyclictest131rcu_preempt09:57:282
514899249,4cyclictest27-21ksoftirqd/210:39:002
514899244,13cyclictest27-21ksoftirqd/212:13:162
514899242,15cyclictest0-21swapper/210:50:052
5148992413,9cyclictest10862-21ssh09:37:332
514899241,15cyclictest2766666ssh10:58:222
5148992411,4cyclictest27-21ksoftirqd/211:53:362
5148992411,4cyclictest27-21ksoftirqd/211:53:352
5148992410,11cyclictest0-21swapper/208:40:252
513899248,11cyclictest21-21ksoftirqd/109:40:131
513899243,18cyclictest27670-21uniq09:30:161
513899243,12cyclictest12186-21ssh10:15:121
5138992415,6cyclictest5000-21cron07:10:051
513299248,11cyclictest12-21ksoftirqd/012:22:090
513299244,4cyclictest131rcu_preempt08:55:110
5132992411,7cyclictest0-21swapper/010:47:300
5132992411,7cyclictest0-21swapper/010:16:240
516199234,10cyclictest0-21swapper/307:25:143
516199233,13cyclictest0-21swapper/311:35:443
514899236,11cyclictest27-21ksoftirqd/209:10:352
5148992314,6cyclictest27-21ksoftirqd/209:50:112
5148992312,9cyclictest0-21swapper/210:20:092
5148992312,7cyclictest32665-21apt-get09:15:132
514899230,15cyclictest19843-21grep07:25:172
513899235,4cyclictest12696-21apt-get08:25:121
513899235,4cyclictest12696-21apt-get08:25:121
513899234,12cyclictest18369-21apt-get09:25:101
513899233,16cyclictest23665-21ssh09:10:141
513899232,16cyclictest21066-21apt-get08:00:111
513299238,9cyclictest12-21ksoftirqd/010:35:120
513299238,7cyclictest12-21ksoftirqd/007:25:130
513299234,3cyclictest131rcu_preempt08:15:170
513299234,13cyclictest23625-21/usr/sbin/munin09:10:250
5132992311,6cyclictest131rcu_preempt09:48:010
5132992310,7cyclictest0-21swapper/009:33:260
5132992310,6cyclictest131rcu_preempt09:50:090
5132992310,4cyclictest12-21ksoftirqd/010:20:560
516199222,10cyclictest0-21swapper/312:05:143
514899228,8cyclictest27-21ksoftirqd/209:00:302
514899227,4cyclictest27-21ksoftirqd/209:05:102
514899223,9cyclictest26141-21grep08:35:272
514899223,11cyclictest7654-21cron08:20:012
514899222,6cyclictest0-21swapper/211:40:452
5148992212,8cyclictest261rcuc/211:20:062
5148992212,8cyclictest0-21swapper/209:42:282
514899221,17cyclictest13913-21kernelversion12:20:172
514899221,11cyclictest16731-21irqrtprio07:55:182
514899221,10cyclictest12374-21fschecks_time10:15:152
5138992214,5cyclictest0-21swapper/107:20:191
513899221,15cyclictest13326-21cut12:20:141
513299229,3cyclictest12-21ksoftirqd/007:15:140
513299228,7cyclictest0-21swapper/010:32:440
513299228,7cyclictest0-21swapper/010:32:440
513299228,4cyclictest131rcu_preempt07:35:120
513299226,6cyclictest131rcu_preempt11:06:240
513299222,3cyclictest131rcu_preempt10:00:220
513299222,3cyclictest131rcu_preempt10:00:210
5132992212,3cyclictest0-21swapper/012:09:090
513299221,14cyclictest15000-21ssh10:50:250
5132992211,3cyclictest0-21swapper/009:43:370
5132992210,4cyclictest12-21ksoftirqd/010:05:170
516199217,9cyclictest35966irq/58-eth010:20:113
516199212,13cyclictest0-21swapper/310:00:143
516199212,13cyclictest0-21swapper/310:00:143
516199211,16cyclictest35966irq/58-eth011:25:133
514899217,11cyclictest4685-21kthreadcore07:40:222
514899213,12cyclictest27-21ksoftirqd/210:09:272
514899213,11cyclictest9476-21fschecks_count09:20:162
514899211,11cyclictest30763-21tune2fs10:25:172
514899211,10cyclictest30002-21irqrtprio07:35:192
514899211,10cyclictest16544-21kthreadcore11:10:192
513899212,14cyclictest23251-21diskmemload11:19:101
5138992114,4cyclictest0-21swapper/110:05:141
5138992112,5cyclictest12159-21kthreadcore11:25:211
513899210,13cyclictest0-21swapper/112:37:121
513899210,13cyclictest0-21swapper/112:37:111
513299218,7cyclictest0-21swapper/011:53:330
513299218,7cyclictest0-21swapper/011:53:330
513299216,7cyclictest12-21ksoftirqd/011:46:150
513299212,3cyclictest131rcu_preempt08:10:200
5132992114,4cyclictest0-21swapper/007:10:270
5132992112,3cyclictest0-21swapper/009:22:210
513299211,17cyclictest8026-21cut08:20:150
513299211,17cyclictest12111-21kthreadcore07:50:180
513299211,16cyclictest29164-21ls08:05:220
516199203,13cyclictest35966irq/58-eth009:25:113
516199203,11cyclictest0-21swapper/311:00:103
5161992010,8cyclictest0-21swapper/312:18:503
514899201,9cyclictest8172-21grep08:50:272
514899201,9cyclictest8172-21grep08:50:272
5148992013,3cyclictest31898-21apt-get08:45:122
514899201,15cyclictest5231-21cp07:10:022
514899201,15cyclictest5231-21cp07:10:022
514899201,15cyclictest31188-21irqrtprio08:10:172
514899201,11cyclictest8896-21kthreadcore07:45:202
514899201,10cyclictest12678-21grep08:55:272
513899201,8cyclictest32011-21irqstats12:30:181
5138992013,3cyclictest21-21ksoftirqd/107:30:271
5138992013,3cyclictest21-21ksoftirqd/107:30:271
5138992011,7cyclictest10878-21cron10:50:011
513899201,15cyclictest25878-21apt-get08:05:121
513299209,3cyclictest12-21ksoftirqd/009:35:130
513299208,10cyclictest4448-21cat11:55:280
5132992014,4cyclictest0-21swapper/012:25:140
5132992011,3cyclictest0-21swapper/009:15:050
513299200,6cyclictest18244-21memory07:20:210
516199192,10cyclictest0-21swapper/307:20:123
5161991910,7cyclictest0-21swapper/310:46:043
5161991910,6cyclictest0-21swapper/310:28:373
514899198,9cyclictest27-21ksoftirqd/207:15:122
514899191,10cyclictest5426-21diskstats07:10:132
513899199,3cyclictest21-21ksoftirqd/108:59:471
513899193,13cyclictest14364-21ssh11:45:121
513899191,8cyclictest3106-21cpuspeed08:15:131
5138991911,6cyclictest1-21systemd11:55:471
513899191,15cyclictest16937-21latency_hist10:00:011
513899191,10cyclictest20710-21kthreadcore07:25:201
513899191,10cyclictest12776-21netstat12:00:231
513299198,5cyclictest12-21ksoftirqd/007:35:010
513299198,5cyclictest12-21ksoftirqd/007:35:010
513299192,11cyclictest0-21swapper/008:00:130
5132991912,5cyclictest0-21swapper/008:45:220
5132991912,4cyclictest20602-21cut09:05:190
513299190,11cyclictest2010-21grep07:40:160
516199189,6cyclictest35966irq/58-eth009:30:133
516199189,6cyclictest35966irq/58-eth009:10:133
516199185,8cyclictest35966irq/58-eth009:20:133
516199183,10cyclictest0-21swapper/312:20:503
516199182,10cyclictest0-21swapper/311:30:103
516199182,10cyclictest0-21swapper/308:00:133
516199182,10cyclictest0-21swapper/307:40:143
516199182,10cyclictest0-21swapper/307:15:133
516199181,6cyclictest0-21swapper/308:35:113
513899188,6cyclictest0-21swapper/107:40:121
513899185,8cyclictest5120-21cyclictest08:50:111
513899185,8cyclictest5120-21cyclictest08:50:111
513899183,12cyclictest19350-21/usr/sbin/munin11:30:181
5138991812,3cyclictest18008-21grep08:30:161
5138991811,4cyclictest4471-21grep12:15:171
5138991810,6cyclictest1-21systemd09:15:201
5138991810,5cyclictest5069-21kthreadcore10:10:211
5138991810,5cyclictest21-21ksoftirqd/108:10:131
513899180,13cyclictest13694-21ssh09:20:311
513299182,11cyclictest0-21swapper/008:30:120
5132991811,5cyclictest24415-21cut08:35:190
5132991810,6cyclictest0-21swapper/007:55:240
513299180,16cyclictest31689-21ls08:45:020
516199179,6cyclictest0-21swapper/310:54:363
516199179,6cyclictest0-21swapper/310:13:373
516199179,6cyclictest0-21swapper/309:18:313
516199178,7cyclictest0-21swapper/310:08:103
516199172,10cyclictest0-21swapper/308:30:123
516199172,10cyclictest0-21swapper/308:15:123
516199172,10cyclictest0-21swapper/308:05:123
516199172,10cyclictest0-21swapper/307:30:053
516199172,10cyclictest0-21swapper/307:30:043
516199171,12cyclictest0-21swapper/307:10:133
513899172,12cyclictest16412-21fschecks_time07:55:151
513899170,8cyclictest0-21swapper/111:01:471
513299178,7cyclictest6608-21apt-get07:45:120
513299173,2cyclictest131rcu_preempt08:50:210
513299173,2cyclictest131rcu_preempt08:50:210
516199162,9cyclictest0-21swapper/308:40:123
516199162,9cyclictest0-21swapper/308:25:133
516199162,9cyclictest0-21swapper/308:25:133
516199162,10cyclictest0-21swapper/308:45:133
513899164,9cyclictest1-21systemd09:05:181
513899163,9cyclictest0-21swapper/108:45:131
513899162,10cyclictest0-21swapper/107:15:121
513899161,8cyclictest29618-21hwlatdetect07:35:171
513899161,3cyclictest131rcu_preempt08:40:121
5138991612,3cyclictest0-21swapper/108:35:121
513299169,5cyclictest12-21ksoftirqd/007:10:020
513299169,5cyclictest12-21ksoftirqd/007:10:020
513299161,10cyclictest0-21swapper/009:00:120
516199152,7cyclictest0-21swapper/307:50:133
516199152,10cyclictest0-21swapper/308:10:133
513899155,7cyclictest11442-21apt-get07:50:131
513899153,10cyclictest8098-21fschecks_time08:20:151
513899151,11cyclictest5215-21latency_hist07:10:021
513899151,11cyclictest5215-21latency_hist07:10:011
516199138,3cyclictest0-21swapper/307:56:293
516199132,7cyclictest0-21swapper/308:50:113
516199132,7cyclictest0-21swapper/308:50:103
516199127,3cyclictest0-21swapper/308:55:213
516199127,3cyclictest0-21swapper/307:39:323
516199123,7cyclictest0-21swapper/308:20:163
516199121,9cyclictest0-21swapper/309:00:193
516199121,8cyclictest0-21swapper/307:45:133
516199111,7cyclictest0-21swapper/307:10:023
516199111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional