You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-21 - 04:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Tue Jul 21, 2026 00:43:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32229911913,82cyclictest29479-21ssh23:25:420
32249911336,36cyclictest26062-21sh00:00:412
32229910986,7cyclictest0-21swapper/023:50:360
32229910817,55cyclictest0-21swapper/021:35:370
3223991035,12cyclictest2471-21sshd21:35:351
3225991023,80cyclictest29389-21taskset23:25:403
3225991013,73cyclictest2479-21ssh21:35:363
32229910072,6cyclictest12456-21sshd23:05:360
322399949,8cyclictest7950irq/56-dwc_otg_23:50:351
3224999328,37cyclictest8050irq/56-dwc_otg_21:35:352
322399933,11cyclictest32436-21apt-get22:50:361
322299937,57cyclictest0-21swapper/020:15:360
322299937,57cyclictest0-21swapper/020:15:360
322599928,60cyclictest17293-21sh23:50:373
3223999212,8cyclictest8050irq/56-dwc_otg_22:05:351
3222999258,7cyclictest0-21swapper/022:50:340
3222999245,28cyclictest0-21swapper/000:00:410
322399913,8cyclictest0-21swapper/100:00:411
3222999116,43cyclictest10861-21fschecks_count19:30:390
3224999028,22cyclictest17281-21ssh23:50:362
112122880,2sleep00-21swapper/000:20:540
3222998738,28cyclictest30919-21grep00:05:500
3224998623,25cyclictest29449-21ssh23:25:412
95412840,2sleep0101ktimersoftd/000:20:220
322399844,73cyclictest0-21swapper/100:05:501
3223998325,52cyclictest29374-21fschecks_time23:25:391
322299831,28cyclictest0-21swapper/000:10:350
322599812,59cyclictest30867-21sed00:05:493
322599812,58cyclictest27418-21ssh22:05:323
3225998110,47cyclictest8050irq/56-dwc_otg_00:00:413
3222998137,22cyclictest0-21swapper/022:05:330
322599807,50cyclictest8050irq/56-dwc_otg_20:55:523
169322800,2sleep20-21swapper/223:50:202
107822790,2sleep20-21swapper/219:30:362
322599783,69cyclictest32459-21ssh22:50:353
3225997819,47cyclictest25507-21fschecks_time20:10:383
322299787,28cyclictest0-21swapper/020:05:350
322299763,9cyclictest0-21swapper/023:55:330
322599751,49cyclictest0-21swapper/300:30:483
163292750,5sleep2322499cyclictest00:27:452
3222997432,3cyclictest0-21swapper/023:30:320
216872740,2sleep30-21swapper/321:19:203
322599721,48cyclictest9421-21/usr/sbin/munin19:25:483
322599721,45cyclictest7361-21/usr/sbin/munin21:40:463
322599711,47cyclictest8051-21/usr/sbin/munin20:50:473
3222997119,12cyclictest0-21swapper/022:15:320
322299711,45cyclictest0-21swapper/021:30:340
322299711,20cyclictest8050irq/56-dwc_otg_19:15:360
322599692,54cyclictest5812-21grep20:45:383
322599688,39cyclictest11602-21cat19:30:513
322599684,61cyclictest10728-21ssh21:45:323
3223996810,5cyclictest26302-21unixbench_multi20:10:501
3222996863,3cyclictest0-21swapper/022:26:110
3222996813,31cyclictest6910-21aten_rbpower_cu19:20:350
322599672,53cyclictest29207-21ssh22:45:473
322599671,53cyclictest21038-21ssh21:57:043
3224996737,25cyclictest30925-21awk00:05:502
322299671,29cyclictest0-21swapper/022:00:330
322599661,54cyclictest13028-21ssh00:24:163
322599661,47cyclictest5576-21/usr/sbin/munin00:15:323
322599661,47cyclictest24044-21ls20:05:443
3224996624,21cyclictest7950irq/56-dwc_otg_22:05:322
322599651,46cyclictest31653-21apt-get22:10:353
322599651,46cyclictest31653-21apt-get22:10:353
322599651,46cyclictest24549-21ssh21:22:113
322399654,5cyclictest24166-21ps20:05:481
322299653,12cyclictest0-21swapper/020:00:310
322299651,41cyclictest0-21swapper/021:10:260
322599644,57cyclictest2275050ssh23:55:563
322599641,48cyclictest21265-21apt-get20:00:283
322599641,46cyclictest6877-21apt-get19:20:333
322399649,13cyclictest6233-21missed_timers00:15:421
322399647,6cyclictest7850irq/56-dwc_otg21:11:471
322299641,5cyclictest18979-21chrt19:50:510
322599635,41cyclictest8050irq/56-dwc_otg_21:30:373
322599632,40cyclictest7950irq/56-dwc_otg_19:20:223
322599631,45cyclictest25274-21chrt22:02:043
322599631,43cyclictest15108-21rm23:08:193
322299633,37cyclictest0-21swapper/019:55:380
3222996317,23cyclictest16858-21fschecks_count23:10:380
3222996317,23cyclictest16858-21fschecks_count23:10:380
322299631,49cyclictest0-21swapper/020:50:210
322599621,43cyclictest17486-21/usr/sbin/munin23:10:463
322599621,43cyclictest17486-21/usr/sbin/munin23:10:463
322599621,43cyclictest15363-21ssh23:47:323
322599621,43cyclictest11963-21ssh23:05:083
322599621,41cyclictest3710-21rm00:12:443
3225996211,31cyclictest0-21swapper/323:20:383
322499623,36cyclictest0-21swapper/222:50:342
322299623,7cyclictest22133-21ssh22:37:220
3222996223,10cyclictest7950irq/56-dwc_otg_21:20:340
322599611,43cyclictest18130-21ssh21:54:203
322599611,42cyclictest24396-21awk22:40:373
322599611,42cyclictest24396-21awk22:40:373
322599611,41cyclictest22171-21sh00:35:253
322599611,40cyclictest20980-21diskstats23:15:393
322499614,23cyclictest5695-21kworker/u8:220:20:282
3224996121,10cyclictest9691-21ls20:55:432
3224996115,24cyclictest25896-21ntpq20:10:442
322399616,5cyclictest0-21swapper/120:45:381
322399614,6cyclictest7950irq/56-dwc_otg_20:45:221
322399611,7cyclictest10315-21users20:55:531
3223996115,6cyclictest198172chrt19:55:351
3223996112,4cyclictest15812-21rm00:26:481
322399611,16cyclictest0-21swapper/119:25:391
322299617,36cyclictest0-21swapper/019:40:350
322299612,46cyclictest0-21swapper/020:50:510
322599602,54cyclictest4157-21apt-get22:55:363
322599601,47cyclictest12493-21runrttasks23:44:513
322599601,47cyclictest12493-21runrttasks23:44:513
322599601,41cyclictest0-21swapper/321:13:233
322399601,16cyclictest1599-21/usr/sbin/munin23:30:461
3223996010,11cyclictest0-21swapper/100:35:211
322299602,36cyclictest0-21swapper/022:25:210
3222996016,29cyclictest0-21swapper/020:20:350
322599591,44cyclictest19293-21ssh22:34:443
322599591,42cyclictest30403-21taskset20:24:213
322599591,42cyclictest1597-21seq20:34:393
322599591,40cyclictest3251-21chrt20:37:363
322399598,8cyclictest7850irq/56-dwc_otg22:40:331
322399598,8cyclictest7850irq/56-dwc_otg22:40:331
322399594,12cyclictest7950irq/56-dwc_otg_21:30:391
322399593,4cyclictest7850irq/56-dwc_otg21:15:381
322399592,14cyclictest7950irq/56-dwc_otg_00:24:251
322399591,5cyclictest7850irq/56-dwc_otg22:45:501
3222995925,16cyclictest7950irq/56-dwc_otg_19:50:210
3222995925,16cyclictest7950irq/56-dwc_otg_19:50:210
322299591,34cyclictest985-21kworker/u8:300:35:320
322299591,31cyclictest1200-21grep20:30:480
322599582,51cyclictest3361-21apt-get22:15:323
322599581,42cyclictest7916-21ssh23:39:043
322599581,41cyclictest13621-21chrt19:36:253
322399589,5cyclictest7850irq/56-dwc_otg19:20:411
322299583,34cyclictest0-21swapper/021:05:400
322299583,34cyclictest0-21swapper/021:05:400
322299582,4cyclictest31681-21ssh22:10:350
322299582,4cyclictest31681-21ssh22:10:350
322599571,44cyclictest15465-21chrt19:41:493
322599571,40cyclictest897-21ssh23:30:363
322599571,40cyclictest16905-21ls19:45:483
322599571,40cyclictest16905-21ls19:45:483
322599571,33cyclictest0-21swapper/322:20:403
3224995718,18cyclictest0-21swapper/222:10:272
3224995718,18cyclictest0-21swapper/222:10:272
322399573,5cyclictest7950irq/56-dwc_otg_21:25:211
322399572,8cyclictest7420-21grep20:50:361
322399572,5cyclictest4149-21ntp_states22:15:441
322299576,36cyclictest9-21ksoftirqd/021:40:380
3222995722,6cyclictest1948-21aten_rbpower_cu20:35:370
322299571,40cyclictest28666-21idleruntime-cro23:25:210
3222995710,11cyclictest7950irq/56-dwc_otg_22:55:340
322599562,10cyclictest8050irq/56-dwc_otg_19:50:333
322599561,38cyclictest5077-21seq20:42:533
322599561,37cyclictest15752-21ssh00:26:423
322599561,36cyclictest8050irq/56-dwc_otg_21:27:383
322399563,5cyclictest16380-21rm23:49:231
322399561,17cyclictest0-21swapper/119:20:221
3222995626,15cyclictest26919-21latency_hist20:15:210
322299562,38cyclictest4830-21grep20:40:500
322299562,33cyclictest9227-21aten_rbpower_po20:55:360
322299561,29cyclictest19254-21missed_timers21:15:440
322599551,50cyclictest31107-21/usr/sbin/munin20:25:393
322399558,9cyclictest7850irq/56-dwc_otg23:05:391
322399554,6cyclictest125-21systemd-journal23:42:241
322399554,6cyclictest125-21systemd-journal23:42:241
322399553,3cyclictest7950irq/56-dwc_otg_21:10:211
322399553,3cyclictest7950irq/56-dwc_otg_21:10:211
322399552,4cyclictest0-21swapper/121:56:591
322399551,5cyclictest23874-21runrttasks22:40:211
322399551,3cyclictest7850irq/56-dwc_otg20:00:311
322299552,25cyclictest0-21swapper/022:30:500
322299551,27cyclictest8605-21grep23:00:400
3222995510,20cyclictest21131-21ls23:15:410
322599541,50cyclictest11575-21ls21:00:433
322599541,49cyclictest19700-21apt-get19:55:343
322599541,46cyclictest14007-21seq21:06:443
322599541,46cyclictest14007-21seq21:06:443
322599541,37cyclictest0-21swapper/322:29:323
3224995412,20cyclictest6955-21cut19:20:352
322399549,4cyclictest7564-21sh22:59:401
322399544,4cyclictest0-21swapper/121:41:131
322399544,4cyclictest0-21swapper/121:26:071
322399541,11cyclictest0-21swapper/123:15:331
322299543,10cyclictest7950irq/56-dwc_otg_21:55:340
322299542,35cyclictest0-21swapper/021:50:420
3222995423,4cyclictest0-21swapper/022:45:340
27292549,9sleep01041-21kworker/0:119:07:270
322599531,37cyclictest22439-21ssh22:37:513
322399533,5cyclictest0-21swapper/122:04:341
322399532,5cyclictest8050irq/56-dwc_otg_20:37:271
322399531,6cyclictest0-21swapper/120:27:091
3223995314,4cyclictest727-21runrttasks23:20:581
322299534,5cyclictest9283-21fschecks_time23:40:370
322299534,5cyclictest9283-21fschecks_time23:40:370
3222995326,6cyclictest7950irq/56-dwc_otg_23:45:430
322299531,29cyclictest27537-21ssh21:25:480
322599521,47cyclictest3252-21apt-get19:10:323
322599521,47cyclictest27619-21missed_timers20:15:443
322599521,47cyclictest27619-21missed_timers20:15:443
322499528,27cyclictest0-21swapper/220:35:362
3224995217,11cyclictest0-21swapper/219:25:342
322399526,4cyclictest7950irq/56-dwc_otg_22:12:061
322399526,4cyclictest7950irq/56-dwc_otg_22:12:061
322399522,3cyclictest7850irq/56-dwc_otg23:37:441
322299527,7cyclictest0-21swapper/000:30:330
322299522,30cyclictest0-21swapper/019:25:520
322299521,31cyclictest24364-21cpuspeed22:40:370
322299521,31cyclictest24364-21cpuspeed22:40:370
322499518,27cyclictest0-21swapper/219:35:352
3224995119,18cyclictest16563-21sh21:12:012
322299512,42cyclictest0-21swapper/021:45:480
322299512,30cyclictest0-21swapper/019:15:210
322499508,27cyclictest0-21swapper/220:00:272
3224995015,19cyclictest18584-21/usr/sbin/munin21:15:322
322399501,4cyclictest7850irq/56-dwc_otg19:46:281
322399501,4cyclictest7850irq/56-dwc_otg19:46:281
3224994921,15cyclictest23978-21memory20:05:432
3224994916,21cyclictest20843-21chrt21:56:432
3224994915,17cyclictest9203-21sh22:21:292
3224994914,20cyclictest7950irq/56-dwc_otg_20:50:522
3224994914,19cyclictest7950irq/56-dwc_otg_23:25:212
322399491,3cyclictest0-21swapper/123:56:291
3222994921,13cyclictest0-21swapper/023:35:350
322299491,22cyclictest0-21swapper/000:25:310
322499483,15cyclictest8050irq/56-dwc_otg_00:35:382
3224994815,22cyclictest25295-21ssh22:02:062
3224994813,12cyclictest7642-21ssh21:40:502
322399488,10cyclictest0-21swapper/100:38:481
322399482,10cyclictest0-21swapper/122:33:121
322399481,19cyclictest0-21swapper/123:04:251
3224994712,23cyclictest14024-21rm23:06:202
3224994711,20cyclictest22977-21ssh23:56:212
322399476,10cyclictest0-21swapper/121:45:221
322399472,5cyclictest17911-21ssh23:10:521
322399472,5cyclictest17911-21ssh23:10:521
322499469,20cyclictest8050irq/56-dwc_otg_19:20:212
3224994619,11cyclictest10459-21ssh23:02:232
3224994616,9cyclictest7950irq/56-dwc_otg_22:15:432
3224994613,12cyclictest5709-21cut00:15:362
3224994612,10cyclictest30723-21ssh21:30:322
322399464,13cyclictest0-21swapper/100:11:191
3224994515,18cyclictest7472-21ssh22:59:292
3224994515,18cyclictest16486-21ssh21:51:202
3224994512,22cyclictest22597-21ssh23:16:512
3224994512,21cyclictest24751-21ssh21:22:352
3224994512,17cyclictest28899-21sort22:45:422
3224994511,17cyclictest17831-21ssh22:32:052
322299456,27cyclictest0-21swapper/021:00:460
3224994422,7cyclictest24166-21apt-get22:40:342
3224994422,7cyclictest24166-21apt-get22:40:342
3224994414,13cyclictest16625-21ls19:45:432
3224994414,13cyclictest16625-21ls19:45:432
3224994412,21cyclictest12442-21ssh23:44:462
3224994412,21cyclictest12442-21ssh23:44:462
322299446,22cyclictest0-21swapper/019:35:350
3224994314,13cyclictest23753-21ssh22:40:142
3224994310,18cyclictest13678-21ssh21:48:472
3222994318,5cyclictest7950irq/56-dwc_otg_20:30:210
322499428,12cyclictest7850irq/56-dwc_otg00:23:512
322499426,23cyclictest0-21swapper/219:10:342
3224994217,13cyclictest21496-21dpkg20:00:222
3224994215,22cyclictest14319-21diskmemload00:11:392
3224994213,12cyclictest1064-21ssh23:30:392
3224994212,14cyclictest5594-21sshd20:45:352
3224994212,11cyclictest19603-21ssh23:13:572
3224994212,11cyclictest19603-21ssh23:13:572
3223994221,5cyclictest1666-21latency_hist20:35:211
147722420,1sleep20-21swapper/222:29:042
322499419,13cyclictest22017-21cut00:35:222
3224994127,11cyclictest28782-21ssh21:27:252
3224994114,7cyclictest29-21rcuc/221:05:392
3224994114,7cyclictest29-21rcuc/221:05:392
3224994110,13cyclictest12156-21taskset21:01:062
322499406,20cyclictest8050irq/56-dwc_otg_23:36:422
322499397,22cyclictest1385-21runrttasks20:32:052
3224993915,12cyclictest750-21snmpd19:41:272
3224993910,18cyclictest125-21systemd-journal20:41:392
322399392,15cyclictest0-21swapper/119:38:551
322499381,18cyclictest31022-21sort20:25:392
322399382,15cyclictest0-21swapper/119:30:571
3224993711,17cyclictest19215-21runrttasks19:52:242
3223993715,7cyclictest28593-21latency_hist20:20:221
3223993715,7cyclictest28593-21latency_hist20:20:221
29502374,18sleep22979-21munin-run19:10:212
3224993418,13cyclictest28137-21grep20:15:522
3224993418,13cyclictest28137-21grep20:15:522
322399347,14cyclictest8050irq/56-dwc_otg_22:25:501
322399344,11cyclictest7950irq/56-dwc_otg_21:00:411
3223993418,5cyclictest2897350wget20:20:351
3223993414,6cyclictest15566-21missed_timers21:50:431
322399326,6cyclictest0-21swapper/122:23:221
322399323,17cyclictest15277-21unixbench-2d19:40:511
322399314,5cyclictest7850irq/56-dwc_otg19:50:371
322399302,11cyclictest0-21swapper/119:10:391
28962279,12sleep30-21swapper/319:09:373
27482273,14sleep10-21swapper/119:07:401
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional