You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-17 - 08:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Fri Jul 17, 2026 00:43:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1999521760,6sleep02965999cyclictest00:00:400
296599910562,23cyclictest7950irq/56-dwc_otg_23:45:400
29659991034,79cyclictest0-21swapper/022:35:300
29659999737,16cyclictest23318-21ssh22:45:470
29659999737,16cyclictest23318-21ssh22:45:470
56772930,2sleep20-21swapper/223:04:262
45372910,2sleep10-21swapper/119:30:201
29659999137,21cyclictest0-21swapper/000:20:430
2965999894,49cyclictest0-21swapper/019:50:300
147252880,2sleep00-21swapper/000:33:300
2965999853,48cyclictest1743-21ssh22:20:320
324002830,2sleep20-21swapper/200:15:402
29659998119,40cyclictest0-21swapper/022:55:360
80632790,5sleep32966299cyclictest22:27:273
2965999733,9cyclictest0-21swapper/000:05:330
317972720,2sleep20-21swapper/222:55:482
29661997110,24cyclictest4370-21ntp_states00:20:442
2966099714,46cyclictest31002-21uniq22:55:371
2965999714,51cyclictest11-21rcu_preempt22:25:310
29661996940,23cyclictest23369-21/usr/sbin/munin22:45:472
29661996940,23cyclictest23369-21/usr/sbin/munin22:45:472
295902690,1sleep30-21swapper/322:15:113
2965999684,43cyclictest9359-21ssh21:50:300
112992680,3sleep1221ktimersoftd/121:13:081
29660996712,36cyclictest4479-21/usr/sbin/munin00:20:461
29661996435,23cyclictest14098-21apt-get22:35:332
29660996413,13cyclictest23-21ksoftirqd/123:45:391
29661996331,26cyclictest7700-21grep23:45:402
29659996312,39cyclictest9-21ksoftirqd/019:20:180
2966099623,16cyclictest23-21ksoftirqd/122:45:471
2966099623,16cyclictest23-21ksoftirqd/122:45:471
2965999621,50cyclictest27821-21grep22:50:500
2966099614,22cyclictest23-21ksoftirqd/122:35:311
2966299597,25cyclictest6305-21kworker/3:123:45:423
29662995928,9cyclictest7950irq/56-dwc_otg_22:35:333
29659995940,9cyclictest2677-21ls23:40:190
2965999591,45cyclictest13046-21ls21:15:350
248502590,2sleep30-21swapper/323:26:453
2965999588,28cyclictest18423-21wget22:40:340
2965999581,4cyclictest16378-21ssh00:35:360
2965999581,45cyclictest0-21swapper/000:10:300
2965999581,45cyclictest0-21swapper/000:10:300
29659995731,5cyclictest7950irq/56-dwc_otg_23:55:480
29659995716,31cyclictest13-21rcuc/020:01:580
2965999571,31cyclictest25938-21wget22:10:320
306052560,1sleep20-21swapper/219:10:452
2966299564,29cyclictest0-21swapper/300:20:453
2965999564,29cyclictest8954-21unixbench_multi00:25:480
2965999562,5cyclictest0-21swapper/023:15:500
2965999562,5cyclictest0-21swapper/023:15:500
2965999562,51cyclictest0-21swapper/000:20:180
29659995618,26cyclictest9-21ksoftirqd/020:35:480
29661995517,6cyclictest8050irq/56-dwc_otg_23:35:322
2965999553,45cyclictest17144-21kworker/0:222:06:580
2965999551,5cyclictest2713-21ssh23:00:380
208622550,2sleep00-21swapper/021:25:190
29662995413,11cyclictest31059-21ssh22:55:383
29661995415,16cyclictest30560-21ls21:35:472
2965999542,45cyclictest12509-21kworker/0:223:15:070
29659995424,4cyclictest3599-21ssh23:40:400
29659995421,15cyclictest1936-21sh21:40:440
106742540,2sleep30-21swapper/300:28:503
2965999536,28cyclictest3940-21unixbench_singl19:25:470
2965999536,28cyclictest3940-21unixbench_singl19:25:470
29659995326,6cyclictest7950irq/56-dwc_otg_21:10:310
29659995323,18cyclictest9-21ksoftirqd/020:55:470
29659995315,28cyclictest9-21ksoftirqd/021:35:200
29659995315,28cyclictest9-21ksoftirqd/021:35:200
29662995211,24cyclictest23250-21ntp_states22:45:453
29662995211,24cyclictest23250-21ntp_states22:45:453
2966199528,26cyclictest19972-21ssh23:20:452
29661995213,19cyclictest7950irq/56-dwc_otg_00:00:542
2965999525,44cyclictest23086-21aten_rbpower_en23:25:340
2965999524,38cyclictest101ktimersoftd/019:56:500
29659995223,5cyclictest8050irq/56-dwc_otg_23:35:190
320882511,7sleep132140-21fschecks_count20:45:361
2966199514,21cyclictest8050irq/56-dwc_otg_21:15:482
29661995114,22cyclictest8715-21cat00:25:452
29659995126,5cyclictest7950irq/56-dwc_otg_22:15:340
2965999511,5cyclictest19894-21ls23:20:440
2965999511,29cyclictest29981-21fschecks_time19:10:360
2965999511,27cyclictest8543-21fschecks_count19:40:360
318352500,2sleep20-21swapper/220:45:202
29661995021,20cyclictest6351-21sh23:44:342
29659995030,11cyclictest1557-21fschecks_time20:50:360
29659995030,11cyclictest1557-21fschecks_time20:50:360
29659995021,7cyclictest7950irq/56-dwc_otg_23:05:340
2965999501,27cyclictest10040-21aten_rbpower_en22:30:320
2965999501,27cyclictest10040-21aten_rbpower_en22:30:320
2965999501,26cyclictest6804-21fschecks_time19:35:360
2965999501,26cyclictest6804-21fschecks_time19:35:360
2965999501,24cyclictest27100-21proc_pri20:30:440
2965999501,12cyclictest13-21rcuc/020:50:200
2966199499,22cyclictest26692-21rm00:09:042
29661994915,13cyclictest6776-21ssh22:25:452
2966099493,26cyclictest0-21swapper/123:35:301
2965999495,32cyclictest0-21swapper/020:27:550
2965999495,32cyclictest0-21swapper/020:27:550
2965999492,6cyclictest0-21swapper/019:30:400
2965999491,30cyclictest0-21swapper/021:55:310
2965999491,27cyclictest0-21swapper/020:15:310
128562490,2sleep10-21swapper/123:51:231
2966199483,4cyclictest7950irq/56-dwc_otg_20:20:342
29661994818,14cyclictest23798-21/usr/sbin/munin23:25:432
29661994817,18cyclictest7850irq/56-dwc_otg22:10:322
29661994815,12cyclictest8219-21apt-get21:10:232
29661994814,16cyclictest13109-21ls00:30:472
29661994810,20cyclictest439-21seq19:18:242
2966299471,4cyclictest8050irq/56-dwc_otg_20:40:293
29661994715,15cyclictest19571-21chrt20:10:382
29661994714,20cyclictest7266-21ssh21:47:072
2965999477,8cyclictest7950irq/56-dwc_otg_19:45:490
2965999477,19cyclictest0-21swapper/020:10:450
2965999472,5cyclictest0-21swapper/021:25:290
2965999472,38cyclictest101ktimersoftd/021:45:530
29659994722,11cyclictest0-21swapper/020:20:370
2965999471,26cyclictest18997-21ssh22:00:530
2966299467,25cyclictest0-21swapper/321:05:323
2966199466,29cyclictest0-21swapper/222:05:332
29661994616,17cyclictest13272-21ssh23:13:152
29661994613,13cyclictest16134-21ssh21:58:122
29659994616,24cyclictest9-21ksoftirqd/021:35:400
29659994615,23cyclictest0-21swapper/023:53:580
29659994615,17cyclictest0-21swapper/019:20:430
29659994615,14cyclictest5158-21/usr/sbin/munin21:00:360
292352463,35sleep00-21swapper/019:08:170
2966299453,11cyclictest8050irq/56-dwc_otg_20:35:313
2966199459,12cyclictest17086-21awk20:05:192
29661994514,18cyclictest29356-21chrt20:35:582
29661994514,13cyclictest10786-21ntp_states19:45:442
29661994510,14cyclictest30135-21rm23:33:512
2966199448,23cyclictest0-21swapper/220:05:332
2966199446,20cyclictest13584-21ssh23:52:442
2966199445,22cyclictest7850irq/56-dwc_otg21:21:282
29661994412,13cyclictest17062-21ssh23:17:372
29661994412,13cyclictest17062-21ssh23:17:372
29661994411,21cyclictest18450-21/usr/sbin/munin22:00:442
2966099441,6cyclictest26113-21rm00:08:001
2966099441,5cyclictest16650-21latency00:35:391
29659994419,13cyclictest0-21swapper/020:41:280
2966299437,20cyclictest8050irq/56-dwc_otg_21:35:333
29661994311,12cyclictest15167-21ssh23:55:202
29661994310,14cyclictest3919-21ssh22:22:242
29660994319,15cyclictest5237-21ssh21:45:311
2966299421,15cyclictest6345-21timerwakeupswit21:45:493
2966199429,10cyclictest19194-21ssh00:39:062
2966199428,12cyclictest27022-21awk20:30:422
2966199427,25cyclictest0-21swapper/221:50:322
2966199423,25cyclictest25249-21ssh21:30:322
2966199423,25cyclictest25249-21ssh21:30:322
29661994217,12cyclictest8050irq/56-dwc_otg_21:40:312
29661994215,13cyclictest26535-21ssh22:50:292
29661994215,11cyclictest29753-21chrt00:12:062
29661994215,11cyclictest29753-21chrt00:12:062
29661994213,11cyclictest1434-21awk20:50:342
29661994213,11cyclictest1434-21awk20:50:342
29661994210,12cyclictest11569-21ssh22:31:172
29661994210,12cyclictest11569-21ssh22:31:172
2966099422,26cyclictest0-21swapper/119:55:321
2966099411,26cyclictest0-21swapper/120:35:321
29660994110,18cyclictest7850irq/56-dwc_otg00:25:451
2965999419,19cyclictest13-21rcuc/020:07:350
2965999416,6cyclictest9-21ksoftirqd/021:10:190
237072410,1sleep30-21swapper/322:06:573
2966299401,3cyclictest8050irq/56-dwc_otg_21:25:203
29661994011,15cyclictest24837-21ssh21:30:142
29661994010,19cyclictest20447-21ssh22:42:182
29660994023,5cyclictest9836-21if_eth021:50:381
2966099402,17cyclictest17831-21ps21:20:451
2966099401,17cyclictest8050irq/56-dwc_otg_19:50:301
189122400,32sleep10-21swapper/123:59:571
2966299392,4cyclictest8050irq/56-dwc_otg_00:15:333
2966199398,13cyclictest8653-21basename19:40:372
2966199393,22cyclictest8050irq/56-dwc_otg_19:21:072
2966099392,5cyclictest0-21swapper/121:05:301
29660993923,4cyclictest25257-21sh22:09:461
2966099391,23cyclictest0-21swapper/120:55:321
2966299387,10cyclictest31948-21/usr/sbin/munin19:15:393
29662993814,10cyclictest8050irq/56-dwc_otg_00:31:113
2966299381,29cyclictest17063-21switchtime00:35:453
29661993815,11cyclictest301ktimersoftd/219:50:192
29661993813,10cyclictest8520-21sh23:07:052
2966099388,16cyclictest0-21swapper/123:25:411
2966099385,4cyclictest0-21swapper/121:32:421
2966099385,4cyclictest0-21swapper/121:32:421
2966099384,11cyclictest0-21swapper/119:40:301
29660993819,12cyclictest3604-21sensors19:25:441
2966099381,32cyclictest25047-21ls20:25:391
2966099381,32cyclictest25047-21ls20:25:391
29660993813,16cyclictest0-21swapper/122:10:491
2966099381,25cyclictest15729-21ssh23:15:471
2966099381,25cyclictest15729-21ssh23:15:471
29662993720,4cyclictest727-21runrttasks23:21:083
29662993714,7cyclictest7950irq/56-dwc_otg_20:30:423
29662993714,5cyclictest28429-21sshd23:30:493
2966199377,12cyclictest32414-21ssh22:17:442
2966099376,11cyclictest8050irq/56-dwc_otg_22:00:341
2966099372,30cyclictest8050irq/56-dwc_otg_23:20:281
2966099371,24cyclictest7950irq/56-dwc_otg_23:05:341
29660993712,23cyclictest0-21swapper/120:40:241
2966299367,15cyclictest381ktimersoftd/300:13:283
2966299367,15cyclictest381ktimersoftd/300:13:283
2966299363,20cyclictest381ktimersoftd/323:39:163
29662993618,7cyclictest7850irq/56-dwc_otg19:45:383
2966299361,22cyclictest5630-21ls22:25:193
29662993611,7cyclictest381ktimersoftd/320:30:203
29662993611,7cyclictest381ktimersoftd/320:30:203
2966199369,12cyclictest5862-21chrt19:31:542
2966199361,4cyclictest8050irq/56-dwc_otg_20:15:462
2966099365,9cyclictest23-21ksoftirqd/122:50:481
2966099362,4cyclictest0-21swapper/122:22:421
29660993616,12cyclictest1049-21/usr/sbin/munin21:40:461
29660993615,13cyclictest0-21swapper/121:35:381
2966099361,25cyclictest0-21swapper/120:50:321
2966099361,25cyclictest0-21swapper/120:50:321
2966099361,24cyclictest0-21swapper/100:15:331
2966099361,23cyclictest18248-21ssh22:40:251
2966299358,8cyclictest39-21ksoftirqd/323:06:483
2966299358,12cyclictest7850irq/56-dwc_otg21:55:073
2966299355,7cyclictest19309-21latency_hist00:00:203
2966299355,12cyclictest7950irq/56-dwc_otg_00:00:533
2966299354,19cyclictest8050irq/56-dwc_otg_23:42:143
2966299352,6cyclictest0-21swapper/321:40:303
2966299352,5cyclictest8050irq/56-dwc_otg_00:05:333
2966299352,3cyclictest8050irq/56-dwc_otg_23:00:333
2966199359,11cyclictest8050irq/56-dwc_otg_19:40:162
2966199359,11cyclictest8050irq/56-dwc_otg_19:40:162
29661993526,6cyclictest301ktimersoftd/220:55:292
29661993511,11cyclictest8050irq/56-dwc_otg_19:30:192
29661993511,11cyclictest8050irq/56-dwc_otg_19:30:192
2966099352,9cyclictest2664-21sh23:00:371
29660993520,6cyclictest13135-21ssh23:12:591
29660993514,12cyclictest0-21swapper/121:15:351
29660993512,4cyclictest0-21swapper/121:00:361
29660993510,5cyclictest0-21swapper/100:30:351
2966299349,15cyclictest7950irq/56-dwc_otg_21:01:373
2966299349,13cyclictest0-21swapper/321:55:483
2966299348,14cyclictest7950irq/56-dwc_otg_22:50:303
2966299347,24cyclictest0-21swapper/319:25:243
2966299347,24cyclictest0-21swapper/319:25:243
2966299347,16cyclictest37-21rcuc/322:45:083
2966299347,15cyclictest0-21swapper/319:30:313
2966299347,14cyclictest0-21swapper/319:20:303
2966299346,19cyclictest37-21rcuc/321:30:273
2966299346,19cyclictest37-21rcuc/321:30:273
2966299345,13cyclictest381ktimersoftd/321:15:433
29662993423,4cyclictest381ktimersoftd/322:00:433
2966299342,21cyclictest0-21swapper/323:12:433
29662993413,2cyclictest0-21swapper/320:45:343
2966299341,18cyclictest0-21swapper/321:10:243
2966299341,18cyclictest0-21swapper/320:10:203
29662993411,5cyclictest15990-21awk23:15:503
29662993411,5cyclictest15990-21awk23:15:503
29662993410,12cyclictest381ktimersoftd/320:10:383
2966299340,12cyclictest8050irq/56-dwc_otg_22:30:403
2966299340,12cyclictest8050irq/56-dwc_otg_22:30:403
2966199346,11cyclictest0-21swapper/220:47:222
2966199345,7cyclictest8050irq/56-dwc_otg_21:05:362
2966099349,7cyclictest6800-21cat23:45:191
2966099348,14cyclictest0-21swapper/100:01:591
29660993419,8cyclictest24425-21rm21:29:281
29660993413,4cyclictest0-21swapper/122:26:301
29660993411,4cyclictest0-21swapper/123:32:131
2966299338,15cyclictest774-21lldpd20:50:293
2966299338,15cyclictest774-21lldpd20:50:293
29662993325,5cyclictest30032-21ssh22:15:323
29662993322,4cyclictest8050irq/56-dwc_otg_19:10:353
29661993316,14cyclictest14422-21/usr/sbin/munin19:55:422
2966099338,9cyclictest8050irq/56-dwc_otg_19:35:401
2966099338,9cyclictest8050irq/56-dwc_otg_19:35:401
2966099338,4cyclictest0-21swapper/122:15:441
2966099332,22cyclictest0-21swapper/120:00:471
29660993315,10cyclictest17452-21cat22:00:201
2966299324,14cyclictest13980-21rm23:53:273
2966299322,16cyclictest0-21swapper/319:56:413
29662993216,4cyclictest8050irq/56-dwc_otg_19:40:323
2966299321,19cyclictest0-21swapper/320:03:103
29661993212,11cyclictest125-21systemd-journal21:01:242
2966099327,12cyclictest10656-21memory19:45:421
2966099325,15cyclictest21-21rcuc/120:10:321
2966099321,5cyclictest27708-21acpi00:10:301
2966099321,5cyclictest27708-21acpi00:10:301
2966099321,5cyclictest0-21swapper/120:15:341
29660993212,9cyclictest27477-21grep20:30:481
29660993210,4cyclictest0-21swapper/122:30:321
29660993210,4cyclictest0-21swapper/122:30:321
2966299312,16cyclictest0-21swapper/320:20:433
29662993117,4cyclictest8050irq/56-dwc_otg_19:37:313
29662993117,4cyclictest8050irq/56-dwc_otg_19:37:313
2966299311,21cyclictest0-21swapper/320:56:243
2966199312,20cyclictest0-21swapper/220:27:532
2966199312,20cyclictest0-21swapper/220:27:532
2966099316,5cyclictest7950irq/56-dwc_otg_19:30:421
2966099306,7cyclictest0-21swapper/119:10:221
2966099302,19cyclictest0-21swapper/119:15:331
213472300,2sleep30-21swapper/321:25:343
2966299291,6cyclictest12165-21fschecks_count19:50:363
2966299291,16cyclictest37-21rcuc/320:19:223
2966099294,3cyclictest0-21swapper/119:20:401
2966099292,21cyclictest0-21swapper/120:09:551
2966099283,8cyclictest8050irq/56-dwc_otg_20:20:381
291252283,16sleep10-21swapper/119:06:531
2783722810,12sleep20-21swapper/219:05:282
277562263,12sleep30-21swapper/319:05:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional