You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 10:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Sun Apr 19, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
534622640,1sleep00-21swapper/004:26:110
940621040,5sleep1639999cyclictest09:34:561
940621040,5sleep1639999cyclictest09:34:561
6398998682,3cyclictest7750irq/56-dwc_otg08:20:210
6398998318,46cyclictest23352-21sed07:15:460
6398998317,58cyclictest27289-21sed09:56:050
6398998317,58cyclictest27289-21sed09:56:050
6398998217,59cyclictest7685-21tail06:01:130
639899814,31cyclictest9960-21kworker/u8:208:05:570
6398998115,40cyclictest12933-21unixbench_multi04:46:150
6398998012,58cyclictest0-21swapper/009:00:530
6398998012,58cyclictest0-21swapper/009:00:530
6398997818,42cyclictest5082-21fschecks_time08:11:020
6398997817,47cyclictest13880-21aten_rbpower_en06:20:570
6398997813,51cyclictest23999-21sh06:37:450
6398997811,55cyclictest0-21swapper/004:36:010
6398997717,47cyclictest15339-21grep07:06:030
6398997716,51cyclictest30595-21taskset05:36:080
6398997712,61cyclictest25743-21fschecks_time08:36:020
6398997712,61cyclictest25743-21fschecks_time08:36:020
6398997617,36cyclictest6809-21awk09:31:080
6398997617,36cyclictest6809-21awk09:31:080
6398997616,48cyclictest8463-21ssh06:57:280
6398997616,48cyclictest8463-21ssh06:57:280
6398997613,44cyclictest32575-21latency_hist08:05:460
6398997523,38cyclictest21220-21/usr/sbin/munin05:11:030
6398997521,48cyclictest0-21swapper/005:45:590
6398997514,42cyclictest0-21swapper/007:25:490
6398997513,50cyclictest31645-21sh06:46:130
639899747,52cyclictest0-21swapper/008:35:460
6398997418,39cyclictest24674-21ssh07:55:590
6398997418,39cyclictest24674-21ssh07:55:590
6398997414,41cyclictest0-21swapper/009:55:370
6398997411,43cyclictest11148-21grep09:36:100
219902740,2sleep021989-21ssh09:11:000
6398997314,50cyclictest20492-21/usr/sbin/munin07:51:100
6398997313,48cyclictest14807-21ssh08:22:070
6398997313,41cyclictest0-21swapper/009:25:580
199712730,1sleep20-21swapper/209:46:372
6398997223,40cyclictest9196-21/usr/sbin/munin07:36:150
6398997217,40cyclictest28005-21runrttasks09:17:210
639899719,51cyclictest0-21swapper/008:49:530
6398997122,24cyclictest0-21swapper/007:45:570
6398997121,39cyclictest20935-21sh09:09:320
6398997116,38cyclictest11897-21runrttasks08:58:090
6398997115,45cyclictest0-21swapper/006:40:590
6398997114,47cyclictest19237-21ssh08:27:400
6398997112,43cyclictest5452-21sh06:54:400
6398997017,37cyclictest27086-21ssh07:20:100
6398997015,44cyclictest20245-21ssh09:47:070
6398997012,42cyclictest0-21swapper/007:21:010
6398997012,42cyclictest0-21swapper/007:21:010
6398997011,43cyclictest7486-21sh08:52:390
6398996917,37cyclictest32416-21taskset09:22:510
6398996912,47cyclictest18714-21timerwakeupswit06:31:140
6398996860,5cyclictest0-21swapper/005:30:580
6398996813,40cyclictest0-21swapper/005:41:400
221372680,3sleep122092-21ssh09:11:011
639899678,40cyclictest709-21kworker/u8:006:00:460
639899678,40cyclictest709-21kworker/u8:006:00:460
6398996718,42cyclictest10559-21latency04:41:060
6398996713,40cyclictest0-21swapper/007:31:320
6398996613,44cyclictest15940-21ssh07:45:360
6398996515,36cyclictest11330-21taskset07:01:060
639899642,46cyclictest0-21swapper/009:40:570
6398996421,34cyclictest24150-21chrt05:19:510
6398996421,34cyclictest24150-21chrt05:19:510
6398996415,35cyclictest0-21swapper/006:26:300
6398996411,41cyclictest0-21swapper/008:41:370
6401996322,22cyclictest2160-21/usr/sbin/munin09:26:002
639999635,23cyclictest7995-21ssh07:35:571
6398996315,37cyclictest19061-21awk05:05:560
6401996219,8cyclictest7950irq/56-dwc_otg_09:50:582
6398996213,29cyclictest25767-21seq05:22:330
639899617,42cyclictest0-21swapper/006:18:260
6398996112,32cyclictest0-21swapper/005:26:050
639899609,39cyclictest0-21swapper/004:35:120
639899608,45cyclictest0-21swapper/005:00:040
6398996013,9cyclictest0-21swapper/004:54:470
6398996010,40cyclictest0-21swapper/005:02:460
6398996010,34cyclictest0-21swapper/006:06:490
6401995821,11cyclictest30837-21/usr/sbin/munin05:36:112
6398995813,23cyclictest11480-21chrt06:12:080
6401995730,12cyclictest32135-21apt-get07:25:572
640199567,30cyclictest0-21swapper/205:30:582
6401995517,19cyclictest8808-21ssh09:33:522
6401995517,19cyclictest8808-21ssh09:33:522
640299548,10cyclictest7850irq/56-dwc_otg_07:35:573
6401995419,20cyclictest16403-21aten_rbpower_cu07:45:592
6401995417,21cyclictest16345-21sh08:24:562
6398995314,36cyclictest4199-21chrt05:52:480
6401995221,7cyclictest12756-21ls04:46:132
6401995216,21cyclictest22487-21rm07:14:372
6401995215,24cyclictest26520-21ssh07:57:152
6401995215,24cyclictest26520-21ssh07:57:152
6401995215,21cyclictest31063-21ssh08:41:352
6401995214,11cyclictest30104-21ssh07:23:012
6401995214,11cyclictest30104-21ssh07:23:012
640299513,10cyclictest7950irq/56-dwc_otg_04:40:583
640199517,28cyclictest486-21ssh08:05:522
6401995117,19cyclictest29772-21rm09:20:352
6401995116,15cyclictest16406-21ssh07:06:262
6401995116,13cyclictest25926-21irqstats08:36:042
6401995116,13cyclictest25926-21irqstats08:36:042
6401995115,22cyclictest25355-21ssh09:15:042
6401995115,14cyclictest27326-21rm07:20:362
6401995113,23cyclictest24220-21sh06:38:072
6401995112,25cyclictest30801-21ssh08:02:322
6401995016,21cyclictest20863-21/usr/sbin/munin05:11:072
6401994925,18cyclictest19071-21apt-get05:05:572
6401994915,10cyclictest9190-21ssh08:55:452
6399994918,15cyclictest7950irq/56-dwc_otg_09:51:111
6399994911,23cyclictest0-21swapper/106:50:031
54992490,6sleep20-21swapper/204:26:132
6401994817,12cyclictest12092-21ssh08:58:302
6401994815,11cyclictest1038-21rm06:49:252
6401994812,21cyclictest7850irq/56-dwc_otg_09:55:472
6401994812,21cyclictest7850irq/56-dwc_otg_09:55:472
640199479,23cyclictest7950irq/56-dwc_otg_06:41:032
640199476,27cyclictest0-21swapper/205:21:012
6401994718,11cyclictest10612-21/usr/sbin/munin04:41:062
6401994715,16cyclictest8492-21ssh06:57:312
6401994715,16cyclictest8492-21ssh06:57:312
6401994715,14cyclictest21850-21ssh07:51:152
6401994715,12cyclictest11604-21sh07:40:242
6401994715,11cyclictest19392-21ssh06:32:092
6401994714,13cyclictest7437-21rm08:13:472
6401994712,13cyclictest13669-21apt-get09:00:562
6401994712,13cyclictest13669-21apt-get09:00:562
639999476,6cyclictest0-21swapper/105:50:571
640199467,25cyclictest0-21swapper/206:20:562
640199467,24cyclictest0-21swapper/206:16:002
6401994618,12cyclictest20732-21ssh09:09:102
6401994616,19cyclictest17554-21ssh09:44:452
6401994614,12cyclictest7096-21ssh07:34:452
6401994612,12cyclictest1615-21latency_hist09:25:462
6401994610,19cyclictest7950irq/56-dwc_otg_09:36:402
639999461,8cyclictest13176-21sed06:16:151
6399994615,13cyclictest0-21swapper/108:25:061
6401994519,12cyclictest23678-21ssh08:33:132
6401994518,18cyclictest19248-21ssh08:27:422
6401994517,17cyclictest4845-21ssh08:50:232
6401994515,17cyclictest6590-21/usr/sbin/munin06:01:122
6401994513,18cyclictest5456-21ssh06:54:402
640299443,9cyclictest0-21swapper/305:55:593
640299443,9cyclictest0-21swapper/305:55:593
640299441,5cyclictest7950irq/56-dwc_otg_08:50:563
6401994417,7cyclictest4689-21apt-get05:55:592
6401994417,7cyclictest4689-21apt-get05:55:592
6401994417,17cyclictest13351-21ssh07:03:332
639999446,25cyclictest0-21swapper/106:30:591
639999441,6cyclictest26170-21aten_rbpower_en06:40:591
6401994317,11cyclictest12117-21taskset08:19:462
6401994313,11cyclictest11512-21taskset06:12:332
6399994332,8cyclictest221ktimersoftd/105:40:561
639999433,21cyclictest0-21swapper/108:05:571
640199425,23cyclictest0-21swapper/205:00:282
639999424,6cyclictest0-21swapper/107:16:031
639999421,7cyclictest32589-21idleruntime-cro08:05:451
640299412,7cyclictest908-21runrttasks06:51:073
640199419,12cyclictest18801-21seq05:05:382
640199417,20cyclictest0-21swapper/207:42:542
6401994112,10cyclictest7850irq/56-dwc_otg_06:29:272
639999411,6cyclictest13361-21awk09:00:461
639999411,5cyclictest917-21snmpd04:56:001
51712410,2sleep30-21swapper/309:29:473
640199408,21cyclictest2575-21chrt05:50:082
639999403,4cyclictest0-21swapper/109:26:021
639999402,10cyclictest21179-21missed_timers07:51:061
639999401,6cyclictest27198-21ssh09:56:041
639999401,6cyclictest27198-21ssh09:56:041
639999401,6cyclictest0-21swapper/106:11:111
639999401,5cyclictest10369-21munin-run07:00:461
639999401,5cyclictest10369-21munin-run07:00:461
204472400,1sleep30-21swapper/307:50:503
6401993911,14cyclictest7850irq/56-dwc_otg_05:42:092
639999397,22cyclictest0-21swapper/104:40:591
639999396,14cyclictest7950irq/56-dwc_otg_09:46:041
639999395,21cyclictest7750irq/56-dwc_otg05:46:001
6399993925,5cyclictest1368550wget04:51:001
6399993919,5cyclictest19181-21aten_rbpower_vo05:06:001
6399993916,5cyclictest25534-21unixbench-2d05:21:151
6399993915,7cyclictest0-21swapper/108:36:101
6399993915,7cyclictest0-21swapper/108:36:101
639999391,25cyclictest28324-21grep05:31:011
6399993911,7cyclictest11295-21ssh09:36:121
639999391,10cyclictest0-21swapper/105:55:581
639999391,10cyclictest0-21swapper/105:55:581
640299385,5cyclictest482050kworker/u9:107:10:573
6402993817,5cyclictest14488-21sh07:42:573
640299381,26cyclictest18188-21ls09:45:463
640199382,25cyclictest0-21swapper/204:30:592
639999383,5cyclictest26745-21missed_timers09:16:081
639999382,6cyclictest29931-21/usr/sbin/munin05:35:591
639999381,9cyclictest8902-21ntpq04:36:081
6399993816,5cyclictest18080-21fschecks_count09:06:001
6399993816,11cyclictest25752-21latency_hist06:40:461
640299379,19cyclictest0-21swapper/306:58:163
640299379,19cyclictest0-21swapper/306:58:163
640299372,3cyclictest0-21swapper/308:32:323
6402993720,10cyclictest2558-21ssh08:07:273
6402993718,13cyclictest39-21ksoftirqd/307:26:023
6401993722,8cyclictest13847-21cpuspeed04:51:012
6399993728,6cyclictest221ktimersoftd/106:00:561
639999372,14cyclictest0-21swapper/107:45:471
6399993711,12cyclictest6455-21ssh08:51:121
6402993612,6cyclictest39-21ksoftirqd/306:20:453
639999362,5cyclictest0-21swapper/108:47:311
6399993623,9cyclictest6397-21cyclictest05:00:571
639999361,6cyclictest28574-21ntp_states07:21:101
639999361,6cyclictest28574-21ntp_states07:21:101
639999361,6cyclictest2802-21memory06:51:071
639999361,24cyclictest11347-21ls07:01:071
6399993611,5cyclictest23-21ksoftirqd/108:11:091
6399993610,9cyclictest21-21rcuc/106:21:041
640299357,13cyclictest0-21swapper/307:46:043
640299355,20cyclictest30243-21ssh09:20:593
640299355,12cyclictest0-21swapper/309:50:273
6402993527,5cyclictest39-21ksoftirqd/307:21:033
6402993527,5cyclictest39-21ksoftirqd/307:21:033
640299352,3cyclictest7950irq/56-dwc_otg_04:35:593
640299351,7cyclictest30608-21ssh06:46:013
6402993517,5cyclictest13222-21sh07:03:213
6402993516,11cyclictest10366-21/usr/sbin/munin08:56:103
6402993513,15cyclictest30608-21latency_hist10:00:463
6402993513,15cyclictest30608-21latency_hist10:00:463
640299351,23cyclictest22872-21grep09:51:033
640199357,13cyclictest0-21swapper/206:10:062
640199355,21cyclictest0-21swapper/205:16:052
640199355,21cyclictest0-21swapper/205:16:052
640199354,11cyclictest9574-21taskset04:38:032
6401993514,11cyclictest4421-21taskset05:55:412
6401993511,15cyclictest27599-21runrttasks05:27:522
639999357,5cyclictest19270-21sh07:10:591
639999353,8cyclictest3665-21latency_hist07:30:461
639999352,6cyclictest7484-21sed04:31:131
6399993525,7cyclictest7656-21munin-plugin-st07:35:451
639999351,7cyclictest10865-21ssh08:17:291
639999351,5cyclictest20935-21cut05:10:591
134822352,29sleep10-21swapper/107:41:151
640299344,11cyclictest0-21swapper/304:55:593
640299343,21cyclictest0-21swapper/308:37:353
640299343,21cyclictest0-21swapper/308:37:353
6402993426,6cyclictest21716-21latency_hist09:10:453
640299341,7cyclictest7545-21tail04:31:133
639999346,13cyclictest7950irq/56-dwc_otg_07:08:441
6399993412,5cyclictest0-21swapper/108:41:101
640299336,10cyclictest7950irq/56-dwc_otg_06:36:143
640299334,22cyclictest7950irq/56-dwc_otg_05:50:553
640299332,15cyclictest7850irq/56-dwc_otg_07:57:203
640299332,15cyclictest7850irq/56-dwc_otg_07:57:203
640299331,6cyclictest0-21swapper/309:18:573
640299331,4cyclictest7850irq/56-dwc_otg_05:36:003
640299331,20cyclictest7950irq/56-dwc_otg_09:12:553
639999337,4cyclictest24984-21ssh07:56:031
639999337,4cyclictest24984-21ssh07:56:031
639999335,17cyclictest0-21swapper/109:22:281
639999335,11cyclictest7850irq/56-dwc_otg_09:41:101
639999331,3cyclictest7750irq/56-dwc_otg08:31:071
6399993312,4cyclictest7950irq/56-dwc_otg_08:25:571
640299326,4cyclictest0-21swapper/308:13:013
640299322,20cyclictest17570-21ssh06:30:593
640299322,18cyclictest7850irq/56-dwc_otg_05:46:003
640299321,6cyclictest37-21rcuc/306:01:023
6402993215,6cyclictest39-21ksoftirqd/309:01:063
6402993215,6cyclictest39-21ksoftirqd/309:01:063
640299321,4cyclictest0-21swapper/307:16:033
6402993212,13cyclictest274772chrt05:26:183
640299321,12cyclictest7950irq/56-dwc_otg_05:15:463
639999327,4cyclictest7750irq/56-dwc_otg09:01:081
639999327,4cyclictest7750irq/56-dwc_otg09:01:081
639999324,7cyclictest7850irq/56-dwc_otg_05:26:001
639999322,6cyclictest12059-21df_inode04:46:021
640299317,12cyclictest0-21swapper/306:43:373
640299315,5cyclictest7850irq/56-dwc_otg_04:51:033
640299312,8cyclictest0-21swapper/305:40:573
6402993120,4cyclictest7950irq/56-dwc_otg_08:46:253
640299311,7cyclictest7950irq/56-dwc_otg_06:10:453
640299311,5cyclictest12158-21fschecks_time04:46:033
6402993115,5cyclictest17196-21diskmemload08:25:533
640299311,23cyclictest0-21swapper/308:02:563
6402993111,14cyclictest9853-21ssh09:35:463
6402993111,14cyclictest9853-21ssh09:35:463
6402993110,11cyclictest381ktimersoftd/309:40:453
639999312,5cyclictest8972-21taskset06:06:061
639999312,15cyclictest7850irq/56-dwc_otg_05:15:591
639999312,15cyclictest7850irq/56-dwc_otg_05:15:591
640299309,9cyclictest0-21swapper/308:19:273
640299309,14cyclictest0-21swapper/308:42:173
640299301,6cyclictest28122-21cut05:30:583
6402993012,10cyclictest908-21runrttasks08:23:593
640299292,5cyclictest7750irq/56-dwc_otg05:06:003
6402992916,8cyclictest17196-21diskmemload07:31:093
640299291,5cyclictest22738-21cut05:15:583
640299291,5cyclictest22738-21cut05:15:583
640299291,18cyclictest16014-21timerandwakeup07:06:133
640299291,15cyclictest7950irq/56-dwc_otg_06:26:133
639999292,3cyclictest0-21swapper/106:26:091
60122296,14sleep10-21swapper/104:29:201
640299281,5cyclictest17341-21aten_rbpower_po05:01:003
6402992812,5cyclictest381ktimersoftd/306:10:573
640299271,17cyclictest917-21snmpd06:21:173
45642273,15sleep30-21swapper/304:25:513
640299251,9cyclictest0-21swapper/305:20:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional