You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Fri May 09, 2025 00:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22081998418,26cyclictest0-21swapper/023:30:350
22083998311,29cyclictest8050irq/56-dwc_otg_23:30:352
207252810,4sleep211-21rcu_preempt19:05:382
37932770,5sleep22208399cyclictest23:27:082
225992770,2sleep2301ktimersoftd/223:09:122
2208199753,55cyclictest0-21swapper/022:40:190
22081997328,10cyclictest0-21swapper/021:05:310
19282720,3sleep21926-21df_inode22:40:372
2208199711,57cyclictest2873-21unixbench_multi22:40:500
241552700,2sleep00-21swapper/021:40:480
2208199703,54cyclictest19512-21cut21:35:370
2208199701,58cyclictest2059-21memory21:55:420
2208199701,57cyclictest4428-21fschecks_time19:50:360
2208199701,54cyclictest8323-21munin-plugin-st00:20:190
2208199696,40cyclictest0-21swapper/023:15:470
22084996712,11cyclictest8050irq/56-dwc_otg_21:50:343
22084996620,8cyclictest8050irq/56-dwc_otg_23:05:333
2208199641,35cyclictest21165-21ls00:40:190
22081996317,34cyclictest27659-21/usr/sbin/munin22:30:450
2208199631,50cyclictest13503-21grep22:10:480
22081996313,35cyclictest9-21ksoftirqd/023:10:330
232592620,2sleep00-21swapper/020:40:460
22082996230,15cyclictest23-21ksoftirqd/100:15:341
2208199621,45cyclictest28617-21sed00:00:530
2208199621,45cyclictest28617-21sed00:00:530
22081996115,36cyclictest12966-21if_eth022:55:390
2208199611,4cyclictest5372-21sh23:30:200
2208199601,49cyclictest32694-21sh19:40:190
2208199601,24cyclictest0-21swapper/020:05:190
2208199595,41cyclictest1941-21cut00:10:360
2208199591,30cyclictest16559-21ls20:25:200
22081995910,27cyclictest12510-21unixbench_singl20:10:490
22082995825,30cyclictest0-21swapper/122:55:341
2208199584,26cyclictest31100-21ps21:50:500
2208199582,27cyclictest0-21swapper/023:50:470
22081995814,34cyclictest0-21swapper/022:15:370
2208199581,31cyclictest10467-21sed23:35:520
2208199576,41cyclictest13168-21uniq23:40:380
2208199569,34cyclictest0-21swapper/021:20:420
2208199565,35cyclictest0-21swapper/022:00:330
2208199563,44cyclictest101ktimersoftd/023:23:490
2208199563,27cyclictest23809-21aten_rbpower_po23:55:340
2208199561,39cyclictest12787-21ntp_states00:25:440
2208199561,24cyclictest3050-21ps19:45:440
2208199561,24cyclictest3050-21ps19:45:440
2208299544,23cyclictest0-21swapper/123:35:321
2208199541,28cyclictest896-21snmpd21:30:430
22081995325,4cyclictest0-21swapper/022:45:380
2208199531,2cyclictest0-21swapper/022:20:290
2208499527,38cyclictest0-21swapper/319:25:343
2208299527,24cyclictest0-21swapper/119:35:341
2208199521,35cyclictest0-21swapper/020:55:190
2208199521,25cyclictest26602-21wc19:20:420
22081995212,22cyclictest12552-21missed_timers21:25:470
2208199515,35cyclictest0-21swapper/019:30:400
2208499507,18cyclictest7950irq/56-dwc_otg_19:20:303
22084995025,11cyclictest8050irq/56-dwc_otg_23:30:203
2208499502,19cyclictest0-21swapper/319:35:323
22081995037,3cyclictest0-21swapper/000:20:410
22081995019,7cyclictest8050irq/56-dwc_otg_21:50:190
2208199501,29cyclictest7951-21latency_hist21:20:190
2208199501,28cyclictest20612-21proc_pri23:05:460
2208499497,27cyclictest0-21swapper/320:25:343
2208499493,8cyclictest11902-21ssh21:25:383
22082994917,7cyclictest1336-21latency_hist21:55:201
2208199492,39cyclictest0-21swapper/021:10:270
22081994923,4cyclictest8050irq/56-dwc_otg_20:15:440
2208199491,32cyclictest17867-21kworker/0:119:10:430
22084994823,11cyclictest8050irq/56-dwc_otg_23:22:223
22081994821,4cyclictest8050irq/56-dwc_otg_00:30:370
22081994821,4cyclictest8050irq/56-dwc_otg_00:30:370
2208199481,25cyclictest0-21swapper/019:55:300
22084994722,10cyclictest7950irq/56-dwc_otg_22:50:363
2208299471,19cyclictest7883-21latency_hist21:20:201
2208199472,26cyclictest0-21swapper/022:54:370
2208199472,25cyclictest0-21swapper/019:16:280
2208199472,23cyclictest0-21swapper/000:05:330
22081994721,5cyclictest20483-21wget20:35:320
22081994716,9cyclictest0-21swapper/021:00:330
2208199471,5cyclictest16233-21cut23:00:340
2208199471,23cyclictest24532-21runrttasks22:25:500
2208199471,23cyclictest24532-21runrttasks22:25:500
2208499467,25cyclictest0-21swapper/319:10:323
2208499467,24cyclictest0-21swapper/320:20:303
22084994624,4cyclictest8050irq/56-dwc_otg_20:45:483
22082994622,15cyclictest23-21ksoftirqd/123:50:421
22081994623,4cyclictest7950irq/56-dwc_otg_20:25:350
22081994621,9cyclictest0-21swapper/020:57:450
2208199461,33cyclictest0-21swapper/023:46:310
2208499457,20cyclictest8050irq/56-dwc_otg_21:22:043
22084994517,9cyclictest7950irq/56-dwc_otg_22:37:263
22081994524,4cyclictest7950irq/56-dwc_otg_22:05:420
216722452,13sleep00-21swapper/019:08:320
2208499447,23cyclictest0-21swapper/300:20:333
2208399441,11cyclictest31687-21timerandwakeup23:20:512
2208299447,25cyclictest0-21swapper/122:20:321
2208299447,25cyclictest0-21swapper/121:55:311
2208499436,25cyclictest0-21swapper/323:30:263
2208499433,11cyclictest0-21swapper/321:30:323
22084994313,14cyclictest8050irq/56-dwc_otg_23:40:353
2208299437,22cyclictest0-21swapper/120:20:301
2208199432,23cyclictest0-21swapper/019:40:380
2208199431,34cyclictest0-21swapper/020:46:580
2208199431,23cyclictest10813-21kworker/0:020:06:580
95762421,3sleep20-21swapper/220:05:352
2208499427,23cyclictest0-21swapper/321:40:333
2208499426,16cyclictest8050irq/56-dwc_otg_00:35:513
22084994214,4cyclictest7850irq/56-dwc_otg21:35:403
2208399426,23cyclictest0-21swapper/223:15:322
2208399426,23cyclictest0-21swapper/220:55:352
2208299428,23cyclictest0-21swapper/121:40:331
2208199425,17cyclictest18660-21grep20:30:350
2208199422,21cyclictest0-21swapper/019:25:450
41302411,36sleep24158-21munin-run21:15:192
2208499417,22cyclictest0-21swapper/319:45:343
2208499417,22cyclictest0-21swapper/319:45:343
2208499413,25cyclictest0-21swapper/323:10:333
22084994115,11cyclictest8050irq/56-dwc_otg_22:02:273
22084994111,12cyclictest7950irq/56-dwc_otg_22:48:003
22084994110,17cyclictest8050irq/56-dwc_otg_00:15:473
2208399414,12cyclictest0-21swapper/221:05:312
22083994113,7cyclictest13663-21rm21:26:532
2208299417,22cyclictest0-21swapper/100:35:341
283372400,2sleep10-21swapper/121:46:491
2208499404,12cyclictest0-21swapper/322:10:313
22084994013,12cyclictest8050irq/56-dwc_otg_22:24:213
2208299408,6cyclictest21-21rcuc/122:15:501
2208299405,20cyclictest5585-21ssh23:30:261
2208299404,17cyclictest21-21rcuc/122:02:191
2208499399,12cyclictest8050irq/56-dwc_otg_22:34:483
2208499399,11cyclictest8050irq/56-dwc_otg_23:03:553
2208499396,16cyclictest14128-21ls20:15:463
22084993916,9cyclictest8050irq/56-dwc_otg_23:15:443
2208499391,24cyclictest0-21swapper/322:25:343
2208499391,24cyclictest0-21swapper/322:25:343
22084993910,14cyclictest8050irq/56-dwc_otg_21:10:433
2208399397,22cyclictest0-21swapper/220:10:342
2208399396,5cyclictest8050irq/56-dwc_otg_21:15:482
2208399394,22cyclictest7850irq/56-dwc_otg20:05:192
2208399391,4cyclictest0-21swapper/222:30:452
2208299398,6cyclictest0-21swapper/121:25:361
2208299396,24cyclictest0-21swapper/119:45:341
2208299396,24cyclictest0-21swapper/119:45:341
2208299395,4cyclictest8684-21ssh22:05:351
2208299395,12cyclictest221ktimersoftd/120:10:191
2208299392,21cyclictest7950irq/56-dwc_otg_21:35:201
22082993910,13cyclictest7950irq/56-dwc_otg_21:35:211
2208499389,5cyclictest9344-21basename23:35:373
2208499387,11cyclictest0-21swapper/321:56:343
22084993819,4cyclictest8050irq/56-dwc_otg_19:32:023
2208399389,18cyclictest22232-21aten_rbpower_en20:40:322
2208399387,23cyclictest7950irq/56-dwc_otg_22:27:312
2208399387,23cyclictest7950irq/56-dwc_otg_22:27:312
22083993816,13cyclictest20356-21fschecks_count23:50:362
22083993813,13cyclictest30923-21aten_rbpower_en00:05:332
2208299388,19cyclictest221ktimersoftd/122:50:381
2208299385,23cyclictest27927-21ls00:00:431
2208299385,23cyclictest27927-21ls00:00:431
2208299383,9cyclictest8050irq/56-dwc_otg_23:00:311
2208299382,24cyclictest23245-21ssh22:25:321
2208299382,24cyclictest23245-21ssh22:25:321
2208299381,14cyclictest28677-21grep19:25:481
2208499373,3cyclictest8050irq/56-dwc_otg_19:55:423
2208399379,19cyclictest13445-21unixbench-2d22:10:472
2208399371,3cyclictest0-21swapper/222:55:412
2208299376,5cyclictest21-21rcuc/120:45:441
22082993716,5cyclictest2338050/usr/sbin/munin23:10:371
2208499369,13cyclictest8050irq/56-dwc_otg_22:58:343
2208499368,15cyclictest0-21swapper/321:05:313
2208499365,19cyclictest30886-21ssh00:05:333
2208499365,15cyclictest14658-21ssh00:29:333
2208499363,19cyclictest28953-21ssh00:01:343
2208499363,19cyclictest28953-21ssh00:01:343
2208499361,4cyclictest8050irq/56-dwc_otg_21:45:413
22084993613,5cyclictest381ktimersoftd/320:40:333
2208399367,17cyclictest0-21swapper/221:00:342
2208399364,5cyclictest8050irq/56-dwc_otg_20:55:202
22083993619,5cyclictest8088-21sh00:19:482
22083993613,5cyclictest3926-21latency_hist19:50:202
22083993613,5cyclictest3926-21latency_hist19:50:202
2208299368,7cyclictest0-21swapper/123:55:311
2208299366,16cyclictest8050irq/56-dwc_otg_23:45:381
2208299364,6cyclictest21-21rcuc/123:17:221
2208299363,13cyclictest21-21rcuc/123:25:501
2208299361,7cyclictest0-21swapper/123:23:141
22082993617,10cyclictest878-21runrttasks20:50:401
2208299361,24cyclictest4273-21ls19:50:331
22084993514,10cyclictest7950irq/56-dwc_otg_23:46:323
2208399357,14cyclictest14259-21grep20:15:472
2208399354,21cyclictest8884-21ls22:05:362
22083993517,4cyclictest16216-21aten_rbpower_po23:00:342
22083993515,13cyclictest17727-21ssh21:32:522
2208399351,4cyclictest23396-21apt-get23:10:312
22083993514,4cyclictest9337-21aten_rbpower_en23:35:372
22083993512,13cyclictest22571-21http21:40:192
2208299356,11cyclictest21-21rcuc/100:26:261
2208299353,7cyclictest0-21swapper/121:20:341
2208299352,5cyclictest7950irq/56-dwc_otg_22:40:321
2208299352,29cyclictest8050irq/56-dwc_otg_23:05:341
2208299352,16cyclictest0-21swapper/120:15:411
2208299352,15cyclictest8050irq/56-dwc_otg_21:10:381
2208299351,5cyclictest23-21ksoftirqd/122:45:341
2208499345,11cyclictest7950irq/56-dwc_otg_23:55:193
2208499344,10cyclictest0-21swapper/323:55:333
2208499342,5cyclictest2881-21ssh00:10:513
2208399349,3cyclictest0-21swapper/200:35:532
2208399342,6cyclictest8050irq/56-dwc_otg_19:35:352
2208399341,5cyclictest5011-21apt22:00:322
22083993415,9cyclictest15252-21fschecks_time20:20:342
2208399341,23cyclictest16915-21switchtime22:15:472
2208399341,22cyclictest0-21swapper/222:50:512
2208299349,22cyclictest0-21swapper/119:10:321
2208299346,24cyclictest12367-21ls20:10:481
2208299345,16cyclictest2936850wget19:30:341
2208299344,5cyclictest23-21ksoftirqd/100:13:011
22082993420,4cyclictest221ktimersoftd/120:35:411
2208299341,21cyclictest26733-21timerwakeupswit19:20:431
22084993319,4cyclictest8050irq/56-dwc_otg_21:15:283
22084993317,4cyclictest8050irq/56-dwc_otg_22:18:373
2208399335,5cyclictest19580-21/usr/sbin/munin22:20:292
2208399334,6cyclictest30006-21grep19:30:422
2208399333,18cyclictest0-21swapper/221:50:492
2208399331,8cyclictest22968-21ssh21:40:312
22083993310,4cyclictest0-21swapper/200:30:492
22083993310,4cyclictest0-21swapper/200:30:492
2208299336,15cyclictest0-21swapper/100:20:331
2208299335,12cyclictest7850irq/56-dwc_otg22:32:061
2208299335,12cyclictest21-21rcuc/120:42:521
2208299333,15cyclictest8050irq/56-dwc_otg_19:55:401
22082993312,9cyclictest8050irq/56-dwc_otg_00:34:351
22082993312,9cyclictest8050irq/56-dwc_otg_00:34:351
2208299331,26cyclictest13084-21ls23:40:351
22082993311,13cyclictest221ktimersoftd/100:07:401
22082993310,7cyclictest8050irq/56-dwc_otg_22:40:041
17032331,27sleep20-21swapper/200:10:212
2208499322,21cyclictest0-21swapper/300:32:463
2208499322,21cyclictest0-21swapper/300:32:463
2208499321,8cyclictest0-21swapper/320:50:313
2208399326,18cyclictest7850irq/56-dwc_otg21:20:402
2208399324,7cyclictest8050irq/56-dwc_otg_19:20:442
2208399324,5cyclictest1466-21latency_hist22:40:192
2208399322,20cyclictest0-21swapper/220:46:072
22083993219,4cyclictest21934-21sh20:40:192
2208399321,4cyclictest23815-21ssh23:55:352
22083993210,10cyclictest0-21swapper/223:40:422
2208299326,15cyclictest7950irq/56-dwc_otg_22:10:471
2208299321,14cyclictest7950irq/56-dwc_otg_19:15:381
2208499317,22cyclictest0-21swapper/321:00:243
2208499317,15cyclictest0-21swapper/319:40:323
2208499316,4cyclictest0-21swapper/322:42:233
2208499312,19cyclictest0-21swapper/320:06:393
22084993112,7cyclictest8050irq/56-dwc_otg_22:05:293
2208399319,4cyclictest0-21swapper/223:45:372
2208399317,10cyclictest8050irq/56-dwc_otg_22:45:382
2208399316,15cyclictest0-21swapper/200:25:342
2208399315,10cyclictest7950irq/56-dwc_otg_00:05:132
2208399315,10cyclictest7950irq/56-dwc_otg_00:05:132
2208399311,5cyclictest4518-21if_eth019:50:372
2208399311,19cyclictest9634-21timerandwakeup00:20:472
2208399311,13cyclictest0-21swapper/221:59:172
2208299317,22cyclictest0-21swapper/119:40:321
2208299316,15cyclictest21-21rcuc/120:04:431
2208499303,12cyclictest12364-21grep20:10:483
2208499301,5cyclictest0-21swapper/320:55:343
2208499301,20cyclictest0-21swapper/320:00:583
2208399302,6cyclictest27524-21/usr/sbin/munin19:25:452
2208399301,5cyclictest7950irq/56-dwc_otg_19:15:482
2208399301,11cyclictest0-21swapper/221:47:342
2208299307,11cyclictest0-21swapper/120:55:421
2208299305,5cyclictest7950irq/56-dwc_otg_21:05:351
2208299305,5cyclictest7950irq/56-dwc_otg_21:00:421
2208499297,17cyclictest0-21swapper/319:50:323
2208499292,19cyclictest0-21swapper/319:18:043
2208399297,4cyclictest8050irq/56-dwc_otg_19:10:452
2208399292,19cyclictest0-21swapper/220:32:292
2208299297,5cyclictest8050irq/56-dwc_otg_20:25:431
2208299291,13cyclictest878-21runrttasks20:32:521
2208499283,5cyclictest7950irq/56-dwc_otg_20:35:433
2208399282,16cyclictest0-21swapper/219:44:062
2208399281,24cyclictest17364-21missed_timers20:25:422
2208399272,6cyclictest0-21swapper/219:59:052
215512273,15sleep30-21swapper/319:06:573
202382276,13sleep10-21swapper/119:05:221
2208499262,6cyclictest8050irq/56-dwc_otg_20:30:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional