You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-30 - 07:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Sat Sep 30, 2023 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2443399945,10cyclictest0-21swapper/023:15:440
24433999420,62cyclictest31037-21timerwakeupswit20:56:030
24433999019,58cyclictest1230-21wget21:05:440
24433999015,57cyclictest32321-21wc21:00:540
24433999015,57cyclictest32321-21wc21:00:540
295582860,5sleep32443799cyclictest21:45:463
295582860,5sleep32443799cyclictest21:45:463
24433998623,40cyclictest17387-21/usr/sbin/munin22:55:540
24433998620,50cyclictest13264-21cron22:50:260
24433998617,53cyclictest8201-21cron23:25:260
24433998617,53cyclictest8201-21cron23:25:260
24433998525,45cyclictest16645-21unixbench_multi22:11:050
24433998518,54cyclictest12675-21/usr/sbin/munin23:30:470
24433998411,60cyclictest14903-21grep20:11:040
24433998411,60cyclictest14903-21grep20:11:040
24433998318,50cyclictest12210-21fschecks_count20:05:470
24433998222,51cyclictest31624-21cat23:55:260
24433998221,50cyclictest10088-21latency_hist20:00:260
24433998217,53cyclictest4536-21grep00:01:040
24433998129,42cyclictest24432-21cyclictest19:10:270
24433998025,38cyclictest5790-21/usr/sbin/munin19:46:020
24433998020,48cyclictest11911-21/usr/sbin/munin22:05:460
24433998016,48cyclictest25276-21runrttasks21:40:190
24433998013,56cyclictest0-21swapper/019:20:580
24433997919,45cyclictest6693-21cron21:15:250
24433997916,49cyclictest2736-21diskmemload22:40:540
24433997916,37cyclictest22694-21ls00:26:010
161942790,1sleep10-21swapper/123:35:341
24433997722,41cyclictest16257-21/usr/sbin/munin23:35:510
24433997619,48cyclictest24105-21dpkg23:45:430
24433997619,48cyclictest24105-21dpkg23:45:430
24433997617,43cyclictest19416-21ssh22:15:500
24433997617,39cyclictest26263-21taskset19:15:420
24433997617,11cyclictest3987-21apt21:55:250
24433997616,46cyclictest0-21swapper/000:31:090
24433997616,38cyclictest32129-21ls23:55:480
24433997519,35cyclictest0-21swapper/021:30:160
24433997422,40cyclictest8648-21wget23:25:460
24433997419,44cyclictest32094-21rm23:13:590
24433997416,48cyclictest22752-21ssh23:01:130
2443399739,40cyclictest0-21swapper/022:50:270
2443399739,40cyclictest0-21swapper/022:50:270
24433997321,40cyclictest21255-21rm00:25:190
24433997320,38cyclictest19163-21/usr/sbin/munin20:26:020
24433997316,46cyclictest0-21swapper/022:38:240
24433997315,41cyclictest0-21swapper/020:00:280
24433997313,48cyclictest0-21swapper/022:34:310
2443399729,49cyclictest0-21swapper/023:09:400
24433997223,32cyclictest1292-21runrttasks21:23:050
24433997217,41cyclictest29856-21ssh22:29:180
24433997214,46cyclictest0-21swapper/000:39:500
24433997118,34cyclictest2999-21basename19:40:450
24433997116,44cyclictest23360-21ssh23:44:460
24433997114,46cyclictest0-21swapper/022:03:130
24433997114,46cyclictest0-21swapper/020:48:380
24433997114,45cyclictest25602-21ssh21:40:340
319952700,1sleep20-21swapper/200:40:082
24435997012,48cyclictest22058-21aten_rbpower_po21:35:452
24433997017,39cyclictest0-21swapper/000:14:430
24433997015,42cyclictest4734-21sh21:55:510
24433997014,48cyclictest2736-21diskmemload21:15:410
24433996922,32cyclictest20336-21ssh21:32:460
24433996922,32cyclictest20336-21ssh21:32:460
24433996914,47cyclictest0-21swapper/000:06:290
24435996821,37cyclictest20387-21ssh21:32:572
24435996821,37cyclictest20387-21ssh21:32:572
2443399689,44cyclictest0-21swapper/020:20:470
24433996817,40cyclictest31516-21taskset21:47:450
24433996817,40cyclictest31516-21taskset21:47:450
24433996813,38cyclictest16314-21chrt00:17:270
24433996813,38cyclictest16314-21chrt00:17:270
24437996727,22cyclictest32455-21ls21:00:563
24437996727,22cyclictest32455-21ls21:00:563
24437996628,21cyclictest6046-21ssh00:04:083
24433996611,37cyclictest25875-21rm22:24:060
24433996517,30cyclictest0-21swapper/019:39:240
2443599641,14cyclictest22049-21fschecks_time00:25:502
2443399649,44cyclictest0-21swapper/019:52:340
24433996420,30cyclictest0-21swapper/020:51:100
24433996415,33cyclictest0-21swapper/020:31:240
24437996325,20cyclictest12127-21/usr/sbin/munin22:05:513
24437996321,26cyclictest17012-21taskset20:19:103
2443599631,15cyclictest0-21swapper/220:05:592
2443599631,11cyclictest0-21swapper/219:25:262
24433996213,36cyclictest0-21swapper/020:41:120
24437996126,21cyclictest14939-21ssh23:33:303
24437996123,22cyclictest11098-21rm23:28:293
2443599618,9cyclictest8312-21fschecks_count22:00:512
2443599612,18cyclictest0-21swapper/222:45:332
24433996115,39cyclictest0-21swapper/020:16:450
24433996112,30cyclictest0-21swapper/020:38:450
24437996028,11cyclictest19031-21ssh23:38:583
2443599606,11cyclictest13584-21ls23:31:012
2443599605,4cyclictest8050irq/56-dwc_otg_23:05:472
2443599601,15cyclictest0-21swapper/221:20:492
2443799599,34cyclictest0-21swapper/321:35:323
2443599591,18cyclictest0-21swapper/219:50:262
2443499597,14cyclictest0-21swapper/120:26:011
24437995829,17cyclictest4576-21df_inode21:55:483
24437995827,11cyclictest20481-21ssh21:33:083
24437995827,11cyclictest20481-21ssh21:33:083
24437995824,13cyclictest28951-21aten_rbpower_vo00:35:463
24437995822,22cyclictest16807-21ssh21:28:083
24437995821,22cyclictest2736-21diskmemload23:09:423
2443599584,11cyclictest14619-21ls22:50:582
2443599584,11cyclictest14619-21ls22:50:582
2443599582,16cyclictest0-21swapper/220:43:512
2443599581,48cyclictest0-21swapper/219:55:252
2443599581,10cyclictest7507-21if_err_eth021:15:512
24433995811,30cyclictest0-21swapper/019:31:290
24437995735,9cyclictest26280-21apt-get19:15:463
24437995728,20cyclictest30201-21ssh23:52:443
24437995725,12cyclictest18080-21ssh22:13:583
24437995723,15cyclictest1341-21snmpd22:21:523
24437995722,20cyclictest13743-21ssh00:14:453
24437995721,11cyclictest19705-21irqstats22:15:543
24437995717,9cyclictest7950irq/56-dwc_otg23:41:483
2443599571,6cyclictest8150irq/56-dwc_otg_21:56:002
2443599571,48cyclictest0-21swapper/222:58:422
2443599571,3cyclictest0-21swapper/222:37:382
2443599571,10cyclictest7950irq/56-dwc_otg22:06:002
24433995719,24cyclictest31311-21taskset19:28:510
24437995627,12cyclictest22629-21/usr/sbin/munin00:26:003
24437995626,12cyclictest13892-21/usr/sbin/munin22:50:473
24437995626,12cyclictest13892-21/usr/sbin/munin22:50:473
24437995620,12cyclictest1305650ssh21:23:093
2443599565,11cyclictest8150irq/56-dwc_otg_21:25:462
2443599562,6cyclictest8050irq/56-dwc_otg_21:50:262
2443599562,6cyclictest8050irq/56-dwc_otg_21:50:262
2443599562,4cyclictest8150irq/56-dwc_otg_00:05:562
2443599561,9cyclictest0-21swapper/220:39:132
2443599561,5cyclictest15229-21grep00:16:012
2443599561,5cyclictest15229-21grep00:16:012
2443599561,45cyclictest0-21swapper/223:28:182
2443599561,20cyclictest0-21swapper/223:44:382
2443599561,17cyclictest0-21swapper/223:39:272
2443599561,13cyclictest0-21swapper/219:15:532
2443599561,12cyclictest25219-21ps19:10:582
24437995523,11cyclictest28751-21ssh22:27:033
24437995522,10cyclictest3023-21ssh21:10:323
24437995521,12cyclictest2782-21ssh21:52:513
24437995520,13cyclictest13360-21cat22:50:263
24437995518,10cyclictest0-21swapper/322:34:573
2443599555,11cyclictest10854-21ntp_states20:00:592
2443599553,8cyclictest8150irq/56-dwc_otg_21:50:452
2443599553,4cyclictest24819-21sh22:22:002
2443599553,11cyclictest28065-21ssh00:34:282
2443599552,11cyclictest0-21swapper/221:10:592
2443599551,3cyclictest8050irq/56-dwc_otg_22:32:062
2443599551,17cyclictest0-21swapper/219:39:552
2443599551,15cyclictest0-21swapper/200:23:082
2443599551,12cyclictest0-21swapper/221:40:442
24437995431,11cyclictest24687-21diskstats19:10:503
24437995427,20cyclictest12548-21missed_timers20:05:523
24437995426,19cyclictest6854-21ssh23:22:473
24437995426,19cyclictest6854-21ssh23:22:473
24437995426,11cyclictest8050irq/56-dwc_otg_21:05:493
24437995426,11cyclictest4077-21ssh22:37:083
24437995425,17cyclictest24338-21ssh23:04:173
24437995422,24cyclictest4140-21ssh23:20:113
24437995421,12cyclictest17965-21taskset00:20:343
24437995420,18cyclictest2736-21diskmemload00:17:503
24437995420,18cyclictest2736-21diskmemload00:17:503
2443599543,14cyclictest0-21swapper/223:59:372
2443599541,4cyclictest0-21swapper/223:01:462
2443599541,12cyclictest19976-21ssh22:16:002
2443599541,11cyclictest0-21swapper/220:34:002
2443599541,10cyclictest1868-21ntp_offset21:05:532
24437995324,11cyclictest8161-21rm22:42:413
24437995321,21cyclictest8549-21ssh00:06:313
24437995321,10cyclictest27387-21ssh23:50:003
24437995321,10cyclictest27387-21ssh23:50:003
24437995317,10cyclictest8150irq/56-dwc_otg_23:55:563
2443499533,5cyclictest18838-21ssh00:20:591
24437995220,18cyclictest8150irq/56-dwc_otg_20:00:553
24437995219,11cyclictest17882-21ssh22:55:513
2443599522,5cyclictest0-21swapper/223:21:032
2443599522,5cyclictest0-21swapper/223:21:032
2443599521,9cyclictest0-21swapper/219:29:022
2443599521,3cyclictest0-21swapper/222:43:502
2443599521,14cyclictest0-21swapper/223:12:132
2443599521,14cyclictest0-21swapper/222:14:092
24437995124,12cyclictest27916-21rm21:43:033
24437995122,12cyclictest27672-21ssh00:33:313
24437995120,12cyclictest8050irq/56-dwc_otg_22:01:043
24437995119,14cyclictest9243-21ssh21:17:503
2443599513,13cyclictest0-21swapper/219:55:482
2443599512,4cyclictest0-21swapper/223:50:262
2443599512,4cyclictest0-21swapper/223:50:262
2443599511,9cyclictest0-21swapper/219:31:592
2443599511,4cyclictest0-21swapper/223:15:542
2443599511,10cyclictest0-21swapper/221:01:222
2443599511,10cyclictest0-21swapper/221:01:222
24437995025,15cyclictest31856-21rm23:13:303
24437995020,14cyclictest28086-21/usr/sbin/munin19:20:573
24437995019,13cyclictest31173-21sed20:56:043
2443599502,10cyclictest8150irq/56-dwc_otg_23:50:532
2443599501,4cyclictest8050irq/56-dwc_otg_22:26:442
24435995010,4cyclictest0-21swapper/200:12:442
24437994923,21cyclictest14884-21unixbench_singl20:11:043
24437994923,21cyclictest14884-21unixbench_singl20:11:043
2443599491,41cyclictest0-21swapper/220:29:072
24437994833,12cyclictest21538-21ls20:30:553
24437994818,9cyclictest0-21swapper/319:54:183
2443599471,12cyclictest0-21swapper/220:24:152
2443599471,11cyclictest6085-21taskset00:04:152
24437994619,16cyclictest8150irq/56-dwc_otg_20:22:073
2443599461,15cyclictest0-21swapper/220:51:342
24437994529,12cyclictest9257-21ls19:55:593
24437994523,11cyclictest20444-21sshd20:26:553
24437994514,10cyclictest0-21swapper/320:51:303
2443599452,10cyclictest0-21swapper/220:19:332
2443599451,13cyclictest0-21swapper/220:56:352
2443499454,30cyclictest0-21swapper/122:40:261
24434994522,5cyclictest21609-21fschecks_count23:00:501
24434994510,5cyclictest8050irq/56-dwc_otg_22:50:431
24434994510,5cyclictest8050irq/56-dwc_otg_22:50:431
4512440,2sleep3453-21grep19:31:023
24437994433,8cyclictest22790-21apt-get20:35:443
24437994431,10cyclictest24591-21munin-node20:40:443
24437994419,15cyclictest466-21rs:main3
239942443,34sleep00-21swapper/019:08:130
24437994329,11cyclictest5487-21/usr/sbin/munin19:45:593
24437994317,11cyclictest0-21swapper/320:46:323
2443499437,23cyclictest23-21ksoftirqd/123:50:441
24437994227,12cyclictest1335-21/usr/sbin/munin19:35:473
24437994117,12cyclictest0-21swapper/319:45:053
2443499414,17cyclictest7950irq/56-dwc_otg22:40:591
24434994115,5cyclictest15108-21latency_hist22:10:261
2443599401,14cyclictest0-21swapper/219:42:122
2443499406,5cyclictest0-21swapper/119:20:461
24434994016,5cyclictest23586-21/usr/sbin/munin22:20:561
24434994013,6cyclictest131-21systemd-journal20:06:251
24434993925,7cyclictest8150irq/56-dwc_otg_19:30:571
24434993920,12cyclictest15874-21fschecks_time20:15:511
24434993919,4cyclictest11111-21/usr/sbin/munin22:46:031
2443499391,3cyclictest0-21swapper/123:55:441
2443499391,27cyclictest15455-21ssh21:25:591
2443499383,5cyclictest8050irq/56-dwc_otg_19:35:541
24434993819,11cyclictest8150irq/56-dwc_otg_00:16:001
24434993819,11cyclictest8150irq/56-dwc_otg_00:16:001
24434993816,5cyclictest24528-21latency_hist20:40:261
2443499381,5cyclictest2053250ssh23:40:491
24434993811,5cyclictest31943-21cpu21:00:481
24434993811,5cyclictest31943-21cpu21:00:481
2443499377,14cyclictest8150irq/56-dwc_otg_23:16:011
2443499375,19cyclictest0-21swapper/122:04:291
2443499374,14cyclictest7950irq/56-dwc_otg21:00:261
24434993724,7cyclictest8150irq/56-dwc_otg_23:45:441
24434993724,7cyclictest8150irq/56-dwc_otg_23:45:441
24434993712,15cyclictest23-21ksoftirqd/120:05:261
2443499368,13cyclictest0-21swapper/100:37:581
2443499366,13cyclictest8150irq/56-dwc_otg_21:41:001
24434993627,6cyclictest221ktimersoftd/119:25:431
2443499362,22cyclictest6986-21ssh21:15:341
2443499361,5cyclictest30341-21switchtime21:45:581
2443499361,5cyclictest30341-21switchtime21:45:581
24434993614,6cyclictest27306-21fschecks_time22:25:501
2443499360,15cyclictest12068-21ssh00:11:041
2443499355,5cyclictest8150irq/56-dwc_otg_21:31:021
2443499355,5cyclictest8150irq/56-dwc_otg_21:31:021
2443499354,20cyclictest21696-21ssh00:25:451
24434993526,6cyclictest221ktimersoftd/119:55:441
2443499349,17cyclictest0-21swapper/121:56:581
2443499349,12cyclictest0-21swapper/121:14:191
2443499342,23cyclictest0-21swapper/122:19:341
24434993412,4cyclictest0-21swapper/100:30:521
2443599331,15cyclictest0-21swapper/220:46:372
2443499337,24cyclictest0-21swapper/121:20:451
2443499336,7cyclictest8050irq/56-dwc_otg_20:45:521
2443499336,6cyclictest8150irq/56-dwc_otg_19:45:261
2443499334,26cyclictest0-21swapper/121:05:441
2443499332,25cyclictest1029-21idleruntime21:50:501
24434993320,7cyclictest8150irq/56-dwc_otg_23:23:301
24434993320,7cyclictest8150irq/56-dwc_otg_23:23:301
24434993316,4cyclictest13513-21sh23:31:001
2443499331,4cyclictest18783-21cron22:15:261
24434993314,9cyclictest7682-21missed_timers00:05:551
24434993310,10cyclictest0-21swapper/123:28:571
240792333,23sleep20-21swapper/219:09:202
2443599321,18cyclictest0-21swapper/220:10:452
2443599321,18cyclictest0-21swapper/220:10:452
2443499328,11cyclictest0-21swapper/121:39:311
2443499325,12cyclictest8150irq/56-dwc_otg_22:58:231
2443499321,4cyclictest8050irq/56-dwc_otg_22:32:541
24434993115,5cyclictest7244-21ntp_states19:50:571
2443499311,16cyclictest7950irq/56-dwc_otg20:31:541
2443499305,13cyclictest8150irq/56-dwc_otg_23:05:481
2443499301,20cyclictest0-21swapper/120:10:451
2443499301,20cyclictest0-21swapper/120:10:451
2443499301,15cyclictest8050irq/56-dwc_otg_20:50:521
2394923010,12sleep30-21swapper/319:07:403
2443499298,9cyclictest0-21swapper/100:02:031
2443499297,6cyclictest131-21systemd-journal20:41:321
2443499297,4cyclictest7950irq/56-dwc_otg23:10:441
2443499295,5cyclictest9650irq/80-mmc019:20:261
24434992921,5cyclictest24753-21fschecks_time19:10:511
24434992911,4cyclictest0-21swapper/120:20:511
2443499282,6cyclictest5140-21if_eth019:45:521
2396822710,10sleep10-21swapper/119:07:541
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional