You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-19 - 22:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 19, 2024 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8156210374,10sleep10-21swapper/107:08:191
8153210390,9sleep30-21swapper/307:08:183
820828363,16sleep00-21swapper/007:08:580
731227666,6sleep20-21swapper/207:05:242
837899250,25cyclictest0-21swapper/212:00:442
837899250,25cyclictest0-21swapper/211:15:592
837899250,24cyclictest0-21swapper/209:00:142
8378992421,2cyclictest10594-21kworker/u8:210:35:302
837899240,23cyclictest0-21swapper/209:18:032
837899230,23cyclictest0-21swapper/212:06:312
8378992220,2cyclictest32-21ksoftirqd/212:29:112
8378992220,2cyclictest0-21swapper/210:08:022
8382992113,7cyclictest1584-21diskmemload10:45:133
838299210,1cyclictest0-21swapper/311:16:453
8378992120,1cyclictest0-21swapper/207:35:152
837899210,21cyclictest0-21swapper/210:45:092
837899210,20cyclictest0-21swapper/210:23:162
836899210,21cyclictest0-21swapper/011:21:410
8382992013,5cyclictest1584-21diskmemload12:36:453
8378992017,3cyclictest32-21ksoftirqd/209:50:132
836899205,6cyclictest224-21systemd-journal11:00:010
8368992017,1cyclictest0-21swapper/009:16:090
8368992017,1cyclictest0-21swapper/009:12:480
836899201,2cyclictest0-21swapper/011:51:520
8382991917,1cyclictest0-21swapper/311:14:323
8382991916,3cyclictest39-21ksoftirqd/312:17:223
8382991916,1cyclictest1584-21diskmemload11:05:263
8382991914,3cyclictest381rcuc/311:57:003
837899194,1cyclictest141rcu_preempt12:15:352
837899194,10cyclictest0-21swapper/207:30:112
837899193,1cyclictest141rcu_preempt08:35:132
8378991918,1cyclictest32-21ksoftirqd/211:23:472
8374991918,1cyclictest0-21swapper/107:10:111
838299188,10cyclictest0-21swapper/312:20:163
838299181,3cyclictest0-21swapper/311:35:133
837899189,3cyclictest32-21ksoftirqd/211:11:202
8378991816,1cyclictest32-21ksoftirqd/211:02:242
837499181,17cyclictest0-21swapper/110:21:391
838299179,5cyclictest39-21ksoftirqd/309:45:503
838299177,9cyclictest280282sleep311:31:313
837899174,3cyclictest141rcu_preempt11:53:552
837899170,2cyclictest0-21swapper/211:49:592
837899170,1cyclictest141rcu_preempt09:40:132
837499171,15cyclictest0-21swapper/109:45:121
836899174,10cyclictest27350-21apt-get09:30:120
838299166,10cyclictest0-21swapper/310:35:593
838299165,10cyclictest27633-21unixbench_singl07:45:273
838299164,7cyclictest39-21ksoftirqd/310:29:553
8382991615,1cyclictest0-21swapper/310:53:573
8382991611,5cyclictest39-21ksoftirqd/312:10:183
838299161,13cyclictest39-21ksoftirqd/311:47:223
837899162,5cyclictest32-21ksoftirqd/210:30:492
837899162,11cyclictest2656-21/usr/sbin/munin10:50:282
8378991613,2cyclictest32-21ksoftirqd/211:30:212
8378991611,4cyclictest32-21ksoftirqd/210:27:352
8378991611,2cyclictest37850irq/131-snd_hda09:23:512
837899160,1cyclictest0-21swapper/211:09:132
837899160,14cyclictest0-21swapper/212:37:122
837499161,8cyclictest29192-21munin-run09:00:001
837499160,16cyclictest0-21swapper/111:31:231
838299154,2cyclictest224-21systemd-journal12:30:003
838299153,3cyclictest0-21swapper/309:15:203
8382991514,1cyclictest0-21swapper/309:52:583
837899159,5cyclictest32-21ksoftirqd/210:01:282
837899155,4cyclictest224-21systemd-journal08:05:002
8374991510,3cyclictest17695-21grep08:35:121
837499150,1cyclictest0-21swapper/109:43:501
837499150,15cyclictest0-21swapper/111:23:441
837499150,15cyclictest0-21swapper/110:55:361
837499150,15cyclictest0-21swapper/110:06:371
837499150,13cyclictest0-21swapper/111:49:431
838299147,6cyclictest0-21swapper/307:20:223
838299142,11cyclictest381rcuc/309:38:593
838299141,2cyclictest39-21ksoftirqd/312:02:143
8382991411,3cyclictest39-21ksoftirqd/312:32:233
8382991411,2cyclictest1584-21diskmemload10:00:163
838299140,2cyclictest0-21swapper/310:21:543
838299140,10cyclictest39-21ksoftirqd/309:41:503
837899147,6cyclictest32-21ksoftirqd/210:57:302
837899142,1cyclictest0-21swapper/209:30:172
8378991412,1cyclictest1584-21diskmemload11:39:172
837899141,1cyclictest141rcu_preempt07:40:242
837899140,1cyclictest1584-21diskmemload10:14:582
837499145,5cyclictest21639-21/usr/sbin/munin11:50:221
837499143,2cyclictest141rcu_preempt10:15:491
837499140,1cyclictest0-21swapper/110:31:011
836899143,7cyclictest0-21swapper/012:05:410
836899141,2cyclictest21044-21apache_processe09:25:110
838299139,3cyclictest27674-21grep08:55:183
838299137,5cyclictest0-21swapper/312:07:163
838299136,5cyclictest39-21ksoftirqd/308:47:123
838299135,6cyclictest17762-21systemctl07:25:243
838299134,4cyclictest39-21ksoftirqd/310:32:483
8382991312,1cyclictest61750irq/125-enp2s0-11:28:033
8382991310,2cyclictest27805-21grep09:55:123
8382991310,2cyclictest1166-21tr08:00:203
837899139,3cyclictest8287-21tr08:15:152
837899138,5cyclictest32-21ksoftirqd/212:10:262
837899138,3cyclictest32-21ksoftirqd/211:28:422
837899138,2cyclictest32-21ksoftirqd/209:57:472
837899136,6cyclictest32-21ksoftirqd/209:38:062
837899136,6cyclictest16445-21grep09:45:282
837899133,2cyclictest0-21swapper/211:44:362
837899131,1cyclictest0-21swapper/212:21:262
8378991310,2cyclictest4910-21sed08:05:272
8378991310,2cyclictest26198-21tr07:45:132
837899130,2cyclictest0-21swapper/212:32:142
837899130,12cyclictest20501-21df08:40:152
837499139,3cyclictest3123-21grep11:15:161
837499131,4cyclictest224-21systemd-journal08:24:541
8374991310,2cyclictest9056-21grep09:15:191
8374991310,2cyclictest4701-21chrt10:00:571
8374991310,2cyclictest24967-21tr08:50:161
8374991310,2cyclictest18912-21cut07:30:121
8374991310,2cyclictest16171-21grep09:20:221
837499130,12cyclictest14574-21cut12:35:151
837499130,11cyclictest0-21swapper/110:43:011
836899137,5cyclictest32144-21perf10:25:000
836899132,6cyclictest224-21systemd-journal09:40:010
8368991310,2cyclictest20389-21chrt07:30:250
8368991310,2cyclictest15492-21cat12:10:210
8368991310,2cyclictest10264-21cat07:10:270
8368991310,2cyclictest0-21swapper/011:58:020
836899130,8cyclictest6357-21idleruntime-cro11:40:000
836899130,8cyclictest29223-21awk09:00:000
836899130,7cyclictest8167-21idleruntime-cro10:05:000
836899130,11cyclictest2098-21cut12:00:190
836899130,10cyclictest8329-21fschecks_count12:30:160
838299129,2cyclictest7580-21ssh10:55:133
838299129,2cyclictest30173-21sed07:50:333
838299129,2cyclictest22308-21sed11:50:303
838299129,2cyclictest22067-21ls08:45:003
838299128,4cyclictest0-21swapper/310:13:493
838299128,3cyclictest29263-21sed09:30:293
838299128,2cyclictest6814-21ssh11:40:123
838299128,2cyclictest14670-21grep10:09:593
838299124,3cyclictest39-21ksoftirqd/310:40:143
837899129,2cyclictest7277-21sed08:10:302
837899129,2cyclictest32549-21sort09:05:162
837899129,2cyclictest12153-21grep07:15:192
837899128,3cyclictest27868-21grep11:55:202
837899128,3cyclictest21337-21tr09:25:132
837899128,3cyclictest15961-21cut08:30:252
837899127,2cyclictest32-21ksoftirqd/209:14:132
837899120,2cyclictest32-21ksoftirqd/210:40:272
837899120,12cyclictest0-21swapper/207:55:152
837899120,11cyclictest22056-21ssh10:16:002
837499129,2cyclictest7488-21basename12:04:591
837499129,2cyclictest4920-21grep08:05:281
837499129,2cyclictest24542-21cat08:50:001
837499129,2cyclictest1934-21cut10:00:001
837499129,2cyclictest17325-21grep08:34:591
837499129,2cyclictest15405-21grep11:00:361
837499128,3cyclictest20732-21sh09:50:011
837499128,3cyclictest15277-21sed10:35:321
837499128,2cyclictest2012-21grep10:25:311
837499124,4cyclictest224-21systemd-journal07:45:001
836899129,2cyclictest7772-21apt08:14:590
836899129,2cyclictest4503-21grep08:05:230
836899129,2cyclictest32651-21grep07:55:290
836899129,2cyclictest29596-21grep09:55:270
836899129,2cyclictest2695-21cut11:15:130
836899129,2cyclictest22132-21wc08:45:000
836899129,2cyclictest21486-21tr12:15:160
836899129,2cyclictest15762-21sed09:20:170
836899129,2cyclictest15274-21cut12:35:210
836899129,2cyclictest14062-21grep07:20:170
836899128,3cyclictest1794-21ls10:25:300
836899125,6cyclictest17308-21munin-run08:34:590
836899124,7cyclictest31653-21awk09:05:000
8368991211,1cyclictest0-21swapper/012:21:240
8368991210,1cyclictest21922-21head07:35:190
838299119,1cyclictest17388-21latency_hist08:35:003
838299119,1cyclictest13092-21sed07:15:293
838299118,2cyclictest9721-21cat07:10:163
838299118,2cyclictest562-21grep09:05:203
838299118,2cyclictest3084-21wc09:10:203
838299118,2cyclictest30220-21head09:00:193
838299118,2cyclictest26703-21taskset08:53:503
838299118,2cyclictest23810-21ssh09:26:163
838299118,2cyclictest22084-21ssh10:16:033
838299118,2cyclictest19554-21ssh09:23:463
838299118,2cyclictest19397-21grep07:30:153
838299117,3cyclictest23271-21munin-run07:40:003
838299114,6cyclictest0-21swapper/308:05:183
838299114,5cyclictest1584-21diskmemload11:20:013
838299114,3cyclictest224-21systemd-journal08:24:523
838299112,6cyclictest39-21ksoftirqd/308:25:153
838299111,5cyclictest39-21ksoftirqd/311:00:313
838299110,9cyclictest0-21swapper/307:55:153
837899119,2cyclictest311rcuc/208:45:132
837899118,2cyclictest18498-21cron07:30:012
837899118,2cyclictest15536-21sed07:20:302
837899117,4cyclictest32-21ksoftirqd/208:20:202
837899114,2cyclictest224-21systemd-journal09:00:002
837899110,10cyclictest30540-21taskset07:54:172
837499119,1cyclictest8454-21tail12:30:171
837499119,1cyclictest17314-21ssh10:10:561
837499118,2cyclictest856-21cut09:05:291
837499118,2cyclictest6906-21grep11:40:131
837499118,2cyclictest6422-21grep08:10:171
837499118,2cyclictest4772-21ssh11:38:161
837499118,2cyclictest32036-21grep07:55:211
837499118,2cyclictest28245-21basename09:30:181
837499118,2cyclictest27305-21grep12:20:191
837499118,2cyclictest27228-21ssh11:55:161
837499118,2cyclictest27216-21ssh11:10:091
837499118,2cyclictest27038-21df_abs10:45:141
837499118,2cyclictest2591-21grep10:50:281
837499118,2cyclictest23879-21ssh11:07:261
837499118,2cyclictest22772-21sed07:35:301
837499118,2cyclictest22163-21sh12:15:201
837499118,2cyclictest21306-21grep09:25:131
837499118,2cyclictest2095-21grep12:25:191
837499118,2cyclictest18110-21cut11:25:171
837499118,2cyclictest14379-21rm12:10:121
837499118,2cyclictest10269-21ssh12:06:131
837499112,2cyclictest0-21swapper/109:10:141
836899119,1cyclictest8638-21tr10:05:120
836899118,2cyclictest9192-21sed09:40:240
836899118,2cyclictest6332-21ssh10:30:060
836899118,2cyclictest26327-21grep08:50:300
836899118,2cyclictest25494-21ssh10:19:560
836899118,2cyclictest2363-21wc10:50:240
836899118,2cyclictest21539-21ssh11:27:240
836899118,2cyclictest20344-21tr10:40:150
836899118,2cyclictest17593-21grep08:35:120
836899118,2cyclictest16995-21sh11:02:210
836899118,2cyclictest1640-21tune2fs12:25:170
836899118,2cyclictest15030-21ntpq11:45:270
836899118,2cyclictest14787-21grep09:45:140
836899118,2cyclictest14631-21grep10:35:230
836899118,1cyclictest22221-21ssh11:05:570
836899118,1cyclictest20346-21apt-get09:50:020
836899117,3cyclictest30601-21cron07:55:000
836899117,3cyclictest11665-21grep08:20:200
836899110,6cyclictest10184-21awk08:20:000
836899110,10cyclictest94482sleep011:41:050
836899110,10cyclictest28453-21sed11:10:190
838299108,1cyclictest241162sleep307:40:153
838299105,4cyclictest381rcuc/308:35:123
838299100,9cyclictest5783-21grep08:10:143
837899108,1cyclictest9675-21if_enp2s007:10:162
837899108,1cyclictest14760-21chrt08:28:052
837899100,10cyclictest0-21swapper/208:50:152
837499108,1cyclictest2170-21ssh09:35:151
837499108,1cyclictest17010-21cut07:25:161
837499108,1cyclictest16023-21cut07:25:001
837499108,1cyclictest1345-21expr08:00:191
837499108,1cyclictest127782sleep108:25:081
837499108,1cyclictest12629-21cut07:15:241
837499107,2cyclictest9345-21ps08:15:251
837499107,2cyclictest30131-21sed07:50:311
837499100,9cyclictest27940-21taskset07:49:051
837499100,9cyclictest217692sleep108:41:221
836899108,1cyclictest28974-21ssh10:46:000
836899108,1cyclictest25877-21taskset07:45:060
836899108,1cyclictest22294-21seq08:45:100
836899108,1cyclictest163592sleep007:25:040
836899108,1cyclictest12291-21cat07:15:190
836899108,1cyclictest1179-21tr08:00:190
836899107,2cyclictest31932-21wget09:05:130
836899107,2cyclictest24681-21sh11:30:020
836899107,2cyclictest14948-21ssh10:10:130
83829997,1cyclictest8024-21basename08:15:113
83749996,2cyclictest30067-21grep09:00:171
83689997,1cyclictest13571-21expr08:25:200
83689996,2cyclictest25660-21sleep07:45:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional