You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-19 - 08:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun May 19, 2024 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4108210276,9sleep10-21swapper/119:05:101
5907210086,10sleep30-21swapper/319:05:443
606728464,16sleep20-21swapper/219:07:482
496628168,8sleep00-21swapper/019:05:220
215982650,0sleep00-21swapper/022:53:260
632899290,1cyclictest0-21swapper/000:27:280
633899273,2cyclictest30153-21diskmemload23:22:473
633899273,23cyclictest30153-21diskmemload22:45:163
633899270,4cyclictest0-21swapper/321:24:503
633899270,3cyclictest0-21swapper/322:06:313
633099270,1cyclictest0-21swapper/123:20:491
633899263,22cyclictest6772-21latency_hist22:00:003
633899262,4cyclictest0-21swapper/322:10:263
633899262,1cyclictest26932-21/usr/sbin/munin22:35:213
633099260,24cyclictest0-21swapper/120:05:161
633899251,3cyclictest6710-21unixbench_singl22:20:423
633899251,3cyclictest30153-21diskmemload21:52:443
633899250,1cyclictest30153-21diskmemload21:26:273
633899242,2cyclictest32200-21/usr/sbin/munin23:00:183
633899240,1cyclictest0-21swapper/323:40:403
6328992424,0cyclictest0-21swapper/000:34:020
633899234,1cyclictest30153-21diskmemload22:28:143
633899233,5cyclictest8351-21munin-run22:45:003
633899232,1cyclictest30153-21diskmemload23:47:463
633899231,4cyclictest30153-21diskmemload00:09:493
633899230,22cyclictest0-21swapper/323:34:063
633899230,1cyclictest0-21swapper/321:44:143
6334992320,2cyclictest10520-21awk00:15:312
633499230,22cyclictest0-21swapper/222:10:392
633099230,22cyclictest0-21swapper/100:00:171
633899222,1cyclictest30153-21diskmemload22:18:253
633899220,21cyclictest0-21swapper/300:10:113
632899220,22cyclictest0-21swapper/021:49:130
633899217,7cyclictest27769-21ssh23:19:503
633899211,6cyclictest9403-21sh21:39:503
633899211,0cyclictest0-21swapper/300:35:153
633899211,0cyclictest0-21swapper/300:24:023
633899210,2cyclictest0-21swapper/300:15:163
633899210,0cyclictest0-21swapper/321:49:573
633899201,2cyclictest6340-21aten2_r7power_v20:20:133
632899200,1cyclictest0-21swapper/023:12:560
633899196,7cyclictest7824-21ssh21:15:353
633899192,2cyclictest0-21swapper/319:35:273
633899191,3cyclictest24553-21timerandwakeup20:55:313
633899191,3cyclictest22777-21cron20:55:013
633899191,3cyclictest0-21swapper/322:56:253
633899190,3cyclictest0-21swapper/322:52:123
6334991917,1cyclictest0-21swapper/220:45:142
6334991917,1cyclictest0-21swapper/220:45:142
633499190,19cyclictest0-21swapper/221:53:152
633499190,18cyclictest0-21swapper/220:55:132
633099190,16cyclictest0-21swapper/122:25:271
633899183,1cyclictest30153-21diskmemload21:10:373
633899181,3cyclictest17617-21tr23:35:153
633899181,15cyclictest30153-21diskmemload22:30:463
633899181,15cyclictest30153-21diskmemload22:30:463
633899180,6cyclictest0-21swapper/319:50:303
633899180,5cyclictest27831-21cut21:30:163
633899180,1cyclictest0-21swapper/323:55:223
633099180,1cyclictest0-21swapper/121:53:011
632899187,6cyclictest4002-21munin-run22:20:010
632899180,1cyclictest0-21swapper/023:27:530
633899178,8cyclictest15879-21cat19:30:303
633899178,8cyclictest11036-21taskset19:20:003
633899176,6cyclictest25546-21ssh00:26:563
633899174,12cyclictest29836-21hwlatdetect20:00:193
633899170,2cyclictest0-21swapper/319:40:133
633899170,2cyclictest0-21swapper/300:01:033
633499170,1cyclictest0-21swapper/221:21:222
633099170,1cyclictest0-21swapper/123:26:211
633099170,1cyclictest0-21swapper/100:18:351
633899168,7cyclictest20010-21ssh23:14:203
6338991615,1cyclictest0-21swapper/320:40:153
633899160,1cyclictest0-21swapper/323:50:143
633899160,11cyclictest0-21swapper/323:27:563
633899160,0cyclictest0-21swapper/323:06:463
633499164,11cyclictest30451-21cups-browsed21:00:272
6334991614,1cyclictest0-21swapper/221:48:462
633499160,16cyclictest0-21swapper/223:12:222
633099169,2cyclictest0-21swapper/100:13:321
6330991614,1cyclictest0-21swapper/121:45:591
633099160,3cyclictest141rcu_preempt21:56:421
6328991611,3cyclictest24214-21kworker/u8:123:45:270
632899160,9cyclictest0-21swapper/023:51:210
632899160,9cyclictest0-21swapper/019:55:130
632899160,2cyclictest0-21swapper/023:55:110
633899158,6cyclictest7323-21head19:10:203
633899157,7cyclictest21723-21expr19:45:163
633899150,4cyclictest224-21systemd-journal20:20:003
633499155,6cyclictest37850irq/131-snd_hda19:25:072
6334991512,2cyclictest12724-21sendmail21:20:012
6334991512,1cyclictest0-21swapper/221:57:532
633499150,4cyclictest0-21swapper/222:07:322
633499150,1cyclictest0-21swapper/221:37:492
633099159,5cyclictest30440-21apache223:52:231
6330991511,3cyclictest14845-21ls20:35:321
633099150,1cyclictest0-21swapper/121:22:361
632899151,8cyclictest27134-21munin-run00:05:000
633899147,6cyclictest8986-21sed22:00:313
633899147,6cyclictest32184-21ssh00:31:353
633899147,6cyclictest25717-21tr21:00:153
633899147,6cyclictest10106-21cat20:25:343
633899146,7cyclictest27136-21tail19:55:183
633899145,2cyclictest2629-21cut20:10:213
633899141,2cyclictest224-21systemd-journal21:10:003
6338991410,3cyclictest13267-21fgrep19:25:183
633499146,7cyclictest11360-21cat23:30:202
633499140,1cyclictest0-21swapper/223:36:552
633099148,6cyclictest0-21swapper/122:46:541
6330991411,2cyclictest23434-21grep22:55:001
6330991410,3cyclictest28300-21ssh21:30:181
6328991411,1cyclictest0-21swapper/022:34:300
6328991411,1cyclictest0-21swapper/022:34:290
6328991410,3cyclictest3231-21grep00:10:220
632899140,2cyclictest0-21swapper/023:03:070
632899140,1cyclictest0-21swapper/023:42:110
632899140,1cyclictest0-21swapper/021:12:580
632899140,1cyclictest0-21swapper/000:18:320
632899140,12cyclictest12717-21sed19:25:130
633899137,5cyclictest15290-21taskset20:38:193
633899134,2cyclictest11694-21idleruntime-cro19:25:013
633899132,2cyclictest224-21systemd-journal20:46:173
633899132,2cyclictest224-21systemd-journal20:46:173
633899131,2cyclictest224-21systemd-journal20:32:143
633899130,12cyclictest32571-21tr20:05:263
6334991310,2cyclictest6064-21wc23:04:592
6334991310,2cyclictest3947-21cut23:25:182
6334991310,2cyclictest3671-21grep22:40:382
6334991310,2cyclictest3559-21cat20:15:002
6334991310,2cyclictest28978-21cat21:30:252
6334991310,2cyclictest25745-21grep23:40:282
6334991310,2cyclictest19013-21cat19:40:002
6334991310,2cyclictest15825-21munin-run22:50:002
6334991310,2cyclictest11550-21sed21:40:262
633499130,7cyclictest0-21swapper/200:24:052
633499130,13cyclictest0-21swapper/222:54:412
633499130,13cyclictest0-21swapper/200:14:392
633099139,3cyclictest17457-21head19:35:221
633099133,1cyclictest224-21systemd-journal20:55:001
6330991310,2cyclictest7813-21grep20:20:281
6330991310,2cyclictest6683-21tail22:20:411
6330991310,2cyclictest30297-21cat21:10:001
6330991310,2cyclictest28938-21grep22:35:441
6330991310,2cyclictest26230-21cat19:55:001
6330991310,2cyclictest16449-21awk19:34:591
6330991310,2cyclictest10585-21sed19:15:471
632899139,3cyclictest15810-21cron22:50:000
632899139,3cyclictest10226-21grep21:40:130
632899139,3cyclictest10044-21grep19:15:250
632899131,2cyclictest28302-21diskstats00:05:210
6328991312,1cyclictest181irq_work/021:21:000
6328991311,1cyclictest13-21ksoftirqd/000:40:000
6328991310,2cyclictest18878-21latency_hist22:30:010
632899130,12cyclictest24669-21sed20:55:310
632899130,0cyclictest0-21swapper/022:37:290
633499129,2cyclictest7850-21ssh23:50:312
633499129,2cyclictest32664-21grep21:10:322
633499129,2cyclictest30338-21wc21:10:002
633499129,2cyclictest29346-21tr00:30:172
633499129,2cyclictest21707-21sed00:00:222
633499129,2cyclictest20581-21cat19:40:312
633499129,2cyclictest16581-21wc19:34:592
633499129,2cyclictest14978-21grep20:35:322
633499129,2cyclictest13097-21grep22:25:242
633499128,3cyclictest28269-21ntpq22:35:342
633499127,4cyclictest11224-21ssh22:24:542
633499124,4cyclictest493-21systemd-logind00:06:562
633499120,11cyclictest0-21swapper/223:21:152
633099129,2cyclictest3834-21tr20:15:151
633099129,2cyclictest29920-21expr00:30:211
633099129,2cyclictest29296-21tr20:00:151
633099129,2cyclictest28886-21cat22:15:001
633099129,2cyclictest25022-21grep23:40:171
633099129,2cyclictest22366-21ls23:15:211
633099129,2cyclictest21957-21df19:45:181
633099129,2cyclictest21051-21tail21:25:161
633099129,2cyclictest20419-21ls20:50:011
633099129,2cyclictest20419-21ls20:50:001
633099129,2cyclictest17194-21chrt00:20:301
633099129,2cyclictest14791-21tr23:55:241
633099129,2cyclictest11659-21ssh23:30:231
633099128,3cyclictest7475-21ssh00:39:111
633099128,3cyclictest4941-21seq23:49:191
633099128,3cyclictest17572-21wc23:35:141
633099121,10cyclictest10232-21grep20:25:341
632899129,2cyclictest7827-21cut22:00:190
632899129,2cyclictest5920-21grep20:20:010
632899129,2cyclictest522-21sort21:55:200
632899129,2cyclictest23481-21chrt23:39:370
632899129,2cyclictest23160-21grep19:45:370
632899129,2cyclictest22052-21tr22:10:150
632899129,2cyclictest22029-21grep23:15:170
632899129,2cyclictest21169-21taskset21:25:170
632899129,2cyclictest17490-21cut19:35:220
632899129,2cyclictest15561-21grep19:30:240
632899127,3cyclictest12529-21wc20:30:290
632899120,1cyclictest141rcu_preempt22:57:240
633499119,1cyclictest20709-21head22:30:272
633499119,1cyclictest20709-21head22:30:272
633499118,2cyclictest9915-21rm22:01:122
633499118,2cyclictest7747-21wc20:20:272
633499118,2cyclictest3325-21expr00:35:162
633499118,2cyclictest32654-21ssh22:16:422
633499118,2cyclictest31161-21taskset23:45:022
633499118,2cyclictest29757-21rm22:59:032
633499118,2cyclictest28916-21dpkg20:00:002
633499118,2cyclictest27651-21ls19:55:262
633499118,2cyclictest26809-21sh00:28:142
633499118,2cyclictest14441-21rm23:55:222
633499118,2cyclictest10638-21ssh23:07:292
633499117,3cyclictest18046-21ls20:45:012
6334991110,1cyclictest7874-21kworker/2:223:15:202
633099118,2cyclictest9660-21ssh22:00:591
633099118,2cyclictest8995-21rm23:06:011
633099118,2cyclictest5350-21ssh21:36:011
633099118,2cyclictest31853-21ssh00:07:461
633099118,2cyclictest30158-21grep22:15:251
633099118,2cyclictest24708-21tail22:55:171
633099118,2cyclictest23839-21tail20:55:191
633099118,2cyclictest23439-21ssh00:25:211
633099118,2cyclictest1878-21ssh23:01:221
633099118,2cyclictest16211-21ssh23:10:451
633099118,2cyclictest13605-21grep19:25:231
633099118,2cyclictest12861-21taskset20:32:331
633099118,2cyclictest12432-21seq21:40:441
633099118,2cyclictest10072-21sh21:17:321
633099117,3cyclictest31553-21sed21:10:211
633099117,3cyclictest19129-21cut19:40:141
633099116,4cyclictest15877-21proc_pri22:05:311
633099112,7cyclictest0-21swapper/122:40:571
633099110,1cyclictest0-21swapper/122:30:051
633099110,1cyclictest0-21swapper/122:30:041
633099110,10cyclictest28658-21sed20:00:011
632899119,1cyclictest7011-21wc21:15:250
632899119,1cyclictest29766-21grep20:00:180
632899118,2cyclictest9421-21ssh21:39:500
632899118,2cyclictest8648-21grep20:25:160
632899118,2cyclictest7726-21ssh23:05:250
632899118,2cyclictest7327-21sed19:10:190
632899118,2cyclictest5139-21sed22:20:210
632899118,2cyclictest30837-21ssh23:20:470
632899118,2cyclictest28582-21tail21:30:210
632899118,2cyclictest28048-21chrt21:51:300
632899118,2cyclictest25290-21grep19:50:230
632899118,2cyclictest2508-21sed22:40:220
632899118,2cyclictest20816-21ssh22:09:330
632899118,2cyclictest18903-21ssh00:22:010
632899118,2cyclictest16640-21grep20:40:190
632899118,2cyclictest15738-21ssh23:33:570
632899117,3cyclictest29104-21wc21:05:250
633499108,2cyclictest21558-21ntpq21:25:232
633499108,1cyclictest8938-21expr20:25:172
633499108,1cyclictest76792sleep219:10:242
633499108,1cyclictest4599-21smartctl20:15:212
633499108,1cyclictest314732sleep220:05:152
633499108,1cyclictest22044-21df19:45:172
633499107,2cyclictest22082-21sensors_temp20:50:322
633499100,9cyclictest24516-21expr19:50:142
633099108,1cyclictest275312sleep121:04:521
633099107,2cyclictest6675-21cut19:10:161
633099100,9cyclictest15810-21apt-key20:40:001
633099100,9cyclictest1401-21basename20:10:081
632899108,1cyclictest3441-21taskset20:14:000
632899108,1cyclictest32310-21basename20:05:240
632899108,1cyclictest20262-21taskset20:48:140
632899108,1cyclictest20262-21taskset20:48:130
632899108,1cyclictest13744-21expr20:35:170
632899107,2cyclictest6803-21sort20:20:150
632899107,2cyclictest26382-21cut21:00:200
632899107,2cyclictest21423-21grep20:50:220
632899107,2cyclictest20920-21chrt19:41:320
63349997,1cyclictest9225-21expr19:15:172
63349997,1cyclictest11823-21cat20:30:182
63349997,1cyclictest11565-21chrt19:23:352
63289997,1cyclictest11501-21chrt19:22:440
63309980,4cyclictest224-21systemd-journal19:23:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional