You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-21 - 13:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Fri Jun 21, 2024 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3017599706,36cyclictest11169-21ssh22:05:102
3015599623,55cyclictest1945-21ssh21:40:140
30175995916,21cyclictest0-21swapper/222:55:142
30175995516,29cyclictest0-21swapper/219:40:122
30175994830,14cyclictest28957-21apt-get22:35:132
3017599481,27cyclictest0-21swapper/223:45:142
30175994721,21cyclictest0-21swapper/223:40:112
30175994615,26cyclictest0-21swapper/221:10:112
30183994311,21cyclictest25866irq/58-eth023:25:133
30175994332,7cyclictest9228-21apt-get00:20:142
3016599430,35cyclictest9838-21ssh00:00:121
3016599430,32cyclictest15942-21ssh23:25:131
3017599422,30cyclictest12422-21sh22:25:142
30175994214,18cyclictest31865-21ssh21:37:012
30175994118,13cyclictest0-21swapper/200:15:122
30175994115,9cyclictest7807-21sh21:23:212
30175994115,7cyclictest23396-21ssh00:28:392
30175994015,8cyclictest688-21rm21:57:332
30175994015,7cyclictest16398-21diskmemload22:10:562
30175994012,13cyclictest18098-21ntp_192.168.11423:05:242
30175993924,10cyclictest0-21swapper/220:45:102
30175993923,12cyclictest0-21swapper/219:45:122
30175993917,17cyclictest0-21swapper/219:20:102
3016599392,30cyclictest13193-21apt-get22:45:121
30175993823,13cyclictest29628-21apt-get20:50:122
3017599381,19cyclictest744866ssh23:20:142
3016599382,25cyclictest20673-21ssh22:30:131
3016599381,30cyclictest9247-21sh00:20:151
3017599379,12cyclictest14330-21grep20:00:192
30175993711,12cyclictest10141-21latency_hist20:30:022
30175993629,4cyclictest15144-21apt-get20:35:132
30175993625,8cyclictest25051-21apt-get21:15:152
30175993625,8cyclictest0-21swapper/223:00:122
30175993620,12cyclictest2743-21sendmail19:15:042
30175993617,15cyclictest0-21swapper/221:45:132
30175993615,8cyclictest16398-21diskmemload21:40:092
3016599362,28cyclictest24348-21sh23:30:121
3017599358,11cyclictest0-21swapper/221:53:312
3017599357,11cyclictest0-21swapper/222:02:412
30175993523,8cyclictest3958-21ssh22:20:132
30175993519,13cyclictest0-21swapper/223:35:112
30175993518,13cyclictest13740-21ntp_192.168.11420:30:272
30175993518,12cyclictest22146irq/40-dwc2_hso20:10:122
30175993516,16cyclictest900-21snmpd19:25:332
30175993510,12cyclictest32418-21kthreadcore23:15:212
30165993513,15cyclictest21-21ksoftirqd/119:20:101
3017599347,22cyclictest27-21ksoftirqd/200:03:582
30175993420,12cyclictest0-21swapper/219:55:112
30175993418,13cyclictest24333-21ssh00:30:152
30175993418,12cyclictest21344-21ssh23:26:152
30175993414,11cyclictest21873-21latency_hist19:35:012
30175993413,8cyclictest10014-21ssh22:40:532
3016599342,26cyclictest32712-21ssh23:35:111
3017599338,11cyclictest25219-21ssh22:50:302
30175993327,3cyclictest22863-21apt-get23:10:122
30175993323,7cyclictest20667-21apt-get22:30:142
30175993323,7cyclictest19290-21irqstats20:05:242
30175993317,12cyclictest11193-21usage_EthcTask21:00:362
30175993316,13cyclictest25774-21ssh00:10:132
30175993312,9cyclictest31181-21ssh23:51:522
30175993218,6cyclictest16898-21ntp_192.168.11422:45:282
30175993217,7cyclictest1356-21/usr/sbin/munin19:10:312
30175993217,12cyclictest1569-21ssh23:55:162
30175993217,12cyclictest1569-21ssh23:55:162
30175993215,8cyclictest24262-21ssh00:08:092
30175993213,16cyclictest12455-21kthreadcore21:05:222
3016599322,19cyclictest16724-21ssh21:10:111
3018399311,17cyclictest25866irq/58-eth021:40:133
30175993118,10cyclictest17743-21df21:30:132
30175993117,12cyclictest17054-21ssh21:29:412
30175993115,13cyclictest0-21swapper/222:15:102
30175993115,12cyclictest1256-21cut20:20:192
30175993113,14cyclictest28359-21ls20:15:132
30175993110,17cyclictest20081-21fschecks_count20:40:182
30175993110,17cyclictest20081-21fschecks_count20:40:182
3016599312,22cyclictest961-21ssh21:20:121
30175993018,10cyclictest7627-21ntp_kernel_pll_19:50:282
30175993016,11cyclictest31489-21/usr/sbin/munin00:35:142
30175993015,12cyclictest22061-21/usr/sbin/munin19:35:312
30175993014,13cyclictest24424-21ssh23:30:122
30175993013,13cyclictest2099-21/usr/sbin/munin20:55:262
3015599302,26cyclictest1167-21ssh00:15:120
3015599302,23cyclictest24349-21ssh23:30:120
3016599292,19cyclictest21-21ksoftirqd/123:50:121
3015599292,24cyclictest29981-21ssh22:55:140
30155992921,6cyclictest12-21ksoftirqd/022:35:140
3015599291,24cyclictest9374-21ssh21:25:140
3015599291,22cyclictest0-21swapper/023:40:100
30183992818,6cyclictest25866irq/58-eth022:55:293
3016599282,19cyclictest21557-21ssh22:50:141
3015599285,7cyclictest131rcu_preempt23:20:120
30183992719,5cyclictest0-21swapper/300:30:163
3015599279,8cyclictest12-21ksoftirqd/000:20:140
3015599279,7cyclictest131rcu_preempt20:10:120
3015599275,20cyclictest12-21ksoftirqd/022:20:140
3015599273,21cyclictest17637-21ssh21:30:110
3015599272,19cyclictest29314-21sh23:12:410
3016599252,3cyclictest131rcu_preempt19:50:111
3016599252,18cyclictest14969-21ssh23:42:421
3015599253,18cyclictest19546-21ssh22:10:130
3015599251,21cyclictest10287-21ssh21:45:140
3015599251,20cyclictest20370-21ssh21:10:270
3015599251,19cyclictest24344-21ssh00:30:170
3015599251,18cyclictest13205-21ssh22:45:120
3018399243,14cyclictest0-21swapper/323:12:413
3016599246,13cyclictest0-21swapper/100:10:111
30165992414,7cyclictest18936-21sshd22:09:411
30165992412,6cyclictest0-21swapper/120:40:141
30165992412,6cyclictest0-21swapper/120:40:141
3015599244,18cyclictest21542-21ssh22:50:120
3015599242,17cyclictest11178-21ssh22:05:110
3015599241,18cyclictest12-21ksoftirqd/000:35:130
3018399232,16cyclictest0-21swapper/322:50:123
3018399231,16cyclictest25866irq/58-eth021:15:163
3017599237,12cyclictest30032-21latency_hist19:10:012
30165992311,7cyclictest0-21swapper/122:27:421
3015599232,16cyclictest25072-21ssh21:15:160
3018399227,9cyclictest0-21swapper/321:55:133
3018399223,15cyclictest25866irq/58-eth000:35:133
3018399222,13cyclictest0-21swapper/323:05:113
30155992214,5cyclictest0-21swapper/023:15:150
3015599221,18cyclictest26931-21ssh21:55:130
3015599221,17cyclictest5430-21ls23:55:350
3015599221,17cyclictest5430-21ls23:55:350
30155992210,3cyclictest16398-21diskmemload22:30:540
3015599220,17cyclictest918-21ls20:50:330
3018399219,8cyclictest0-21swapper/321:10:223
3018399212,15cyclictest0-21swapper/320:50:133
30183992111,7cyclictest25866irq/58-eth021:45:153
3016599215,4cyclictest131rcu_preempt21:30:191
30165992112,7cyclictest761-21ssh22:35:371
3016599211,13cyclictest6035-21ssh23:00:121
3016599210,18cyclictest17393-21ls21:30:011
3015599212,15cyclictest1062-21df_inode21:20:130
30155992112,6cyclictest0-21swapper/022:03:310
3015599211,17cyclictest4882-21sh22:40:120
3015599211,17cyclictest10003-21fschecks_count00:00:170
3015599211,15cyclictest20086-21cut20:40:190
3015599211,15cyclictest20086-21cut20:40:190
3015599210,18cyclictest12516-21ls22:25:160
3018399209,4cyclictest0-21swapper/323:05:013
3016599209,8cyclictest307-21cat22:15:361
3016599209,6cyclictest0-21swapper/123:47:411
3016599205,12cyclictest21-21ksoftirqd/121:35:101
3016599202,12cyclictest0-21swapper/123:10:111
30165992012,6cyclictest5000-21ssh22:59:111
30165992012,5cyclictest25338-21munin-plugin-st00:10:011
3016599201,15cyclictest12435-21apt-get19:25:121
3016599200,13cyclictest10300-21ls21:00:251
3015599205,12cyclictest25554-21cut23:50:170
3015599203,14cyclictest20778-21tr20:05:260
30155992011,7cyclictest16384-21cron21:10:010
30155992011,6cyclictest29440-21latency21:35:240
3015599201,15cyclictest32705-21apt-get23:35:100
3015599201,15cyclictest24790-21apt-get20:45:100
3015599201,12cyclictest5294-21latency_hist20:25:010
3018399199,6cyclictest25866irq/58-eth022:30:153
3018399195,11cyclictest25866irq/58-eth020:55:123
3018399193,11cyclictest0-21swapper/323:59:413
3018399193,11cyclictest0-21swapper/323:59:413
3018399193,11cyclictest0-21swapper/323:20:143
3018399192,14cyclictest0-21swapper/322:35:143
3018399192,13cyclictest0-21swapper/300:15:133
30183991914,3cyclictest25866irq/58-eth022:40:253
30183991913,4cyclictest0-21swapper/322:20:333
30183991911,5cyclictest25866irq/58-eth023:40:103
3018399191,14cyclictest0-21swapper/323:45:203
3018399191,13cyclictest0-21swapper/320:25:353
30183991910,6cyclictest0-21swapper/323:35:323
30183991910,6cyclictest0-21swapper/323:32:293
30183991910,6cyclictest0-21swapper/322:00:053
30183991910,6cyclictest0-21swapper/321:20:553
30183991910,6cyclictest0-21swapper/300:06:533
3016599196,4cyclictest3521-21irqrtprio22:00:211
3016599194,11cyclictest0-21swapper/119:40:111
3016599192,14cyclictest18586-21apt-get21:50:141
3016599191,9cyclictest17182-21/usr/sbin/munin00:25:371
3016599191,9cyclictest10017-21latency_hist21:45:021
30165991912,5cyclictest7958-21threads22:20:341
30165991912,5cyclictest28799-21ssh21:15:331
30165991912,4cyclictest9506-21idleruntime-cro00:00:011
30165991912,4cyclictest9506-21idleruntime-cro00:00:011
30165991911,6cyclictest15071-21ssh23:24:291
30165991910,7cyclictest23072-21memory22:10:261
30165991910,6cyclictest0-21swapper/100:30:201
30165991910,5cyclictest0-21swapper/123:15:121
3015599198,8cyclictest5984-21grep20:55:340
3015599195,11cyclictest19986-21kthreadcore00:25:270
3015599192,10cyclictest5005-21kthreadcore19:15:210
30155991915,3cyclictest12-21ksoftirqd/023:25:050
30155991912,4cyclictest18643-21kthreadcore00:05:220
30155991911,6cyclictest31521-21sshd19:43:010
3015599191,16cyclictest14859-21/usr/sbin/munin23:05:170
3015599191,15cyclictest9844-21grep23:00:290
3015599191,15cyclictest29728-21ssh22:15:210
3015599191,15cyclictest16954-21apt23:45:130
30155991911,5cyclictest16398-21diskmemload21:50:190
3015599191,10cyclictest30633-21cut19:10:210
30155991910,6cyclictest0-21swapper/019:30:200
3018399189,6cyclictest25866irq/58-eth021:50:143
3018399188,7cyclictest25866irq/58-eth021:30:113
3018399183,11cyclictest0-21swapper/322:26:413
3018399183,10cyclictest25866irq/58-eth022:05:113
3018399183,10cyclictest0-21swapper/321:39:413
3018399182,11cyclictest0-21swapper/322:49:423
3018399182,11cyclictest0-21swapper/300:02:413
3018399181,14cyclictest0-21swapper/323:15:333
3018399181,13cyclictest0-21swapper/319:20:023
3016599188,7cyclictest0-21swapper/120:45:301
3016599188,2cyclictest131rcu_preempt21:55:211
3016599186,7cyclictest21-21ksoftirqd/119:55:111
3016599182,13cyclictest17982-21memory23:05:221
3016599181,9cyclictest4849-21/usr/sbin/munin22:40:201
3016599181,9cyclictest28786-21irqrtprio20:15:191
3016599181,8cyclictest10253-21/usr/sbin/munin21:45:291
3016599181,7cyclictest26716-21idleruntime-cro19:40:021
3016599181,3cyclictest131rcu_preempt00:15:231
30165991811,5cyclictest0-21swapper/120:05:351
3015599189,7cyclictest12765-21tail19:25:200
3015599185,10cyclictest8711-21kthreadcore20:25:260
3015599184,10cyclictest11693-21kthreadcore20:30:220
3015599182,13cyclictest7210-21idleruntime21:00:150
3015599182,13cyclictest4159-21apt-get19:50:120
3015599182,13cyclictest11491-21grep19:20:320
3015599182,11cyclictest0-21swapper/019:35:140
3015599181,8cyclictest1217-21kthreadcore19:45:240
3015599181,15cyclictest11494-21kthreadcore19:55:230
3015599181,14cyclictest25673-21ssh00:10:110
3015599181,12cyclictest18562-21cat20:05:010
3015599181,12cyclictest18275-21kthreadcore20:35:260
3018399179,6cyclictest0-21swapper/321:25:073
3018399179,6cyclictest0-21swapper/300:22:113
3018399179,6cyclictest0-21swapper/300:11:363
3018399179,5cyclictest25866irq/58-eth000:25:153
3018399178,7cyclictest0-21swapper/323:51:093
30183991711,4cyclictest25866irq/58-eth022:15:363
3018399171,12cyclictest0-21swapper/320:40:193
3018399171,12cyclictest0-21swapper/320:40:193
3018399171,11cyclictest0-21swapper/322:13:333
3016599178,7cyclictest9254-21sed20:25:351
3016599178,6cyclictest18159-21cut19:30:201
3016599177,7cyclictest21-21ksoftirqd/121:05:121
3016599174,10cyclictest806-21cut20:20:141
3016599172,11cyclictest0-21swapper/120:30:111
3016599172,10cyclictest0-21swapper/119:10:131
3016599171,8cyclictest808-21kthreadcore00:35:231
30165991710,5cyclictest24084-21irqstats20:10:211
30165991710,4cyclictest0-21swapper/120:50:301
3015599171,13cyclictest28468-21fschecks_time20:15:160
3018399162,10cyclictest0-21swapper/320:00:113
3018399161,9cyclictest0-21swapper/319:25:023
3016599168,6cyclictest7347-21sshd19:16:411
3016599161,7cyclictest18428-21munin-run20:05:011
3016599161,11cyclictest17098-21kthreadcore20:35:251
30165991610,4cyclictest0-21swapper/119:45:331
3015599161,10cyclictest30225-21latency_hist19:10:020
3018399152,10cyclictest0-21swapper/321:00:113
3018399152,10cyclictest0-21swapper/319:55:123
3018399152,10cyclictest0-21swapper/319:25:143
3018399151,8cyclictest0-21swapper/320:25:013
3018399151,8cyclictest0-21swapper/320:20:013
3018399151,11cyclictest0-21swapper/321:10:023
3016599151,11cyclictest2771-21sed20:55:211
3018399149,3cyclictest0-21swapper/320:30:263
3018399149,3cyclictest0-21swapper/319:40:163
3018399144,4cyclictest0-21swapper/320:10:123
3018399142,8cyclictest0-21swapper/319:15:013
3018399141,10cyclictest0-21swapper/319:50:023
3016599141,9cyclictest30214-21latency_hist19:10:011
3018399131,9cyclictest0-21swapper/319:30:163
3018399131,7cyclictest0-21swapper/320:45:103
3018399131,7cyclictest0-21swapper/319:35:193
3018399131,4cyclictest0-21swapper/320:09:043
3018399122,8cyclictest0-21swapper/320:35:143
3018399121,7cyclictest0-21swapper/319:50:243
3018399101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional