You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-13 - 22:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 13, 2025 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1983099674,47cyclictest10226-21ssh09:55:140
19839995120,27cyclictest21-21ksoftirqd/110:20:151
19839995120,27cyclictest21-21ksoftirqd/110:20:151
1984999492,5cyclictest131rcu_preempt09:30:142
1983999491,39cyclictest17445-21ssh12:00:151
1983099492,35cyclictest18004-21ssh10:00:170
1983099482,6cyclictest131rcu_preempt12:25:140
19849994725,12cyclictest27-21ksoftirqd/207:50:162
1983999472,34cyclictest27826-21ssh11:10:141
1983099462,39cyclictest29161-21ssh11:30:140
1986499444,26cyclictest32466irq/58-eth010:50:133
1984999442,27cyclictest3169-21ssh11:15:132
19864994320,16cyclictest32466irq/58-eth012:35:153
1986499423,11cyclictest32466irq/58-eth011:15:133
1984999421,28cyclictest0-21swapper/211:25:162
19864994017,14cyclictest32466irq/58-eth011:20:153
1983999406,25cyclictest32012-21ssh11:50:141
1983999399,22cyclictest21-21ksoftirqd/110:05:161
19839993923,12cyclictest0-21swapper/112:35:151
1983999382,8cyclictest131rcu_preempt09:05:141
1983999382,28cyclictest12047-21ssh11:00:131
1983999381,23cyclictest4488-21ssh10:55:141
1984999371,18cyclictest8486-21ssh11:55:132
19849993710,5cyclictest0-21swapper/208:10:132
1983999374,10cyclictest131rcu_preempt08:00:141
19830993718,13cyclictest13681-21diskmemload09:15:170
1986499362,19cyclictest32466irq/58-eth011:25:163
19849993619,13cyclictest261rcuc/210:40:142
1983999352,24cyclictest21-21ksoftirqd/108:55:141
1984999342,25cyclictest916-21snmpd08:50:162
1986499332,18cyclictest32466irq/58-eth008:40:173
19864993318,7cyclictest32466irq/58-eth010:25:133
19864993318,7cyclictest32466irq/58-eth010:25:133
1984999339,8cyclictest131rcu_preempt07:40:132
1984999333,18cyclictest23346irq/40-dwc2_hso08:35:152
19849993319,8cyclictest27-21ksoftirqd/212:06:102
1983999331,26cyclictest20012-21ssh11:05:121
1983999331,19cyclictest13707-21ssh10:40:131
19839993211,9cyclictest21-21ksoftirqd/107:25:151
1983099321,25cyclictest23547-21sh10:25:150
1983099321,25cyclictest23547-21sh10:25:150
19864993011,13cyclictest32466irq/58-eth011:10:143
1983999302,23cyclictest5565-21diskmemload12:20:141
1984999295,12cyclictest23346irq/40-dwc2_hso07:25:152
1984999294,16cyclictest23346irq/40-dwc2_hso07:15:142
1984999292,21cyclictest0-21swapper/207:20:162
1984999291,23cyclictest30098-21sh12:25:142
1984999291,18cyclictest916-21snmpd09:00:142
1983999291,5cyclictest131rcu_preempt07:55:131
1983999291,21cyclictest29152-21ssh11:30:141
1983099292,22cyclictest20132-21ssh09:40:150
1983099291,21cyclictest3178-21sh11:15:140
1984999282,5cyclictest27369-21sh09:45:142
1984999282,18cyclictest23346irq/40-dwc2_hso12:10:162
1984999281,17cyclictest8673-21apt-get10:15:142
1986499272,16cyclictest32466irq/58-eth011:55:133
1986499271,12cyclictest32466irq/58-eth007:10:153
1984999273,15cyclictest23346irq/40-dwc2_hso09:40:142
1984999272,15cyclictest23346irq/40-dwc2_hso08:20:152
1984999272,15cyclictest21412-21ssh09:20:152
19849992717,5cyclictest0-21swapper/208:00:152
1984999271,15cyclictest15047-21sshd09:55:342
1983099272,19cyclictest2173-21ssh12:09:340
1984999263,15cyclictest23346irq/40-dwc2_hso11:50:132
1984999262,17cyclictest23346irq/40-dwc2_hso08:40:172
1984999262,15cyclictest23346irq/40-dwc2_hso10:05:162
1983999268,11cyclictest0-21swapper/109:45:151
1983999262,18cyclictest21215-21apt-get07:45:151
1983099263,19cyclictest11585-21ssh11:20:160
1983099262,20cyclictest15798-21ssh11:58:340
1983099261,4cyclictest131rcu_preempt12:00:150
1986499251,17cyclictest32466irq/58-eth010:15:163
1986499251,15cyclictest32466irq/58-eth010:35:153
1984999258,10cyclictest0-21swapper/211:05:132
1984999253,15cyclictest23346irq/40-dwc2_hso07:30:142
1984999252,16cyclictest23346irq/40-dwc2_hso07:45:162
1984999252,15cyclictest23346irq/40-dwc2_hso08:45:142
1984999251,15cyclictest12099-21ssh09:35:042
1983999256,14cyclictest5565-21diskmemload11:35:161
1983999241,15cyclictest12021-21rm12:15:171
1983099249,8cyclictest31137-21ssh10:30:140
1983099249,8cyclictest31137-21ssh10:30:140
19830992415,6cyclictest22804-21sh11:45:020
1986499234,9cyclictest32466irq/58-eth011:00:133
1984999238,9cyclictest131rcu_preempt12:38:262
1984999237,13cyclictest11705-21grep07:35:182
1984999235,15cyclictest17933-21fschecks_time08:15:192
1984999232,6cyclictest0-21swapper/212:21:152
1984999232,6cyclictest0-21swapper/212:01:092
1984999232,16cyclictest0-21swapper/212:15:342
19849992314,7cyclictest0-21swapper/210:30:232
19849992314,7cyclictest0-21swapper/210:30:232
19849992312,9cyclictest27-21ksoftirqd/210:51:372
19849992312,8cyclictest261rcuc/210:20:152
19849992312,8cyclictest261rcuc/210:20:152
1984999231,19cyclictest27420-21ls08:25:152
1984999231,18cyclictest14398-21ssh11:40:152
19849992311,10cyclictest16943-21ssh11:00:352
1983099234,15cyclictest769-21ssh10:10:140
1983099232,17cyclictest29566-21apt-get07:20:150
1983099231,16cyclictest13717-21ssh10:40:140
1983099231,15cyclictest32286-21ls11:50:170
1984999227,10cyclictest27-21ksoftirqd/210:00:162
1984999223,15cyclictest23346irq/40-dwc2_hso10:45:172
1984999223,11cyclictest23346irq/40-dwc2_hso07:10:152
1984999221,9cyclictest23811-21ssh10:25:182
1984999221,9cyclictest23811-21ssh10:25:182
19849992212,4cyclictest23346irq/40-dwc2_hso09:10:152
19849992211,9cyclictest9312-21wc11:35:292
19849992211,9cyclictest11120-21ssh10:59:092
1984999221,17cyclictest17329-21grep09:15:302
1983999224,10cyclictest131rcu_preempt09:55:241
19839992211,6cyclictest5565-21diskmemload09:20:151
1983999221,15cyclictest29557-21find07:20:141
1983999221,15cyclictest21-21ksoftirqd/111:20:151
19839992210,8cyclictest21-21ksoftirqd/110:45:181
1983099222,17cyclictest27836-21ssh11:10:150
1986499212,12cyclictest32466irq/58-eth011:40:163
1986499211,11cyclictest0-21swapper/312:20:143
1984999219,9cyclictest156-21systemd-udevd09:05:222
1984999219,4cyclictest27-21ksoftirqd/211:21:002
1984999217,11cyclictest8042-21apt-get08:05:152
1984999211,9cyclictest441966ssh10:10:272
19849992111,8cyclictest0-21swapper/209:50:312
1984999211,16cyclictest26536-21kthreadcore08:55:222
1984999211,15cyclictest1034-21kthreadcore08:30:202
1984999211,15cyclictest1034-21kthreadcore08:30:202
1984999211,12cyclictest7618-21kthreadcore12:30:202
1984999211,12cyclictest31719-21kthreadcore07:55:192
1984999211,11cyclictest30635-21kthreadcore11:10:232
1984999211,10cyclictest30773-21cut11:30:222
1983999213,13cyclictest21-21ksoftirqd/112:06:101
1983999212,14cyclictest8678-21ssh10:15:151
1983999211,16cyclictest26209-21switchtime08:20:291
1983999211,11cyclictest0-21swapper/108:50:021
19839992110,3cyclictest0-21swapper/109:30:131
1983099217,3cyclictest131rcu_preempt07:40:010
1983099211,9cyclictest6130-21/usr/sbin/munin09:10:180
19830992112,6cyclictest12696-21tail10:15:310
19830992110,8cyclictest3364-21sed12:10:200
1983099210,14cyclictest12754-21ls08:10:020
1986499209,6cyclictest0-21swapper/311:52:233
19864992011,6cyclictest32466irq/58-eth012:00:143
1984999202,9cyclictest8296-21kthreadcore10:35:242
1984999202,9cyclictest25488-21kthreadcore11:45:232
1984999202,10cyclictest29338-21fschecks_time09:25:182
1983999209,8cyclictest25972-21apt-get07:50:161
1983999202,11cyclictest5565-21diskmemload09:35:161
19839992013,4cyclictest9426-21cut11:55:201
19839992010,7cyclictest29345-21basename07:20:021
1983099208,8cyclictest0-21swapper/009:25:150
1983099207,10cyclictest612-21dbus-daemon08:10:210
1983099207,10cyclictest20547-21sh12:20:010
1983099202,15cyclictest8755-21cut07:30:220
1983099202,14cyclictest662-21ntpd12:30:130
19830992011,6cyclictest28830-21cron09:25:010
1986499199,8cyclictest0-21swapper/310:34:463
1986499199,8cyclictest0-21swapper/310:34:463
1986499198,8cyclictest0-21swapper/309:52:033
1986499193,11cyclictest0-21swapper/312:06:103
1986499192,11cyclictest0-21swapper/310:45:173
1986499192,10cyclictest0-21swapper/307:50:163
19864991912,5cyclictest0-21swapper/312:25:213
1986499191,12cyclictest0-21swapper/308:55:153
1986499191,12cyclictest0-21swapper/308:10:153
1983999197,3cyclictest0-21swapper/110:35:171
1983999192,12cyclictest3219-21perl12:10:191
1983999192,11cyclictest18005-21ssh10:00:171
1983999191,15cyclictest8301-21cron10:15:011
1983999191,14cyclictest6561-21ssh12:30:131
1983999191,14cyclictest23548-21ssh10:25:151
1983999191,14cyclictest23548-21ssh10:25:151
1983999191,10cyclictest14993-21kthreadcore08:10:211
19839991910,6cyclictest30693-21cron10:30:051
19839991910,6cyclictest30693-21cron10:30:051
1983099192,11cyclictest23228-21ssh11:45:160
1983099191,15cyclictest19279-21tr08:50:210
1983099190,12cyclictest20223-21ls07:10:190
1986499189,7cyclictest0-21swapper/311:47:033
1986499189,6cyclictest0-21swapper/310:22:323
1986499189,6cyclictest0-21swapper/310:22:323
1986499189,6cyclictest0-21swapper/309:10:193
1986499188,6cyclictest0-21swapper/309:57:283
1986499187,7cyclictest32466irq/58-eth011:30:153
1986499183,10cyclictest0-21swapper/312:15:343
1986499182,10cyclictest0-21swapper/309:30:153
1986499182,10cyclictest0-21swapper/309:00:143
1986499182,10cyclictest0-21swapper/308:20:153
1986499182,10cyclictest0-21swapper/308:15:043
1986499182,10cyclictest0-21swapper/307:55:143
1986499182,10cyclictest0-21swapper/307:35:163
1986499181,13cyclictest0-21swapper/307:25:023
1986499181,12cyclictest0-21swapper/309:20:243
1986499181,12cyclictest0-21swapper/308:30:233
1986499181,12cyclictest0-21swapper/308:30:233
1986499181,10cyclictest0-21swapper/311:05:133
19864991810,6cyclictest0-21swapper/311:36:343
19864991810,6cyclictest0-21swapper/309:41:353
19864991810,5cyclictest32466irq/58-eth012:10:163
19864991810,5cyclictest32466irq/58-eth009:45:143
1984999181,8cyclictest19934-21latency_hist07:10:012
1983999182,13cyclictest17420-21apt-get08:15:031
1983999181,6cyclictest16402-21sed11:40:231
19839991811,5cyclictest3746-21sh11:15:191
19839991810,6cyclictest23508-21sshd08:53:341
1983099188,7cyclictest19812-21cyclictest12:20:150
1983099186,3cyclictest131rcu_preempt08:30:010
1983099183,12cyclictest24862-21kthreadcore08:55:200
1983099182,14cyclictest662-21ntpd10:35:180
1983099182,12cyclictest2529-21ssh09:50:190
1983099182,11cyclictest12101-21ssh09:35:040
1983099181,7cyclictest131rcu_preempt12:35:230
19830991812,4cyclictest0-21swapper/011:00:120
1983099181,14cyclictest13183-21unixbench_multi08:40:320
1983099181,13cyclictest19547-21ssh10:20:280
1983099181,13cyclictest19547-21ssh10:20:280
1986499179,6cyclictest0-21swapper/310:55:523
1986499179,6cyclictest0-21swapper/310:42:003
1986499179,6cyclictest0-21swapper/310:05:213
1986499179,6cyclictest0-21swapper/309:35:123
1986499179,6cyclictest0-21swapper/309:27:333
1986499179,6cyclictest0-21swapper/309:18:383
1986499179,5cyclictest32466irq/58-eth012:30:133
1986499178,6cyclictest0-21swapper/310:11:093
1986499177,8cyclictest0-21swapper/310:01:533
1986499172,9cyclictest0-21swapper/307:25:163
1986499172,10cyclictest0-21swapper/309:05:153
1986499172,10cyclictest0-21swapper/308:00:153
1986499172,10cyclictest0-21swapper/307:45:173
1986499172,10cyclictest0-21swapper/307:40:153
1986499171,10cyclictest0-21swapper/308:10:023
1983999179,6cyclictest9372-21apt-get08:40:151
1983999179,5cyclictest23484-21grep07:10:291
1983999178,7cyclictest1369-21awk12:25:281
1983999176,7cyclictest0-21swapper/108:05:151
1983999175,9cyclictest14535-21kthreadcore09:15:231
1983999173,10cyclictest21152-21latency_hist07:45:021
1983999172,12cyclictest29053-21ssh10:50:141
1983999171,6cyclictest16200-21latency_hist07:40:011
1983999171,13cyclictest9768-21kthreadcore07:30:231
1983999171,13cyclictest31181-21kthreadcore09:25:221
1983999171,13cyclictest29285-21cut09:00:191
1983999171,10cyclictest20525-21diskstats11:25:181
19839991710,4cyclictest6064-21diskstats09:10:171
19839991710,4cyclictest31815-21latency_hist11:50:021
1983099179,4cyclictest5006-21cut10:55:190
1983099176,8cyclictest25540-21cat10:45:330
1983099174,10cyclictest30014-21kthreadcore10:50:200
1983099173,11cyclictest27204-21kthreadcore07:15:230
1983099173,11cyclictest19812-21cyclictest09:00:140
1983099172,3cyclictest131rcu_preempt09:30:140
1983099172,11cyclictest5185-21cat07:25:240
1983099172,11cyclictest0-21swapper/008:30:140
1983099172,11cyclictest0-21swapper/008:30:130
1983099172,11cyclictest0-21swapper/007:45:170
1983099172,10cyclictest0-21swapper/008:15:170
1983099171,7cyclictest27343-21/usr/sbin/munin09:45:240
19830991711,4cyclictest856-21sshd11:26:020
1983099171,13cyclictest4566-21kthreadcore08:00:210
1983099171,13cyclictest2158-21timerwakeupswit07:55:280
1983099171,13cyclictest14083-21latency_hist11:40:010
1983099171,12cyclictest20172-21fschecks_count11:05:170
1986499162,10cyclictest0-21swapper/308:35:173
1986499161,12cyclictest0-21swapper/307:20:023
1983999166,7cyclictest4516-21sort08:35:151
1983999161,12cyclictest27365-21apt-get08:25:151
1983999161,12cyclictest2611-21ls09:50:201
1983999161,12cyclictest1613-21tr08:30:201
1983999161,12cyclictest1613-21tr08:30:201
1983999161,11cyclictest20220-21df09:40:161
1983099169,4cyclictest5743-21latency_hist09:10:020
1983099162,9cyclictest0-21swapper/008:45:150
1983099162,10cyclictest0-21swapper/008:35:170
19830991611,3cyclictest22946-21cut08:20:190
1983099161,11cyclictest25907-21df_inode10:05:170
19830991610,4cyclictest16576-21fschecks_time07:40:170
1983999151,10cyclictest19745-21latency_hist07:10:011
1983099151,11cyclictest19936-21wc07:10:010
1983099151,10cyclictest26202-21hwlatdetect07:50:190
1986499123,7cyclictest0-21swapper/308:25:233
1986499122,7cyclictest0-21swapper/308:45:303
1986499122,6cyclictest0-21swapper/307:30:153
1986499112,5cyclictest0-21swapper/308:50:173
1986499111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional