You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-07 - 09:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 07, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
202599503,39cyclictest28790-21ssh00:30:171
2049994814,20cyclictest29566irq/58-eth021:10:133
202599472,3cyclictest131rcu_preempt23:15:141
202599443,33cyclictest16889-21ssh22:05:121
2025994426,6cyclictest21-21ksoftirqd/123:25:131
202599426,20cyclictest19198-21ssh23:05:131
204999411,27cyclictest29566irq/58-eth020:00:133
2025994012,18cyclictest21-21ksoftirqd/100:00:181
203899371,20cyclictest28943-21sh23:50:142
201499375,26cyclictest5281-21ssh21:20:130
2025993519,5cyclictest21-21ksoftirqd/120:50:161
2014993529,3cyclictest131rcu_preempt19:15:100
2049993322,6cyclictest29666irq/59-eth021:20:133
201499333,7cyclictest131rcu_preempt00:05:130
2049993212,16cyclictest0-21swapper/323:15:153
203899322,18cyclictest15420-21cat23:00:282
203899322,16cyclictest30106-21ssh22:30:262
203899314,24cyclictest261rcuc/223:35:142
203899312,17cyclictest5892-21ssh22:35:282
202599313,19cyclictest201rcuc/121:40:141
2049993013,12cyclictest29566irq/58-eth022:40:133
204999301,12cyclictest29566irq/58-eth022:35:163
203899303,19cyclictest26346irq/40-dwc2_hso19:35:132
203899301,18cyclictest9804-21ssh22:20:142
2025993011,13cyclictest21-21ksoftirqd/119:10:131
201499302,23cyclictest3734-21ssh23:35:150
204999292,17cyclictest29566irq/58-eth022:45:143
203899294,18cyclictest26346irq/40-dwc2_hso20:40:162
203899293,18cyclictest26346irq/40-dwc2_hso00:25:132
202599293,21cyclictest10552-21kworker/u8:322:45:151
201499298,4cyclictest131rcu_preempt19:35:140
201499293,7cyclictest131rcu_preempt19:50:130
201499291,25cyclictest18225-21ssh22:25:130
201499291,24cyclictest12177-21ssh23:40:140
204999283,18cyclictest0-21swapper/322:26:283
204999283,16cyclictest0-21swapper/323:55:283
204999280,17cyclictest0-21swapper/323:50:153
203899286,17cyclictest27-21ksoftirqd/221:53:272
203899283,19cyclictest8429-21ssh22:00:152
203899283,11cyclictest16560-21kworker/2:222:25:132
202599282,20cyclictest6645-21grep22:55:281
202599282,20cyclictest13743-21ssh21:25:161
201499286,9cyclictest131rcu_preempt20:45:130
201499285,6cyclictest131rcu_preempt20:15:150
201499281,6cyclictest131rcu_preempt20:40:160
201499281,24cyclictest17001-21ssh00:00:280
201499281,23cyclictest26403-21ssh22:30:130
204999273,16cyclictest0-21swapper/323:00:283
2049992713,11cyclictest0-21swapper/323:40:143
203899275,8cyclictest10607-21ssh22:40:132
203899273,19cyclictest0-21swapper/221:34:272
203899273,18cyclictest0-21swapper/221:55:272
203899272,9cyclictest25676-21ssh23:27:272
203899272,19cyclictest0-21swapper/219:45:142
203899271,16cyclictest5285-21ssh21:20:132
203899271,11cyclictest11775-21ssh23:58:232
202599273,19cyclictest0-21swapper/122:43:021
202599273,15cyclictest20990-21ssh21:10:131
202599272,20cyclictest27474-21ssh23:48:281
202599272,20cyclictest27474-21ssh23:48:281
202599272,19cyclictest8835-21timerandwakeup23:55:281
202599272,19cyclictest16045-21ssh22:22:281
201499278,16cyclictest11003-21sh23:00:130
201499271,23cyclictest8424-21ssh22:00:150
204999263,15cyclictest0-21swapper/323:20:283
204999261,14cyclictest29566irq/58-eth023:05:123
203899261,17cyclictest26346irq/40-dwc2_hso19:15:112
201499261,22cyclictest20632-21diskmemload23:50:140
2049992513,7cyclictest29566irq/58-eth021:35:123
204999251,18cyclictest29566irq/58-eth023:25:143
204999251,13cyclictest29566irq/58-eth000:00:183
203899259,7cyclictest15099-21/usr/sbin/munin23:20:272
203899253,16cyclictest26346irq/40-dwc2_hso22:50:142
203899252,19cyclictest20632-21diskmemload00:02:272
203899252,18cyclictest19922-21ssh21:27:282
203899252,17cyclictest26346irq/40-dwc2_hso23:10:132
203899251,11cyclictest21155-21grep21:10:162
2025992514,7cyclictest0-21swapper/123:40:151
201499254,8cyclictest131rcu_preempt20:30:150
201499253,16cyclictest27503-21ssh23:10:130
201499251,21cyclictest13024-21ssh00:20:160
201499251,19cyclictest29502-21ssh00:10:140
201499251,18cyclictest15264-21expr23:20:280
204999241,17cyclictest0-21swapper/300:33:283
204999241,14cyclictest29566irq/58-eth021:25:143
203899249,12cyclictest19517-21apt-get20:35:152
203899242,19cyclictest0-21swapper/219:50:152
203899242,16cyclictest0-21swapper/200:15:282
203899241,8cyclictest1473666ssh21:44:472
203899241,11cyclictest2926266ssh23:30:202
2038992411,10cyclictest0-21swapper/200:20:192
202599242,17cyclictest6257-21ssh22:15:341
202599241,18cyclictest9976-21ssh23:37:281
201499244,8cyclictest12-21ksoftirqd/023:05:130
201499244,13cyclictest0-21swapper/022:40:130
201499243,5cyclictest131rcu_preempt22:50:130
201499243,16cyclictest12-21ksoftirqd/020:10:130
201499242,19cyclictest25377-21sh22:10:140
204999233,13cyclictest0-21swapper/322:22:283
203899238,7cyclictest155-21systemd-udevd00:37:132
203899232,16cyclictest0-21swapper/200:10:152
2025992313,7cyclictest0-21swapper/123:55:001
202599231,18cyclictest14776-21cut20:30:191
201499231,6cyclictest0-21swapper/023:55:280
201499231,16cyclictest31135-21find19:40:120
201499231,16cyclictest31135-21find19:40:120
2014992310,8cyclictest121ksoftirqd/019:45:120
201499230,2cyclictest131rcu_preempt22:20:140
204999222,13cyclictest0-21swapper/321:55:273
203899229,10cyclictest0-21swapper/223:40:172
203899227,7cyclictest29258-21apt-get21:15:152
203899221,17cyclictest24027-21kthreadcore00:05:212
203899221,14cyclictest20377-21ls22:05:232
2038992211,4cyclictest19579-21sh23:05:182
203899221,11cyclictest28952-21fschecks_count00:30:202
203899221,11cyclictest25373-21ssh22:10:132
202599222,16cyclictest2241-21ssh22:35:161
202599221,18cyclictest29411-21ls20:45:171
201499226,3cyclictest131rcu_preempt22:15:300
201499221,9cyclictest21085-21kthreadcore20:00:220
2014992213,6cyclictest0-21swapper/021:35:010
204999212,14cyclictest29566irq/58-eth022:05:113
203899219,5cyclictest21019-21cat19:25:332
203899218,6cyclictest21478-21ntpq20:00:252
203899213,8cyclictest13346-21kthreadcore19:20:222
203899213,15cyclictest7014-21kthreadcore20:20:212
203899212,4cyclictest27-21ksoftirqd/222:56:342
203899212,10cyclictest21763-21kthreadcore23:45:202
203899212,10cyclictest21763-21kthreadcore23:45:202
203899212,10cyclictest1618-21ssh22:15:172
2038992115,4cyclictest0-21swapper/219:10:012
203899211,12cyclictest3288-21ssh23:15:142
203899211,11cyclictest30682-21kthreadcore20:10:222
203899211,11cyclictest19017-21ssh22:45:172
2038992110,8cyclictest16459-21kthreadcore21:45:232
202599211,9cyclictest29249-21switchtime22:10:291
202599211,17cyclictest19261-21ls23:25:011
2025992111,7cyclictest11380-21cron00:40:001
2025992111,5cyclictest8768-21/usr/sbin/munin00:15:291
201499216,10cyclictest508-21ssh21:15:300
201499211,3cyclictest131rcu_preempt21:55:140
201499211,3cyclictest131rcu_preempt21:10:130
201499211,17cyclictest24762-21ls00:25:270
201499211,15cyclictest18907-21rm22:45:160
204999205,9cyclictest0-21swapper/300:15:173
204999202,12cyclictest0-21swapper/300:27:283
203899208,5cyclictest2107-21/usr/sbin/munin19:40:232
203899208,5cyclictest2107-21/usr/sbin/munin19:40:232
203899201,9cyclictest29379-21fschecks_time20:45:162
203899201,9cyclictest2026-21irqcore20:50:212
203899201,16cyclictest9856-21cut20:25:172
203899201,15cyclictest21652-21fschecks_time19:30:192
203899201,11cyclictest6468-21sort20:55:152
203899201,10cyclictest17316-21kthreadcore21:05:182
203899201,10cyclictest1635-21kthreadcore21:35:212
203899201,10cyclictest11745-21tr21:00:192
203899201,10cyclictest1104-21kthreadcore20:15:222
203899201,10cyclictest0-21swapper/220:30:152
203899200,17cyclictest13840-21kthreadcore19:55:212
202599209,8cyclictest25491-21sed00:05:281
202599209,8cyclictest10238-21grep20:55:291
202599202,12cyclictest1880-21ssh23:13:281
202599201,16cyclictest23753-21ssh21:50:141
202599201,15cyclictest32654-21cut20:15:191
202599201,15cyclictest32228-21ssh21:55:141
2025992010,7cyclictest24546-21cut19:30:251
201499201,8cyclictest0-21swapper/000:39:490
201499201,5cyclictest0-21swapper/021:36:470
201499201,16cyclictest2344-21sh22:35:180
201499201,12cyclictest21527-21ls19:30:170
204999198,8cyclictest0-21swapper/322:13:543
204999193,11cyclictest0-21swapper/323:48:283
204999193,11cyclictest0-21swapper/323:48:283
204999193,11cyclictest0-21swapper/323:38:153
204999193,11cyclictest0-21swapper/323:30:283
204999193,11cyclictest0-21swapper/322:58:283
204999193,11cyclictest0-21swapper/322:51:273
204999193,11cyclictest0-21swapper/322:30:273
204999193,11cyclictest0-21swapper/321:50:283
204999193,11cyclictest0-21swapper/300:37:273
204999193,10cyclictest29566irq/58-eth021:30:143
204999193,10cyclictest0-21swapper/300:05:273
203899191,9cyclictest2668-21sed19:10:192
203899191,7cyclictest22976-21fschecks_count20:05:202
202599199,7cyclictest1-21systemd20:22:291
2025991913,4cyclictest26500-21sh22:30:141
202599191,16cyclictest16599-21ls22:05:011
202599191,15cyclictest24346-21apt-get20:40:161
2025991910,7cyclictest31369-21ntp_kernel_err23:30:241
2025991910,6cyclictest15076-21grep21:00:291
202599190,15cyclictest21677-21ls22:25:211
201499198,8cyclictest9701-21apt-get20:25:130
201499197,9cyclictest1-21systemd23:28:140
201499195,2cyclictest131rcu_preempt21:41:450
201499192,12cyclictest15347-21ssh21:45:170
201499192,11cyclictest8783-21ssh00:15:290
201499191,8cyclictest10093-21find20:55:290
2014991911,5cyclictest0-21swapper/023:45:170
2014991911,5cyclictest0-21swapper/023:45:170
201499191,12cyclictest2839-21ssh00:33:270
201499191,10cyclictest12829-21cut21:00:210
201499191,10cyclictest10552-21kworker/u8:322:55:210
201499190,2cyclictest131rcu_preempt21:50:140
204999189,6cyclictest0-21swapper/321:15:443
204999188,6cyclictest29566irq/58-eth022:15:123
204999183,9cyclictest0-21swapper/323:10:283
204999183,10cyclictest0-21swapper/300:11:283
204999182,11cyclictest0-21swapper/322:00:153
204999182,10cyclictest0-21swapper/321:05:123
204999182,10cyclictest0-21swapper/320:40:173
204999182,10cyclictest0-21swapper/320:35:043
204999182,10cyclictest0-21swapper/320:30:043
204999182,10cyclictest0-21swapper/320:25:153
204999182,10cyclictest0-21swapper/320:10:143
204999181,8cyclictest29566irq/58-eth019:35:133
2049991811,4cyclictest0-21swapper/321:40:233
204999181,13cyclictest0-21swapper/320:55:013
204999181,11cyclictest0-21swapper/321:45:233
204999181,10cyclictest0-21swapper/319:45:173
2049991810,6cyclictest0-21swapper/300:20:163
202599185,3cyclictest131rcu_preempt19:15:161
202599184,9cyclictest2001-21cyclictest21:35:121
202599183,10cyclictest0-21swapper/123:03:281
202599183,10cyclictest0-21swapper/122:52:281
202599183,10cyclictest0-21swapper/120:25:151
202599183,10cyclictest0-21swapper/100:26:281
202599181,9cyclictest24173-21kthreadcore21:30:211
202599181,8cyclictest13115-21df_inode00:20:171
202599181,14cyclictest27835-21tune2fs20:10:161
201499188,5cyclictest5064-21kthreadcore23:15:210
201499182,5cyclictest12-21ksoftirqd/020:20:290
201499181,7cyclictest13844-21/usr/sbin/munin21:25:170
2014991812,3cyclictest17777-21idleruntime-cro20:00:010
201499181,13cyclictest16991-21df22:05:140
201499181,12cyclictest932-21snmpd20:05:180
2014991810,6cyclictest6179-21cron20:55:010
204999172,7cyclictest0-21swapper/319:25:163
202599176,2cyclictest131rcu_preempt21:45:151
202599172,10cyclictest0-21swapper/119:20:151
202599171,6cyclictest6118-21kthreadcore21:20:191
202599171,13cyclictest25097-21kthreadcore20:05:251
202599171,12cyclictest11769-21ntp_192.168.11419:50:251
202599171,10cyclictest22699-21ls20:05:011
2025991710,4cyclictest30217-21kthreadcore00:10:201
202599170,15cyclictest0-21swapper/121:15:151
201499179,6cyclictest20141-21sshd19:25:290
201499171,7cyclictest15532-21find19:20:300
2014991711,3cyclictest29735-21kthreadcore23:30:200
204999162,7cyclictest0-21swapper/320:20:143
204999162,10cyclictest0-21swapper/320:55:153
204999162,10cyclictest0-21swapper/319:30:173
202599168,6cyclictest32327-21cut19:40:201
202599168,6cyclictest32327-21cut19:40:201
202599166,5cyclictest19907-21irqcore20:35:201
202599163,9cyclictest2001-21cyclictest19:25:161
202599162,10cyclictest8144-21idleruntime-cro19:50:001
202599161,7cyclictest13752-21irqrtprio19:55:201
202599161,13cyclictest26681-21cut19:35:181
2025991611,3cyclictest19510-21ntp_offset21:05:231
201499169,5cyclictest6820-21latency_hist19:15:010
201499162,11cyclictest20240-21kthreadcore20:35:210
201499161,10cyclictest20830-21latency_hist21:10:020
204999159,4cyclictest29566irq/58-eth021:00:303
204999152,9cyclictest0-21swapper/319:15:123
204999151,11cyclictest0-21swapper/320:15:263
204999151,11cyclictest0-21swapper/319:10:173
201499151,11cyclictest2032-21grep19:10:010
204999149,3cyclictest0-21swapper/320:08:273
204999146,6cyclictest0-21swapper/320:49:413
202599141,9cyclictest2074-21wc19:10:011
204999138,3cyclictest0-21swapper/319:58:113
204999138,3cyclictest0-21swapper/319:54:303
204999133,6cyclictest0-21swapper/319:40:143
204999133,6cyclictest0-21swapper/319:40:143
204999122,7cyclictest0-21swapper/319:20:133
204999101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional