You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 05:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Wed Aug 27, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
303299643,54cyclictest7641-21ssh22:40:191
303299643,54cyclictest7641-21ssh22:40:181
302499621,13cyclictest0-21swapper/020:35:190
303299551,45cyclictest5329-21ssh21:20:141
304099532,32cyclictest21717-21ssh21:30:132
304099505,17cyclictest0-21swapper/223:55:152
302499487,33cyclictest25413-21ssh23:50:150
303299473,35cyclictest32239-21ssh22:55:131
303299473,35cyclictest32239-21ssh22:55:121
303299466,5cyclictest131rcu_preempt00:35:131
304899454,33cyclictest25366irq/58-eth000:30:153
3040994519,14cyclictest27-21ksoftirqd/223:35:152
302499454,33cyclictest15273-21ssh22:25:160
302499443,33cyclictest29995-21ssh21:35:170
304899433,24cyclictest25366irq/58-eth022:40:203
304899433,24cyclictest25366irq/58-eth022:40:193
3032994215,18cyclictest21-21ksoftirqd/122:20:171
304899412,25cyclictest25366irq/58-eth021:20:143
3040994125,7cyclictest261rcuc/222:10:122
303299418,27cyclictest21-21ksoftirqd/119:15:141
304899402,25cyclictest25366irq/58-eth021:40:173
3048993918,13cyclictest25366irq/58-eth022:25:163
304099382,28cyclictest19703-21cat22:45:282
304899373,24cyclictest25366irq/58-eth022:10:143
304099363,17cyclictest7977-21ssh23:00:142
303299364,25cyclictest837-21munin-node23:40:191
302499366,12cyclictest12-21ksoftirqd/023:45:190
304899352,20cyclictest25366irq/58-eth022:30:173
302499354,22cyclictest20996-21diskmemload00:10:130
304899344,19cyclictest25366irq/58-eth021:25:153
304099346,17cyclictest8730-21ssh23:20:192
3040993410,8cyclictest27-21ksoftirqd/222:20:162
302499342,23cyclictest31355-21ssh00:15:150
302499341,25cyclictest0-21swapper/019:25:140
304899334,15cyclictest25366irq/58-eth022:15:153
3048993310,11cyclictest0-21swapper/323:57:523
304099331,4cyclictest131rcu_preempt20:05:152
304099331,4cyclictest131rcu_preempt20:05:142
304099331,17cyclictest18281-21ssh22:05:282
303299331,23cyclictest20996-21diskmemload22:45:141
302499333,13cyclictest20996-21diskmemload23:15:170
3024993318,12cyclictest29400-21kworker/u8:222:15:160
304099321,27cyclictest24163-21sh22:50:142
303299322,24cyclictest14069-21ssh21:45:141
304099313,21cyclictest0-21swapper/223:40:182
3032993118,8cyclictest3012-21cyclictest23:55:141
304899303,12cyclictest0-21swapper/322:45:283
303299304,18cyclictest21-21ksoftirqd/119:40:151
303299302,22cyclictest29627-21ssh22:13:291
3032993016,10cyclictest629-21Async08:52:161
302499302,20cyclictest0-21swapper/022:35:150
304099293,11cyclictest6372-21ssh22:00:162
304099292,21cyclictest5610-21ssh22:37:452
304099292,20cyclictest0-21swapper/221:45:132
3040992910,11cyclictest3012-21cyclictest23:05:162
3032992915,7cyclictest3012-21cyclictest00:20:141
303299291,22cyclictest0-21swapper/120:30:131
302499296,18cyclictest12-21ksoftirqd/021:25:150
302499291,7cyclictest12-21ksoftirqd/020:40:130
3024992915,9cyclictest12-21ksoftirqd/022:50:160
304899281,13cyclictest25366irq/58-eth021:45:143
304099284,16cyclictest21846irq/40-dwc2_hso20:25:142
3040992816,7cyclictest261rcuc/223:45:192
304099281,18cyclictest4524-21find19:45:152
302499281,21cyclictest21389-21ssh21:10:150
302499281,21cyclictest21389-21ssh21:10:140
304099272,10cyclictest11110-21ssh22:40:282
304099272,10cyclictest11110-21ssh22:40:272
303299273,16cyclictest25418-21ssh23:50:161
303299272,20cyclictest29839-21ssh21:54:291
302499274,5cyclictest131rcu_preempt00:00:140
302499272,18cyclictest9996-21ntp_kernel_pll_22:00:280
302499272,18cyclictest16153-21/usr/sbin/munin23:05:290
304099263,16cyclictest21846irq/40-dwc2_hso19:40:132
304099261,2cyclictest0-21swapper/222:25:152
303299262,18cyclictest19849-21ssh23:05:281
302499265,16cyclictest12-21ksoftirqd/022:05:150
302499263,7cyclictest131rcu_preempt22:32:460
302499262,18cyclictest11791-21/usr/sbin/munin23:00:270
302499262,14cyclictest5892-21ssh21:40:170
3024992614,8cyclictest12-21ksoftirqd/019:10:120
304899252,17cyclictest0-21swapper/323:07:503
304899252,17cyclictest0-21swapper/322:22:273
304099259,12cyclictest19192-21sshd20:30:292
304099254,15cyclictest21846irq/40-dwc2_hso20:35:192
304099254,13cyclictest21846irq/40-dwc2_hso20:55:132
304099252,8cyclictest569-21kworker/2:120:15:172
304099252,16cyclictest21846irq/40-dwc2_hso20:45:172
3040992513,5cyclictest20350-21ssh00:28:442
304099251,11cyclictest31220-21irqrtprio21:55:212
304099251,11cyclictest31220-21irqrtprio21:55:202
303299253,15cyclictest0-21swapper/119:35:151
303299252,18cyclictest20996-21diskmemload22:02:281
304899243,14cyclictest0-21swapper/322:59:283
304899243,14cyclictest0-21swapper/322:59:273
3048992411,8cyclictest25366irq/58-eth023:20:193
304899241,12cyclictest25366irq/58-eth023:00:153
304099245,14cyclictest21846irq/40-dwc2_hso21:40:182
304099241,18cyclictest23566-21ssh22:30:182
304099241,14cyclictest18795-21kthreadcore19:25:212
302499242,16cyclictest15867-21sh22:45:140
3024992411,4cyclictest151rcuc/021:20:240
3024992410,3cyclictest0-21swapper/000:07:020
304899233,13cyclictest0-21swapper/322:53:283
304899233,13cyclictest0-21swapper/321:54:293
304899232,11cyclictest25366irq/58-eth023:35:143
304099238,12cyclictest20996-21diskmemload21:35:232
304099232,19cyclictest0-21swapper/200:05:152
3040992312,9cyclictest14702-21ssh00:01:472
3040992312,9cyclictest0-21swapper/220:00:142
304099231,18cyclictest17135-21ssh21:25:262
304099231,12cyclictest30309-21ssh21:55:022
304099230,15cyclictest31955-21ls19:40:022
304899221,15cyclictest0-21swapper/321:55:153
304899221,15cyclictest0-21swapper/321:55:143
304099225,12cyclictest27-21ksoftirqd/222:15:162
304099224,10cyclictest17553-21kernelversion23:25:242
304099222,15cyclictest0-21swapper/200:13:282
304099222,14cyclictest21846irq/40-dwc2_hso21:00:142
304099221,7cyclictest32307-21ls22:55:152
304099221,7cyclictest32307-21ls22:55:142
3040992211,9cyclictest12400-21ssh00:22:432
303299229,10cyclictest0-21swapper/122:50:141
303299226,4cyclictest131rcu_preempt00:25:221
303299222,16cyclictest17358-21ssh23:45:181
303299221,17cyclictest3324-21cut19:10:161
302499228,5cyclictest131rcu_preempt22:55:120
302499228,5cyclictest131rcu_preempt22:55:110
302499222,8cyclictest12-21ksoftirqd/019:50:140
302499222,18cyclictest25940-21ntp_192.168.11421:50:300
3024992211,3cyclictest0-21swapper/021:17:420
3024992210,8cyclictest12-21ksoftirqd/022:12:450
3024992210,3cyclictest12-21ksoftirqd/023:10:450
304899217,7cyclictest25366irq/58-eth022:00:173
3048992113,5cyclictest0-21swapper/321:35:333
304899211,13cyclictest0-21swapper/323:40:193
304899211,12cyclictest0-21swapper/323:10:173
304099217,11cyclictest6646-21kthreadcore21:20:212
304099216,6cyclictest6819-21kthreadcore20:20:202
304099214,9cyclictest26175-21cat00:30:342
304099212,10cyclictest942-21snmpd19:15:162
304099211,6cyclictest32726-21sh23:15:172
304099211,17cyclictest31636-21kthreadcore00:35:222
304099211,16cyclictest28099-21awk23:10:302
304099211,12cyclictest4918-21cron21:20:002
304099211,11cyclictest21463-21cpuspeed21:10:162
304099211,11cyclictest21463-21cpuspeed21:10:152
304099211,10cyclictest17544-21latency_hist19:25:022
3040992110,9cyclictest136-21systemd-journal00:17:532
303299219,8cyclictest21-21ksoftirqd/123:35:141
303299212,15cyclictest21445-21ssh00:30:161
303299211,16cyclictest16857-21ssh23:25:181
3032992111,5cyclictest0-21swapper/120:55:121
303299210,15cyclictest27005-21ls22:30:261
302499218,10cyclictest17928-21grep19:55:310
302499212,3cyclictest131rcu_preempt00:23:090
302499211,17cyclictest73-21kswapd000:30:150
302499211,14cyclictest13131-21ssh22:22:270
302499211,13cyclictest13880-21find00:25:140
302499210,18cyclictest707-21ls20:15:010
304899209,6cyclictest0-21swapper/323:46:023
3048992010,8cyclictest0-21swapper/300:10:063
3048992010,4cyclictest0-21swapper/300:00:223
304099203,14cyclictest21846irq/40-dwc2_hso20:10:152
304099202,11cyclictest25055-21ssh23:30:142
304099202,10cyclictest10955-21kthreadcore19:50:202
304099201,9cyclictest2139-21fschecks_time20:50:172
304099201,9cyclictest19781-21wc21:05:252
304099201,15cyclictest3114-21latency_hist19:10:022
304099201,10cyclictest4219-21kthreadcore19:10:192
304099201,10cyclictest28218-21sed20:40:242
304099201,10cyclictest17967-21grep19:55:312
304099200,17cyclictest22462-21ls19:30:152
303299208,9cyclictest12861-21kthreadcore21:00:221
303299204,4cyclictest0-21swapper/121:35:171
303299203,11cyclictest9003-21idleruntime23:20:241
303299201,15cyclictest7982-21ssh23:00:151
3032992010,7cyclictest16680-21df_inode00:05:151
3032992010,5cyclictest21725-21sh21:30:141
302499207,6cyclictest131rcu_preempt20:20:230
302499206,5cyclictest12-21ksoftirqd/000:35:320
302499203,12cyclictest0-21swapper/021:00:160
3024992014,3cyclictest12-21ksoftirqd/020:20:020
3024992011,7cyclictest0-21swapper/023:20:340
302499200,16cyclictest23607-21ls20:05:010
302499200,15cyclictest20113-21timerandwakeup21:05:300
304899199,7cyclictest0-21swapper/322:35:163
304899199,3cyclictest0-21swapper/321:32:003
304899196,7cyclictest25366irq/58-eth023:15:183
304899191,13cyclictest0-21swapper/320:30:013
304899191,13cyclictest0-21swapper/319:45:303
304899191,12cyclictest0-21swapper/320:30:173
304899191,12cyclictest0-21swapper/320:20:173
3048991910,6cyclictest0-21swapper/300:35:263
304099192,8cyclictest25569-21fschecks_count23:50:172
303299192,13cyclictest21370-21ssh21:10:131
303299192,13cyclictest21370-21ssh21:10:131
303299192,12cyclictest4987-21ssh22:17:431
303299192,12cyclictest21-21ksoftirqd/123:10:191
3032991911,6cyclictest6669-21sshd23:17:491
3032991911,6cyclictest6515-21sshd20:54:291
303299191,12cyclictest25266-21ssh23:30:171
3032991910,7cyclictest15257-21apt-get22:25:141
303299190,12cyclictest28239-21latency_hist20:10:011
303299190,12cyclictest28239-21latency_hist20:10:001
302499199,6cyclictest32470-21ps19:40:190
302499192,14cyclictest12-21ksoftirqd/023:55:160
3024991913,3cyclictest12-21ksoftirqd/023:35:220
302499191,14cyclictest2254-21/usr/sbin/munin21:55:310
302499191,14cyclictest2254-21/usr/sbin/munin21:55:310
302499191,12cyclictest26016-21ls19:30:290
302499191,12cyclictest16965-21ssh23:25:200
304899188,7cyclictest0-21swapper/322:05:263
304899182,10cyclictest0-21swapper/323:50:163
304899182,10cyclictest0-21swapper/321:05:153
304899182,10cyclictest0-21swapper/321:00:163
304899182,10cyclictest0-21swapper/320:50:153
304899182,10cyclictest0-21swapper/320:45:183
304899182,10cyclictest0-21swapper/320:40:143
304899182,10cyclictest0-21swapper/320:10:183
304899182,10cyclictest0-21swapper/319:55:163
304899182,10cyclictest0-21swapper/319:50:153
304899182,10cyclictest0-21swapper/319:40:153
304899182,10cyclictest0-21swapper/319:35:053
304899182,10cyclictest0-21swapper/319:25:153
304899182,10cyclictest0-21swapper/319:10:133
3048991812,4cyclictest0-21swapper/300:20:133
304899181,13cyclictest25366irq/58-eth000:15:143
3048991810,6cyclictest0-21swapper/321:15:423
3048991810,3cyclictest0-21swapper/321:10:193
3048991810,3cyclictest0-21swapper/321:10:183
303299189,7cyclictest30600-21apt-get21:55:171
303299189,7cyclictest30600-21apt-get21:55:161
303299186,8cyclictest21-21ksoftirqd/121:40:181
303299184,4cyclictest3411-21usage_Anybus22:35:291
303299183,13cyclictest1719-21grep20:50:001
303299181,8cyclictest131rcu_preempt00:15:141
302499189,3cyclictest12-21ksoftirqd/020:30:130
302499188,3cyclictest131rcu_preempt22:42:050
302499188,3cyclictest131rcu_preempt22:42:040
302499188,2cyclictest12-21ksoftirqd/023:41:310
302499186,3cyclictest12-21ksoftirqd/023:32:080
302499185,9cyclictest151rcuc/020:05:150
302499185,9cyclictest151rcuc/020:05:140
302499185,10cyclictest0-21swapper/021:45:120
3024991813,3cyclictest0-21swapper/019:15:200
3024991812,4cyclictest0-21swapper/019:45:310
302499181,14cyclictest21904-21cut21:30:150
302499181,11cyclictest16522-21timerandwakeup19:20:290
304899179,6cyclictest0-21swapper/323:28:273
304899179,6cyclictest0-21swapper/300:05:093
304899178,6cyclictest0-21swapper/300:26:153
304899172,9cyclictest0-21swapper/320:00:143
304899172,10cyclictest0-21swapper/320:05:173
304899172,10cyclictest0-21swapper/320:05:163
303299179,6cyclictest0-21swapper/100:00:151
303299178,6cyclictest6204-21kthreadcore19:45:221
303299174,11cyclictest24519-21kthreadcore00:10:221
303299174,10cyclictest21418-21grep19:25:291
303299171,8cyclictest12789-21/usr/sbin/munin19:20:131
303299171,5cyclictest28651-21grep20:10:201
303299171,12cyclictest23462-21latency_hist20:05:011
3032991710,5cyclictest30103-21grep21:15:221
302499178,2cyclictest12-21ksoftirqd/019:35:250
302499173,7cyclictest12-21ksoftirqd/020:45:230
3024991711,4cyclictest0-21swapper/020:55:190
3024991711,4cyclictest0-21swapper/020:25:200
302499171,13cyclictest6619-21latency_hist20:55:010
304899161,12cyclictest0-21swapper/323:30:153
304899161,10cyclictest0-21swapper/320:15:163
303299162,12cyclictest24989-21fschecks_time20:40:161
303299161,8cyclictest25566-21kthreadcore19:30:241
303299161,12cyclictest16714-21tr22:05:221
303299161,12cyclictest14319-21fschecks_time19:55:171
303299161,11cyclictest24659-21cat20:40:011
303299161,11cyclictest16975-21latency21:25:251
303299161,10cyclictest6207-21irqrtprio20:20:191
303299161,10cyclictest10623-21fschecks_time20:25:161
303299160,13cyclictest12004-21kthreadcore19:50:231
303299152,10cyclictest18568-21kthreadcore21:05:221
303299141,10cyclictest3060-21latency_hist19:10:011
302499141,10cyclictest2951-21latency_hist19:10:020
304899138,3cyclictest0-21swapper/320:37:213
304899132,9cyclictest0-21swapper/319:20:143
304899123,4cyclictest0-21swapper/319:15:163
304899122,6cyclictest0-21swapper/320:55:143
304899121,5cyclictest0-21swapper/319:35:023
304899101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional