You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-18 - 03:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Oct 18, 2025 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10540996010,8cyclictest131rcu_preempt23:55:161
1055499501,5cyclictest131rcu_preempt20:55:132
10566994923,14cyclictest25366irq/58-eth023:35:163
1055499491,29cyclictest11928-21ssh23:55:152
10554994910,6cyclictest131rcu_preempt21:45:112
10566994722,17cyclictest25366irq/58-eth021:20:143
10566994722,17cyclictest25366irq/58-eth021:20:133
1053199441,40cyclictest26349-21ssh00:25:140
1053199424,33cyclictest4655-21ssh21:15:140
1056699413,26cyclictest25366irq/58-eth019:50:163
1053199410,11cyclictest12-21ksoftirqd/022:30:160
1056699405,18cyclictest25366irq/58-eth022:15:123
1056699405,18cyclictest25366irq/58-eth022:15:123
10566994018,10cyclictest25366irq/58-eth020:00:143
1056699383,19cyclictest25366irq/58-eth023:30:183
1055499382,19cyclictest7945-21sh22:53:372
1054099384,25cyclictest11157-21ssh23:35:161
1056699373,20cyclictest25366irq/58-eth023:00:143
1055499372,29cyclictest14713-21ssh22:18:362
1055499372,29cyclictest14713-21ssh22:18:352
10566993518,12cyclictest25366irq/58-eth022:50:133
10566993510,18cyclictest25366irq/58-eth000:30:163
1055499340,17cyclictest319-21ssh23:08:382
1053199342,23cyclictest9274-21ssh22:55:140
1056699331,24cyclictest0-21swapper/321:45:093
1056699322,19cyclictest25366irq/58-eth000:35:173
1055499322,17cyclictest131rcu_preempt22:03:352
10554993219,8cyclictest261rcuc/222:59:372
1056699315,8cyclictest25466irq/59-eth022:55:133
1056699311,22cyclictest25366irq/58-eth000:20:153
1053199312,5cyclictest12-21ksoftirqd/022:00:020
1053199312,19cyclictest27924-21ssh00:05:110
1056699303,11cyclictest0-21swapper/322:10:143
10554993010,16cyclictest131rcu_preempt21:00:162
1054099303,20cyclictest21-21ksoftirqd/120:55:151
10540993013,5cyclictest13290-21fschecks_time20:20:171
1054099301,23cyclictest31872-21sh22:10:161
1053199307,6cyclictest12-21ksoftirqd/022:45:290
10531993019,8cyclictest15126-21apt-get22:00:170
1056699292,16cyclictest25366irq/58-eth022:00:173
1055499293,19cyclictest21846irq/40-dwc2_hso00:10:142
1055499292,20cyclictest0-21swapper/219:25:052
10531992917,7cyclictest28332-21diskmemload21:50:130
1056699283,12cyclictest25366irq/58-eth020:50:153
10566992812,13cyclictest25366irq/58-eth021:25:163
1055499283,20cyclictest629-21Async08:52:162
1055499282,11cyclictest0-21swapper/221:30:132
1055499281,18cyclictest942-21snmpd20:15:132
1054099285,17cyclictest28332-21diskmemload22:15:141
1054099285,17cyclictest28332-21diskmemload22:15:131
1054099283,4cyclictest131rcu_preempt23:00:121
1053199286,7cyclictest12-21ksoftirqd/023:55:290
1053199284,4cyclictest0-21swapper/021:05:140
1053199281,7cyclictest12-21ksoftirqd/023:30:310
1053199281,6cyclictest12-21ksoftirqd/020:35:290
1053199281,6cyclictest12-21ksoftirqd/000:00:150
1056699272,18cyclictest0-21swapper/321:37:383
1055499272,13cyclictest28332-21diskmemload00:15:142
10540992711,10cyclictest21-21ksoftirqd/121:30:141
1053199276,7cyclictest131rcu_preempt00:22:150
1053199276,3cyclictest131rcu_preempt22:09:440
1053199274,5cyclictest12-21ksoftirqd/021:10:320
1053199272,4cyclictest12-21ksoftirqd/022:45:020
1053199272,4cyclictest12-21ksoftirqd/022:20:290
10531992718,6cyclictest12-21ksoftirqd/021:40:130
1053199270,3cyclictest12-21ksoftirqd/000:30:310
1056699263,15cyclictest0-21swapper/321:57:413
1056699263,14cyclictest25366irq/58-eth000:15:133
10566992613,8cyclictest25366irq/58-eth023:55:173
1055499261,12cyclictest0-21swapper/221:25:052
1054099266,9cyclictest131rcu_preempt21:42:251
1053199269,5cyclictest12-21ksoftirqd/023:25:010
1053199266,3cyclictest131rcu_preempt21:45:110
1053199265,8cyclictest12-21ksoftirqd/021:30:150
1053199264,5cyclictest12-21ksoftirqd/023:35:210
1053199264,5cyclictest12-21ksoftirqd/000:15:270
1053199264,15cyclictest0-21swapper/023:50:180
1053199263,6cyclictest12-21ksoftirqd/022:25:280
1053199263,6cyclictest12-21ksoftirqd/020:05:150
1053199262,19cyclictest11245-21rm21:36:370
1053199261,8cyclictest20676-21kthreadcore23:40:210
1053199261,5cyclictest12-21ksoftirqd/023:10:160
1053199261,5cyclictest12-21ksoftirqd/023:05:020
10566992510,9cyclictest25366irq/58-eth021:10:173
1055499256,11cyclictest15632-21cut21:20:222
1055499256,11cyclictest15632-21cut21:20:222
1055499255,3cyclictest131rcu_preempt19:45:212
1055499253,15cyclictest21846irq/40-dwc2_hso21:35:142
1055499251,7cyclictest131rcu_preempt23:15:152
10554992515,4cyclictest27-21ksoftirqd/223:30:342
1054099254,11cyclictest1737-21ssh23:10:151
1054099252,17cyclictest21228-21kworker/u8:121:55:141
1053199259,6cyclictest12-21ksoftirqd/020:50:000
1053199254,5cyclictest12-21ksoftirqd/023:15:300
1053199254,5cyclictest12-21ksoftirqd/021:20:300
1053199254,5cyclictest12-21ksoftirqd/021:20:290
1053199254,4cyclictest12-21ksoftirqd/020:15:270
1053199254,12cyclictest26763-21ssh23:25:150
1053199254,10cyclictest12-21ksoftirqd/022:35:320
1053199253,5cyclictest12-21ksoftirqd/020:40:110
1053199253,4cyclictest12-21ksoftirqd/023:20:270
1053199253,4cyclictest12-21ksoftirqd/022:15:290
1053199253,4cyclictest12-21ksoftirqd/022:15:280
1053199252,6cyclictest12-21ksoftirqd/022:50:130
1053199252,4cyclictest12-21ksoftirqd/020:55:280
1053199252,4cyclictest12-21ksoftirqd/000:35:280
1053199251,4cyclictest2875-21memory22:10:230
10531992510,4cyclictest131rcu_preempt00:10:240
1053199250,7cyclictest25976-21fschecks_count23:05:170
1055499248,7cyclictest131rcu_preempt19:15:222
1055499247,14cyclictest5632-21awk23:10:292
1055499242,13cyclictest131rcu_preempt21:43:192
1055499241,8cyclictest28332-21diskmemload22:44:362
10554992414,5cyclictest27-21ksoftirqd/222:14:572
1055499241,3cyclictest131rcu_preempt00:30:252
10554992413,9cyclictest30514-21ssh22:08:402
1054099244,10cyclictest5655-21ssh21:35:131
1054099242,4cyclictest131rcu_preempt22:00:231
10540992410,11cyclictest21-21ksoftirqd/100:33:541
1053199244,9cyclictest12-21ksoftirqd/020:30:220
1053199243,5cyclictest12-21ksoftirqd/019:10:170
1053199242,6cyclictest11101-21kworker/u8:023:45:110
1053199241,6cyclictest12-21ksoftirqd/020:10:260
1056699239,10cyclictest0-21swapper/321:30:203
1055499237,3cyclictest131rcu_preempt00:00:312
1055499233,12cyclictest21846irq/40-dwc2_hso20:35:142
1055499231,4cyclictest0-21swapper/219:40:222
10554992314,4cyclictest27-21ksoftirqd/223:25:162
1055499231,3cyclictest0-21swapper/200:20:172
10554992313,4cyclictest27-21ksoftirqd/200:38:282
1055499231,17cyclictest17564-21ssh23:00:122
1055499231,15cyclictest0-21swapper/222:48:202
1055499230,14cyclictest131rcu_preempt21:50:182
1054099236,3cyclictest131rcu_preempt23:20:151
1054099236,3cyclictest131rcu_preempt23:20:151
1054099231,8cyclictest21-21ksoftirqd/120:40:121
1054099231,17cyclictest73-21kswapd000:15:151
1053199234,5cyclictest12-21ksoftirqd/021:05:010
1053199234,5cyclictest12-21ksoftirqd/020:55:010
1053199234,4cyclictest12-21ksoftirqd/020:25:190
1053199234,4cyclictest12-21ksoftirqd/019:35:230
1053199233,4cyclictest12-21ksoftirqd/021:25:200
1053199233,4cyclictest12-21ksoftirqd/019:30:210
1053199233,4cyclictest12-21ksoftirqd/019:25:010
1053199232,7cyclictest12-21ksoftirqd/019:25:140
1053199231,7cyclictest0-21swapper/019:40:140
1053199231,3cyclictest12-21ksoftirqd/019:55:250
1055499228,12cyclictest29564-21kthreadcore23:45:192
1055499225,8cyclictest32367-21fschecks_count20:40:162
1055499222,12cyclictest0-21swapper/222:32:002
1055499221,5cyclictest30502-21cat20:00:322
1055499221,3cyclictest22218-21kthreadcore23:40:242
1055499221,18cyclictest93-21pr/ttyS019:10:022
10554992211,8cyclictest0-21swapper/220:30:302
1055499221,12cyclictest1367666ssh22:35:472
1055499221,12cyclictest131rcu_preempt21:13:122
1055499221,12cyclictest131rcu_preempt19:10:252
1055499221,11cyclictest26471-21ssh00:25:152
10540992214,4cyclictest5345-21kernelversion21:15:201
1054099221,18cyclictest10063-21ssh23:15:141
1054099220,18cyclictest21145-21ls20:25:221
1053199225,4cyclictest0-21swapper/020:00:140
1053199224,5cyclictest12-21ksoftirqd/019:50:330
10531992211,2cyclictest131rcu_preempt19:45:240
1056699219,9cyclictest0-21swapper/323:11:043
1056699219,9cyclictest0-21swapper/323:05:263
1056699217,7cyclictest0-21swapper/322:05:123
1056699214,11cyclictest0-21swapper/323:15:123
1056699212,15cyclictest0-21swapper/321:00:143
1056699212,13cyclictest0-21swapper/323:45:133
1055499213,2cyclictest131rcu_preempt22:20:252
1055499213,14cyclictest21846irq/40-dwc2_hso23:35:142
1055499211,4cyclictest131rcu_preempt19:35:302
1055499211,3cyclictest131rcu_preempt20:10:212
1055499211,2cyclictest131rcu_preempt23:50:322
10554992112,4cyclictest27-21ksoftirqd/222:26:532
10554992112,2cyclictest21811-21sort19:55:192
1055499210,7cyclictest19698-21kthreadcore23:20:222
1055499210,7cyclictest19698-21kthreadcore23:20:212
1055499210,3cyclictest131rcu_preempt00:05:212
1055499210,12cyclictest0-21swapper/219:20:192
1054099219,10cyclictest0-21swapper/122:41:441
1054099216,4cyclictest131rcu_preempt23:45:141
10540992116,3cyclictest21-21ksoftirqd/122:49:211
10540992115,4cyclictest0-21swapper/121:20:271
10540992115,4cyclictest0-21swapper/121:20:261
1054099211,11cyclictest0-21swapper/100:27:561
1053199211,17cyclictest13296-21cut20:20:170
1056699203,9cyclictest25366irq/58-eth022:35:133
1056699203,10cyclictest0-21swapper/323:20:153
1056699203,10cyclictest0-21swapper/323:20:143
1056699202,13cyclictest0-21swapper/323:40:143
10566992012,5cyclictest0-21swapper/322:25:293
1056699201,14cyclictest0-21swapper/321:15:173
1056699201,11cyclictest25366irq/58-eth019:45:123
1055499207,10cyclictest12666-21latency_hist21:20:012
1055499203,9cyclictest29870-21/usr/sbin/munin19:30:172
1055499201,6cyclictest1831-21grep20:05:232
1055499201,2cyclictest131rcu_preempt19:50:152
1055499201,15cyclictest28439-21ls21:10:012
1055499200,12cyclictest131rcu_preempt20:50:222
1054099203,8cyclictest131rcu_preempt21:11:181
1054099203,14cyclictest23466-21ssh22:05:101
1054099203,10cyclictest8073-21kworker/1:022:36:061
1054099202,8cyclictest131rcu_preempt22:55:131
1054099202,2cyclictest131rcu_preempt23:33:151
1054099202,2cyclictest131rcu_preempt00:08:501
1054099202,10cyclictest18065-21kthreadcore19:15:211
10540992013,4cyclictest21-21ksoftirqd/121:25:241
10540992012,5cyclictest0-21swapper/119:45:181
1054099201,16cyclictest2444-21cut19:35:161
1054099201,14cyclictest10717-21ls19:10:141
1053199201,6cyclictest12-21ksoftirqd/019:15:120
1056699193,11cyclictest0-21swapper/322:44:373
1056699193,11cyclictest0-21swapper/321:52:413
1056699193,11cyclictest0-21swapper/321:40:363
1056699192,11cyclictest0-21swapper/320:05:153
1056699192,11cyclictest0-21swapper/319:25:153
1056699192,10cyclictest0-21swapper/320:20:153
1056699191,11cyclictest0-21swapper/321:05:173
1056699191,11cyclictest0-21swapper/319:55:193
10566991910,6cyclictest0-21swapper/300:05:133
1055499191,9cyclictest4786-21fschecks_time20:45:162
1055499191,14cyclictest18003-21cut20:25:152
1055499191,10cyclictest12786-21apt-get20:20:032
1054099199,8cyclictest322-21apt-get22:30:141
1054099198,7cyclictest0-21swapper/120:00:271
1054099196,11cyclictest24454-21ssh22:25:161
1054099194,5cyclictest23505-21latency19:20:241
1054099193,9cyclictest131rcu_preempt23:29:081
1054099193,13cyclictest18703-21ssh00:20:151
1054099192,14cyclictest3807-21irqcore20:10:191
1054099192,12cyclictest9217-21ssh00:35:181
1054099191,9cyclictest131rcu_preempt23:05:151
1054099191,9cyclictest131rcu_preempt19:30:201
1054099191,3cyclictest131rcu_preempt00:13:341
10540991913,3cyclictest21-21ksoftirqd/121:45:171
10540991910,7cyclictest7086-21apt-get19:40:151
1054099190,3cyclictest131rcu_preempt20:15:211
1054099190,11cyclictest131rcu_preempt20:30:231
1056699189,6cyclictest0-21swapper/323:52:543
1056699182,10cyclictest0-21swapper/320:45:153
1056699182,10cyclictest0-21swapper/320:25:143
1056699182,10cyclictest0-21swapper/320:10:153
1056699182,10cyclictest0-21swapper/319:35:143
1056699182,10cyclictest0-21swapper/319:10:133
1056699181,9cyclictest0-21swapper/319:25:023
10566991812,4cyclictest0-21swapper/300:10:133
1056699181,12cyclictest0-21swapper/322:30:183
1056699181,12cyclictest0-21swapper/319:40:173
1056699181,11cyclictest0-21swapper/322:45:243
1056699181,10cyclictest0-21swapper/319:30:233
10566991810,5cyclictest25366irq/58-eth000:25:143
1054099189,7cyclictest28561-21ntp_states19:25:261
1054099183,4cyclictest131rcu_preempt21:50:271
1054099182,8cyclictest131rcu_preempt20:45:291
1054099182,13cyclictest19876-21ssh00:00:141
10540991812,4cyclictest0-21swapper/122:20:501
1054099181,15cyclictest26540-21kthreadcore21:05:231
1054099181,15cyclictest1375-21if_eth122:50:161
1053199182,4cyclictest12-21ksoftirqd/019:10:010
1056699179,6cyclictest0-21swapper/323:26:473
1056699171,12cyclictest0-21swapper/322:20:283
1056699171,12cyclictest0-21swapper/319:15:233
1056699171,11cyclictest0-21swapper/300:00:133
1054099176,9cyclictest32111-21cut20:05:201
1054099172,12cyclictest20572-21sed19:50:321
1054099171,8cyclictest21-21ksoftirqd/123:50:201
1054099171,3cyclictest131rcu_preempt21:00:241
1054099171,2cyclictest131rcu_preempt19:55:201
10540991712,3cyclictest0-21swapper/120:50:181
1054099171,13cyclictest19933-21sed23:40:201
1056699161,12cyclictest0-21swapper/320:30:233
1054099165,8cyclictest0-21swapper/120:35:151
1056699142,9cyclictest0-21swapper/320:20:013
1056699142,8cyclictest0-21swapper/320:55:153
1056699142,10cyclictest0-21swapper/320:45:013
1056699141,10cyclictest0-21swapper/320:35:163
1054099141,10cyclictest10565-21latency_hist19:10:011
105669991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional