You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 07:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Mon Dec 01, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
355199634,11cyclictest131rcu_preempt21:05:151
356099561,40cyclictest1-21systemd23:55:132
356099561,40cyclictest1-21systemd23:55:122
3551995510,10cyclictest131rcu_preempt21:10:141
3540995434,13cyclictest22207-21diskmemload21:40:140
355199492,35cyclictest30864-21ssh21:15:131
354099463,37cyclictest9417-21ssh22:00:140
3540994617,11cyclictest11732-21ssh22:58:500
3540994614,11cyclictest4285-21ssh22:55:020
354099445,33cyclictest27417-21ssh22:30:140
3540994417,11cyclictest18861-21ssh22:25:020
354099441,36cyclictest21330-21ssh23:05:140
3540994317,11cyclictest14492-21ssh23:57:350
3540994317,11cyclictest14492-21ssh23:57:340
3540994316,11cyclictest9996-21ssh22:19:430
3540994215,9cyclictest16528-21apt-get00:00:130
3540994215,10cyclictest9616-21ssh00:35:490
3540994128,9cyclictest1042-21ssh21:55:150
3540994117,8cyclictest18908-21ssh22:43:350
3540994117,8cyclictest18908-21ssh22:43:340
3540994117,8cyclictest17449-21sed23:20:230
3540994114,10cyclictest32150-21ssh00:10:200
3540994113,10cyclictest669-21cut20:45:220
357199404,24cyclictest29566irq/58-eth022:50:163
3540994015,9cyclictest22947-21ssh22:05:530
3540994015,9cyclictest18078-21cut21:05:190
3540994013,16cyclictest12-21ksoftirqd/000:15:090
3560993927,9cyclictest27-21ksoftirqd/222:00:122
3540993917,7cyclictest22207-21diskmemload21:37:120
3540993917,7cyclictest22207-21diskmemload21:14:390
3540993916,8cyclictest5871-21ssh23:34:030
3540993916,8cyclictest12352-21ssh21:21:260
3540993916,12cyclictest1615-21ls20:15:000
3540993915,8cyclictest23103-21ssh21:47:490
3540993915,10cyclictest29777-21ssh21:32:080
3540993913,8cyclictest953-21ssh23:10:280
357199383,21cyclictest29566irq/58-eth022:40:143
357199383,21cyclictest29566irq/58-eth022:40:133
354099382,28cyclictest32275-21ssh23:50:150
3540993817,7cyclictest25603-21ssh00:06:230
3540993816,7cyclictest22207-21diskmemload23:42:190
3540993814,11cyclictest10618-21ntp_192.168.11423:35:260
3540993814,11cyclictest10618-21ntp_192.168.11423:35:250
3540993814,10cyclictest5614-21cpu23:15:150
3540993813,9cyclictest29835-21ntp_192.168.11422:10:230
3540993813,10cyclictest27431-21cat19:30:300
3540993810,13cyclictest29019-21kthreadcore19:35:200
3540993726,7cyclictest29218-21ssh00:30:160
3540993717,6cyclictest22207-21diskmemload23:47:400
3540993715,7cyclictest19890-21sh00:23:200
3540993715,10cyclictest22294-21sh23:25:030
3540993715,10cyclictest17529-21usage_EthcTask23:00:320
3540993714,8cyclictest0-21swapper/022:36:410
3540993714,7cyclictest0-21swapper/021:17:540
3540993713,8cyclictest25386-21ssh22:45:530
3540993713,10cyclictest18959-21netstat21:25:250
3540993711,10cyclictest30126-21ssh21:50:550
3571993615,10cyclictest29566irq/58-eth022:30:143
356099362,17cyclictest5431-21tr22:35:222
3540993617,8cyclictest6360-21sshd20:19:220
3540993614,8cyclictest28963-21cat20:05:300
3540993614,8cyclictest28963-21cat20:05:290
3540993613,7cyclictest22207-21diskmemload00:28:020
3540993612,8cyclictest17447-21ssh22:23:220
3571993511,16cyclictest0-21swapper/323:35:243
3571993511,16cyclictest0-21swapper/323:35:233
3540993514,8cyclictest14131-21kthreadcore19:20:220
3540993513,8cyclictest21769-21ntp_states19:25:230
3540993512,8cyclictest6826-21fw_packets20:20:190
3540993512,10cyclictest11406-21kthreadcore19:50:230
355199342,26cyclictest32378-21rm22:12:241
3540993429,2cyclictest11460-21apt-get20:25:150
3540993414,12cyclictest19463-21sshd19:59:220
3560993326,3cyclictest26255-21ssh22:10:132
355199337,3cyclictest131rcu_preempt19:50:151
355199332,23cyclictest24251-21tr23:05:241
3540993317,12cyclictest19703-21/usr/sbin/munin20:00:260
3540993314,6cyclictest0-21swapper/019:45:160
356099322,19cyclictest17451-21ssh22:23:222
355199327,7cyclictest131rcu_preempt22:34:241
355199322,24cyclictest15358-21sort22:40:241
355199322,24cyclictest15358-21sort22:40:231
354099322,24cyclictest0-21swapper/020:55:150
354099322,24cyclictest0-21swapper/020:55:140
3540993220,9cyclictest13112-21sshd19:17:210
3540993218,10cyclictest4203-21sed19:40:290
3540993212,7cyclictest16909-21irqstats20:30:200
3540993210,18cyclictest12-21ksoftirqd/019:10:160
357199311,17cyclictest29566irq/58-eth021:20:123
356099317,8cyclictest27-21ksoftirqd/221:35:142
3540993120,8cyclictest6790-21sed20:50:300
356099302,23cyclictest3145-21sshd00:13:362
356099302,15cyclictest9076-21ssh22:18:232
355199303,9cyclictest131rcu_preempt20:25:151
3540993020,7cyclictest21057-21munin-node20:35:140
3540993018,8cyclictest28586-21cut20:40:230
3540993012,14cyclictest12792-21cpu21:00:150
357199297,14cyclictest0-21swapper/321:40:153
357199292,17cyclictest29566irq/58-eth021:55:163
356099298,15cyclictest27-21ksoftirqd/200:18:152
356099293,20cyclictest0-21swapper/223:53:202
355199293,19cyclictest28698-21ssh22:50:161
355199292,21cyclictest9756-21ssh22:19:221
355199292,21cyclictest27339-21ssh00:08:501
357199283,16cyclictest0-21swapper/323:15:223
357199283,12cyclictest0-21swapper/322:08:243
356099288,11cyclictest27-21ksoftirqd/223:20:202
356099283,18cyclictest26346irq/40-dwc2_hso23:40:132
356099282,20cyclictest0-21swapper/219:35:152
356099281,5cyclictest27-21ksoftirqd/221:15:142
3560992815,7cyclictest0-21swapper/221:45:172
3560992812,4cyclictest27-21ksoftirqd/222:25:432
356099281,13cyclictest155-21systemd-udevd20:05:132
356099281,13cyclictest155-21systemd-udevd20:05:122
355199283,17cyclictest201rcuc/120:10:141
355199282,20cyclictest24212-21ssh21:49:231
3551992819,7cyclictest17461-21cron22:05:011
3540992817,8cyclictest3632-21cut19:10:020
357199273,18cyclictest0-21swapper/322:12:233
357199273,16cyclictest29566irq/58-eth022:00:133
357199273,16cyclictest0-21swapper/322:29:243
357199273,15cyclictest0-21swapper/321:49:233
3571992715,9cyclictest0-21swapper/300:10:183
3571992713,12cyclictest29566irq/58-eth000:00:133
356099279,4cyclictest27-21ksoftirqd/221:56:062
356099279,4cyclictest131rcu_preempt19:25:272
356099276,15cyclictest27-21ksoftirqd/223:19:482
356099273,16cyclictest26346irq/40-dwc2_hso19:45:132
356099273,14cyclictest26346irq/40-dwc2_hso21:05:162
356099273,13cyclictest26346irq/40-dwc2_hso20:55:142
356099273,13cyclictest26346irq/40-dwc2_hso20:55:132
356099273,10cyclictest0-21swapper/222:48:242
356099272,20cyclictest5384-21ssh23:33:222
356099272,17cyclictest26346irq/40-dwc2_hso21:40:132
356099272,17cyclictest26346irq/40-dwc2_hso19:20:152
356099271,5cyclictest27-21ksoftirqd/221:20:162
356099271,17cyclictest0-21swapper/200:20:112
3560992710,11cyclictest27-21ksoftirqd/220:00:142
355199272,19cyclictest9016-21latency23:15:231
355199272,19cyclictest355-21kthreadcore23:10:221
355199272,19cyclictest26414-21ssh22:47:241
355199272,19cyclictest10020-21grep23:35:241
355199272,19cyclictest10020-21grep23:35:241
355199271,4cyclictest131rcu_preempt19:10:151
3551992713,11cyclictest21-21ksoftirqd/121:40:131
355199271,19cyclictest21393-21ssh00:25:131
357199268,12cyclictest29566irq/58-eth021:30:163
357199263,16cyclictest0-21swapper/322:15:243
357199262,17cyclictest0-21swapper/322:23:223
357199262,14cyclictest29566irq/58-eth021:50:153
356099268,11cyclictest27-21ksoftirqd/221:34:162
356099267,12cyclictest27-21ksoftirqd/222:51:272
356099264,14cyclictest27-21ksoftirqd/223:05:202
356099263,13cyclictest27-21ksoftirqd/222:07:452
355199265,13cyclictest629-21Async14:00:481
355199261,22cyclictest25031-21rm21:50:141
357199255,11cyclictest0-21swapper/323:50:153
357199254,16cyclictest0-21swapper/323:10:163
357199253,16cyclictest0-21swapper/323:56:243
357199253,16cyclictest0-21swapper/323:56:233
357199252,13cyclictest29566irq/58-eth022:35:153
356099258,8cyclictest27-21ksoftirqd/223:25:142
356099258,11cyclictest27-21ksoftirqd/220:45:192
356099255,7cyclictest27-21ksoftirqd/222:57:172
356099254,6cyclictest27-21ksoftirqd/223:46:102
356099252,15cyclictest26346irq/40-dwc2_hso19:55:132
356099251,18cyclictest0-21swapper/219:40:152
357199243,13cyclictest0-21swapper/322:47:233
357199241,19cyclictest29566irq/58-eth000:15:083
356099249,5cyclictest27-21ksoftirqd/222:40:062
356099249,5cyclictest27-21ksoftirqd/222:40:062
356099248,10cyclictest131rcu_preempt21:12:412
356099247,8cyclictest131rcu_preempt00:30:222
356099247,7cyclictest27-21ksoftirqd/223:10:332
356099247,7cyclictest27-21ksoftirqd/221:51:572
356099247,2cyclictest131rcu_preempt23:03:092
3560992417,4cyclictest0-21swapper/223:35:302
3560992417,4cyclictest0-21swapper/223:35:292
356099241,3cyclictest0-21swapper/221:25:242
356099241,11cyclictest131rcu_preempt00:00:122
3560992410,4cyclictest27-21ksoftirqd/200:09:552
355199242,19cyclictest3535-21kthreadcore21:55:231
355199241,15cyclictest14011-21ssh23:20:131
356099239,8cyclictest27-21ksoftirqd/222:34:502
356099236,11cyclictest27-21ksoftirqd/200:26:332
356099233,8cyclictest27-21ksoftirqd/200:35:502
356099233,11cyclictest131rcu_preempt19:15:292
356099231,16cyclictest0-21swapper/220:30:142
356099230,7cyclictest26108-21uniq20:40:182
355199237,12cyclictest21-21ksoftirqd/123:55:141
355199237,12cyclictest21-21ksoftirqd/123:55:131
355199237,11cyclictest21-21ksoftirqd/100:13:371
355199232,17cyclictest4642-21ssh00:35:151
355199232,15cyclictest14166-21ntp_192.168.11422:20:231
355199232,12cyclictest131rcu_preempt23:28:231
355199231,3cyclictest0-21swapper/120:05:121
355199231,3cyclictest0-21swapper/120:05:111
355199231,14cyclictest131rcu_preempt22:05:291
355199230,11cyclictest131rcu_preempt21:38:191
357199222,17cyclictest29566irq/58-eth021:10:143
357199222,14cyclictest0-21swapper/323:30:143
357199222,12cyclictest29566irq/58-eth023:00:143
356099224,8cyclictest27-21ksoftirqd/220:35:262
356099224,11cyclictest27-21ksoftirqd/221:00:272
356099223,13cyclictest26346irq/40-dwc2_hso20:15:122
356099221,4cyclictest131rcu_preempt20:25:292
356099221,14cyclictest0-21swapper/220:50:172
355199229,3cyclictest0-21swapper/120:30:221
355199228,4cyclictest131rcu_preempt00:15:161
355199227,9cyclictest21-21ksoftirqd/123:52:381
355199227,2cyclictest131rcu_preempt00:31:231
355199222,17cyclictest3414-21ssh22:35:161
355199222,14cyclictest26591-21sh22:29:231
355199222,14cyclictest15361-21sh21:25:131
355199221,15cyclictest28110-21grep19:35:161
3551992210,7cyclictest0-21swapper/100:00:141
3551992210,3cyclictest0-21swapper/123:47:321
357199213,13cyclictest0-21swapper/323:05:233
357199212,9cyclictest29566irq/58-eth000:30:153
357199212,13cyclictest0-21swapper/323:25:143
3571992113,5cyclictest29566irq/58-eth000:25:133
356099212,7cyclictest0-21swapper/219:30:142
356099212,16cyclictest30293-21kthreadcore20:10:222
3551992113,5cyclictest0-21swapper/119:20:191
355199211,2cyclictest131rcu_preempt00:20:241
355199211,16cyclictest11897-21ls19:15:251
355199211,14cyclictest23801-21rm21:30:161
357199203,11cyclictest0-21swapper/300:05:243
357199202,11cyclictest29566irq/58-eth000:20:123
356099205,4cyclictest27-21ksoftirqd/219:50:222
356099203,15cyclictest8964-21kthreadcore20:20:232
355199202,9cyclictest131rcu_preempt23:04:521
355199202,2cyclictest131rcu_preempt23:44:481
355199202,14cyclictest12789-21apt-get21:00:131
3551992011,6cyclictest0-21swapper/121:20:131
355199200,13cyclictest131rcu_preempt20:50:171
357199194,11cyclictest29566irq/58-eth021:15:133
357199193,11cyclictest0-21swapper/323:20:243
357199192,11cyclictest0-21swapper/321:25:043
357199192,10cyclictest0-21swapper/319:20:053
357199191,13cyclictest0-21swapper/320:50:023
357199191,13cyclictest0-21swapper/320:25:173
357199191,13cyclictest0-21swapper/320:20:173
357199191,11cyclictest0-21swapper/320:00:243
357199191,10cyclictest0-21swapper/320:40:023
356099191,9cyclictest4209-21ps19:10:212
355199197,2cyclictest131rcu_preempt23:30:291
355199191,8cyclictest131rcu_preempt22:56:051
357199189,6cyclictest0-21swapper/300:36:263
357199183,10cyclictest0-21swapper/323:41:233
357199183,10cyclictest0-21swapper/320:10:163
357199182,12cyclictest0-21swapper/323:45:203
357199182,10cyclictest0-21swapper/321:35:153
357199182,10cyclictest0-21swapper/319:40:163
357199182,10cyclictest0-21swapper/319:35:143
357199181,13cyclictest0-21swapper/320:55:013
357199181,12cyclictest0-21swapper/319:25:103
356099181,9cyclictest3523-21cyclictest19:10:012
355199189,6cyclictest24322-21cron20:05:001
355199188,7cyclictest1782-21apt-get20:15:141
355199182,2cyclictest131rcu_preempt20:55:291
355199182,2cyclictest131rcu_preempt20:55:281
3551991813,3cyclictest0-21swapper/119:25:251
355199181,2cyclictest131rcu_preempt19:45:261
3551991810,6cyclictest25666-21sshd20:39:231
357199179,6cyclictest0-21swapper/322:55:453
357199172,10cyclictest0-21swapper/320:30:153
357199172,10cyclictest0-21swapper/320:15:153
357199172,10cyclictest0-21swapper/319:30:153
357199171,10cyclictest0-21swapper/321:00:023
357199171,10cyclictest0-21swapper/321:00:013
3571991710,4cyclictest29566irq/58-eth019:45:123
355199172,3cyclictest131rcu_preempt20:40:221
355199171,13cyclictest7716-21cut20:20:211
355199171,12cyclictest23201-21diskstats19:30:171
355199171,11cyclictest514-21grep19:40:181
357199162,10cyclictest0-21swapper/320:05:143
357199162,10cyclictest0-21swapper/320:05:133
357199162,10cyclictest0-21swapper/319:55:133
357199161,9cyclictest0-21swapper/319:15:233
357199161,11cyclictest0-21swapper/321:00:143
357199161,11cyclictest0-21swapper/320:40:183
355199161,12cyclictest14904-21apt-get19:55:131
357199142,9cyclictest0-21swapper/319:15:023
357199142,5cyclictest0-21swapper/321:10:013
355199141,10cyclictest3467-21latency_hist19:10:021
357199132,7cyclictest0-21swapper/319:55:023
357199111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional