You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 22:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Dec 03, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13441210478,9sleep10-21swapper/107:08:001
1325329780,12sleep30-21swapper/307:05:333
1352628469,10sleep00-21swapper/007:09:020
1336428164,13sleep20-21swapper/207:07:002
1368199321,29cyclictest6338-21idleruntime-cro09:10:003
1368199271,25cyclictest0-21swapper/311:53:393
1368199271,24cyclictest6330-21diskmemload11:07:393
1368199261,24cyclictest6330-21diskmemload11:27:003
1368199248,14cyclictest6330-21diskmemload11:37:453
1368199241,17cyclictest6330-21diskmemload09:37:333
1368199240,20cyclictest0-21swapper/308:15:123
1368199240,16cyclictest0-21swapper/310:00:103
1367499240,1cyclictest0-21swapper/012:16:260
1368199234,16cyclictest6330-21diskmemload09:59:203
1368199232,20cyclictest6330-21diskmemload09:27:433
1368199232,20cyclictest6330-21diskmemload09:27:433
1367799230,1cyclictest0-21swapper/207:30:282
1367799230,1cyclictest0-21swapper/207:30:272
1367799230,19cyclictest0-21swapper/211:25:522
1368199226,15cyclictest6330-21diskmemload10:43:523
1368199226,14cyclictest12790-21ssh11:33:383
13681992221,1cyclictest0-21swapper/312:17:353
1368199221,18cyclictest6330-21diskmemload09:46:283
1368199221,17cyclictest6330-21diskmemload10:46:443
1368199220,17cyclictest0-21swapper/307:15:253
1367799220,22cyclictest0-21swapper/209:24:162
1367799220,21cyclictest0-21swapper/209:25:142
1367799220,21cyclictest0-21swapper/209:25:142
1367599224,13cyclictest224-21systemd-journal11:22:281
1367499220,1cyclictest0-21swapper/009:56:300
1368199217,11cyclictest28249-21expr10:25:143
1368199213,14cyclictest6330-21diskmemload10:39:003
1368199212,18cyclictest6330-21diskmemload09:53:583
1368199210,17cyclictest6330-21diskmemload09:16:003
1367799212,2cyclictest0-21swapper/210:50:182
1367799210,1cyclictest0-21swapper/210:20:342
13674992120,1cyclictest0-21swapper/012:24:240
1368199209,8cyclictest29840-21find07:40:383
1368199205,14cyclictest26663-21sh11:45:183
1368199202,15cyclictest6330-21diskmemload12:24:213
1368199200,19cyclictest6330-21diskmemload10:34:403
1368199200,19cyclictest0-21swapper/309:24:313
1367799200,3cyclictest18744-21cut07:20:002
1367799200,20cyclictest0-21swapper/212:24:242
1367799200,20cyclictest0-21swapper/211:13:042
1367799200,13cyclictest0-21swapper/211:00:412
1367599200,18cyclictest0-21swapper/112:21:471
1368199199,9cyclictest61750irq/125-enp2s0-11:43:013
1368199198,10cyclictest6330-21diskmemload10:10:123
1368199192,16cyclictest0-21swapper/308:50:003
1368199191,17cyclictest6330-21diskmemload12:37:483
1368199191,17cyclictest6330-21diskmemload12:31:003
1368199190,18cyclictest0-21swapper/312:14:483
1368199190,18cyclictest0-21swapper/310:25:003
1368199190,17cyclictest0-21swapper/312:09:553
1368199190,17cyclictest0-21swapper/309:31:283
1368199190,17cyclictest0-21swapper/309:00:143
1368199190,14cyclictest0-21swapper/311:10:383
1367799190,19cyclictest0-21swapper/210:40:482
13674991918,1cyclictest0-21swapper/010:59:500
13674991916,2cyclictest13-21ksoftirqd/010:44:420
1368199185,12cyclictest25387-21grep07:30:283
1368199185,12cyclictest25387-21grep07:30:273
1368199181,16cyclictest0-21swapper/307:50:123
1368199180,16cyclictest6330-21diskmemload11:59:203
1368199180,16cyclictest0-21swapper/309:44:573
1368199180,15cyclictest6330-21diskmemload12:25:443
1368199180,15cyclictest6330-21diskmemload12:25:443
1368199180,12cyclictest0-21swapper/311:00:113
1367799188,7cyclictest224-21systemd-journal10:08:222
1367799180,2cyclictest0-21swapper/212:19:522
1367599180,16cyclictest0-21swapper/110:20:441
1367499182,2cyclictest141rcu_preempt11:05:140
1367499180,2cyclictest0-21swapper/012:01:560
1368199179,5cyclictest32750-21taskset07:47:113
1368199175,11cyclictest7744-21snmpd10:55:213
1368199171,15cyclictest6330-21diskmemload10:51:073
1368199171,14cyclictest0-21swapper/307:25:143
1368199170,14cyclictest0-21swapper/312:03:033
1368199170,14cyclictest0-21swapper/310:15:203
1367799170,17cyclictest0-21swapper/212:01:562
1367799170,17cyclictest0-21swapper/210:30:322
1367799170,0cyclictest0-21swapper/210:59:512
1367599172,8cyclictest0-21swapper/111:40:141
1367599170,2cyclictest0-21swapper/111:54:111
13674991716,1cyclictest0-21swapper/011:49:480
13674991716,1cyclictest0-21swapper/008:27:190
13674991714,2cyclictest17329-21diskstats10:15:200
1367499170,17cyclictest0-21swapper/010:27:030
1368199168,7cyclictest22924-21sort08:35:173
1368199161,14cyclictest10100-21munin-run08:10:003
1368199160,15cyclictest0-21swapper/307:35:163
1368199160,13cyclictest0-21swapper/311:19:353
1368199160,13cyclictest0-21swapper/311:19:343
1368199160,11cyclictest0-21swapper/310:07:213
13677991615,1cyclictest0-21swapper/209:50:082
1367799160,16cyclictest0-21swapper/211:54:162
1367799160,15cyclictest0-21swapper/210:25:362
1367599166,6cyclictest18261-21sh12:05:491
1367499161,14cyclictest0-21swapper/011:13:490
1368199154,10cyclictest6330-21diskmemload11:20:243
1368199154,10cyclictest13939-21gpgv07:10:003
1368199150,14cyclictest6330-21diskmemload09:10:223
1367799151,1cyclictest6330-21diskmemload11:46:142
1367799150,4cyclictest0-21swapper/209:47:342
1367799150,3cyclictest0-21swapper/210:15:122
1367599156,7cyclictest14100-21/usr/sbin/munin07:10:131
1367499154,4cyclictest5327-21grep09:35:260
13674991512,2cyclictest31156-21http11:50:010
1368199147,6cyclictest32088-21tr08:55:213
1368199147,6cyclictest22170-21cat08:34:593
1368199142,6cyclictest19085-21aten2_r7power_v07:20:143
1368199140,12cyclictest17633-21basename08:25:143
13677991413,1cyclictest0-21swapper/210:35:122
13677991412,2cyclictest32-21ksoftirqd/209:44:542
1367799140,13cyclictest0-21swapper/212:06:122
1367599149,3cyclictest5483-21ls08:00:011
1367599149,3cyclictest3007-21tail11:25:221
13675991411,2cyclictest9192-21ls08:05:221
13675991411,2cyclictest16504-21wc11:10:001
13675991410,3cyclictest16431-21grep08:20:241
13675991410,2cyclictest18531-21wc08:25:191
1367499148,2cyclictest9660-21munin-run12:00:000
1367499141,2cyclictest0-21swapper/010:09:430
13674991411,2cyclictest26541-21grep09:25:280
13674991411,2cyclictest26541-21grep09:25:270
1367499140,2cyclictest0-21swapper/009:22:310
237872130,1sleep20-21swapper/211:42:132
1368199137,5cyclictest4683-21wc07:55:263
1368199137,5cyclictest17055-21taskset08:21:563
1368199131,11cyclictest29674-21grep08:50:153
13681991310,2cyclictest24948-21grep08:40:143
1367799139,3cyclictest32-21ksoftirqd/208:10:122
1367799138,4cyclictest11245-21grep11:05:152
1367799134,8cyclictest1929-21latency_hist11:25:012
13677991310,2cyclictest15688-21grep11:35:222
1367799130,9cyclictest0-21swapper/208:05:142
1367799130,13cyclictest0-21swapper/209:18:372
1367599139,3cyclictest31222-21grep09:30:161
1367599139,3cyclictest25485-21cut09:25:181
1367599139,3cyclictest25485-21cut09:25:181
1367599139,2cyclictest25747-21cat08:40:191
1367599135,4cyclictest224-21systemd-journal07:15:011
1367599133,6cyclictest22455-21grep12:10:191
13675991310,2cyclictest20394-21sort10:45:181
13675991310,2cyclictest12994-21ssh12:01:531
1367599130,2cyclictest24458-21perf08:40:001
1367599130,12cyclictest18505-21grep10:15:291
1367499138,1cyclictest0-21swapper/008:45:130
1367499133,2cyclictest0-21swapper/010:45:140
1367499131,8cyclictest0-21swapper/011:40:190
1367499130,4cyclictest0-21swapper/009:11:380
1368199126,5cyclictest5896-21cat08:00:173
1367799129,2cyclictest5998-21chrt09:06:082
1367799129,2cyclictest589-21sort10:00:182
1367799129,2cyclictest5854-21grep08:00:172
1367799129,2cyclictest2503-21sort09:00:172
1367799129,2cyclictest22556-21cut12:10:202
1367799129,2cyclictest16530-21which08:20:252
1367799123,6cyclictest5328-21mailstats09:35:262
13677991211,1cyclictest0-21swapper/209:14:382
1367799120,1cyclictest0-21swapper/207:25:002
1367599129,2cyclictest9206-21head10:35:281
1367599129,2cyclictest8551-21grep11:30:161
1367599129,2cyclictest7294-21grep10:05:271
1367599129,2cyclictest7077-21grep09:10:161
1367599129,2cyclictest29787-21ls12:15:261
1367599129,2cyclictest26242-21grep07:35:161
1367599129,2cyclictest2054-21tr09:00:151
1367599129,2cyclictest1609-21sed07:50:171
1367599129,2cyclictest13443-21cut10:40:161
1367599128,3cyclictest11318-21wc10:10:161
1367599128,2cyclictest5832-21tr08:00:181
1367599122,5cyclictest0-21swapper/109:40:211
13675991212,0cyclictest0-21swapper/111:46:521
1367599120,6cyclictest0-21swapper/110:25:191
1367599120,11cyclictest0-21swapper/109:45:181
1367499129,2cyclictest7615-21ssh11:01:320
1367499129,2cyclictest31469-21ssh11:21:280
1367499129,2cyclictest30921-21wc08:50:260
1367499129,2cyclictest28766-21tr07:40:310
1367499129,2cyclictest26970-21sshd10:50:230
1367499129,2cyclictest24381-21ssh12:11:150
1367499129,2cyclictest22761-21head10:20:170
1367499129,2cyclictest21780-21expr07:25:420
1367499129,2cyclictest20030-21basename08:30:180
1367499129,2cyclictest17232-21taskset08:24:110
1367499129,2cyclictest14451-21expr07:10:150
1367499129,2cyclictest1392-21ssh10:30:020
1367499129,2cyclictest13794-21cat08:15:190
1367499129,2cyclictest12323-21ssh10:39:180
1367499129,2cyclictest1227-21cat07:50:140
1367499122,5cyclictest5288-21latency_hist10:04:590
13674991210,1cyclictest2448-21grep09:00:170
1367499120,1cyclictest6330-21diskmemload12:35:050
1367499120,1cyclictest0-21swapper/012:06:120
1367499120,12cyclictest0-21swapper/009:50:070
1368199110,8cyclictest0-21swapper/308:10:243
1367799119,1cyclictest9494-21tr11:30:212
1367799119,1cyclictest5966-21sort11:55:192
1367799119,1cyclictest304862sleep207:43:382
1367799119,1cyclictest3011-21sort07:55:002
1367799119,1cyclictest26427-21grep09:55:172
1367799119,1cyclictest246842sleep208:40:082
1367799119,1cyclictest20083-21hostname08:30:182
1367799119,1cyclictest14915-21cat12:30:232
1367799119,1cyclictest11534-21sh12:28:112
1367799119,1cyclictest11534-21sh12:28:102
1367799118,2cyclictest32007-21wc09:30:252
1367799118,2cyclictest26581-21expr07:35:182
1367799118,2cyclictest22608-21ssh11:15:022
1367799118,2cyclictest22608-21ssh11:15:022
1367799118,2cyclictest22063-21ssh12:36:072
1367799118,2cyclictest14622-21grep07:10:162
1367799111,7cyclictest1531-21cron09:00:002
1367799110,6cyclictest13855-21seq10:11:462
1367799110,11cyclictest21519-21grep07:25:422
1367799110,11cyclictest0-21swapper/210:45:102
1367799110,10cyclictest3640-21expr07:55:162
1367599118,2cyclictest7926-21cut12:25:171
1367599118,2cyclictest7926-21cut12:25:161
1367599118,2cyclictest7904-21ssh11:57:081
1367599118,2cyclictest6825-21ssh11:00:341
1367599118,2cyclictest6487-21ssh10:34:221
1367599118,2cyclictest4667-21grep09:35:201
1367599118,2cyclictest31609-21wc10:55:141
1367599118,2cyclictest3146-21ssh10:02:281
1367599118,2cyclictest27143-21tail09:55:221
1367599118,2cyclictest26521-21sed10:50:201
1367599118,2cyclictest24959-21grep07:30:231
1367599118,2cyclictest24959-21grep07:30:231
1367599118,2cyclictest22764-21ssh11:15:131
1367599118,2cyclictest22764-21ssh11:15:131
1367599118,2cyclictest19901-21uname09:20:191
1367599118,2cyclictest19707-21cut12:35:161
1367599118,2cyclictest18437-21ssh12:34:121
1367599118,2cyclictest16601-21chrt11:35:501
1367599118,2cyclictest15311-21ssh09:16:121
1367599110,10cyclictest16693-21rm11:10:061
1367499119,2cyclictest10130-21uniq09:40:180
1367499119,1cyclictest8495-21basename12:25:200
1367499119,1cyclictest8495-21basename12:25:200
1367499119,1cyclictest3640-21wc11:25:270
1367499119,1cyclictest3181-21acpi07:55:120
1367499119,1cyclictest31307-21expr07:45:160
1367499119,1cyclictest28090-21taskset07:38:390
1367499119,1cyclictest23895-21cat11:15:260
1367499119,1cyclictest23895-21cat11:15:250
1367499119,1cyclictest19971-21cat07:20:200
1367499119,1cyclictest18318-21ssh12:34:050
1367499119,1cyclictest14639-21grep09:45:160
1367499119,1cyclictest14632-21grep09:15:260
1367499119,1cyclictest12038-21grep08:10:270
1367499119,1cyclictest10312-21ssh11:30:390
1367499118,2cyclictest7277-21grep08:00:310
1367499118,2cyclictest4688-21df_abs09:05:140
1367499118,2cyclictest25266-21cat07:30:270
1367499118,2cyclictest25266-21cat07:30:270
1367499110,6cyclictest0-21swapper/011:35:020
1367499110,10cyclictest12322-21ssh10:10:240
1367799109,1cyclictest327002sleep207:46:312
1367799109,1cyclictest315182sleep208:54:182
1367799109,1cyclictest12933-21cat08:15:152
1367799108,1cyclictest27742-21grep08:45:172
1367799108,1cyclictest24638-21wc08:40:002
1367799107,2cyclictest17713-21grep08:25:142
1367599108,1cyclictest31937-21cut08:55:191
1367599108,1cyclictest29622-21egrep08:50:151
1367599108,1cyclictest199882sleep108:30:171
1367599108,1cyclictest12812-21cut08:15:141
1367599107,2cyclictest4694-21grep09:05:141
1367599107,2cyclictest32229-21sendmail_mailtr07:45:231
1367599107,2cyclictest21696-21ssh09:50:251
1367599107,2cyclictest12198-21taskset08:11:451
1367599106,3cyclictest27844-21grep08:45:171
1367599104,5cyclictest31148-21munin-node07:24:411
1367499109,1cyclictest26746-21sshd08:44:040
1367499108,1cyclictest31810-21chrt08:55:030
1367499108,1cyclictest30804-21egrep09:30:150
1367499108,1cyclictest22647-21expr08:35:150
1367499104,5cyclictest16542-21/usr/sbin/munin07:15:440
1367499100,7cyclictest0-21swapper/008:05:140
136759997,1cyclictest213072sleep107:25:061
136759985,2cyclictest28609-21wget07:40:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional