You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 19:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Nov 20, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17990211182,10sleep10-21swapper/107:08:571
1779429885,9sleep30-21swapper/307:06:253
1593328368,10sleep00-21swapper/007:05:020
1774828265,13sleep20-21swapper/207:05:472
1817299240,5cyclictest0-21swapper/310:40:223
1817299240,18cyclictest39-21ksoftirqd/312:06:143
18164992312,4cyclictest10790-21diskmemload12:08:591
186822220,1sleep211402-21kworker/2:211:00:302
1817299210,1cyclictest0-21swapper/311:35:013
1817299204,2cyclictest141rcu_preempt10:33:493
1816499200,19cyclictest0-21swapper/109:14:381
1816299208,6cyclictest689-21munin-run09:54:590
1816299208,6cyclictest24086-21awk10:40:000
1816299200,3cyclictest0-21swapper/012:20:410
1816299200,3cyclictest0-21swapper/012:20:400
1816299200,2cyclictest0-21swapper/012:11:300
1816299200,1cyclictest0-21swapper/009:27:220
1817299192,4cyclictest39-21ksoftirqd/312:17:563
1816499193,8cyclictest0-21swapper/109:35:121
1816499191,13cyclictest0-21swapper/111:40:131
1816299195,5cyclictest18166-21turbostat10:35:000
1816799187,7cyclictest26857-21turbostat.cron09:49:592
1816799182,15cyclictest0-21swapper/210:20:132
18162991817,1cyclictest0-21swapper/011:38:150
1817299173,4cyclictest0-21swapper/310:29:483
1817299172,1cyclictest141rcu_preempt12:34:563
18172991715,2cyclictest0-21swapper/309:41:593
18172991714,3cyclictest39-21ksoftirqd/310:38:383
18167991713,4cyclictest10546-21apache210:11:272
18167991712,3cyclictest28644-21chrt08:40:012
18162991716,0cyclictest0-21swapper/010:45:130
1817299165,11cyclictest10790-21diskmemload10:10:263
1817299160,1cyclictest0-21swapper/312:29:453
1817299160,1cyclictest0-21swapper/311:11:483
1817299160,1cyclictest0-21swapper/310:57:593
1817299160,15cyclictest39-21ksoftirqd/308:30:123
18167991616,0cyclictest0-21swapper/211:30:062
18162991615,1cyclictest0-21swapper/012:32:450
1817299158,5cyclictest39-21ksoftirqd/310:01:153
18167991514,1cyclictest0-21swapper/212:29:092
1816299154,7cyclictest21775-21apt-get08:25:100
1816299153,6cyclictest7772-21perf09:34:590
18162991514,1cyclictest0-21swapper/011:33:170
1817299143,1cyclictest141rcu_preempt11:34:263
18172991412,1cyclictest0-21swapper/311:03:013
1817299141,10cyclictest39-21ksoftirqd/307:35:133
18172991410,3cyclictest24145-21cat09:20:193
18172991410,3cyclictest10052-21grep08:00:143
1817299140,1cyclictest0-21swapper/309:45:563
1816799146,1cyclictest10790-21diskmemload12:23:502
1816799146,1cyclictest10790-21diskmemload12:23:492
1816799145,8cyclictest0-21swapper/212:16:342
18167991410,2cyclictest26392-21head10:40:282
1816799140,14cyclictest0-21swapper/210:57:442
1816499148,4cyclictest5892-21apt10:25:001
18164991410,3cyclictest6473-21cat07:50:251
18164991410,2cyclictest23662-21grep11:30:201
1816499140,14cyclictest0-21swapper/109:55:371
1816499140,13cyclictest0-21swapper/110:40:031
1816299141,7cyclictest9527-21munin-run08:00:000
1816299140,14cyclictest0-21swapper/010:18:230
18172991311,1cyclictest381rcuc/311:57:413
18172991310,2cyclictest9564-21wc07:59:593
18172991310,2cyclictest4756-21cron07:49:593
18172991310,2cyclictest23243-21grep07:20:143
18172991310,2cyclictest18869-21sed09:15:363
18172991310,2cyclictest10790-21diskmemload09:30:253
1817299130,7cyclictest39-21ksoftirqd/311:15:133
1816799137,2cyclictest224-21systemd-journal08:00:002
1816799131,1cyclictest10790-21diskmemload10:46:312
18167991310,2cyclictest28126-21grep10:15:262
1816799130,13cyclictest0-21swapper/212:10:392
1816499139,3cyclictest819-21grep12:25:231
1816499139,3cyclictest18147-21grep11:25:301
1816499139,3cyclictest10072-21cat11:20:181
1816499139,2cyclictest28752-21cut09:25:151
1816499133,5cyclictest0-21swapper/109:20:121
18164991310,2cyclictest29647-21head08:40:141
18164991310,2cyclictest29011-21wc12:00:001
18164991310,2cyclictest28649-21grep07:30:171
18164991310,2cyclictest22774-21ssh11:05:001
1816499130,12cyclictest18192-21sleep10:35:001
1816299139,3cyclictest4840-21tr10:50:160
1816299139,2cyclictest10934-21cat09:10:000
1816299138,3cyclictest22201-21wc09:45:240
1816299136,2cyclictest29221-21munin-run11:10:000
1816299135,7cyclictest26275-21munin-run08:35:010
1816299135,4cyclictest2877-21sh11:14:340
18162991310,2cyclictest4588-21sh11:40:260
18162991310,2cyclictest31398-21cut07:35:200
18162991310,2cyclictest18027-21grep12:15:170
18162991310,2cyclictest17745-21cut08:15:200
1816299130,8cyclictest16830-21perf09:15:000
1816299130,1cyclictest0-21swapper/012:01:530
1816299130,11cyclictest20697-21wc07:15:000
1817299129,2cyclictest8262-21tr09:35:113
1817299129,2cyclictest5167-21grep07:50:143
1817299129,2cyclictest2775-21cat09:55:263
1817299129,2cyclictest22939-21tr08:25:373
1817299129,2cyclictest18019-21ssh10:07:543
1817299128,3cyclictest5405-21head10:50:193
1817299128,3cyclictest30030-21find09:25:223
1817299128,3cyclictest21500-21cron08:25:003
1817299128,3cyclictest1-21systemd10:24:223
1817299127,4cyclictest381rcuc/311:44:593
1817299125,2cyclictest224-21systemd-journal08:15:013
1817299123,5cyclictest39-21ksoftirqd/309:54:423
1817299123,1cyclictest10790-21diskmemload11:05:113
1817299122,4cyclictest0-21swapper/310:15:133
1817299121,7cyclictest8605-21runrttasks09:05:093
1817299120,0cyclictest0-21swapper/312:37:243
1816799129,2cyclictest5238-21cut10:50:182
1816799129,2cyclictest3494-21tr12:05:142
1816799129,2cyclictest2939-21grep09:55:272
1816799129,2cyclictest23948-21ssh10:39:552
1816799129,2cyclictest22539-21tr11:55:132
1816799129,2cyclictest16228-21grep08:10:272
1816799129,2cyclictest12432-21tr09:10:202
1816799129,2cyclictest10448-21grep08:00:162
1816799128,3cyclictest9824-21ls09:05:202
1816799128,3cyclictest4187-21kworker/2:112:32:402
1816799122,9cyclictest2385-21ssh12:04:402
1816799121,10cyclictest4010-21cstates08:55:362
1816799120,11cyclictest0-21swapper/210:00:192
1816499129,2cyclictest4203-21grep08:55:361
1816499129,2cyclictest26290-21grep12:20:251
1816499129,2cyclictest26290-21grep12:20:251
1816499129,2cyclictest25249-21grep08:30:191
1816499129,2cyclictest23289-21grep11:05:131
1816499129,2cyclictest21845-21tr08:25:141
1816499129,2cyclictest21526-21perf08:25:011
1816499129,2cyclictest17641-21df09:15:171
1816499129,2cyclictest15836-21grep08:10:221
1816499129,2cyclictest15770-21wc09:40:221
1816499129,2cyclictest14300-21sh12:35:001
1816499128,3cyclictest9720-21cut09:05:181
1816499128,3cyclictest8124-21cron09:05:011
1816499128,3cyclictest14624-21head10:05:171
1816499128,2cyclictest7732-21grep10:00:141
1816499124,5cyclictest1238-21apache_accesses08:50:121
1816499123,8cyclictest0-21swapper/110:50:581
1816499123,7cyclictest10790-21diskmemload09:45:221
18164991210,2cyclictest30019-21ssh10:17:291
18164991210,2cyclictest0-21swapper/111:48:191
18164991210,1cyclictest10546-21apache210:56:001
1816299129,2cyclictest6253-21grep09:00:150
1816299129,2cyclictest4747-21tail08:55:410
1816299129,2cyclictest2982-21tail09:55:280
1816299129,2cyclictest18632-21grep11:00:280
1816299129,2cyclictest17596-21sed11:25:250
1816299129,2cyclictest15103-21tr12:35:150
1816299129,2cyclictest12531-21cut08:05:140
1816299129,2cyclictest11561-21grep08:00:260
1816299128,3cyclictest992-21basename08:50:010
1816299128,3cyclictest10941-21uniq10:55:170
1816299125,6cyclictest14172-21sendmail09:39:590
18162991210,1cyclictest10283-21diskstats11:20:180
1816299120,11cyclictest28179-21tr07:30:140
294482110,1sleep10-21swapper/112:00:161
1817299119,2cyclictest39-21ksoftirqd/312:23:533
1817299119,2cyclictest39-21ksoftirqd/312:23:523
1817299119,1cyclictest381rcuc/310:45:553
1817299118,2cyclictest6271-21grep09:00:143
1817299118,2cyclictest39-21ksoftirqd/311:50:183
1817299118,2cyclictest31028-21cron08:45:003
1817299118,2cyclictest29097-21ssh12:00:023
1817299118,2cyclictest21858-21taskset11:29:293
1817299118,2cyclictest20640-21cut07:15:013
1817299118,2cyclictest13267-21cat08:05:183
1817299118,2cyclictest13126-21ssh11:47:493
1817299118,2cyclictest12851-21ssh12:11:063
1817299118,2cyclictest12214-21uname09:10:203
1817299117,3cyclictest381rcuc/311:20:583
1816799119,1cyclictest29154-21grep08:40:122
1816799118,2cyclictest9921-21ssh11:20:182
1816799118,2cyclictest590-21ssh11:11:482
1816799118,2cyclictest5397-21expr07:50:152
1816799118,2cyclictest383-21cat08:45:242
1816799118,2cyclictest3734-21wget11:15:142
1816799118,2cyclictest32202-21taskset07:37:262
1816799118,2cyclictest3028-21expr11:40:142
1816799118,2cyclictest29900-21taskset07:33:222
1816799118,2cyclictest27170-21rm11:07:462
1816799118,2cyclictest26203-21cut07:25:162
1816799118,2cyclictest25880-21sh09:21:222
1816799118,2cyclictest22686-21sshd07:17:352
1816799118,2cyclictest22018-21grep08:25:312
1816799118,2cyclictest1981-21cut09:30:152
1816799118,2cyclictest18531-21rm10:08:282
1816799118,2cyclictest13881-21sed08:05:252
1816799118,2cyclictest10700-21rm10:28:202
1816799118,2cyclictest10206-21grep11:45:232
1816799116,4cyclictest3129-21cat07:45:162
1816799116,4cyclictest16643-21ls12:35:272
1816799116,2cyclictest493-21systemd-logind09:04:382
1816799112,8cyclictest62050irq/128-enp2s0-09:40:162
1816799112,8cyclictest0-21swapper/209:50:412
1816799110,7cyclictest0-21swapper/208:55:012
1816799110,11cyclictest0-21swapper/209:19:252
1816499118,2cyclictest7092-21head10:25:171
1816499118,2cyclictest3656-21cut11:15:131
1816499118,2cyclictest30848-21ssh10:45:111
1816499118,2cyclictest29586-21cat11:35:431
1816499118,2cyclictest28469-21chrt08:37:581
1816499118,2cyclictest28160-21head09:50:331
1816499118,2cyclictest2499-21head09:30:171
1816499118,2cyclictest24033-21if_enp2s007:20:171
1816499118,2cyclictest22528-21ssh10:10:581
1816499118,2cyclictest21057-21tr07:15:191
1816499118,2cyclictest20730-21ssh10:35:431
1816499118,2cyclictest19515-21ssh12:15:411
1816499118,2cyclictest18430-21cut07:10:141
1816499118,2cyclictest16856-21tr11:50:221
1816499118,2cyclictest16771-21head12:35:271
1816499118,2cyclictest13854-21grep08:05:251
1816499118,2cyclictest12044-21ls08:05:001
1816499118,2cyclictest10287-21cut12:10:141
1816299118,2cyclictest7624-21ssh12:07:520
1816299118,2cyclictest5808-21head07:50:170
1816299118,2cyclictest5332-21sed11:15:270
1816299118,2cyclictest4473-21ssh12:28:280
1816299118,2cyclictest4282-21grep07:45:280
1816299118,2cyclictest28515-21taskset08:38:350
1816299118,2cyclictest28505-21ssh11:59:420
1816299118,2cyclictest24574-21grep09:20:220
1816299118,2cyclictest23614-21expr07:20:140
1816299118,2cyclictest23541-21ssh10:12:090
1816299118,2cyclictest2028-21sed08:50:180
1816299118,2cyclictest18851-21grep09:15:370
1816299118,2cyclictest16707-21cat11:50:210
1816299118,2cyclictest16509-21ssh10:06:160
1816299118,2cyclictest14646-21taskset11:49:390
1816299118,2cyclictest14598-21ssh09:40:140
1816299118,2cyclictest1027-21cut07:40:180
1816299117,3cyclictest7425-21wc10:25:200
1816299110,10cyclictest32563-21sh10:20:020
1817299108,1cyclictest21942sleep307:42:403
1817299107,2cyclictest28268-21expr07:30:153
1817299107,2cyclictest2679-21cat08:50:273
1817299103,3cyclictest224-21systemd-journal08:36:383
1817299101,2cyclictest39-21ksoftirqd/308:58:403
1816799108,1cyclictest31222-21ssh11:36:072
1816799108,1cyclictest30514-21sh09:25:292
1816799108,1cyclictest23495-21cat07:20:142
1816799108,1cyclictest21032sleep207:41:302
1816799108,1cyclictest20797-21taskset08:20:272
1816799107,2cyclictest9977-21tail09:35:242
1816799107,2cyclictest18654-21tr07:10:152
1816799107,2cyclictest17793-21rm11:50:462
1816799107,2cyclictest12736-21aten2_r7power_p10:30:152
1816799100,9cyclictest18124-21ntpq08:15:232
1816799100,9cyclictest16040-21ssh11:25:022
1816499108,1cyclictest46022sleep107:48:071
1816499108,1cyclictest32741-21grep07:40:141
1816499108,1cyclictest318332sleep111:10:291
1816499108,1cyclictest272962sleep107:25:271
1816499108,1cyclictest17024-21cut08:15:171
1816299108,1cyclictest26538-21wc07:25:190
1816299108,1cyclictest25758-21ntp_states10:40:210
1816299107,2cyclictest20242-21if_enp0s31f608:20:210
1816299107,2cyclictest15701-21needreboot08:10:200
1816299107,2cyclictest10795-21ssh10:01:530
1816299103,3cyclictest224-21systemd-journal08:40:000
181729997,1cyclictest26903-21cut07:25:213
181729997,1cyclictest20882-21wget07:15:173
181729994,2cyclictest224-21systemd-journal08:19:383
181729993,4cyclictest381rcuc/308:45:153
181679997,1cyclictest24609-21expr08:30:132
181649997,1cyclictest7408-21dpkg07:55:111
181649997,1cyclictest31864-21sed08:45:151
181649997,1cyclictest303052sleep107:35:131
181629997,1cyclictest22821-21taskset07:19:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional