You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-14 - 00:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Sep 13, 2025 12:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28633210679,8sleep30-21swapper/307:08:223
28596210480,10sleep10-21swapper/107:07:501
2851328367,9sleep00-21swapper/007:06:460
2861727666,6sleep20-21swapper/207:08:072
28858992625,1cyclictest0-21swapper/207:40:012
2885899242,3cyclictest0-21swapper/208:00:152
2885599230,23cyclictest0-21swapper/110:56:061
2885899220,20cyclictest0-21swapper/212:38:312
2885399226,14cyclictest7744-21snmpd12:35:300
2886399212,1cyclictest0-21swapper/309:17:413
28858992119,1cyclictest0-21swapper/210:00:142
28858992018,2cyclictest37850irq/131-snd_hda11:30:152
2885899200,20cyclictest0-21swapper/212:25:102
28855992020,0cyclictest0-21swapper/111:59:041
2885899192,10cyclictest0-21swapper/208:30:142
2885599195,12cyclictest0-21swapper/112:35:351
162602190,0sleep20-21swapper/210:24:502
2885899180,1cyclictest0-21swapper/209:53:582
2885399180,4cyclictest0-21swapper/012:22:590
57092170,0sleep00-21swapper/011:35:250
28863991715,1cyclictest39-21ksoftirqd/312:23:563
2886399171,4cyclictest17596-21kworker/u8:010:02:153
2886399171,1cyclictest141rcu_preempt09:12:073
2886399170,2cyclictest0-21swapper/312:29:213
2885599175,4cyclictest32637-21perf08:25:011
2885599170,16cyclictest0-21swapper/110:15:301
28863991613,3cyclictest0-21swapper/312:16:103
2885399165,5cyclictest6412-21ssh09:50:010
28853991611,2cyclictest22855-21cut09:10:190
28863991514,1cyclictest39-21ksoftirqd/307:40:173
28863991513,1cyclictest39-21ksoftirqd/311:52:593
28863991513,1cyclictest39-21ksoftirqd/309:38:583
2886399150,2cyclictest0-21swapper/310:43:523
2885899157,7cyclictest37850irq/131-snd_hda11:56:522
2885899151,1cyclictest0-21swapper/209:39:072
28858991511,2cyclictest16424-21sed08:55:292
28858991510,4cyclictest1638-21sh10:40:002
2885899150,1cyclictest0-21swapper/211:23:292
2885899150,15cyclictest0-21swapper/211:09:042
28855991512,2cyclictest0-21swapper/112:24:401
2885399151,8cyclictest18320-21munin-run07:55:010
230782150,1sleep1241rcuc/111:50:001
2886399149,4cyclictest6404-21munin-run09:50:013
2886399143,0cyclictest141rcu_preempt09:26:473
28863991411,2cyclictest31126-21ssh10:10:173
28858991413,1cyclictest0-21swapper/212:30:132
28858991413,1cyclictest0-21swapper/209:33:272
28858991413,0cyclictest0-21swapper/212:23:562
28858991411,2cyclictest31876-21wc10:10:242
2885599140,2cyclictest0-21swapper/107:25:151
2885399143,5cyclictest21976-21perf12:30:010
28853991413,1cyclictest11288-21apache210:53:490
28853991411,2cyclictest22213-21head08:00:370
2885399140,13cyclictest0-21swapper/011:32:140
2886399138,4cyclictest241792sleep312:10:223
2886399138,4cyclictest241792sleep312:10:223
2886399138,2cyclictest39-21ksoftirqd/309:59:223
28863991311,1cyclictest32284-21sed12:35:283
28863991311,1cyclictest19962-21chmod09:35:013
28863991310,3cyclictest39-21ksoftirqd/310:59:163
28863991310,2cyclictest9593-21sort07:35:173
28863991310,2cyclictest6527-21cut11:10:183
28863991310,2cyclictest6297-21grep10:15:283
28863991310,2cyclictest32639-21basename08:25:013
28863991310,2cyclictest27436-21grep11:00:233
28863991310,2cyclictest16945-21sed12:05:263
28863991310,2cyclictest15937-21munin-plugin-st07:50:013
2886399130,13cyclictest0-21swapper/311:34:453
2885899139,3cyclictest300-21cat08:25:012
2885899132,5cyclictest224-21systemd-journal12:20:002
28858991310,2cyclictest9276-21grep07:35:162
28858991310,2cyclictest27091-21wc09:40:192
28858991310,2cyclictest25747-21cut10:05:182
28858991310,2cyclictest24943-21grep11:50:222
28858991310,2cyclictest22505-21ls12:09:592
28858991310,2cyclictest20664-21tr10:55:182
28858991310,2cyclictest14689-21ssh10:50:152
28858991310,2cyclictest1223-21tail09:45:212
2885899130,0cyclictest0-21swapper/212:00:562
2885599139,3cyclictest19300-21cron09:05:001
2885599139,3cyclictest11342-21taskset11:14:121
28855991313,0cyclictest0-21swapper/109:23:411
28855991310,2cyclictest30513-21ls07:10:231
28855991310,2cyclictest30184-21cat10:09:591
28855991310,2cyclictest17299-21grep07:50:211
28855991310,2cyclictest1373-21cron09:20:001
2885599130,12cyclictest7744-21snmpd09:50:011
2885399139,3cyclictest26754-21sed11:25:280
2885399139,3cyclictest22529-21cron10:30:000
2885399139,2cyclictest13626-21ssh11:15:240
28853991310,2cyclictest23951-21sh11:50:150
2886399129,3cyclictest11288-21apache211:08:263
2886399129,2cyclictest949-21grep07:15:303
2886399129,2cyclictest9283-21sed08:41:043
2886399129,2cyclictest8838-21taskset11:38:333
2886399129,2cyclictest7576-21cron09:24:593
2886399129,2cyclictest5332-21grep07:25:213
2886399129,2cyclictest27625-21ssh09:40:223
2886399129,2cyclictest26927-21chrt11:25:373
2886399129,2cyclictest25993-21grep08:10:143
2886399129,2cyclictest18230-21grep10:25:223
2886399129,2cyclictest15988-21grep11:45:003
2886399128,3cyclictest4963-21apt08:35:003
2886399128,3cyclictest28816-21tr08:15:193
2886399128,3cyclictest19377-21grep09:05:003
2886399128,2cyclictest23319-21tr10:30:173
2886399128,2cyclictest20067-21wc10:55:013
2886399127,3cyclictest12436-21grep11:15:173
2886399126,3cyclictest0-21swapper/310:45:143
2886399126,2cyclictest224-21systemd-journal08:40:003
2886399124,4cyclictest224-21systemd-journal08:02:313
2886399122,9cyclictest11296-21apache211:24:113
28863991210,1cyclictest39-21ksoftirqd/312:34:533
28863991210,1cyclictest0-21swapper/311:55:213
2886399120,1cyclictest0-21swapper/311:45:353
2885899129,3cyclictest61750irq/125-enp2s0-10:43:362
2885899129,2cyclictest9294-21grep09:25:222
2885899129,2cyclictest8035-21cat07:30:252
2885899129,2cyclictest5250-21grep08:35:142
2885899129,2cyclictest30909-21grep07:10:272
2885899129,2cyclictest21686-21turbostat.cron09:10:002
2885899129,2cyclictest17532-21grep11:45:232
2885899129,2cyclictest15941-21turbostat07:50:002
2885899129,2cyclictest13918-21dpkg10:50:002
2885899129,2cyclictest13236-21cut09:55:162
2885899128,3cyclictest8101-21rm10:17:292
2885899128,3cyclictest28605-21tr09:15:182
2885899128,3cyclictest2258-21taskset08:26:032
2885899128,2cyclictest19456-21wc09:05:002
2885899127,4cyclictest22829-21tail09:10:202
2885899120,11cyclictest0-21swapper/211:35:062
2885599129,2cyclictest8667-21tr10:45:151
2885599129,2cyclictest8290-21grep08:40:551
2885599129,2cyclictest31948-21grep12:15:191
2885599129,2cyclictest2629-21grep07:20:181
2885599129,2cyclictest25016-21cut11:25:161
2885599129,2cyclictest23177-21grep12:30:171
2885599129,2cyclictest16362-21grep10:50:271
2885599128,3cyclictest19159-21grep11:20:181
2885599128,2cyclictest6030-21grep11:35:281
2885599125,6cyclictest21681-21diskmemload10:35:191
2885599125,2cyclictest22510-21apt12:09:591
2885599123,3cyclictest21681-21diskmemload09:35:181
2885599120,1cyclictest0-21swapper/108:47:531
2885599120,10cyclictest0-21swapper/108:10:141
2885399129,2cyclictest71112sleep008:37:250
2885399129,2cyclictest6853-21date12:00:010
2885399129,2cyclictest4381-21tr07:25:160
2885399129,2cyclictest3982-21tr08:30:210
2885399129,2cyclictest30145-21cat10:09:590
2885399129,2cyclictest30073-21sed07:10:180
2885399129,2cyclictest27463-21head11:00:220
2885399129,2cyclictest24540-21wc11:25:000
2885399129,2cyclictest23318-21tail12:30:170
2885399129,2cyclictest13115-21grep07:40:280
2885399128,3cyclictest29414-21grep10:35:170
2885399128,3cyclictest1076-21sed11:55:270
228592120,1sleep2902-21kworker/2:010:30:062
2886399118,2cyclictest9547-21taskset12:00:493
2886399118,2cyclictest8684-21chrt07:34:363
2886399118,2cyclictest3976-21sh07:25:003
2886399118,2cyclictest39-21ksoftirqd/309:50:143
2886399118,2cyclictest387-21ssh10:38:273
2886399118,2cyclictest31354-21taskset07:14:313
2886399118,2cyclictest29881-21ssh10:09:233
2886399118,2cyclictest1215-21tail08:25:183
2886399118,2cyclictest11876-21grep10:20:223
2885899118,2cyclictest6933-21grep11:10:202
2885899118,2cyclictest6220-21sshd07:28:312
2885899118,2cyclictest32480-21hwlatdetect07:15:212
2885899118,2cyclictest31239-21ssh11:03:522
2885899118,2cyclictest25834-21aten2_r7power_v08:10:132
2885899118,2cyclictest24977-21tr11:25:152
2885899118,2cyclictest23778-21cat08:05:172
2885899118,2cyclictest23500-21expr12:10:172
2885899118,2cyclictest23500-21expr12:10:172
2885899118,2cyclictest17634-21tail10:25:182
2885899118,2cyclictest13592-21sh11:15:252
2885899118,2cyclictest11105-21cut11:40:192
2885899110,10cyclictest0-21swapper/209:22:172
2885599119,1cyclictest11976-21wc11:40:341
2885599118,2cyclictest9381-21ssh12:00:401
2885599118,2cyclictest7823-21ssh09:50:201
2885599118,2cyclictest6926-21grep08:35:361
2885599118,2cyclictest30191-21ssh11:02:251
2885599118,2cyclictest28144-21tr08:15:151
2885599118,2cyclictest24327-21ssh10:04:451
2885599118,2cyclictest2411-21grep10:40:161
2885599118,2cyclictest23858-21sed10:30:201
2885599118,2cyclictest23572-21ssh11:50:131
2885599118,2cyclictest2339-21ssh10:13:081
2885599118,2cyclictest22584-21ssh09:10:171
2885599118,2cyclictest21057-21cut08:00:291
2885599118,2cyclictest20776-21ssh10:27:491
2885599118,2cyclictest19911-21readlink07:55:231
2885599118,2cyclictest17670-21ssh12:26:261
2885599118,2cyclictest16800-21ssh11:18:341
2885599118,2cyclictest16422-21cron10:25:001
2885599118,2cyclictest15316-21cat07:45:271
2885599118,2cyclictest14810-21cat09:30:201
2885599118,2cyclictest13200-21sshd07:40:321
2885599118,2cyclictest1229-21ssh11:31:331
2885599118,2cyclictest10336-21ssh09:25:571
2885599117,3cyclictest32362-21cron09:45:011
2885599116,3cyclictest9131-21grep07:35:151
2885599110,10cyclictest15125-21ssh09:55:481
2885399118,2cyclictest9617-21cut07:35:170
2885399118,2cyclictest8840-21sh09:25:160
2885399118,2cyclictest8189-21ssh11:10:350
2885399118,2cyclictest7237-21cut07:30:170
2885399118,2cyclictest6991-21ssh09:24:170
2885399118,2cyclictest4126-21ssh10:40:370
2885399118,2cyclictest32640-21wc12:15:250
2885399118,2cyclictest31985-21ssh10:10:240
2885399118,2cyclictest31588-21ssh09:17:030
2885399118,2cyclictest28837-21grep08:15:180
2885399118,2cyclictest27776-21chrt08:14:210
2885399118,2cyclictest25490-21ssh12:10:520
2885399118,2cyclictest25490-21ssh12:10:510
2885399118,2cyclictest25164-21ssh10:30:440
2885399118,2cyclictest20484-21cut09:35:150
2885399118,2cyclictest20199-21tr10:00:220
2885399118,2cyclictest20106-21cat07:55:260
2885399118,2cyclictest19837-21ssh12:07:460
2885399118,2cyclictest18830-21grep09:00:330
2885399118,2cyclictest16310-21ssh09:31:020
2885399118,2cyclictest15478-21fschecks_time08:55:190
2885399118,2cyclictest15192-21ssh09:55:530
2885399118,2cyclictest13554-21ssh11:42:270
2885399118,2cyclictest13026-21sort08:50:190
2885399118,2cyclictest1175-21cat11:05:210
2885399117,3cyclictest14822-21seq10:23:070
2885399117,3cyclictest11684-21grep08:45:270
2885399115,2cyclictest224-21systemd-journal10:50:000
2885399112,5cyclictest26792-21aten2_r7power_c09:40:170
2885399110,11cyclictest0-21swapper/011:45:270
2886399108,1cyclictest191692sleep307:55:163
2886399108,1cyclictest180492sleep307:51:483
2886399108,1cyclictest16564-21taskset08:55:403
2886399107,2cyclictest25210-21chrt08:06:593
2886399107,2cyclictest10736-21tail08:45:183
2886399100,3cyclictest0-21swapper/308:50:163
2885899108,1cyclictest142872sleep208:51:582
2885899107,2cyclictest19215-21grep07:55:172
2885899107,1cyclictest3225-21sendmail_mailtr07:20:242
2885899104,3cyclictest16979-21gltestperf07:50:182
2885899102,4cyclictest224-21systemd-journal08:16:462
2885599108,1cyclictest7436-21basename07:30:171
2885599108,1cyclictest4190-21cut08:30:241
2885599108,1cyclictest22810-21ssh12:10:051
2885599108,1cyclictest22810-21ssh12:10:051
2885599108,1cyclictest20639-21cat09:05:211
2885599108,1cyclictest16069-21cut08:55:251
2885599108,1cyclictest1230-21chrt07:18:331
2885599107,2cyclictest32372-21ssh11:05:061
2885599107,2cyclictest25041-21grep08:05:281
2885599107,2cyclictest1223-21head08:25:191
2885399108,1cyclictest9530-21taskset08:42:400
2885399108,1cyclictest7171-21expr09:50:160
2885399108,1cyclictest4402-21cat10:15:130
2885399108,1cyclictest23711-21expr08:05:170
2885399108,1cyclictest22572sleep008:26:030
2885399108,1cyclictest15670-21chrt07:46:440
2885399107,2cyclictest435-21ls07:15:260
2885399107,2cyclictest21868-21python10:55:290
2885399107,2cyclictest20305-21grep09:05:180
288639996,2cyclictest21523-21taskset09:08:053
288589998,1cyclictest11395-21mailstats08:45:232
288589997,1cyclictest7636-21aten2_r7power_c08:40:162
288559997,1cyclictest12882-21expr08:50:171
288539996,2cyclictest31257-21grep08:20:180
288539986,1cyclictest3263-21python07:20:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional