You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-14 - 21:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot7s.osadl.org (updated Tue Oct 14, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15088210878,10sleep10-21swapper/107:05:181
16576210289,9sleep30-21swapper/307:09:343
1447228164,13sleep20-21swapper/207:05:042
1638327965,9sleep00-21swapper/007:07:050
1669599230,0cyclictest0-21swapper/107:55:131
1669899220,6cyclictest0-21swapper/211:55:132
16698992118,2cyclictest17270-21/usr/sbin/munin09:15:242
1670199201,11cyclictest0-21swapper/311:20:163
1670199201,11cyclictest0-21swapper/311:20:163
16698991913,5cyclictest32-21ksoftirqd/211:40:002
172842180,0sleep00-21swapper/010:13:500
1669899180,17cyclictest0-21swapper/210:25:272
1669599180,18cyclictest0-21swapper/111:02:171
1669599179,3cyclictest18120-21ssh10:15:121
1669199177,5cyclictest6140-21perf10:05:000
16701991614,2cyclictest39-21ksoftirqd/309:56:463
16698991611,1cyclictest32-21ksoftirqd/212:05:122
1669599160,3cyclictest0-21swapper/112:20:141
1669199165,6cyclictest19055-21munin-run07:15:000
1670199159,4cyclictest0-21swapper/310:50:143
16698991514,1cyclictest0-21swapper/209:00:122
16698991510,5cyclictest37850irq/131-snd_hda09:29:332
1669899150,1cyclictest0-21swapper/209:54:152
1669899150,15cyclictest0-21swapper/211:26:172
1670199141,12cyclictest18443-21wc11:10:013
16701991411,2cyclictest17881-21sed09:15:283
16701991410,3cyclictest21752-21grep08:25:323
16698991410,2cyclictest12822-21ssh12:00:152
16698991410,2cyclictest12314-21grep09:40:212
1669899140,1cyclictest0-21swapper/212:12:502
1669899140,13cyclictest0-21swapper/209:11:152
1669899140,10cyclictest0-21swapper/208:55:152
1669599149,4cyclictest25-21ksoftirqd/110:35:001
1669599142,12cyclictest24141-21ssh10:20:141
16695991410,3cyclictest22980-21grep10:45:271
16695991410,3cyclictest19567-21grep12:05:211
1669199144,10cyclictest19622-21smartctl11:10:220
1669199142,6cyclictest31024-21munin-run07:40:000
1669199142,6cyclictest31024-21munin-run07:40:000
16691991411,2cyclictest2712-21head09:30:310
16691991411,2cyclictest2712-21head09:30:310
16691991411,2cyclictest24506-21ssh10:47:090
1669199141,10cyclictest22765-21cut09:20:200
16691991410,3cyclictest23683-21head08:30:200
1670199139,3cyclictest6048-21wc07:55:003
1670199139,3cyclictest25532-21grep08:35:163
1670199139,2cyclictest25956-21tr12:10:233
1670199134,3cyclictest9536-21diskmemload10:00:033
1670199133,8cyclictest0-21swapper/311:45:193
16701991310,2cyclictest8742-21grep08:00:153
16701991310,2cyclictest8398-21grep10:05:283
16701991310,2cyclictest5132-21cat12:20:153
16701991310,2cyclictest28464-21grep08:40:203
16701991310,2cyclictest24994-21cron08:34:593
16701991310,2cyclictest20719-21munin-plugin-st10:45:003
1670199130,1cyclictest141rcu_preempt10:16:013
1670199130,1cyclictest0-21swapper/310:58:333
1670199130,13cyclictest0-21swapper/311:42:413
1669899139,3cyclictest32703-21tr08:50:152
1669899139,3cyclictest20774-21tr08:25:182
1669899139,3cyclictest1728-21sed11:50:162
1669899139,3cyclictest11939-21cron10:09:592
1669899131,3cyclictest224-21systemd-journal10:24:162
16698991310,2cyclictest5987-21grep12:20:222
16698991310,2cyclictest32224-21perf08:50:002
16698991310,2cyclictest18619-21grep07:10:262
16698991310,2cyclictest16053-21grep08:15:422
1669599139,3cyclictest992-21grep10:00:151
1669599139,3cyclictest3596-21wc07:50:001
1669599139,2cyclictest32761-21sh10:00:011
1669599139,2cyclictest29663-21head07:35:201
1669599139,2cyclictest29663-21head07:35:191
1669599138,4cyclictest12055-21dpkg10:10:001
1669599133,8cyclictest0-21swapper/107:50:141
1669599133,6cyclictest10990-21apt-get08:05:151
16695991311,2cyclictest0-21swapper/111:38:451
16695991310,2cyclictest505-21cron11:50:001
16695991310,2cyclictest26716-21tr07:30:141
16695991310,2cyclictest21971-21tr07:20:141
16695991310,2cyclictest13139-21sed12:25:181
16695991310,2cyclictest11121-21turbostat09:40:011
1669599130,2cyclictest0-21swapper/110:55:141
1669599130,13cyclictest0-21swapper/109:26:591
1669599130,12cyclictest32059-21grep07:40:181
1669599130,12cyclictest24567-21tail08:30:291
1669199139,2cyclictest4956-21seq09:00:130
1669199138,3cyclictest28832-21grep08:40:230
1669199131,7cyclictest12269-21munin-run11:05:000
16691991310,2cyclictest826-21sed08:50:170
16691991310,2cyclictest28010-21ssh09:25:150
16691991310,2cyclictest15346-21cut10:40:200
16691991310,2cyclictest13281-21cat08:10:010
16691991310,2cyclictest12427-21sed09:40:210
16691991310,2cyclictest11102-21apt09:40:000
1670199129,2cyclictest4617-21grep11:25:243
1670199129,2cyclictest3713-21wc07:50:003
1670199129,2cyclictest31911-21tr12:15:223
1670199129,2cyclictest31334-21head08:45:263
1670199129,2cyclictest30166-21basename11:20:003
1670199129,2cyclictest29858-21dump-pmu-power10:24:593
1670199129,2cyclictest2698-21sed11:50:273
1670199129,2cyclictest2177-21head09:30:273
1670199129,2cyclictest2177-21head09:30:263
1670199129,2cyclictest21429-21apt07:20:003
1670199129,2cyclictest20417-21wc11:10:313
1670199129,2cyclictest14372-21ssh10:10:283
1670199129,2cyclictest13690-21sed12:25:243
1670199128,3cyclictest32107-21sed10:25:273
1670199128,2cyclictest18622-21rm12:05:153
1670199121,7cyclictest224-21systemd-journal07:25:003
1670199120,11cyclictest23776-21wc09:50:243
1669899129,2cyclictest4403-21cat12:20:002
1669899129,2cyclictest28424-21cut11:45:212
1669899129,2cyclictest25564-21cut11:15:172
1669899129,2cyclictest20438-21cat08:25:002
1669899129,2cyclictest1421-21sh11:22:062
1669899129,2cyclictest1421-21sh11:22:062
1669899128,2cyclictest12254-21dump-pmu-power11:05:002
1669899127,4cyclictest30760-21taskset07:37:032
1669899127,4cyclictest30760-21taskset07:37:022
16698991210,1cyclictest18924-21sh11:10:192
1669899120,11cyclictest0-21swapper/210:30:382
1669599129,3cyclictest165572sleep110:40:281
1669599129,2cyclictest3685-21wc08:55:231
1669599129,2cyclictest26846-21sed12:35:271
1669599129,2cyclictest26846-21sed12:35:271
1669599129,2cyclictest24731-21sort07:25:171
1669599129,2cyclictest22942-21ssh11:40:241
1669599129,2cyclictest2133-21chrt11:50:201
1669599129,2cyclictest13792-21sed12:00:211
1669599129,2cyclictest12656-21grep10:10:171
1669599128,3cyclictest3320-21cron10:30:001
1669599128,3cyclictest24221-21cron11:15:001
1669599128,3cyclictest21938-21munin-plugin-st09:50:011
1669599128,3cyclictest19410-21tune2fs12:30:181
1669599128,3cyclictest12709-21grep11:05:121
1669599128,3cyclictest11562-21uname11:30:211
1669599128,3cyclictest10550irq/122-ahci[0011:55:121
1669599120,8cyclictest7744-21snmpd10:52:591
1669599120,12cyclictest0-21swapper/109:22:041
1669199129,2cyclictest6029-21tr12:20:210
1669199129,2cyclictest4477-21grep07:50:170
1669199129,2cyclictest30902-21tr11:20:190
1669199129,2cyclictest30902-21tr11:20:180
1669199129,2cyclictest14593-21tr08:10:230
1669199128,3cyclictest26739-21tr07:30:140
1669199128,3cyclictest21621-21grep11:40:150
1669199128,3cyclictest19838-21grep08:20:280
1669199128,3cyclictest10325-21sed08:00:270
1669199126,2cyclictest224-21systemd-journal11:20:000
1669199121,6cyclictest224-21systemd-journal12:15:000
1669199120,7cyclictest993-21idleruntime-cro07:45:000
1669199120,6cyclictest32753-21cron10:00:000
1669199120,5cyclictest27982-21fschecks_time11:45:180
1670199118,2cyclictest8338-21perf08:00:003
1670199118,2cyclictest8219-21cut11:00:223
1670199118,2cyclictest6927-21ssh11:55:093
1670199118,2cyclictest6632-21taskset09:01:193
1670199118,2cyclictest5848-21ssh10:31:143
1670199118,2cyclictest4379-21chrt08:58:123
1670199118,2cyclictest32761-21cut08:50:153
1670199118,2cyclictest31966-21cut07:40:173
1670199118,2cyclictest31488-21ssh09:28:133
1670199118,2cyclictest29272-21ssh12:37:183
1670199118,2cyclictest29272-21ssh12:37:173
1670199118,2cyclictest21965-21ssh09:20:123
1670199118,2cyclictest19528-21head12:30:183
1670199118,2cyclictest17783-21sh11:35:473
1670199118,2cyclictest16491-21taskset09:45:093
1670199118,2cyclictest13203-21cat08:10:003
1670199118,2cyclictest12622-21taskset11:30:513
1670199118,2cyclictest11511-21ssh09:40:113
1670199118,2cyclictest10383-21expr09:10:163
1670199111,9cyclictest8588-21ssh09:36:263
1669899118,2cyclictest9251-21fschecks_time08:00:182
1669899118,2cyclictest8928-21cat09:05:302
1669899118,2cyclictest8184-21ssh09:35:412
1669899118,2cyclictest4622-21ssh09:33:212
1669899118,2cyclictest4622-21ssh09:33:212
1669899118,2cyclictest3737-21date07:50:002
1669899118,2cyclictest30724-21ssh09:56:462
1669899118,2cyclictest29659-21rm10:50:542
1669899118,2cyclictest25780-21head12:35:182
1669899118,2cyclictest25780-21head12:35:182
1669899118,2cyclictest25544-21grep08:35:162
1669899118,2cyclictest2480-21cat10:55:262
1669899118,2cyclictest22479-21ssh12:32:302
1669899118,2cyclictest21725-21sh10:45:172
1669899118,2cyclictest20990-21ssh09:48:412
1669899118,2cyclictest18972-21smartctl10:15:162
1669899118,2cyclictest15897-21sh11:07:092
1669899118,2cyclictest15106-21taskset10:11:102
1669899118,2cyclictest13625-21grep08:10:152
1669899118,2cyclictest12210-21head08:05:242
1669899118,2cyclictest11899-21ssh10:36:282
1669899118,2cyclictest1030-21tr10:00:152
1669899118,1cyclictest9536-21diskmemload11:33:582
1669899117,3cyclictest12391-21expr12:25:132
1669899110,10cyclictest0-21swapper/210:40:172
1669599118,2cyclictest4217-21taskset11:25:041
1669599118,2cyclictest31929-21ssh12:15:221
1669599118,2cyclictest27029-21taskset08:35:401
1669599118,2cyclictest27001-21ssh11:15:481
1669599118,2cyclictest24708-21ssh12:10:121
1669599118,2cyclictest24430-21ssh09:50:431
1669599118,2cyclictest2049-21ssh11:22:501
1669599118,2cyclictest2049-21ssh11:22:501
1669599118,2cyclictest19840-21head08:20:281
1669599118,2cyclictest19573-21tr07:15:151
1669599118,2cyclictest1666-21grep09:30:201
1669599118,2cyclictest1666-21grep09:30:201
1669599118,2cyclictest14710-21ssh10:39:421
1669599118,2cyclictest11364-21ssh09:40:021
1669599117,3cyclictest27832-21apache209:11:451
1669599116,3cyclictest8378-21wc09:05:201
1669599113,2cyclictest0-21swapper/108:15:011
1669199119,1cyclictest9536-21diskmemload11:31:000
1669199118,2cyclictest8985-21ssh10:35:070
1669199118,2cyclictest5096-21grep11:25:260
1669199118,2cyclictest4498-21ssh11:52:370
1669199118,2cyclictest32703-21ssh10:26:050
1669199118,2cyclictest2735-21tail10:55:270
1669199118,2cyclictest27158-21ssh10:50:060
1669199118,2cyclictest2706-21ssh12:18:240
1669199118,2cyclictest25029-21ssh09:51:260
1669199118,2cyclictest24891-21ssh12:35:060
1669199118,2cyclictest24891-21ssh12:35:060
1669199118,2cyclictest24843-21sed07:25:160
1669199118,2cyclictest21824-21ssh10:17:460
1669199118,2cyclictest21128-21taskset12:06:180
1669199118,2cyclictest20980-21sed07:15:310
1669199118,2cyclictest18818-21grep12:30:150
1669199118,2cyclictest17546-21sed11:35:280
1669199118,2cyclictest15934-21ssh09:15:110
1669199118,2cyclictest14840-21ssh12:26:250
1669199118,2cyclictest13444-21head11:05:150
1669199118,2cyclictest11976-21ssh09:10:390
1669199118,2cyclictest10012-21ssh11:56:470
1669199115,2cyclictest224-21systemd-journal09:05:000
1669199115,2cyclictest224-21systemd-journal08:50:000
1669199115,2cyclictest224-21systemd-journal08:00:000
1670199108,1cyclictest21567-21cat10:45:163
1670199108,1cyclictest18628-21df08:20:163
1670199108,1cyclictest15829-21aten2_r7power_p08:15:173
1670199108,1cyclictest13370-21cat12:00:183
1670199107,2cyclictest29235-21expr07:35:173
1670199107,2cyclictest29235-21expr07:35:173
1670199107,2cyclictest26498-21cut07:30:133
1670199107,2cyclictest26162-21chrt07:29:443
1670199107,2cyclictest17273-21expr07:10:143
1670199107,2cyclictest11259-21ssh10:35:473
1670199102,4cyclictest0-21swapper/308:10:273
1669899108,1cyclictest28553-21taskset07:34:132
1669899108,1cyclictest26538-21ssh09:23:542
1669899108,1cyclictest26049-21taskset07:28:202
1669899107,2cyclictest32525-21sed07:40:352
1669899107,2cyclictest27866-21wget08:40:182
1669899107,2cyclictest23541-21taskset07:21:362
1669899107,2cyclictest21353-21ssh11:40:042
1669899106,3cyclictest32-21ksoftirqd/207:15:152
1669899105,4cyclictest32-21ksoftirqd/208:30:192
1669899101,2cyclictest224-21systemd-journal07:55:252
1669599108,1cyclictest28525-21smartctl08:40:211
1669599108,1cyclictest18055-21cat08:20:001
1669599108,1cyclictest16463-21expr09:15:171
1669599107,2cyclictest773-21grep08:50:171
1669599107,2cyclictest7156-21cut09:05:001
1669599107,2cyclictest22513-21taskset08:28:431
1669599107,2cyclictest18834-21taskset07:12:331
1669199108,1cyclictest6533-21basename10:05:130
1669199108,1cyclictest23242-21cat07:20:250
1669199108,1cyclictest21234-21grep08:25:200
1669199107,2cyclictest4970-21head10:30:460
1669199107,2cyclictest4544-21munin-plugin-st09:00:000
1669199107,2cyclictest26101-21cat08:35:180
1669199107,2cyclictest24516-21cat10:20:140
1669199107,2cyclictest17457-21head09:45:220
1669199107,2cyclictest1657-21grep07:45:150
1669199107,2cyclictest16017-21grep08:15:420
1669199107,2cyclictest12667-21ssh12:00:110
167019997,1cyclictest7447-21hostname09:05:163
166989997,1cyclictest3898-21wget07:50:142
166959996,2cyclictest10227-21grep08:00:281
166959992,6cyclictest31162-21missed_timers08:45:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional