You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 15:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue May 13, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16620210678,10sleep10-21swapper/107:09:571
16315210592,9sleep30-21swapper/307:06:013
1656528165,9sleep00-21swapper/007:09:140
1650628065,11sleep20-21swapper/207:08:292
137752500,0sleep20-21swapper/211:25:092
1671499250,5cyclictest0-21swapper/211:30:162
1671499250,3cyclictest0-21swapper/212:23:552
1670899251,13cyclictest0-21swapper/012:20:160
1671799241,22cyclictest0-21swapper/309:10:133
16714992422,1cyclictest32-21ksoftirqd/210:04:332
16714992321,2cyclictest32-21ksoftirqd/212:30:162
44382220,1sleep210400-21kworker/2:109:58:062
1671799220,3cyclictest0-21swapper/309:47:033
16714992217,5cyclictest11414-21kworker/u8:209:00:182
1671499220,22cyclictest0-21swapper/212:05:492
1670899220,2cyclictest0-21swapper/012:38:190
1671799218,1cyclictest141rcu_preempt11:46:473
16712992119,2cyclictest25-21ksoftirqd/110:52:351
1670899217,7cyclictest4282-21awk09:00:000
1670899210,1cyclictest0-21swapper/011:56:220
1671799204,1cyclictest141rcu_preempt11:35:403
16717992017,2cyclictest39-21ksoftirqd/312:40:003
1670899200,2cyclictest0-21swapper/011:51:280
1671499190,2cyclictest141rcu_preempt09:33:252
1670899190,1cyclictest0-21swapper/010:31:570
1670899182,12cyclictest0-21swapper/009:50:030
1670899180,17cyclictest0-21swapper/009:22:290
1671499179,2cyclictest141rcu_preempt10:53:442
1671499174,3cyclictest32-21ksoftirqd/212:15:392
1671499172,15cyclictest0-21swapper/209:14:202
1671499170,16cyclictest0-21swapper/212:26:392
16712991716,1cyclictest0-21swapper/107:20:151
1671299170,13cyclictest6594-21apt-get10:00:161
1670899170,2cyclictest0-21swapper/011:41:140
283442160,0sleep00-21swapper/010:16:150
1671499166,2cyclictest141rcu_preempt09:16:132
1671499166,2cyclictest141rcu_preempt09:16:132
1671499163,7cyclictest32-21ksoftirqd/210:46:322
1671499161,4cyclictest9247-21diskmemload10:11:192
16714991613,2cyclictest1093-21perf11:15:002
1671499160,15cyclictest0-21swapper/208:30:212
1670899164,6cyclictest19116-21munin-run07:15:000
1671799151,9cyclictest0-21swapper/311:40:153
1671799150,1cyclictest141rcu_preempt10:44:523
1671499158,7cyclictest32-21ksoftirqd/210:20:562
1671499158,2cyclictest141rcu_preempt10:32:472
1671499157,2cyclictest32-21ksoftirqd/211:40:202
16714991514,1cyclictest311rcuc/210:57:292
16714991512,2cyclictest32-21ksoftirqd/210:25:122
1671499150,15cyclictest0-21swapper/211:21:452
1671299154,11cyclictest0-21swapper/111:40:291
1671299150,14cyclictest5852-21seq09:59:441
1671799143,10cyclictest361irq_work/311:18:523
16717991412,1cyclictest39-21ksoftirqd/310:57:503
16717991411,2cyclictest29007-21wc08:40:293
16717991410,3cyclictest15636-21grep09:15:203
16717991410,3cyclictest15636-21grep09:15:203
1671799140,13cyclictest0-21swapper/312:25:313
1671499149,4cyclictest32-21ksoftirqd/208:35:222
1671499147,6cyclictest32-21ksoftirqd/207:10:152
16714991412,1cyclictest37850irq/131-snd_hda09:48:362
16714991410,3cyclictest7423-21tr09:05:182
1671499140,3cyclictest0-21swapper/211:16:042
1671499140,2cyclictest0-21swapper/209:26:552
1671499140,13cyclictest0-21swapper/210:05:292
1671299142,8cyclictest224-21systemd-journal07:50:211
16712991410,2cyclictest7393-21tr09:05:181
1670899147,5cyclictest0-21swapper/012:05:000
1670899144,8cyclictest27358-21munin-node09:16:150
1670899144,8cyclictest27358-21munin-node09:16:140
16708991410,3cyclictest19501-21cat09:45:010
1671799139,3cyclictest23868-21cron07:24:593
1671799139,3cyclictest23342-21sort08:30:183
1671799139,3cyclictest18008-21grep08:20:233
1671799139,3cyclictest1364-21wc09:55:213
1671799138,4cyclictest39-21ksoftirqd/310:27:563
1671799131,0cyclictest141rcu_preempt10:16:103
16717991310,2cyclictest5264-21grep09:00:193
16717991310,2cyclictest4246-21cron09:00:003
16717991310,2cyclictest31532-21tr07:40:153
16717991310,2cyclictest28617-21cut11:10:383
1671799130,1cyclictest141rcu_preempt09:50:163
1671799130,12cyclictest2920-21tr12:05:163
1671799130,11cyclictest0-21swapper/312:20:163
1671499139,3cyclictest4697-21tail07:50:222
1671499139,3cyclictest26044-21tr10:15:162
1671499139,3cyclictest15133-21grep11:50:182
1671499138,4cyclictest24359-21ssh12:39:592
1671499136,4cyclictest32-21ksoftirqd/211:56:412
16714991310,2cyclictest29990-21sed12:00:282
16714991310,2cyclictest1889-21ls08:55:002
16714991310,2cyclictest17630-21grep11:00:272
1671299139,3cyclictest23990-21ssh10:13:331
1671299139,3cyclictest14730-21grep11:25:181
1671299139,2cyclictest27281-21ssh12:20:241
1671299136,6cyclictest18980-21apache210:30:451
1671299136,3cyclictest224-21systemd-journal10:44:591
16712991310,2cyclictest31091-21ssh10:45:211
16712991310,2cyclictest2947-21cat09:30:271
16712991310,2cyclictest27581-21ls07:30:201
16712991310,2cyclictest13901-21grep08:10:191
16712991310,2cyclictest1046-21ls11:40:011
1671299130,12cyclictest31928-21cron08:50:001
1670899133,4cyclictest0-21swapper/011:10:140
1670899131,7cyclictest16560-21munin-run12:34:590
16708991313,0cyclictest0-21swapper/010:10:580
16708991311,2cyclictest13-21ksoftirqd/010:48:590
16708991311,1cyclictest18980-21apache212:10:300
16708991310,2cyclictest2646-21ssh11:15:240
1670899130,11cyclictest0-21swapper/007:25:150
138382130,0chrt0-21swapper/110:58:231
1671799129,2cyclictest9637-21expr12:30:163
1671799129,2cyclictest22518-21tr09:20:243
1671799129,2cyclictest20003-21grep12:15:283
1671799128,3cyclictest20094-21grep07:15:203
1671799128,2cyclictest27380-21cat08:40:003
1671799128,2cyclictest20139-21wc11:30:013
1671799126,4cyclictest39-21ksoftirqd/309:28:033
1671799124,4cyclictest224-21systemd-journal08:55:003
1671799123,8cyclictest2147-21grep09:30:183
1671799122,7cyclictest13424-21sh11:25:003
1671799121,6cyclictest22264-21grep11:55:183
16717991210,1cyclictest39-21ksoftirqd/310:53:043
16717991210,1cyclictest13014-21sort10:30:223
1671799120,9cyclictest0-21swapper/312:12:183
1671799120,2cyclictest0-21swapper/307:50:143
1671499129,2cyclictest32-21ksoftirqd/209:20:202
1671499129,2cyclictest2881-21grep07:45:282
1671499129,2cyclictest17210-21grep08:15:292
1671499129,2cyclictest13849-21grep08:10:192
1671499128,3cyclictest32-21ksoftirqd/211:45:232
1671499127,2cyclictest31929-21cron08:50:002
1671499125,4cyclictest311rcuc/207:15:012
1671499121,2cyclictest141rcu_preempt07:36:352
1671299129,2cyclictest9058-21wc09:35:291
1671299129,2cyclictest9013-21if_err_enp0s31f11:45:221
1671299129,2cyclictest483-21grep08:50:181
1671299129,2cyclictest3838-21sed08:55:271
1671299129,2cyclictest3480-21sed12:05:181
1671299129,2cyclictest29981-21grep07:35:211
1671299129,2cyclictest27105-21head09:50:181
1671299129,2cyclictest23461-21grep08:30:201
1671299129,2cyclictest21560-21grep11:30:231
1671299129,2cyclictest21056-21head08:25:191
1671299129,2cyclictest20143-21ssh09:45:161
1671299129,2cyclictest19542-21cut12:15:221
1671299129,2cyclictest17557-21grep07:10:291
1671299129,2cyclictest17274-21cut12:35:191
1671299129,2cyclictest11420-21grep09:10:261
1671299129,2cyclictest10079-21grep08:00:281
1671299128,4cyclictest18980-21apache209:44:561
1671299128,2cyclictest18579-21grep08:20:261
1671299127,4cyclictest9247-21diskmemload12:04:001
1671299127,4cyclictest19007-21apache211:13:221
1671299120,11cyclictest1719-21ssh11:15:121
1671299120,0cyclictest0-21swapper/109:28:571
1670899129,2cyclictest8715-21cat09:05:370
1670899129,2cyclictest8658-21grep08:00:160
1670899129,2cyclictest6196-21grep09:00:280
1670899129,2cyclictest6091-21cron10:00:000
1670899129,2cyclictest6084-21tr07:55:160
1670899129,2cyclictest4558-21grep10:50:200
1670899129,2cyclictest3358-21grep12:25:280
1670899129,2cyclictest21486-21apt07:20:000
1670899129,2cyclictest18811-21basename10:10:000
1670899129,2cyclictest18673-21sed08:20:270
1670899129,2cyclictest15307-21basename11:00:000
1670899129,2cyclictest13419-21cron11:25:000
16708991211,1cyclictest0-21swapper/012:18:090
16708991210,1cyclictest3375-21sort12:05:180
1671799119,1cyclictest22018-21sed08:25:293
1671799119,1cyclictest20565-21chrt10:36:013
1671799118,2cyclictest8818-21tail09:05:383
1671799118,2cyclictest6904-21tr10:00:163
1671799118,2cyclictest3691-21ssh10:23:173
1671799118,2cyclictest30613-21head07:35:273
1671799118,2cyclictest21895-21taskset11:30:263
1671799118,2cyclictest21088-21sed10:10:283
1671799118,2cyclictest14993-21cut11:50:173
1671799118,2cyclictest1413-21ssh10:47:333
1671799118,2cyclictest13571-21tail10:05:203
1671799118,2cyclictest11233-21grep08:05:163
1671799118,2cyclictest10260-21ssh09:36:383
1671799115,5cyclictest28404-21cpuspeed_turbos12:00:163
1671799115,2cyclictest224-21systemd-journal07:59:563
1671799114,6cyclictest381rcuc/309:40:193
1671799113,4cyclictest224-21systemd-journal08:20:003
1671499119,1cyclictest218772sleep211:05:142
1671499118,2cyclictest9516-21grep08:00:222
1671499118,2cyclictest761-21taskset07:42:352
1671499118,2cyclictest29809-21taskset09:52:182
1671499118,2cyclictest27470-21expr11:35:162
1671499118,2cyclictest22478-21grep07:20:202
1671499118,2cyclictest19362-21grep10:35:232
1671499118,1cyclictest311rcuc/210:42:132
1671499117,3cyclictest32-21ksoftirqd/209:36:092
1671499117,3cyclictest311rcuc/212:11:012
1671499117,3cyclictest21044-21head08:25:192
1671499117,3cyclictest12006-21wc08:05:212
1671499113,5cyclictest29579-21idleruntime-cro08:44:592
1671499111,9cyclictest0-21swapper/207:25:192
1671499110,1cyclictest141rcu_preempt08:55:152
1671499110,10cyclictest15380-21sed09:40:292
1671299119,1cyclictest10056-21head12:30:191
1671299118,2cyclictest6909-21open_files10:25:311
1671299118,2cyclictest6490-21taskset09:03:061
1671299118,2cyclictest5246-21sh10:24:591
1671299118,2cyclictest31594-21cat07:40:161
1671299118,2cyclictest29171-21ssh10:17:321
1671299118,2cyclictest25844-21sed07:25:291
1671299118,2cyclictest25304-21taskset11:57:331
1671299118,2cyclictest21484-21tr09:20:171
1671299118,2cyclictest20713-21cron09:20:001
1671299118,2cyclictest20713-21cron09:20:001
1671299118,2cyclictest20108-21head07:15:201
1671299118,2cyclictest18581-21ssh10:09:471
1671299118,2cyclictest1846-21expr12:25:161
1671299118,2cyclictest18043-21ssh10:35:111
1671299118,2cyclictest16111-21grep11:50:231
1671299118,2cyclictest15738-21tr08:15:151
1671299118,2cyclictest11238-21sh12:10:191
1671299114,4cyclictest0-21swapper/111:00:161
1670899118,2cyclictest9944-21chrt10:02:120
1670899118,2cyclictest8785-21wc09:35:240
1670899118,2cyclictest4443-21ssh10:24:060
1670899118,2cyclictest3504-21rm09:30:430
1670899118,2cyclictest31321-21tr07:40:150
1670899118,2cyclictest30110-21ls07:35:230
1670899118,2cyclictest28056-21cat07:30:260
1670899118,2cyclictest27736-21tr08:40:180
1670899118,2cyclictest27344-21cat11:35:160
1670899118,2cyclictest24387-21sh10:40:150
1670899118,2cyclictest23481-21ssh09:47:300
1670899118,2cyclictest23201-21sh11:31:380
1670899118,2cyclictest23114-21expr08:30:170
1670899118,2cyclictest22044-21sed08:25:280
1670899118,2cyclictest21708-21rm11:05:040
1670899118,2cyclictest17966-21ssh10:35:050
1670899118,2cyclictest15766-21ssh11:00:120
1670899118,2cyclictest14706-21head11:25:190
1670899118,2cyclictest10142-21expr09:10:140
1670899118,2cyclictest10028-21ssh10:28:250
1670899117,3cyclictest22831-21cat07:20:220
1670899110,1cyclictest0-21swapper/011:46:060
1671799108,1cyclictest30498-21grep08:45:173
1671799108,1cyclictest22995-21sh11:05:193
1671799108,1cyclictest18435-21wc07:10:373
1671799108,1cyclictest15029-21taskset08:12:373
1671799107,2cyclictest8838-21expr08:00:163
1671799107,2cyclictest25853-21head07:25:293
1671799107,2cyclictest17084-21ssh11:00:223
1671799107,2cyclictest1481-21tr07:45:163
1671799102,7cyclictest26522-21apt-get07:30:133
1671499108,1cyclictest19828-21chrt08:22:132
1671499107,2cyclictest27754-21tr07:30:212
1671299108,1cyclictest25787-21head08:35:181
1671299108,1cyclictest2303-21cat07:45:211
1671299108,1cyclictest21722-21ssh11:05:051
1671299107,2cyclictest8960-21ssh11:20:231
1671299107,2cyclictest8104-21perf08:00:011
1671299104,2cyclictest224-21systemd-journal08:45:001
1670899108,1cyclictest3009-21chrt07:45:530
1670899108,1cyclictest15640-21wc08:15:160
1670899108,1cyclictest11223-21df08:05:160
1670899107,2cyclictest29894-21hostname08:45:140
1670899107,2cyclictest27296-21ssh09:25:030
1670899107,2cyclictest1993-21cat08:55:000
1670899100,9cyclictest5357-21taskset07:50:340
167149997,1cyclictest7104-21awk07:55:222
167129990,4cyclictest0-21swapper/108:05:141
167089997,1cyclictest14192-21cat08:10:210
167089986,1cyclictest25387-21expr08:35:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional