You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 01:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Sep 13, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
638429743,50sleep10-21swapper/107:08:101
627229267,9sleep20-21swapper/207:06:442
620829178,9sleep00-21swapper/007:05:520
562528369,9sleep30-21swapper/307:05:243
661699310,30cyclictest0-21swapper/111:36:371
661999292,26cyclictest20799-21systemctl10:39:592
661999290,27cyclictest0-21swapper/208:05:152
661999280,26cyclictest29674-21ssh10:45:142
661999280,1cyclictest0-21swapper/212:21:062
661999271,24cyclictest32084-21diskmemload12:28:082
6616992720,6cyclictest22381-21irqstats10:40:211
661999261,24cyclictest0-21swapper/209:10:162
661999260,25cyclictest10749-21cut12:10:202
661999260,25cyclictest0-21swapper/211:33:162
6616992618,2cyclictest8525-21apache_processe11:50:141
661999251,23cyclictest27580-21aten2_r7power_v10:05:142
661999251,22cyclictest0-21swapper/210:00:122
661999250,25cyclictest0-21swapper/209:43:332
661999250,24cyclictest0-21swapper/211:07:482
6616992519,5cyclictest27290-21hddtemp_smartct12:20:171
6622992422,2cyclictest1745-21kworker/u8:007:55:213
661999241,21cyclictest0-21swapper/209:22:322
661999240,1cyclictest0-21swapper/212:36:422
661999240,1cyclictest0-21swapper/210:29:392
661999240,18cyclictest0-21swapper/211:00:112
6616992419,4cyclictest32502-21users11:05:261
6622992320,2cyclictest8718-21users11:10:303
6622992320,2cyclictest31609-21apache_processe11:25:123
6622992320,2cyclictest30264-21df_inode10:45:163
6622992320,2cyclictest2884-21diskstats09:50:203
6622992320,2cyclictest26405-21apache_processe12:20:123
6622992320,2cyclictest15817-21hddtemp_smartct11:15:183
6622992320,2cyclictest10954-21ntp_states09:35:223
6622992320,2cyclictest10954-21ntp_states09:35:223
6622992320,2cyclictest10731-21apache_accesses12:30:153
6622992320,2cyclictest10505-21apache_accesses09:55:133
6619992320,2cyclictest32265-21diskstats11:25:152
6619992320,2cyclictest25393-21apache_accesses09:25:122
6619992320,2cyclictest23290-21apache_accesses11:20:132
661999231,21cyclictest32084-21diskmemload12:15:372
661999230,22cyclictest0-21swapper/210:30:282
661999230,1cyclictest0-21swapper/211:45:272
661999230,19cyclictest32084-21diskmemload11:10:252
661999230,0cyclictest0-21swapper/211:52:522
6616992320,2cyclictest7343-21df_inode11:10:191
6616992320,2cyclictest3119-21hddtemp_smartct09:50:211
6616992320,2cyclictest1366-21apache_accesses09:30:161
6616992320,2cyclictest11518-21perl12:30:191
6614992320,2cyclictest9068-21df11:50:170
6614992320,2cyclictest30508-21apache_volume11:05:110
6614992320,2cyclictest27405-21apache_volume10:05:130
6614992320,2cyclictest26506-21ntp_states12:00:220
6614992320,2cyclictest2325-21apache_accesses12:25:110
6614992320,2cyclictest21743-21ntp_states10:20:220
6614992320,2cyclictest16458-21df_inode11:35:160
6614992320,2cyclictest1388-21apache_volume09:30:170
6614992320,2cyclictest10330-21diskstats12:10:170
6622992220,1cyclictest5570-21apache_accesses10:50:163
6622992220,1cyclictest2228-21hddtemp_smartct12:05:183
6622992219,2cyclictest757-21diskstats11:45:143
6622992219,2cyclictest26314-21apache_accesses09:45:143
6622992219,2cyclictest23314-21apache_volume11:20:123
6622992219,2cyclictest20406-21apache_volume10:20:123
6622992219,2cyclictest18866-21apache_volume10:00:113
6622992219,2cyclictest18594-21diskstats09:40:163
6622992219,2cyclictest18105-21apache_volume12:15:143
6622992219,2cyclictest17087-21apache_accesses09:20:113
661999228,12cyclictest31521-21sort09:05:272
661999222,19cyclictest25051-21grep11:40:172
6619992219,2cyclictest3381-21users09:30:312
6619992210,11cyclictest32084-21diskmemload11:16:512
661999220,9cyclictest0-21swapper/210:23:152
661999220,22cyclictest0-21swapper/211:39:212
661999220,21cyclictest0-21swapper/210:41:342
661999220,1cyclictest0-21swapper/210:57:212
661999220,1cyclictest0-21swapper/209:55:142
661999220,17cyclictest0-21swapper/209:19:322
661999220,0cyclictest0-21swapper/209:47:022
6616992220,1cyclictest2655-21ntp_states12:05:211
6616992219,2cyclictest26443-21irqstats09:25:201
6616992219,2cyclictest10944-21ntp_states09:35:221
6616992219,2cyclictest10944-21ntp_states09:35:211
6614992220,1cyclictest6918-21ntp_states10:50:240
6614992220,1cyclictest6706-21apache_processe11:10:170
6614992220,1cyclictest4574-21apache_processe10:30:120
6614992220,1cyclictest21320-21apache_accesses10:40:140
6614992220,1cyclictest18840-21apache_accesses10:00:100
6614992220,1cyclictest13801-21apache_volume10:55:120
6614992219,2cyclictest29647-21apache_volume10:45:130
6614992219,2cyclictest27028-21df_inode12:20:150
6614992219,2cyclictest24259-21apache_volume11:40:120
6614992219,2cyclictest17734-21/usr/sbin/munin11:55:170
6614992219,2cyclictest13564-21diskstats10:35:160
661499220,19cyclictest0-21swapper/008:05:140
661499220,0cyclictest0-21swapper/009:45:320
6622992118,2cyclictest9388-21hddtemp_smartct09:15:173
6622992118,2cyclictest24887-21df_inode11:40:163
6622992118,1cyclictest0-21swapper/310:31:493
661999211,19cyclictest32084-21diskmemload11:59:102
661999210,1cyclictest0-21swapper/209:53:312
6616992118,2cyclictest27940-21df10:05:151
6616992118,2cyclictest23302-21apache_processe11:20:131
6614992119,1cyclictest5625-21users10:10:310
6614992119,1cyclictest31621-21apache_volume11:25:120
6614992118,2cyclictest9154-21diskstats09:15:150
6614992118,2cyclictest25413-21apache_volume09:25:140
6614992118,2cyclictest2217-21apache_accesses09:50:150
6614992118,2cyclictest17112-21apache_volume09:20:110
6622992018,1cyclictest32573-21apache_processe09:10:123
6622992017,2cyclictest13781-21apache_accesses10:55:103
661999202,17cyclictest32084-21diskmemload10:14:432
6619992017,2cyclictest10259-21diskstats09:35:172
6619992017,2cyclictest10259-21diskstats09:35:162
661999201,16cyclictest9290-21grep08:20:222
661999200,3cyclictest0-21swapper/210:19:372
661999200,1cyclictest0-21swapper/212:31:162
661999200,19cyclictest0-21swapper/212:06:312
661999200,18cyclictest0-21swapper/212:04:112
661999200,17cyclictest0-21swapper/207:35:152
6616992019,1cyclictest38150irq/131-snd_hda12:38:141
6616992017,2cyclictest13547-21df_inode10:35:151
661699200,19cyclictest0-21swapper/111:58:511
6622991917,1cyclictest7470-21apache_accesses11:30:113
6622991916,2cyclictest23245-21irqstats11:00:183
6622991916,1cyclictest0-21swapper/310:35:233
661999190,17cyclictest0-21swapper/208:50:202
661499197,6cyclictest20205-21perf08:44:590
661999180,15cyclictest0-21swapper/207:10:172
6616991817,1cyclictest0-21swapper/108:50:131
6616991817,0cyclictest0-21swapper/107:45:151
6616991817,0cyclictest0-21swapper/107:45:141
661699180,3cyclictest0-21swapper/110:48:461
6614991816,1cyclictest19957-21hddtemp_smartct12:35:180
87952170,0sleep00-21swapper/012:09:420
662299177,6cyclictest26696-21ntpq09:25:213
6622991714,2cyclictest39-21ksoftirqd/310:20:003
6622991714,2cyclictest17213-21ntp_states11:35:203
661999171,11cyclictest0-21swapper/208:15:002
661999170,16cyclictest17353-21grep07:30:162
661999170,16cyclictest0-21swapper/207:15:152
661699172,6cyclictest0-21swapper/108:26:351
6616991716,1cyclictest0-21swapper/109:44:471
6616991714,2cyclictest3390-21irqstats12:25:191
6614991714,2cyclictest18845-21hddtemp_smartct09:40:170
6622991615,1cyclictest0-21swapper/308:00:523
662299161,3cyclictest141rcu_preempt08:50:253
6622991613,2cyclictest4365-21users12:25:273
661999161,14cyclictest0-21swapper/208:43:282
661999160,1cyclictest0-21swapper/210:52:072
661699167,8cyclictest17733-21sort12:15:011
6616991616,0cyclictest0-21swapper/111:48:281
6616991613,2cyclictest17874-21perl09:20:161
661699161,1cyclictest32084-21diskmemload10:59:411
6622991515,0cyclictest0-21swapper/311:57:343
661999159,5cyclictest14970-21sed07:25:172
661999158,6cyclictest11751-21basename07:20:172
6619991512,2cyclictest0-21swapper/208:55:132
661999151,13cyclictest0-21swapper/207:50:242
661999150,14cyclictest0-21swapper/207:40:202
661699151,1cyclictest32084-21diskmemload11:34:371
661699150,14cyclictest0-21swapper/109:49:521
661499153,7cyclictest997-21ssh11:45:170
662299144,5cyclictest29690-21kworker/3:110:08:223
662299144,2cyclictest141rcu_preempt11:50:053
6622991412,2cyclictest39-21ksoftirqd/309:34:203
6622991410,3cyclictest5449-21cat10:10:303
661999142,7cyclictest0-21swapper/208:25:142
661999140,13cyclictest0-21swapper/208:00:292
661699149,3cyclictest0-21swapper/110:10:261
6616991411,2cyclictest26077-21which12:00:171
6616991410,3cyclictest29923-21wc10:25:201
6616991410,3cyclictest27703-21wc07:50:261
6616991410,2cyclictest28567-21cut07:55:121
6616991410,2cyclictest15856-21tr08:35:161
661499149,3cyclictest18941-21head12:15:190
661499147,6cyclictest22824-21sendmail-msp11:20:000
661499142,6cyclictest32101-21awk09:09:590
6614991410,2cyclictest11190-21uniq09:55:190
661499140,1cyclictest0-21swapper/010:26:080
662299138,2cyclictest141rcu_preempt12:02:053
662299132,1cyclictest141rcu_preempt10:27:033
6622991311,1cyclictest1142-21runrttasks12:35:303
6622991310,2cyclictest2398-21cat08:05:283
661999137,5cyclictest24867-21sed07:45:212
661999137,5cyclictest24867-21sed07:45:202
661999136,6cyclictest30811-21wc08:00:012
6619991310,2cyclictest17368-21grep08:35:272
661699139,2cyclictest13238-21tr08:30:151
661699138,4cyclictest11373-21sh10:53:281
6616991310,2cyclictest20683-21grep08:45:141
6616991310,2cyclictest19761-21grep07:35:191
661699130,1cyclictest32084-21diskmemload11:19:081
661499139,3cyclictest11530-21grep09:35:280
661499139,3cyclictest11530-21grep09:35:270
661499139,2cyclictest11888-21tr07:20:170
661499135,7cyclictest15153-21munin-run08:34:590
661499131,4cyclictest0-21swapper/008:35:150
6614991310,2cyclictest4886-21grep08:10:280
6614991310,2cyclictest12704-21grep10:15:160
662299129,2cyclictest6387-21grep08:15:183
662299129,2cyclictest4933-21tail08:10:273
662299129,2cyclictest28623-21cat09:00:193
662299129,2cyclictest25006-21grep07:45:233
662299129,2cyclictest25006-21grep07:45:223
662299129,2cyclictest15302-21cat07:25:193
662299129,2cyclictest10461-21cut08:25:143
662299121,2cyclictest141rcu_preempt08:40:193
6622991210,2cyclictest39-21ksoftirqd/307:20:163
662299120,2cyclictest0-21swapper/310:44:193
662299120,10cyclictest0-21swapper/308:55:153
661999129,2cyclictest5903-21grep08:15:152
661699129,2cyclictest9928-21tr09:15:211
661699129,2cyclictest25899-21ssh11:40:251
661699129,2cyclictest22465-21wc07:40:191
661699129,2cyclictest18914-21grep12:15:191
661699129,2cyclictest15312-21head07:25:191
661699128,3cyclictest9485-21grep07:15:131
661699128,3cyclictest21931-21wc10:20:221
661699128,3cyclictest10137-21cron08:25:001
661699120,11cyclictest30009-21cut09:05:151
661699120,11cyclictest23006-21head11:00:161
661499129,2cyclictest8015-21awk09:15:010
661499129,2cyclictest26914-21sed07:50:170
661499129,2cyclictest16426-21wc07:30:000
662299119,1cyclictest18767-21awk07:35:013
662299119,1cyclictest10612-21sensors07:15:223
662299118,2cyclictest39-21ksoftirqd/308:20:133
662299118,2cyclictest30907-21uname09:05:213
662299118,2cyclictest3068-21rm11:07:473
662299118,2cyclictest19784-21grep07:35:183
662299111,2cyclictest141rcu_preempt12:11:513
661999111,6cyclictest225-21systemd-journal08:49:572
661999110,5cyclictest0-21swapper/209:00:152
661999110,1cyclictest0-21swapper/208:30:132
661699118,2cyclictest947-21cat11:25:251
661699118,2cyclictest7575-21chrt08:17:541
661699118,2cyclictest5442-21cut10:30:171
661699118,2cyclictest5347-21chrt08:14:541
661699118,2cyclictest425-21tr09:10:141
661699118,2cyclictest20887-21vmstat10:00:251
661699118,2cyclictest12975-21chrt07:20:251
661699118,2cyclictest11920-21ssh10:15:101
661699118,2cyclictest1140-21tr08:05:161
661699117,3cyclictest28698-21strings09:00:201
661699117,3cyclictest25236-21grep08:55:141
661699111,9cyclictest32632-21head08:00:301
661499118,2cyclictest7095-21expr07:10:160
661499118,2cyclictest5918-21cut08:15:150
661499118,2cyclictest31071-21tr08:00:180
661499118,2cyclictest26261-21ls08:55:200
661499118,2cyclictest25288-21grep11:20:280
661499118,2cyclictest24198-21grep08:50:220
661499118,2cyclictest22355-21chrt11:00:130
661499118,2cyclictest21206-21tail08:45:160
661499118,2cyclictest13578-21ssh12:31:080
661499118,2cyclictest10514-21ssh11:31:130
662299108,1cyclictest8052-21cut07:10:223
662299108,1cyclictest281002sleep307:53:003
662299108,1cyclictest23185-21seq07:40:543
662299108,1cyclictest16821-21grep08:35:223
662299107,2cyclictest13314-21cat08:30:153
662299100,1cyclictest141rcu_preempt08:45:253
661699108,1cyclictest7991-21ntp_offset07:10:221
661699108,1cyclictest12569-21rm09:55:311
661699107,2cyclictest20016-21chrt08:42:581
661699107,2cyclictest18641-21chrt07:33:461
661499108,1cyclictest9828-21chrt08:21:060
661499108,1cyclictest28895-21uname09:00:230
661499108,1cyclictest28512-21basename07:55:100
661499108,1cyclictest11567-21ntpq08:25:210
661499108,1cyclictest10647-21cat07:15:230
661499107,2cyclictest24029-21grep07:45:160
661499107,2cyclictest24029-21grep07:45:160
661499107,2cyclictest17632-21cat07:30:170
66149997,1cyclictest19391-21cat07:35:170
66149990,5cyclictest0-21swapper/007:40:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional