You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 22:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Mar 10, 2026 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
413266799503496,4cyclictest38028-21turbostat11:54:542
413267199475456,16cyclictest4149052-21turbostat07:55:003
413267199472451,18cyclictest4158010-21turbostat08:24:543
413267199471443,24cyclictest55649-21turbostat12:39:593
413267199468451,14cyclictest4136536-21turbostat07:20:003
413267199465456,6cyclictest2744-21turbostat10:10:003
413267199462451,8cyclictest31105-21turbostat11:30:003
413267199461439,19cyclictest4164981-21turbostat08:40:003
413267199460439,18cyclictest39762-21turbostat11:55:013
413267199459434,22cyclictest6232-21turbostat10:20:013
413267199458436,19cyclictest48728-21turbostat12:20:013
413267199457450,4cyclictest4166710-21turbostat08:45:013
413267199456436,17cyclictest34563-21turbostat11:40:003
413267199453446,4cyclictest4187491-21turbostat09:49:553
413267199451444,4cyclictest44955-21turbostat12:14:543
413267199449427,19cyclictest20374-21turbostat11:04:543
413267199445438,4cyclictest4141721-21turbostat07:35:003
413267199445438,4cyclictest34563-21turbostat11:44:553
413266799443435,5cyclictest4192981-21turbostat10:04:562
413267199442435,4cyclictest2744-21turbostat10:14:553
413267199442423,16cyclictest7956-21turbostat10:25:003
413267199442423,16cyclictest7956-21turbostat10:25:003
413267199442421,18cyclictest20374-21turbostat10:59:593
413267199440431,6cyclictest4184015-21turbostat09:35:013
413266799439432,4cyclictest4161475-21turbostat08:34:542
413267199438431,4cyclictest9682-21turbostat10:30:003
413267199435426,6cyclictest4171892-21turbostat09:00:003
413267199435414,18cyclictest4171892-21turbostat09:04:553
413266199435427,5cyclictest43223-21turbostat12:09:550
413267199434426,5cyclictest4177086-21turbostat09:19:553
413267199432425,4cyclictest707-21turbostat10:04:593
413267199432425,4cyclictest11722-21turbostat10:34:593
413267199432408,21cyclictest4173622-21turbostat09:09:553
413266799431423,5cyclictest4184015-21turbostat09:39:562
413266799431423,5cyclictest4184015-21turbostat09:39:562
413267199429422,4cyclictest4134321-21turbostat07:15:013
413267199426419,4cyclictest16914-21turbostat10:54:553
413266799426419,4cyclictest4152813-21turbostat08:09:542
413267199425400,22cyclictest4159747-21turbostat08:29:563
413267199423416,4cyclictest16914-21turbostat10:50:003
413267199421414,4cyclictest4185749-21turbostat09:44:553
413267199421414,4cyclictest36295-21turbostat11:49:543
413267199417393,21cyclictest48728-21turbostat12:24:563
413267199416408,5cyclictest4154549-21turbostat08:10:003
413267199415408,4cyclictest4143556-21turbostat07:44:553
413267199414405,6cyclictest25557-21turbostat11:15:003
413267199411403,5cyclictest4156278-21turbostat08:19:553
413266499411404,4cyclictest4149052-21turbostat07:59:551
413267199405398,4cyclictest29367-21turbostat11:25:003
413267199404395,6cyclictest13448-21turbostat10:40:013
413267199401393,5cyclictest4182275-21turbostat09:30:003
413267199401392,6cyclictest4177086-21turbostat09:14:593
413267199401392,6cyclictest4177086-21turbostat09:14:593
413267199394370,21cyclictest4166710-21turbostat08:49:563
413267199390383,4cyclictest4168437-21turbostat08:54:553
413266499390383,4cyclictest13448-21turbostat10:44:561
413267199388381,4cyclictest4145592-21turbostat07:49:553
413266799388381,4cyclictest41492-21turbostat12:04:562
413267199386379,4cyclictest4190950-21turbostat09:59:563
413267199379371,5cyclictest4163209-21turbostat08:35:003
413267199375367,5cyclictest4152813-21turbostat08:05:003
413267199375367,5cyclictest4152813-21turbostat08:04:593
413267199370362,5cyclictest4150779-21turbostat08:00:013
413267199360353,4cyclictest4190950-21turbostat09:55:013
413266799359352,4cyclictest15180-21turbostat10:49:562
413267199356349,4cyclictest4178821-21turbostat09:24:553
413266799353345,5cyclictest4164981-21turbostat08:44:552
413266499353346,4cyclictest11722-21turbostat10:39:551
413266499350344,3cyclictest22098-21turbostat11:09:561
413267199348341,4cyclictest44955-21turbostat12:10:003
413266799333326,4cyclictest27339-21turbostat11:24:552
413267199332325,4cyclictest4143556-21turbostat07:39:593
413267199331324,4cyclictest52185-21turbostat12:30:003
413267199330324,3cyclictest31105-21turbostat11:34:563
413267199330324,3cyclictest31105-21turbostat11:34:563
413266799327320,4cyclictest4175350-21turbostat09:14:562
413266799327320,4cyclictest4175350-21turbostat09:14:562
413267199326319,4cyclictest53916-21turbostat12:35:003
413267199324317,4cyclictest4156278-21turbostat08:15:003
413266799324317,4cyclictest4138265-21turbostat07:29:552
413267199323316,4cyclictest4138265-21turbostat07:25:003
413266799320313,4cyclictest52185-21turbostat12:34:552
413267199319311,5cyclictest41492-21turbostat12:00:013
413267199317310,4cyclictest25557-21turbostat11:19:553
413266199317309,5cyclictest707-21turbostat10:09:550
413266799315308,4cyclictest39762-21turbostat11:59:552
413250722690,8sleep04497-21PK-Backend07:10:000
413267199262246,11cyclictest4185749-21turbostat09:40:003
413267199262246,11cyclictest4185749-21turbostat09:39:593
413267199246230,12cyclictest4139989-21turbostat07:30:003
413267199182170,8cyclictest23831-21turbostat11:10:013
413267199126113,9cyclictest15180-21turbostat10:45:013
4132444212470,44sleep20-21swapper/207:09:152
4132415210976,23sleep30-21swapper/307:08:523
4131023210572,23sleep10-21swapper/107:05:041
41326719910291,8cyclictest43223-21turbostat12:05:003
4132667999689,4cyclictest4154549-21turbostat08:14:552
55032840,6sleep10-21swapper/110:15:231
197399720,5rtkit-daemon0-21swapper/010:40:030
197399710,5rtkit-daemon0-21swapper/007:51:020
4132661995914,12cyclictest438-20systemd-journal10:04:590
197399580,9rtkit-daemon1972-21rtkit-daemon12:28:132
4132661995617,10cyclictest4177471-21tr09:15:120
4132661995616,12cyclictest24964-21python311:10:270
197399560,8rtkit-daemon1972-21rtkit-daemon11:30:162
197399560,8rtkit-daemon1972-21rtkit-daemon11:30:152
4132661995521,30cyclictest4158014-21latency_hist08:20:020
4132661995512,13cyclictest8864-21cat10:25:210
4132661995421,29cyclictest4138501-21/usr/sbin/munin07:25:300
4132661995419,7cyclictest1137-21runrttasks11:02:510
4132661995417,11cyclictest4161011-21sed08:25:300
4132661995412,12cyclictest16206-21awk10:45:280
4132661995318,7cyclictest4164970-21latency_hist08:40:020
4132661995318,10cyclictest4175430-21grep09:10:020
4132661995316,11cyclictest7107-21grep10:20:200
4132661995316,11cyclictest7107-21grep10:20:200
4132661995315,10cyclictest4182178-21sshd09:28:570
4132661995313,11cyclictest49660-21strings12:20:220
4132661995313,11cyclictest4184475-21expr09:35:140
4132661995313,11cyclictest4184475-21expr09:35:130
4132661995311,13cyclictest4191789-21expr09:55:130
413266199525,18cyclictest55676-21systemd12:40:010
4132661995214,13cyclictest28231-40apt-check11:20:220
4132661995211,11cyclictest4132953-21cat07:10:120
4132661995119,7cyclictest4159030-21sh08:20:250
4132661995113,10cyclictest4154265-21seq08:06:390
4132661995112,11cyclictest4167423-21if_enp2s008:45:200
4132661995112,10cyclictest43200-21seq12:04:570
4132661995111,12cyclictest52849-21hddtemp_smartct12:30:180
4132661995110,12cyclictest4166159-21wc08:40:280
4132661995019,8cyclictest33270-21tr11:35:140
4132661995018,7cyclictest1137-21runrttasks11:33:350
4132661995018,7cyclictest1137-21runrttasks11:33:350
4132661995016,7cyclictest1137-21runrttasks07:36:320
4132661995014,7cyclictest38007-21cron11:49:590
4132661995013,8cyclictest4169412-21ntpq08:50:230
41825512490,8sleep10-21swapper/109:30:111
413266199499,12cyclictest23239-21python311:05:280
413266199499,10cyclictest11113-21cat10:30:280
4132661994919,7cyclictest38943-21needreboot11:50:220
4132661994916,7cyclictest4188261-21latency09:45:180
4132661994915,7cyclictest4190061-21missed_timers09:50:210
4132661994915,6cyclictest4149525-21cstates07:55:140
4132661994914,11cyclictest4176097-21awk09:10:190
4132661994914,11cyclictest4176097-21awk09:10:180
4132661994913,11cyclictest4162464-21ntp_kernel_pll_08:30:230
4132661994816,7cyclictest4136663-21latency_hist07:20:010
4132661994814,7cyclictest4146670-21processes07:45:270
4132661994813,7cyclictest4183503-21timerandwakeup09:30:280
4132661994812,7cyclictest0-21swapper/011:25:220
413266199472,16cyclictest2069-21gnome-shell11:58:590
413266199472,16cyclictest2069-21gnome-shell10:38:590
413266199472,16cyclictest2069-21gnome-shell07:21:590
413266199462,21cyclictest4155250-21if_enp2s008:10:190
413266199462,16cyclictest4179957-21python309:20:280
197399460,33rtkit-daemon4497-21PK-Backend07:18:511
413266199452,17cyclictest4173077-21chrt09:00:280
413266199452,16cyclictest4152576-40lsb_release08:02:280
413266199452,16cyclictest4152576-40lsb_release08:02:270
41513802440,21sleep234-21ksoftirqd/208:00:132
41513802440,21sleep234-21ksoftirqd/208:00:132
413266199449,6cyclictest0-21swapper/012:25:260
4132667994219,18cyclictest4190148-21strings09:50:212
4132664994214,23cyclictest28522-40lsb_release11:20:251
4132664994118,19cyclictest4154558-21sleep08:10:001
4132664994114,6cyclictest41690872forks08:50:181
413266199416,8cyclictest0-21swapper/008:55:240
4132661994114,8cyclictest2069-21gnome-shell10:17:090
4132667994022,13cyclictest34-21ksoftirqd/208:45:132
4132667994022,13cyclictest34-21ksoftirqd/207:57:562
4132664994024,11cyclictest438-20systemd-journal11:15:001
413266499402,32cyclictest4162108-21df_inode08:30:151
4132661994013,7cyclictest461442grep12:10:210
41853562390,11chrt4184255-21/usr/sbin/munin09:35:301
41853562390,11chrt4184255-21/usr/sbin/munin09:35:301
4132667993929,6cyclictest3692992-21snapd10:08:342
4132667993921,13cyclictest34-21ksoftirqd/211:13:392
4132664993924,11cyclictest4161452-21cron08:30:001
413266499392,32cyclictest4493-21munin-run10:15:001
4132664993923,12cyclictest923-21dbus-daemon07:35:001
4132664993921,13cyclictest438-20systemd-journal08:17:181
4132667993823,11cyclictest923-21dbus-daemon07:20:002
4132667993821,5cyclictest34-21ksoftirqd/210:32:182
4132667993821,13cyclictest34-21ksoftirqd/210:26:552
4132667993820,6cyclictest34-21ksoftirqd/208:16:472
4132664993827,7cyclictest51468-21ntp_states12:25:241
4132664993820,13cyclictest40762-21ntpq11:55:221
197399380,25rtkit-daemon0-21swapper/108:35:201
197399380,24rtkit-daemon0-21swapper/109:02:261
413266799377,4cyclictest141rcu_preempt09:21:172
4132667993724,9cyclictest4186154-21tr09:40:132
4132667993719,6cyclictest34-21ksoftirqd/210:39:292
413266799371,13cyclictest31085-21cron11:30:012
4132664993720,13cyclictest4158838-21sed08:20:211
4132664993719,13cyclictest1-21systemd07:55:001
4132661993722,11cyclictest47986-21awk12:15:240
197399370,25rtkit-daemon0-21swapper/110:22:551
197399370,25rtkit-daemon0-21swapper/110:22:541
4132667993623,7cyclictest4175383-21systemd09:10:022
4132667993621,11cyclictest4178560-21cron09:17:002
4132667993621,11cyclictest26699-21python311:15:272
4132667993619,12cyclictest438-20systemd-journal11:50:002
4132664993626,7cyclictest1033-21gdbus07:36:001
4132664993624,8cyclictest2095-21JS10:49:101
4132664993624,8cyclictest2095-21JS10:49:101
4132664993621,11cyclictest52850-21sshd12:30:181
4132664993620,11cyclictest44725-21sshd12:07:181
4132664993619,12cyclictest438-20systemd-journal10:25:191
413266499361,30cyclictest4497-21PK-Backend07:45:131
413266499361,14cyclictest0-21swapper/109:47:211
4132661993619,12cyclictest3760-21ntpq10:10:240
197399360,24rtkit-daemon0-21swapper/109:22:291
4132667993520,11cyclictest6208-21cron10:20:002
4132667993520,11cyclictest33869-21awk11:35:272
413266799351,9cyclictest4497-21PK-Backend08:25:122
4132667993519,12cyclictest19685-21awk10:55:272
4132667993518,12cyclictest438-20systemd-journal10:53:192
413266799351,11cyclictest4497-21PK-Backend11:06:192
413266499357,9cyclictest26-21ksoftirqd/109:13:291
413266499357,9cyclictest26-21ksoftirqd/109:13:291
4132664993522,9cyclictest30309-21grep11:25:231
4132664993521,10cyclictest4133324-21awk07:10:201
4132664993519,12cyclictest4182126-21sshd09:28:171
4132664993519,11cyclictest21344-21awk11:00:231
4132664993519,11cyclictest16121-21strings10:45:231
4132664993518,13cyclictest4178055-21awk09:15:211
4132664993518,13cyclictest11502-21gdbus10:32:321
4132664993515,16cyclictest2080-21gdbus11:17:371
413266499351,18cyclictest923-21dbus-daemon07:41:321
413266799346,9cyclictest34-21ksoftirqd/209:25:332
4132667993418,12cyclictest923-21dbus-daemon08:38:122
4132667993418,11cyclictest438-20systemd-journal07:15:012
4132667993418,11cyclictest4141420-21anacron07:31:182
413266799341,29cyclictest0-21swapper/207:47:212
413266499349,7cyclictest141rcu_preempt11:39:591
413266499347,5cyclictest141rcu_preempt10:51:351
413266499346,9cyclictest26-21ksoftirqd/112:38:421
413266499342,9cyclictest44947-21apt-get12:10:081
4132664993426,4cyclictest4497-21PK-Backend10:55:021
4132664993422,8cyclictest4174480-21grep09:05:211
4132664993420,10cyclictest4187566-21sed09:45:011
4132664993419,11cyclictest4155259-21cat08:10:181
4132661993417,12cyclictest923-21dbus-daemon11:45:000
197399340,27rtkit-daemon0-21swapper/212:18:232
197399340,27rtkit-daemon0-21swapper/212:13:002
197399340,23rtkit-daemon0-21swapper/109:52:261
4132667993327,4cyclictest0-21swapper/210:13:232
4132667993326,4cyclictest4497-21PK-Backend08:22:422
4132667993325,5cyclictest4497-21PK-Backend09:02:052
4132667993323,6cyclictest1137-21runrttasks07:50:172
4132667993316,5cyclictest34-21ksoftirqd/209:49:482
4132667993316,13cyclictest54925-21ntp_states12:35:252
4132667993316,13cyclictest35500-21grep11:40:222
4132664993319,10cyclictest4137699-21sed07:20:271
4132664993318,10cyclictest4167690-21ntp_kernel_pll_08:45:251
4132664993317,12cyclictest4193974-21awk10:00:241
4132664993316,13cyclictest42125-21df_inode12:00:141
413266499331,29cyclictest0-21swapper/111:47:161
413266499331,29cyclictest0-21swapper/109:56:331
413266499331,28cyclictest4497-21PK-Backend07:26:461
413266799325,22cyclictest4171158-21grep08:55:242
4132667993220,8cyclictest4192510-21timerwakeupswit09:55:292
4132667993218,10cyclictest4170242-21ls08:55:022
4132667993217,11cyclictest21522-21awk11:00:292
413266799321,6cyclictest4497-21PK-Backend07:40:092
4132667993216,12cyclictest4137152-21expr07:20:122
413266799321,4cyclictest4497-21PK-Backend12:21:182
413266799321,28cyclictest4497-21PK-Backend07:37:222
413266799321,15cyclictest4497-21PK-Backend12:08:592
4132664993219,8cyclictest2069-21gnome-shell08:55:091
4132664993218,10cyclictest48214-21grep12:15:301
4132664993217,11cyclictest1996-21cat10:05:291
4132664993217,10cyclictest32289-21grep11:30:281
4132664993217,10cyclictest32289-21grep11:30:281
4132664993215,12cyclictest49405-21ls12:20:191
4132664993214,13cyclictest438-20systemd-journal08:44:541
4132661993220,8cyclictest4186134-21grep09:40:140
4132661993218,10cyclictest18623-21cron10:55:000
4132661993210,6cyclictest141rcu_preempt10:59:590
413266799311,12cyclictest4497-21PK-Backend10:22:422
413266799311,12cyclictest4497-21PK-Backend10:22:422
413266499311,8cyclictest4497-21PK-Backend11:54:061
4132664993116,11cyclictest35596-21awk11:40:251
4132661993122,5cyclictest4497-21PK-Backend11:15:470
4132661993121,7cyclictest13-21ksoftirqd/007:40:270
4132667993017,9cyclictest4183500-21xargs09:30:282
4132667992918,7cyclictest14301-21missed_timers10:40:202
4132661992915,10cyclictest4141713-21cron07:35:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional