You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 21:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Jun 04, 2026 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
324922221196,17sleep10-21swapper/107:05:411
324982180108,28sleep20-21swapper/207:05:472
324862177114,49sleep00-21swapper/007:05:370
3142177105,26sleep30-21swapper/307:09:293
169521620,1sleep20-21swapper/210:55:232
2632721110,1sleep00-21swapper/009:38:070
111472570,3sleep334-21ksoftirqd/307:34:393
80252560,2sleep10-21swapper/111:00:161
286952550,2sleep30-21swapper/310:51:343
148652510,2sleep00-21swapper/010:17:240
207512500,2sleep30-21swapper/309:10:143
28482490,1sleep20-21swapper/208:30:122
189412490,1sleep20-21swapper/210:20:202
128602460,2sleep10-21swapper/109:28:101
299772440,1sleep20-21swapper/210:05:022
101502440,2sleep00-21swapper/009:25:270
67599435,21cyclictest17176-21sh10:43:322
675994310,23cyclictest11430-21sh11:02:492
67599429,23cyclictest30445-21sh10:53:222
675994212,17cyclictest23941-21sh11:35:382
285752420,2sleep10-21swapper/112:28:041
67599414,27cyclictest20044-21diskmemload12:20:012
675994010,16cyclictest24286-21sh10:00:182
319992400,2sleep00-21swapper/008:20:230
67599395,18cyclictest22697-21sh11:10:392
282432390,2sleep10-21swapper/111:15:171
159202390,1sleep20-21swapper/211:05:332
67599388,17cyclictest11519-21kworker/u8:212:06:142
67599387,18cyclictest31327-21sh09:41:212
67599381,20cyclictest13060-21ssh11:52:012
67599377,17cyclictest20044-21diskmemload09:16:482
67599371,6cyclictest10450-21sh10:14:442
67699361,15cyclictest8877-21grep11:00:223
67599364,17cyclictest0-21swapper/210:16:322
67599359,13cyclictest20044-21diskmemload11:40:432
673993511,6cyclictest16338-21ntp_states11:30:210
675993413,19cyclictest468-21ls08:25:012
67599341,18cyclictest0-21swapper/209:50:112
67599340,12cyclictest195292sh11:32:562
321742340,2sleep00-21swapper/010:55:000
67699338,8cyclictest0-21swapper/309:20:123
67699332,11cyclictest4437-21ssh10:10:123
67599332,26cyclictest24202-21apt-get12:25:112
675993312,18cyclictest7333-21seq12:11:532
675993312,18cyclictest2852-21find07:15:112
67599331,18cyclictest14269-21sh09:29:342
67699322,9cyclictest24230-21ssh12:25:103
67699315,8cyclictest52750irq/139-enp3s0-10:55:123
67699312,13cyclictest7022-21apt-get08:40:103
676993112,6cyclictest34-21ksoftirqd/307:45:123
67599316,22cyclictest18289-21expr07:50:132
675993114,14cyclictest6117-21sh09:23:112
675993113,15cyclictest19288-21sh11:56:342
675993113,15cyclictest12762-21sh12:15:372
67599310,6cyclictest3908-21rm10:33:422
67599310,6cyclictest3908-21rm10:33:422
67599310,5cyclictest12069-21rm11:27:142
67599310,4cyclictest27868-21/usr/sbin/munin11:15:132
67599307,20cyclictest18363-21expr09:05:122
675993013,14cyclictest23503-21sh12:00:132
675993011,17cyclictest7390-21seq11:24:182
675993011,17cyclictest7390-21seq11:24:172
67599300,8cyclictest3356-21ssh12:33:372
67599300,8cyclictest26108-21ssh09:14:182
67599300,6cyclictest5369-21entropy07:20:142
67599300,6cyclictest3963-21/usr/sbin/munin09:45:172
67599300,6cyclictest29014-21ssh10:28:062
67599300,4cyclictest19667-21taskset09:56:412
67599300,3cyclictest7458-21expr07:25:132
67599300,3cyclictest7424-21ssh11:48:062
67599300,3cyclictest24769-21sh09:36:332
67599300,3cyclictest10410-21ssh10:38:302
67599291,7cyclictest22235-21ssh10:46:502
675992912,14cyclictest17862-21sh09:31:252
67599290,4cyclictest908-21df08:25:132
67599290,4cyclictest1738-21ps07:10:222
67599290,3cyclictest25398-21pmu-power08:05:212
67599290,3cyclictest20044-21diskmemload12:38:222
287612290,2sleep30-21swapper/310:27:503
676992812,3cyclictest20044-21diskmemload10:18:093
67599280,6cyclictest11963-21gltestperf07:35:132
67599280,3cyclictest45-21kcompactd007:45:132
67499281,25cyclictest0-21swapper/108:38:331
67599279,16cyclictest6335-21grep08:35:252
67599275,19cyclictest10041-21/usr/sbin/munin08:45:182
67599270,4cyclictest8886-21taskset08:44:542
67599270,1cyclictest0-21swapper/208:00:142
67699261,9cyclictest66050lldpd10:23:303
676992611,3cyclictest83162sh10:36:233
67599266,17cyclictest13622-21/usr/sbin/munin08:55:092
67599265,18cyclictest9968-21kernelversion07:30:182
67599265,18cyclictest29346-21kernelversion08:15:162
67599262,21cyclictest13902-21cat07:40:152
67599261,9cyclictest27142-21/usr/sbin/munin08:10:172
67499260,24cyclictest0-21swapper/109:15:301
67399261,10cyclictest30392-21users09:40:260
676992518,5cyclictest14705-21/usr/sbin/munin07:40:223
67599256,17cyclictest15932-21expr09:00:122
67599251,9cyclictest20402-21cstates07:55:122
674992515,6cyclictest53050irq/142-enp3s0-11:20:121
674992515,6cyclictest53050irq/142-enp3s0-11:20:121
674992511,11cyclictest13899-1kworker/u9:010:00:111
673992514,9cyclictest27989-21ssh12:03:340
67599248,13cyclictest0-21swapper/208:50:152
674992414,6cyclictest278-21systemd-journal09:30:211
674992413,8cyclictest31179-21df08:20:131
142982240,2sleep120044-21diskmemload11:05:131
67699234,3cyclictest20044-21diskmemload12:33:023
67699233,8cyclictest0-21swapper/311:50:263
67699233,6cyclictest34-21ksoftirqd/311:41:473
67699231,8cyclictest4759-21ls07:20:003
67699231,19cyclictest30112-21/usr/sbin/munin10:05:213
67499233,12cyclictest20747-21ssh11:10:121
674992321,1cyclictest0-21swapper/108:45:111
674992314,6cyclictest560-21dbus-daemon08:00:001
67399231,7cyclictest5939-21kernelversion12:35:200
673992315,6cyclictest459-21sh11:19:110
673992314,6cyclictest15541-21ls11:05:240
6372230,2sleep30-21swapper/311:19:203
316002230,1sleep00-21swapper/010:30:130
316002230,1sleep00-21swapper/010:30:120
67699226,3cyclictest141rcu_preempt12:10:253
67699226,3cyclictest141rcu_preempt10:30:223
67699226,3cyclictest141rcu_preempt10:30:213
67699225,14cyclictest2614-21cron07:14:593
67699224,16cyclictest17700-21ntpq09:55:213
67699221,5cyclictest21593-21turbostat.cron11:35:003
676992214,5cyclictest52750irq/139-enp3s0-09:45:273
676992213,7cyclictest9089-21sed07:30:013
676992212,6cyclictest52750irq/139-enp3s0-12:16:263
676992210,9cyclictest5264-21cpuspeed_turbos12:35:143
674992214,6cyclictest13835-21sh11:52:501
67399223,17cyclictest9337-21munin-run11:50:010
673992214,6cyclictest22691-21cron12:00:000
673992211,8cyclictest10079-21cat07:30:190
99022210,1sleep00-21swapper/010:37:590
69512210,13sleep012-21ksoftirqd/008:40:010
67699218,3cyclictest34-21ksoftirqd/311:39:293
67699214,3cyclictest141rcu_preempt09:30:193
67699211,6cyclictest23513-21chrt11:11:283
676992113,6cyclictest12747-21ls07:35:223
676992113,4cyclictest17389-21grep11:55:183
676992111,7cyclictest22016-21systemctl08:00:003
67499212,8cyclictest797050kworker/u8:110:55:421
67499212,7cyclictest0-21swapper/112:33:411
674992119,1cyclictest0-21swapper/108:44:591
674992112,6cyclictest4405-21seq07:15:421
67399218,10cyclictest15125-21basename09:30:110
67399211,5cyclictest19072-21apt-get10:45:130
673992113,6cyclictest19607-21users10:20:250
673992113,5cyclictest278-21systemd-journal09:50:070
673992112,7cyclictest8088-21cron11:25:000
673992112,7cyclictest8088-21cron11:25:000
673992112,7cyclictest24744-21df_inode08:05:140
673992112,6cyclictest4321-21seq08:31:270
265312211,1sleep10-21swapper/110:25:371
67699209,9cyclictest31190-21python312:05:233
67699204,14cyclictest15661-21grep11:05:253
67699202,14cyclictest28667-21apt-get09:40:133
676992012,6cyclictest1360-21sed08:25:193
676992012,5cyclictest28461-21basename08:14:593
676992011,7cyclictest24883-21wc10:00:243
67699201,14cyclictest3120-21ssh11:20:233
67699201,14cyclictest3120-21ssh11:20:223
676992010,7cyclictest23141-21sendmail_mailqu09:35:223
67499209,7cyclictest53050irq/142-enp3s0-09:35:121
67499207,11cyclictest22-21ksoftirqd/110:55:011
67499202,12cyclictest16433-21apt-get11:55:131
67499201,6cyclictest5550-21ssh12:35:161
674992014,4cyclictest22335-21sh12:23:311
674992013,5cyclictest6992-21seq09:24:021
674992013,4cyclictest31425-21sh10:05:231
674992013,4cyclictest30584-21sendmail08:20:011
674992013,4cyclictest20044-21diskmemload09:50:181
674992013,4cyclictest11922-21cron10:40:001
674992012,6cyclictest8456-21sed07:25:251
674992012,6cyclictest3625-21grep09:45:151
674992012,6cyclictest27071-21seq12:02:391
674992012,6cyclictest11364-21seq11:26:311
674992011,6cyclictest17967-21grep09:55:231
674992010,7cyclictest560-21dbus-daemon09:10:001
67399209,7cyclictest7165-21apt-get07:25:130
67399208,10cyclictest52950irq/141-enp3s0-10:05:150
67399202,12cyclictest2866-21apt-get07:15:130
67399201,5cyclictest25622-21ls12:25:210
67399201,4cyclictest3209-21apt-get09:45:110
673992013,4cyclictest5535-21sshd12:10:250
673992013,4cyclictest31822-21sh12:30:170
673992013,4cyclictest31297-21sh09:17:420
673992012,6cyclictest5857-21seq09:22:550
673992012,6cyclictest560-21dbus-daemon08:15:000
673992012,6cyclictest5476-21cat10:10:200
673992012,6cyclictest24043-21seq08:03:370
673992012,6cyclictest23229-21which11:35:210
673992012,6cyclictest15829-21find07:45:110
673992011,7cyclictest11829-21df08:50:130
673992011,7cyclictest1060-21cat07:10:140
673992011,6cyclictest16037-21expr09:00:120
673992011,6cyclictest11183-21df12:15:140
67699197,2cyclictest34-21ksoftirqd/310:48:333
67699192,5cyclictest8510-21apt-get11:25:113
67699192,12cyclictest30857-21apt-get08:20:123
676991912,5cyclictest23085-21sed08:00:203
676991912,5cyclictest12995-21awk08:50:243
676991911,6cyclictest28148-21sed09:15:183
676991911,5cyclictest20044-21diskmemload12:22:363
676991910,6cyclictest278-21systemd-journal09:05:013
67499192,12cyclictest7970-21kworker/u8:110:45:121
67499192,12cyclictest29853-21apt-get12:05:131
67499192,11cyclictest3039-21apt-get11:45:121
67499191,8cyclictest15542-21awk09:00:001
674991911,6cyclictest31247-21seq09:41:161
674991911,6cyclictest28953-21expr11:40:141
674991911,6cyclictest22913-21cat08:00:181
674991911,6cyclictest17888-21cut09:05:001
67399193,3cyclictest20044-21diskmemload11:10:570
67399193,13cyclictest17727-21ntp_states09:55:200
67399192,15cyclictest13769-21cpuspeed_turbos08:55:120
673991914,3cyclictest24233-21latency10:00:180
673991913,4cyclictest14619-21sh11:29:470
673991912,5cyclictest8273-21cut11:00:170
673991911,6cyclictest29214-21sed08:15:150
673991911,6cyclictest15404-21seq10:41:440
673991911,6cyclictest12-21ksoftirqd/007:40:110
673991911,4cyclictest20044-21diskmemload12:08:460
673991910,7cyclictest663-21df10:55:140
673991910,5cyclictest52950irq/141-enp3s0-10:25:230
67699183,12cyclictest10070-21cpuspeed_turbos09:50:133
67699182,11cyclictest52750irq/139-enp3s0-08:45:113
67699181,12cyclictest8407-21ssh09:25:123
676991810,6cyclictest24877-21perl08:05:163
67699180,5cyclictest0-21swapper/311:48:403
67499182,11cyclictest24416-21apt-get08:05:131
67499182,10cyclictest12366-21sh10:40:121
674991813,3cyclictest16934-21sendmail_mailst07:45:221
674991812,4cyclictest16024-21sh11:30:181
674991811,5cyclictest4004-21ls08:30:221
674991811,5cyclictest12797-21sed08:50:221
674991810,6cyclictest9898-1kworker/u9:210:15:161
674991810,6cyclictest859-21expr08:25:131
674991810,6cyclictest6569-21seq07:20:451
674991810,6cyclictest24309-21grep10:24:591
674991810,6cyclictest21274-21cat09:10:171
674991810,6cyclictest13307-21anacron07:39:521
67399189,7cyclictest9135-21/usr/sbin/munin08:45:160
673991813,3cyclictest19274-21sh12:20:260
673991812,4cyclictest20039-21idleruntime-cro09:10:010
673991812,3cyclictest13447-21dump-pmu-power07:40:010
673991811,5cyclictest6042-21ps07:20:200
673991811,5cyclictest21791-21mailstats09:10:230
673991811,4cyclictest20044-21diskmemload11:41:400
673991810,6cyclictest20798-21cut07:55:180
673991810,6cyclictest18677-21cat07:50:160
673991810,6cyclictest10611-21cut11:50:180
289132180,4sleep3331rcuc/308:15:133
67699179,6cyclictest16597-21wc09:00:173
67699173,3cyclictest62282switchtime07:20:213
67699172,12cyclictest13799-21ssh10:40:233
67699171,10cyclictest640-21snmpd12:00:143
676991710,5cyclictest5092-21expr08:35:143
676991710,5cyclictest3910-21sed08:30:223
67499179,6cyclictest560-21dbus-daemon07:54:591
67499179,6cyclictest13999-21expr07:40:141
67499179,3cyclictest11433-21gltestperf12:15:161
67499176,3cyclictest141rcu_preempt10:30:071
67499176,3cyclictest141rcu_preempt10:30:061
67499171,7cyclictest5631-21sort12:10:261
674991710,4cyclictest20044-21diskmemload10:10:101
673991710,5cyclictest7608-21basename08:40:150
281582170,9sleep122-21ksoftirqd/111:39:531
67699168,5cyclictest18456-21grep07:50:143
676991610,4cyclictest14568-21ntp_states08:55:203
67499169,5cyclictest1279-21hwlatdetect07:10:171
67499168,6cyclictest560-21dbus-daemon07:35:011
674991610,4cyclictest28441-21latency_hist08:15:001
67399161,12cyclictest2054-21sed08:25:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional