You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-01 - 01:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Apr 30, 2026 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19293217096,24sleep00-21swapper/007:09:440
189682152131,15sleep30-21swapper/307:05:303
190382144108,25sleep10-21swapper/107:06:251
189622141105,25sleep20-21swapper/207:05:252
1959199132127,3cyclictest22657-21lspci09:45:571
1959199132127,3cyclictest22657-21lspci09:45:561
1931927456,10sleep00-21swapper/007:10:000
71542640,2sleep20-21swapper/209:10:352
258502620,2sleep00-21swapper/007:25:000
208332570,2sleep00-21swapper/012:34:470
204062570,2sleep00-21swapper/010:56:410
13722570,2sleep20-21swapper/211:55:212
37402560,2sleep30-21swapper/309:05:183
252692530,1sleep30-21swapper/311:25:133
186352510,2sleep00-21swapper/011:44:430
239202500,2sleep019588-21cyclictest11:24:140
5382470,2chrt0-21swapper/309:54:103
52952460,1sleep10-21swapper/111:58:461
4942400,3sleep10-21swapper/110:18:081
1959399378,8cyclictest18420-21apt-get11:20:093
301242350,10sleep122-21ksoftirqd/111:04:441
1959399326,10cyclictest3168-21sh10:44:333
1959399323,9cyclictest32760-21grep11:55:123
1959399316,8cyclictest3283-21rm11:08:373
1959399315,9cyclictest5158-21sh12:22:373
1959399313,19cyclictest8688-21ssh11:12:203
1959399311,10cyclictest20310-21ls09:20:323
1959399305,10cyclictest21357-21sh10:57:373
1959399302,10cyclictest4848-21diskmemload10:53:583
19593993018,7cyclictest4023-21apt-get10:45:123
19590993020,7cyclictest131rcuc/011:25:150
1959399294,14cyclictest0-21swapper/311:15:273
1959399294,14cyclictest0-21swapper/311:15:263
1959399293,17cyclictest18378-21ssh10:06:533
1959399293,10cyclictest9647-21rm09:13:053
1959399292,10cyclictest4848-21diskmemload11:53:163
1959399292,10cyclictest13900-21rm09:15:383
1959399291,15cyclictest4848-21diskmemload09:59:303
1959399285,9cyclictest27401-21sh12:39:403
1959399285,10cyclictest7106-21gltestperf09:35:153
1959399283,9cyclictest2962-21/usr/sbin/munin10:20:233
1959399283,8cyclictest15117-21ssh10:29:093
1959399283,16cyclictest11976-21sh12:03:463
1959399283,16cyclictest11976-21sh12:03:453
1959399283,14cyclictest0-21swapper/311:43:443
1959399283,10cyclictest27765-21rm10:38:273
1959399282,9cyclictest26376-21ssh10:13:143
1959399282,9cyclictest19241-21rm12:09:223
1959399282,9cyclictest18195-21ssh09:43:443
1959399281,9cyclictest4848-21diskmemload12:25:243
241132270,3sleep122-21ksoftirqd/109:24:111
1959399273,9cyclictest827-21ssh10:18:263
1959399272,9cyclictest24568-21ssh12:13:013
1959399272,8cyclictest28504-21ls12:15:233
1959399272,10cyclictest5096-21ssh09:34:113
1959199271,25cyclictest0-21swapper/107:13:011
1959399262,9cyclictest4848-21diskmemload11:32:353
1959399262,9cyclictest11656-21date08:05:183
1959399261,15cyclictest0-21swapper/307:10:133
1959399261,10cyclictest0-21swapper/311:45:503
1959199261,13cyclictest242112ssh12:36:261
1959399254,8cyclictest8260-21sh11:35:593
1959399252,20cyclictest15268-21cut12:30:143
1959399251,9cyclictest19633-21sh10:31:573
1959399251,9cyclictest14088-21ssh10:04:193
1959399251,8cyclictest23971-21ssh09:46:483
1959399251,8cyclictest23971-21ssh09:46:473
1959399251,10cyclictest287302sed07:30:163
19592992523,1cyclictest0-21swapper/207:13:052
1959099242,15cyclictest11981-21ssh09:15:120
19590992416,6cyclictest12-21ksoftirqd/009:15:000
93832230,8sleep228-21ksoftirqd/210:00:202
315112230,2sleep10-21swapper/109:52:401
270882231,2sleep10-21swapper/108:40:221
1959399232,8cyclictest29822-21ssh09:28:123
1959399232,6cyclictest23936-21apt-get07:20:113
19593992313,7cyclictest9103-21df_inode08:00:143
1959299239,8cyclictest52950irq/141-enp3s0-09:35:032
19591992314,7cyclictest6870-21df09:35:131
80042220,1sleep10-21swapper/109:11:271
242102220,2sleep20-21swapper/212:36:272
1959399221,7cyclictest4848-21diskmemload11:01:033
1959399220,9cyclictest0-21swapper/308:20:213
19592992212,7cyclictest278-21systemd-journal10:10:002
1959299221,17cyclictest640-21snmpd12:13:472
1959299220,5cyclictest0-21swapper/209:46:242
1959299220,5cyclictest0-21swapper/209:46:242
19591992213,7cyclictest12381-21smartctl11:15:161
19591992213,7cyclictest12381-21smartctl11:15:151
19591992213,6cyclictest25725-21basename08:40:001
1959099228,5cyclictest12-21ksoftirqd/009:30:290
19590992216,4cyclictest12-21ksoftirqd/011:50:010
19590992212,7cyclictest560-21dbus-daemon10:40:000
1959099221,17cyclictest28087-21grep09:50:160
1959399217,11cyclictest30689-21/usr/sbin/munin07:35:133
19592992116,3cyclictest28-21ksoftirqd/209:55:012
19592992113,6cyclictest4814-21cut11:10:002
19592992111,7cyclictest25496-21diskstats11:25:132
19592992111,6cyclictest52950irq/141-enp3s0-11:15:112
19592992111,6cyclictest52950irq/141-enp3s0-11:15:102
19591992115,3cyclictest786-21grep09:30:241
19591992113,6cyclictest29951-21seq08:49:441
19591992113,6cyclictest27800-21memory12:15:171
19591992113,6cyclictest25702-21seq07:23:391
19591992112,7cyclictest22724-21pluginstate07:15:201
19591992112,7cyclictest13768-21seq10:27:471
1959099215,5cyclictest0-21swapper/012:00:090
1959099215,5cyclictest0-21swapper/012:00:080
1959099212,13cyclictest25356-21apt-get09:25:120
19590992116,3cyclictest12-21ksoftirqd/011:15:000
19590992114,4cyclictest22226-21sed09:45:570
19590992114,4cyclictest22226-21sed09:45:560
1959099211,14cyclictest13637-21sh12:05:110
175332210,1sleep00-21swapper/008:20:130
154612210,1sleep00-21swapper/010:05:280
51302200,2sleep20-21swapper/211:34:352
1959399209,9cyclictest13307-21cat08:10:113
1959399208,9cyclictest825-21/usr/sbin/munin07:40:183
1959399205,9cyclictest29996-21apt-get08:50:033
1959399203,4cyclictest0-21swapper/307:15:123
1959399201,7cyclictest34-21ksoftirqd/308:35:223
19593992010,7cyclictest26426-21awk07:25:153
1959299201,5cyclictest0-21swapper/211:48:282
1959299201,4cyclictest0-21swapper/209:19:522
19592992012,6cyclictest6953-21seq09:58:532
19592992012,6cyclictest32077-21seq12:18:502
19592992012,6cyclictest20315-21date08:25:192
19592992012,5cyclictest17239-21grep08:20:012
19592992012,5cyclictest1715-21sh10:43:062
19592992011,7cyclictest560-21dbus-daemon08:39:592
19592992011,6cyclictest4921-21date09:10:002
19592992011,6cyclictest32344-21timerandwakeup08:55:212
1959299201,13cyclictest6990-21ssh12:00:132
1959299201,13cyclictest6990-21ssh12:00:122
1959299200,9cyclictest189972rm12:09:082
1959199206,5cyclictest5106-21ntpq10:45:211
1959199206,5cyclictest11889-21ls08:05:211
1959199206,11cyclictest8081-21acpi12:25:111
1959199201,7cyclictest0-21swapper/110:23:551
19591992014,4cyclictest22-21ksoftirqd/111:54:591
19591992014,4cyclictest13856-21sh09:40:161
19591992013,5cyclictest20898-21unixbench_multi08:25:241
19591992012,6cyclictest25122-21dpkg11:25:111
19591992012,5cyclictest21904-21sh10:10:141
19591992011,7cyclictest25466-21sort09:25:141
19591992011,7cyclictest21452-21ls10:10:001
19591992011,7cyclictest13632-21apt-get12:05:101
19591992011,6cyclictest30810-21tune2fs07:35:141
1959199200,6cyclictest1190-21sh11:30:361
1959099208,4cyclictest131rcuc/009:20:250
1959099203,10cyclictest4848-21diskmemload12:10:130
1959099202,13cyclictest26025-21apt-get11:50:130
1959099202,11cyclictest9734-21ssh10:25:120
1959099202,11cyclictest4848-21diskmemload10:20:100
1959099201,5cyclictest0-21swapper/012:37:260
19590992013,5cyclictest2300-21awk09:00:250
19590992013,5cyclictest12-21ksoftirqd/010:55:000
19590992013,5cyclictest12-21ksoftirqd/009:40:210
19590992012,6cyclictest27316-21cat08:40:240
49362190,1sleep30-21swapper/307:50:153
252112190,2sleep228-21ksoftirqd/209:25:042
1959399198,9cyclictest28615-21grep08:45:173
1959399192,11cyclictest2338-21apt-get07:45:113
1959399191,7cyclictest34-21ksoftirqd/309:01:363
1959399191,7cyclictest34-21ksoftirqd/308:58:563
1959399191,7cyclictest34-21ksoftirqd/308:25:233
1959399191,7cyclictest34-21ksoftirqd/307:56:363
19593991911,6cyclictest21707-21cat08:30:133
1959399190,8cyclictest34-21ksoftirqd/308:17:483
1959299199,7cyclictest22477-21grep08:30:192
1959299198,7cyclictest278-21systemd-journal12:25:052
1959299197,4cyclictest258542wc07:25:002
19592991913,4cyclictest7138-21sh11:10:472
19592991912,5cyclictest24450-21seq11:24:462
19592991912,5cyclictest23585-21expr10:35:142
19592991912,5cyclictest23446-21ls10:10:262
19592991912,5cyclictest17566-21ls08:20:132
19592991911,6cyclictest9327-21expr08:00:172
19592991911,6cyclictest27915-21seq07:29:372
19592991911,6cyclictest18955-21grep09:20:202
19592991911,6cyclictest15225-21seq09:40:442
19592991911,6cyclictest11695-21sed08:05:182
19592991911,5cyclictest4848-21diskmemload11:39:562
1959299191,12cyclictest17412-21ssh10:55:122
1959199196,10cyclictest23208-21tr10:35:121
1959199191,8cyclictest278-21systemd-journal09:05:001
19591991914,3cyclictest26787-21latency07:25:181
19591991913,4cyclictest3990-21sh09:55:541
19591991913,4cyclictest30863-21cpu11:05:131
19591991913,4cyclictest20572-21sh12:34:321
19591991913,4cyclictest15331-21sh11:41:231
19591991912,5cyclictest4763-21seq09:09:171
19591991912,5cyclictest12067-21sed10:50:221
19591991912,5cyclictest1125-21awk07:40:211
19591991912,4cyclictest20122-21sh12:10:021
19591991911,6cyclictest560-21dbus-daemon10:56:371
19591991911,6cyclictest30594-21sed10:40:171
19591991911,6cyclictest17445-21expr08:20:141
19591991910,7cyclictest6796-21cat07:55:121
19591991910,7cyclictest1635-21grep12:20:131
19591991910,6cyclictest15653-21gltestperf08:15:151
1959099198,9cyclictest4848-21diskmemload11:17:370
1959099198,9cyclictest4848-21diskmemload11:17:370
1959099192,4cyclictest0-21swapper/011:04:480
1959099191,6cyclictest45-21kcompactd011:58:460
1959099191,5cyclictest0-21swapper/011:38:130
19590991910,7cyclictest10374-21sed10:50:010
19590991910,6cyclictest52750irq/139-enp3s0-09:35:230
19590991910,6cyclictest2866-21grep12:20:240
19590991910,6cyclictest278-21systemd-journal08:55:000
1959099190,8cyclictest23581-21unixbench_singl10:10:270
1959099190,2cyclictest141rcu_preempt10:41:130
1959399189,6cyclictest26117-21grep08:40:113
1959299188,7cyclictest52950irq/141-enp3s0-09:30:112
1959299187,9cyclictest0-21swapper/211:41:092
1959299182,11cyclictest21774-21apt-get07:15:112
19592991813,3cyclictest5706-21unixbench_singl10:45:252
19592991813,3cyclictest4219-21sh07:49:592
19592991813,3cyclictest24045-21sh11:00:052
19592991813,3cyclictest13289-21sh10:27:182
19592991812,4cyclictest5729-21sh10:21:232
19592991812,4cyclictest4539-21sh12:22:012
19592991812,4cyclictest32562-21sh10:17:462
19592991812,4cyclictest18563-21sh10:30:542
19592991811,5cyclictest15686-21sed12:30:172
19592991811,5cyclictest1-21systemd09:00:222
19592991810,6cyclictest4447-21seq07:50:092
19592991810,6cyclictest26950-21ls11:50:182
1959299180,5cyclictest16997-21ls10:55:002
1959199188,6cyclictest52850irq/140-enp3s0-10:30:111
1959199184,12cyclictest5184-21tr11:10:121
19591991813,3cyclictest5264-21missed_timers07:50:181
19591991813,3cyclictest3676-21timerwakeupswit07:45:241
19591991813,3cyclictest27724-21munin-node10:00:101
19591991813,3cyclictest22433-21sh11:46:471
19591991812,4cyclictest23986-21sh11:24:161
19591991812,4cyclictest22-21ksoftirqd/109:20:001
19591991812,4cyclictest11470-21sh11:39:131
19591991811,5cyclictest7446-21tune2fs12:00:151
19591991811,5cyclictest7446-21tune2fs12:00:141
19591991811,5cyclictest31440-21grep08:55:121
19591991810,6cyclictest29499-21cat07:30:251
19591991810,5cyclictest22782-21mailstats08:30:221
1959099183,4cyclictest0-21swapper/011:09:510
1959099182,11cyclictest4848-21diskmemload10:15:220
19590991813,3cyclictest32129-210anacron07:38:450
19590991813,3cyclictest2977-21interrupts07:45:160
19590991811,4cyclictest26925-21sh12:15:090
1959099181,13cyclictest0-21swapper/007:55:100
19590991810,6cyclictest9377-21cat08:00:160
19590991810,6cyclictest8156-21grep10:00:000
19590991810,6cyclictest5756-21grep07:50:250
1959299179,6cyclictest32332-21cron07:40:012
1959299179,6cyclictest323-21cat07:40:132
19592991711,4cyclictest26894-21open_files08:40:202
19592991710,5cyclictest13033-21find08:10:092
19591991712,3cyclictest13401-21fschecks_count08:10:131
19591991710,5cyclictest10683-21wc08:05:001
1959099179,6cyclictest28680-21fw_packets07:30:150
1959099178,6cyclictest22956-21cat08:30:250
1959099172,12cyclictest31121-21cpuspeed_turbos08:55:110
1959099172,10cyclictest8108-21apt-get12:25:130
1959099171,7cyclictest21749-21/usr/sbin/munin07:15:160
1959099171,2cyclictest141rcu_preempt11:31:290
19590991712,3cyclictest4848-21diskmemload10:30:090
19590991710,5cyclictest9816-21grep10:00:240
19590991710,5cyclictest15016-21sed08:15:000
19590991710,5cyclictest11139-21expr08:05:140
1959299169,5cyclictest28773-21which07:30:162
1959299168,6cyclictest28873-21grep08:45:192
1959299161,10cyclictest30415-21apt-get08:50:042
1959099169,5cyclictest952-21cut07:40:180
1959099168,6cyclictest19585-21cut08:25:120
1959099168,5cyclictest25077-21sed08:35:240
19590991612,2cyclictest2922-21/usr/sbin/munin09:05:210
19590991610,4cyclictest26759-21latency07:25:190
19592991511,2cyclictest6935-21df07:55:122
1959199157,6cyclictest30400-21gdbus08:50:001
1959099151,10cyclictest15192-21taskset08:15:130
19590991510,3cyclictest28867-21missed_timers08:45:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional