You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 20:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed May 06, 2026 12:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
84362198147,28sleep10-21swapper/107:06:491
85762173121,44sleep20-21swapper/207:08:402
84902162107,29sleep00-21swapper/007:07:330
84882162106,28sleep30-21swapper/307:07:323
3047821470,2sleep00-21swapper/010:22:060
237482660,1sleep00-21swapper/010:17:090
237482660,1sleep00-21swapper/010:17:080
235332620,1sleep00-21swapper/011:05:120
157812600,2sleep00-21swapper/007:25:140
82602550,4sleep07206-1kworker/u9:010:30:130
280042530,1sleep00-21swapper/009:33:100
280042530,1sleep00-21swapper/009:33:100
212982530,2sleep30-21swapper/308:50:243
151442490,1sleep10-21swapper/110:10:301
253172470,2sleep1211rcuc/111:05:371
210722460,1sleep10-21swapper/108:50:221
85232450,1sleep00-21swapper/010:05:410
39652440,1sleep00-21swapper/010:26:160
279232440,1sleep30-21swapper/309:56:133
187642370,1sleep10-21swapper/111:00:481
187642370,1sleep10-21swapper/111:00:481
171882360,3sleep10-21swapper/109:49:121
895699301,27cyclictest0-21swapper/308:09:293
8954993020,7cyclictest27767-21/usr/sbin/munin09:05:241
8954992920,7cyclictest7764-21if_err_enp3s008:20:171
895399291,26cyclictest0-21swapper/008:09:290
895399289,12cyclictest52750irq/139-enp3s0-10:40:120
895399281,25cyclictest0-21swapper/008:10:380
895399280,25cyclictest0-21swapper/008:33:060
123682280,9sleep334-21ksoftirqd/309:45:173
895699271,25cyclictest0-21swapper/308:19:543
895699271,25cyclictest0-21swapper/308:10:393
895699271,1cyclictest0-21swapper/308:29:323
895499271,24cyclictest0-21swapper/108:33:001
895499271,24cyclictest0-21swapper/108:29:341
895499271,24cyclictest0-21swapper/108:19:101
895499271,24cyclictest0-21swapper/108:07:351
8953992714,9cyclictest32617-21memory11:35:190
895399271,24cyclictest0-21swapper/008:20:150
895399271,24cyclictest0-21swapper/008:19:520
895399271,1cyclictest0-21swapper/008:36:570
895399271,1cyclictest0-21swapper/008:00:160
58892270,7sleep012-21ksoftirqd/012:03:230
254352270,3sleep20-21swapper/211:30:142
895699264,2cyclictest141rcu_preempt11:55:023
895699261,1cyclictest0-21swapper/308:21:013
895699260,24cyclictest0-21swapper/308:33:283
895699260,24cyclictest0-21swapper/308:01:373
895599261,24cyclictest0-21swapper/208:11:272
895599261,1cyclictest0-21swapper/208:23:232
895499261,1cyclictest0-21swapper/108:11:071
895499261,1cyclictest0-21swapper/108:01:051
8953992623,2cyclictest0-21swapper/008:25:380
42782260,2sleep10-21swapper/109:39:531
244932260,2sleep20-21swapper/209:30:192
244932260,2sleep20-21swapper/209:30:182
220242260,2sleep00-21swapper/011:04:040
220242260,2sleep00-21swapper/011:04:040
8956992512,10cyclictest17103-21diskstats09:25:143
895699250,1cyclictest0-21swapper/308:38:273
8955992523,1cyclictest0-21swapper/208:19:072
8955992514,8cyclictest32537-21df_inode10:00:132
895599251,23cyclictest0-21swapper/208:06:272
895599250,23cyclictest0-21swapper/208:38:292
895599250,23cyclictest0-21swapper/208:25:052
8954992523,1cyclictest0-21swapper/108:37:381
895499251,17cyclictest28586-21ssh10:45:111
8954992510,4cyclictest66050lldpd08:43:201
8953992510,3cyclictest141rcu_preempt12:20:000
26502250,1sleep10-21swapper/109:14:571
895699242,4cyclictest34-21ksoftirqd/309:10:533
8956992419,3cyclictest22953-21idleruntime-cro11:05:003
8956992419,3cyclictest22953-21idleruntime-cro11:05:003
8955992422,1cyclictest0-21swapper/208:32:462
8955992422,1cyclictest0-21swapper/208:01:242
8953992416,6cyclictest28326-21diskmemload11:51:290
274082240,8chrt12-21ksoftirqd/011:55:170
215692240,2sleep20-21swapper/209:28:312
103052240,9sleep012-21ksoftirqd/009:44:060
895699233,12cyclictest141rcu_preempt09:24:233
895599231,7cyclictest2494-21sh10:50:002
895399234,16cyclictest21188-21sed07:35:250
895399231,6cyclictest15295-21diskstats12:10:140
50022230,2sleep30-21swapper/310:03:553
45932230,3sleep14588-21ssh10:50:241
300942230,3sleep10-21swapper/109:58:231
895699228,4cyclictest82622cpuspeed_turbos10:30:133
895699223,13cyclictest19744-21ssh11:50:123
8956992215,4cyclictest15350-21sh12:10:143
8956992212,8cyclictest31556-21awk11:10:223
8956992212,8cyclictest10827-21ssh09:44:373
895699220,11cyclictest14235-21ls10:10:183
8955992214,6cyclictest6252-21ls09:40:232
8955992212,7cyclictest320-21systemd-udevd09:58:212
8955992212,7cyclictest27585-21python309:05:232
895499222,15cyclictest20138-21ssh10:15:031
895499222,15cyclictest20138-21ssh10:15:031
895499222,14cyclictest16338-21ssh12:35:121
895499221,18cyclictest9319-21df07:10:141
895499221,17cyclictest6687-21sed11:40:151
895499221,14cyclictest141rcu_preempt12:19:081
895399228,9cyclictest13167-21df11:45:140
895399228,9cyclictest13167-21df11:45:140
895399227,4cyclictest31617-1kworker/u9:112:39:570
8953992213,7cyclictest18409-21smartctl08:45:150
8953992213,6cyclictest23064-21cron09:30:000
56382220,1sleep30-21swapper/310:27:553
214492220,8sleep012-21ksoftirqd/008:50:250
213382220,1sleep10-21swapper/111:26:441
895699214,4cyclictest141rcu_preempt11:30:183
895699214,15cyclictest11612-21awk10:55:243
895699213,13cyclictest4653-21ssh11:15:123
8956992115,3cyclictest8926-21irqstats12:05:173
8956992112,7cyclictest23219-21python310:40:253
8956992112,7cyclictest21668-31updatedb.mlocat07:39:543
895699210,8cyclictest244982netstat09:30:193
895699210,8cyclictest244982netstat09:30:183
895599212,11cyclictest3114-21ssh09:15:112
8955992114,5cyclictest52850irq/140-enp3s0-11:15:152
8955992114,4cyclictest1297-21sh11:12:052
8955992112,7cyclictest6494-21ssh12:27:322
8955992112,7cyclictest23902-21fw_packets11:05:152
8955992112,6cyclictest16607-21ntp_kernel_pll_08:40:192
895599211,10cyclictest28326-21diskmemload12:05:032
8955992110,7cyclictest28355-21apt-get12:20:122
8954992112,6cyclictest16828-21grep07:25:251
895499211,13cyclictest141rcu_preempt10:22:251
895499210,6cyclictest13708-21ssh10:57:321
895399219,10cyclictest9891-21expr12:30:120
895399218,9cyclictest52750irq/139-enp3s0-10:35:300
895399218,2cyclictest12-21ksoftirqd/011:14:370
895399217,10cyclictest52750irq/139-enp3s0-10:10:570
895399213,5cyclictest12-21ksoftirqd/009:55:140
8953992116,3cyclictest18894-21tail11:25:150
8953992115,4cyclictest12-21ksoftirqd/009:40:000
8953992113,6cyclictest30340-21seq07:58:160
8953992113,6cyclictest18467-21ntpq07:30:190
8953992112,7cyclictest6472-21cat11:40:140
69732210,2sleep00-21swapper/010:52:440
161402210,2sleep30-21swapper/310:35:223
151842210,1sleep20-21swapper/209:47:132
895699207,9cyclictest52950irq/141-enp3s0-11:43:453
895699207,3cyclictest34-21ksoftirqd/312:18:083
895699204,3cyclictest28326-21diskmemload09:51:103
895699204,14cyclictest3331-21grep10:50:143
895699203,3cyclictest28326-21diskmemload10:47:193
895699202,4cyclictest34-21ksoftirqd/309:00:173
895699202,13cyclictest18298-21ssh11:25:103
895699202,13cyclictest12803-21apt-get11:45:123
895699202,13cyclictest12803-21apt-get11:45:123
895699201,8cyclictest16288-21/usr/sbin/munin12:35:143
895699201,6cyclictest0-21swapper/310:20:103
895699201,5cyclictest0-21swapper/312:33:213
895699201,5cyclictest0-21swapper/312:21:483
8956992013,4cyclictest28326-21diskmemload12:28:543
8956992010,7cyclictest14232-21ntpq07:20:203
895599209,8cyclictest10295-21df_inode09:20:132
895599208,9cyclictest52850irq/140-enp3s0-12:10:112
895599208,9cyclictest0-21swapper/209:52:172
895599207,10cyclictest52850irq/140-enp3s0-11:45:252
895599207,10cyclictest52850irq/140-enp3s0-11:45:252
895599202,6cyclictest18281-21apt-get11:25:112
895599202,12cyclictest24292-21apt-get09:00:122
8955992014,3cyclictest18145-21hwlatdetect07:30:162
8955992013,4cyclictest29174-21hwlatdetect07:55:162
8955992013,4cyclictest28738-21apt-get09:10:112
8955992012,6cyclictest28152-21cat11:55:232
8955992012,6cyclictest28129-21awk10:20:212
8955992012,6cyclictest18761-21ntpq08:45:192
8955992012,6cyclictest10497-21smartctl10:55:152
8955992011,7cyclictest20751-21memory08:50:182
8955992010,6cyclictest21757-21timerandwakeup10:15:242
8955992010,6cyclictest21757-21timerandwakeup10:15:242
895499202,12cyclictest26508-21apt-get11:55:031
8954992014,4cyclictest22-21ksoftirqd/111:35:001
8954992013,5cyclictest18143-21seq10:36:551
8954992012,6cyclictest23554-21seq11:52:321
8954992012,6cyclictest23218-21seq09:53:291
8954992012,6cyclictest16353-21date09:25:001
895399207,9cyclictest52750irq/139-enp3s0-11:32:000
895399207,9cyclictest52750irq/139-enp3s0-09:50:170
895399206,3cyclictest52750irq/139-enp3s0-11:21:170
895399203,3cyclictest28326-21diskmemload09:17:360
895399202,4cyclictest28326-21diskmemload11:17:370
895399201,6cyclictest9873-21apt-get10:55:120
895399201,6cyclictest12-21ksoftirqd/008:58:470
8953992014,4cyclictest15800-21sh09:24:310
8953992012,6cyclictest9281-21expr07:10:130
8953992012,6cyclictest28625-21ls12:20:130
8953992012,6cyclictest12375-21date07:15:250
8953992012,3cyclictest30692-21sh10:45:480
8953992011,6cyclictest23356-21expr07:40:240
895699194,13cyclictest31729-21/usr/sbin/munin11:35:233
895699193,3cyclictest141rcu_preempt08:45:173
895699193,2cyclictest141rcu_preempt10:05:133
895699192,2cyclictest141rcu_preempt11:23:553
895699191,7cyclictest6961-40unattended-upgr12:04:153
895699191,4cyclictest0-21swapper/311:07:313
895699191,3cyclictest245932tail07:45:143
8956991913,4cyclictest4726-21grep09:15:243
8956991912,4cyclictest28326-21diskmemload09:35:143
8956991910,7cyclictest27536-21tr09:05:223
895599198,9cyclictest9700-21cpuspeed_turbos12:30:122
895599198,8cyclictest0-21swapper/210:35:222
895599195,3cyclictest141rcu_preempt11:04:262
895599195,3cyclictest141rcu_preempt11:04:262
895599194,3cyclictest141rcu_preempt11:40:192
895599192,12cyclictest13300-21apt-get10:10:122
895599191,9cyclictest28326-21diskmemload11:35:332
895599191,6cyclictest28326-21diskmemload12:35:112
8955991914,3cyclictest8211-21cpuspeed_turbos10:30:132
8955991914,3cyclictest3134-21sh10:25:262
8955991914,3cyclictest23205-21sh11:52:112
8955991913,4cyclictest8650-21sh10:54:252
8955991913,4cyclictest27398-21sh10:44:152
8955991913,4cyclictest16361-21sh07:25:202
8955991913,4cyclictest16044-21sh11:23:102
8955991911,6cyclictest4554-21cut09:40:002
8955991911,6cyclictest20043-21df07:35:132
8955991911,6cyclictest15269-21date07:25:012
8955991911,6cyclictest1490-21expr12:00:142
895499195,12cyclictest22-21ksoftirqd/111:10:151
895499193,9cyclictest541-21apt-get12:00:041
895499192,11cyclictest23516-21sh09:30:121
895499192,11cyclictest23516-21sh09:30:121
895499192,11cyclictest21717-21apt-get10:40:041
8954991917,1cyclictest0-21swapper/107:55:191
895499191,5cyclictest0-21swapper/112:21:511
895499191,5cyclictest0-21swapper/112:12:491
895499191,4cyclictest0-21swapper/110:09:191
8954991913,4cyclictest11790-21interrupts07:15:191
8954991911,6cyclictest27703-21grep07:50:241
8954991911,6cyclictest11456-21seq12:06:561
8954991911,6cyclictest10909-21swap12:30:221
8954991910,6cyclictest5308-21seq09:15:531
895399192,12cyclictest2665-21ssh12:25:120
895399191,5cyclictest0-21swapper/012:07:360
8953991913,4cyclictest14709-21sh09:46:450
8953991911,6cyclictest30254-21ls09:10:240
8953991911,6cyclictest28462-21cat07:55:010
8953991911,6cyclictest24355-21expr07:45:120
895699185,4cyclictest52950irq/141-enp3s0-08:55:233
895699183,3cyclictest141rcu_preempt10:15:013
895699183,3cyclictest141rcu_preempt10:15:013
8956991812,4cyclictest34-21ksoftirqd/307:55:003
8956991812,4cyclictest19482-21perf07:35:013
8956991811,5cyclictest12058-21ntp_states07:15:223
895599184,5cyclictest6562-21cpu10:05:122
8955991813,3cyclictest26494-21sh12:18:352
8955991811,5cyclictest24278-21cat07:45:132
8955991811,5cyclictest22225-21cat07:40:132
8955991811,4cyclictest22779-21interrupts08:55:172
8955991810,6cyclictest27482-21cat07:50:232
895499188,2cyclictest22-21ksoftirqd/111:15:101
895499187,9cyclictest28326-21diskmemload11:35:181
895499185,10cyclictest2620-21sh10:25:201
895499183,9cyclictest11422-21apt-get11:20:041
895499182,13cyclictest2633-21apt-get12:25:111
895499182,11cyclictest17856-21apt-get08:45:121
895499181,4cyclictest0-21swapper/109:41:281
8954991813,3cyclictest16683-21/usr/sbin/munin09:25:121
8954991813,3cyclictest15579-21sh11:46:281
8954991813,3cyclictest15579-21sh11:46:281
8954991811,5cyclictest278-21systemd-journal07:25:001
8953991813,3cyclictest1001-21sendmail_mailst10:00:210
8953991811,5cyclictest27246-21sed09:05:190
8953991811,5cyclictest26018-21seq09:03:090
8953991810,5cyclictest17064-21timerwakeupswit08:40:240
895699179,5cyclictest16146-21cut08:40:153
895699172,11cyclictest29762-21taskset07:55:223
8956991710,5cyclictest23321-21sed07:40:243
8956991710,5cyclictest17509-21wc07:30:013
8956991710,5cyclictest10348-21sed07:10:243
8955991710,5cyclictest11210-21cat07:15:152
8955991710,5cyclictest11058-21date07:15:012
895499179,6cyclictest23537-21unixbench_multi08:55:251
895499179,5cyclictest23161-21grep07:40:221
8954991712,3cyclictest5820-21sh10:04:441
8954991712,3cyclictest25058-21open_files07:45:201
8954991710,5cyclictest24866-21cat09:00:161
8954991710,5cyclictest17896-21expr07:30:121
895499170,8cyclictest8205-21ssh10:30:131
8953991711,4cyclictest636-21runrttasks07:20:040
8954991611,3cyclictest20645-21sh07:35:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional