You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 10:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Apr 28, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
120112172102,63sleep20-21swapper/219:06:452
121542164115,28sleep10-21swapper/119:08:351
121062163104,29sleep30-21swapper/319:07:573
120612160104,29sleep00-21swapper/019:07:220
5702860,1sleep20-21swapper/223:58:562
1226527556,11sleep10-21swapper/119:10:011
184242690,1sleep10-21swapper/121:47:331
184242690,1sleep10-21swapper/121:47:331
155742610,1sleep20-21swapper/220:30:182
309932560,3sleep3331rcuc/321:32:463
161382520,2sleep0131rcuc/020:30:240
19732510,2sleep122-21ksoftirqd/100:00:041
244312500,2sleep00-21swapper/022:40:140
236602500,2sleep00-21swapper/023:51:170
161662490,2chrt0-21swapper/322:34:063
9982470,1sleep20-21swapper/221:58:582
183562460,2sleep00-21swapper/022:10:560
70222450,1chrt0-21swapper/222:03:152
16422440,3chrt0-21swapper/219:59:592
35702410,1sleep00-21swapper/023:36:200
283972410,1sleep10-21swapper/100:19:551
12002410,3chrt22-21ksoftirqd/100:23:231
12372400,1sleep30-21swapper/300:23:263
228542380,2chrt0-21swapper/021:26:220
147732330,6sleep3331rcuc/300:09:423
147732330,6sleep3331rcuc/300:09:413
153902310,3sleep00-21swapper/019:15:190
1253899302,14cyclictest31794-21diskmemload22:05:122
12536992924,3cyclictest52750irq/139-enp3s0-21:40:110
12538992718,7cyclictest28-21ksoftirqd/222:20:002
1253699279,3cyclictest141rcu_preempt00:19:010
1253699269,5cyclictest12-21ksoftirqd/021:34:220
12536992617,6cyclictest15380-21ssh22:57:130
12536992613,4cyclictest12-21ksoftirqd/021:24:210
12536992611,3cyclictest141rcu_preempt22:06:430
234452250,2chrt0-21swapper/122:39:431
234452250,2chrt0-21swapper/122:39:431
138292250,2sleep20-21swapper/200:08:452
138292250,2sleep20-21swapper/200:08:442
1253899250,12cyclictest131742rm21:43:582
12537992514,8cyclictest13838-21df_inode21:20:131
12537992513,10cyclictest1470-21timerandwakeup21:10:241
1253799251,12cyclictest288212rm22:43:211
1253699259,9cyclictest12-21ksoftirqd/020:02:450
1253699258,5cyclictest12-21ksoftirqd/023:58:390
1253699258,4cyclictest12-21ksoftirqd/000:02:200
1253699257,2cyclictest141rcu_preempt21:16:250
1253699255,11cyclictest12-21ksoftirqd/022:02:580
1253699253,5cyclictest12-21ksoftirqd/023:30:150
12536992513,3cyclictest12-21ksoftirqd/022:15:370
12536992510,8cyclictest12-21ksoftirqd/023:49:410
12536992510,8cyclictest12-21ksoftirqd/021:57:060
12539992419,3cyclictest6858-21/usr/sbin/munin22:50:263
1253899242,15cyclictest3257-21ssh00:25:122
1253799241,16cyclictest6327-21ssh23:15:121
1253699249,3cyclictest12-21ksoftirqd/022:49:310
1253699248,4cyclictest12-21ksoftirqd/023:24:450
1253699248,4cyclictest12-21ksoftirqd/023:24:450
1253699248,2cyclictest141rcu_preempt23:01:010
1253699246,8cyclictest12-21ksoftirqd/000:07:420
1253699246,8cyclictest12-21ksoftirqd/000:07:410
1253699246,11cyclictest12-21ksoftirqd/021:36:090
1253699245,5cyclictest12-21ksoftirqd/023:42:030
1253699245,10cyclictest12-21ksoftirqd/021:50:250
1253699244,9cyclictest19752-21apt-get23:25:130
1253699244,10cyclictest4871-21rm23:13:590
1253699243,10cyclictest12-21ksoftirqd/023:06:200
12536992419,4cyclictest131rcuc/000:28:490
12536992419,3cyclictest131rcuc/023:15:170
12536992417,5cyclictest12-21ksoftirqd/020:19:590
12536992411,3cyclictest12-21ksoftirqd/020:22:400
12536992411,2cyclictest141rcu_preempt22:21:170
196232230,10sleep334-21ksoftirqd/321:24:513
12539992314,7cyclictest560-21dbus-daemon23:45:013
12539992314,6cyclictest717-21awk23:10:213
12539992314,6cyclictest25845-21fw_packets23:05:173
1253899234,12cyclictest29614-21ssh22:20:042
1253799231,5cyclictest20923-21munin-run19:30:011
1253799231,5cyclictest20923-21munin-run19:30:001
12537992315,6cyclictest12326-21ssh22:06:491
1253699239,8cyclictest12-21ksoftirqd/020:28:590
1253699239,4cyclictest12-21ksoftirqd/020:48:410
1253699238,4cyclictest12-21ksoftirqd/022:35:310
1253699238,4cyclictest12-21ksoftirqd/022:35:300
1253699238,4cyclictest12-21ksoftirqd/019:22:420
1253699237,9cyclictest12-21ksoftirqd/022:30:020
1253699235,9cyclictest12-21ksoftirqd/000:20:150
1253699234,4cyclictest12-21ksoftirqd/000:10:230
1253699234,11cyclictest12-21ksoftirqd/021:46:120
1253699234,11cyclictest12-21ksoftirqd/021:46:110
12536992319,2cyclictest16207-21idleruntime-cro00:34:590
163812220,19sleep216398-21ssh23:45:422
12539992213,7cyclictest7324-21ntpq23:15:203
12538992217,4cyclictest28-21ksoftirqd/223:54:592
1253799229,8cyclictest53050irq/142-enp3s0-21:50:131
1253699228,8cyclictest14454-21cron19:15:000
1253699228,4cyclictest12-21ksoftirqd/021:10:360
1253699227,2cyclictest141rcu_preempt22:25:460
12536992210,2cyclictest141rcu_preempt00:37:020
68592210,3sleep26864-21sed22:50:272
1253999217,11cyclictest20317-21cpuspeed_turbos21:25:123
12539992117,3cyclictest52950irq/141-enp3s0-23:34:373
12539992114,4cyclictest6877-21sh22:03:073
12539992113,5cyclictest52950irq/141-enp3s0-00:10:143
12539992112,6cyclictest278-21systemd-journal20:15:003
12539992112,6cyclictest22582-21grep19:30:253
1253999211,18cyclictest2185-21needreboot21:35:193
1253899219,8cyclictest12761-21apt-get20:25:122
12538992113,5cyclictest52850irq/140-enp3s0-00:15:152
12538992113,4cyclictest31794-21diskmemload21:35:132
12538992112,6cyclictest560-21dbus-daemon23:30:012
12538992112,6cyclictest15247-21sed19:15:192
12538992110,7cyclictest560-21dbus-daemon22:15:002
12538992110,7cyclictest4074-21apt-get22:25:102
1253799218,3cyclictest141rcu_preempt19:40:011
1253799217,5cyclictest29059-21date21:55:181
1253799212,7cyclictest9177-21ls23:40:241
12537992114,4cyclictest6046-21sh00:26:321
12537992113,6cyclictest8706-21fw_packets20:15:171
12537992113,6cyclictest22782-21seq23:02:531
12537992113,6cyclictest20829-21awk23:25:221
12537992113,5cyclictest26614-21apt-get23:30:101
12537992112,7cyclictest19710-21seq00:36:471
12537992112,7cyclictest13883-21sshd21:44:441
12537992111,7cyclictest15542-21pluginstate23:45:241
1253699218,4cyclictest12-21ksoftirqd/021:01:540
1253699218,3cyclictest12-21ksoftirqd/020:14:050
1253699217,4cyclictest12-21ksoftirqd/020:57:560
1253699215,7cyclictest12-21ksoftirqd/019:30:570
1253699213,6cyclictest12-21ksoftirqd/019:50:250
12536992113,6cyclictest28704-21ntpq19:45:200
12536992112,7cyclictest5808-21seq20:09:410
12536992110,3cyclictest12-21ksoftirqd/022:50:050
91562200,2sleep10-21swapper/121:16:071
184992200,2sleep00-21swapper/020:35:240
136282200,2sleep10-21swapper/100:08:311
136282200,2sleep10-21swapper/100:08:301
12539992015,3cyclictest7169-21sh00:27:393
12539992014,4cyclictest30352-21sh23:56:153
12539992014,4cyclictest1914-21sh21:59:523
12539992014,3cyclictest23423-21sh23:03:313
12539992013,4cyclictest32413-21/usr/sbin/munin21:10:203
12539992012,6cyclictest28896-21readlink21:00:203
12539992012,6cyclictest26378-21ntp_states19:40:213
12539992012,6cyclictest23399-21wc20:50:013
12539992012,6cyclictest21454-21uptime23:25:273
12539992011,7cyclictest30319-21tune2fs19:50:153
12539992011,7cyclictest25663-21awk20:55:123
12539992011,6cyclictest24584-21timerandwakeup19:35:243
1253899209,9cyclictest16213-21latency_hist00:35:002
1253899205,6cyclictest636-21runrttasks00:23:402
12538992012,6cyclictest28376-21sed19:45:172
12538992011,6cyclictest26188-21sed23:30:002
1253799209,7cyclictest25321-21apt-get23:05:121
12537992014,4cyclictest5506-21sh23:14:401
12537992013,4cyclictest3451-21sh23:36:111
12537992013,4cyclictest1870-21sh22:23:131
12537992013,4cyclictest15499-21/usr/sbin/munin00:10:181
12537992012,6cyclictest868-21cut21:35:011
12537992012,6cyclictest8148-21date20:15:011
12537992012,6cyclictest6272-21seq22:25:531
12537992012,6cyclictest30326-21grep19:50:151
12537992012,6cyclictest26981-21seq19:40:271
12537992012,6cyclictest25203-21seq20:53:111
12537992012,6cyclictest23864-21ntpq22:15:211
12537992011,6cyclictest27730-21seq21:00:001
12537992011,6cyclictest15306-21idleruntime19:15:191
12537992010,8cyclictest560-21dbus-daemon19:59:591
12536992012,6cyclictest31502-21seq21:06:260
12536992012,6cyclictest12-21ksoftirqd/019:35:150
1253999192,13cyclictest15974-21ssh22:10:043
12539991914,3cyclictest6430-21sh22:26:033
12539991914,3cyclictest30775-21netstat22:20:193
12539991914,3cyclictest17814-21sh22:35:123
12539991914,3cyclictest17814-21sh22:35:113
12539991913,4cyclictest16824-21sh23:22:313
12539991913,4cyclictest16824-21sh23:22:303
12539991912,4cyclictest17986-21sh22:59:513
12539991911,6cyclictest4078-21expr20:05:143
12539991911,6cyclictest30127-21cat21:05:133
12539991911,6cyclictest18137-21run-parts20:35:233
12539991911,5cyclictest52950irq/141-enp3s0-21:17:453
1253899199,6cyclictest52850irq/140-enp3s0-21:25:112
1253899198,9cyclictest31994-21ssh22:45:232
12538991914,3cyclictest31985-21acpi23:10:112
12538991913,4cyclictest28-21ksoftirqd/223:05:012
12538991912,5cyclictest8353-21cat23:40:152
12538991912,4cyclictest31794-21diskmemload00:35:172
12538991912,3cyclictest25864-21unixbench_singl22:40:262
12538991911,6cyclictest6748-21cut23:15:162
12538991911,6cyclictest17034-21awk20:35:102
12538991911,6cyclictest12899-21ntpq22:55:192
12538991910,5cyclictest52850irq/140-enp3s0-22:30:122
1253899190,6cyclictest7978-21cron20:14:592
1253899190,6cyclictest4388-21fschecks_time20:05:172
1253799191,8cyclictest2264-21ntp_states21:35:201
1253799191,5cyclictest0-21swapper/122:00:151
12537991913,3cyclictest19921-21sh22:12:301
12537991912,5cyclictest20945-21seq20:43:231
12537991912,4cyclictest16811-21sh22:58:401
12537991912,4cyclictest15870-21sh23:21:331
12537991912,4cyclictest15870-21sh23:21:321
12537991912,4cyclictest11854-21ntp_states22:30:211
12537991911,6cyclictest26378-21expr20:55:181
12537991911,6cyclictest2239-21cut20:00:161
12537991911,6cyclictest21560-21cat21:25:221
1253799191,15cyclictest5180-21ssh22:50:121
12537991910,6cyclictest53050irq/142-enp3s0-23:50:111
12537991910,6cyclictest22066-21ntpq19:30:201
12537991910,5cyclictest0-21swapper/123:55:111
1253699199,2cyclictest141rcu_preempt19:40:250
12536991911,6cyclictest19478-21cut19:25:150
12536991911,6cyclictest19478-21cut19:25:140
12536991911,2cyclictest141rcu_preempt20:53:120
1253999189,7cyclictest14497-21ls19:15:003
1253999189,6cyclictest52950irq/141-enp3s0-00:00:133
1253999189,6cyclictest15982-21cat21:45:253
1253999189,6cyclictest15982-21cat21:45:243
1253999188,8cyclictest23570-21ssh22:15:173
1253999187,7cyclictest52950irq/141-enp3s0-00:35:113
1253999185,10cyclictest9972-21ssh00:30:123
1253999182,11cyclictest9186-21apt-get22:05:123
1253999182,11cyclictest14899-21apt-get20:30:113
1253999181,3cyclictest28724-21ntp_states19:45:213
12539991813,3cyclictest24940-21sh21:52:093
12539991813,3cyclictest15689-21sendmail_mailqu23:45:243
12539991812,4cyclictest29161-21sh22:43:433
12539991812,4cyclictest10616-21sh21:41:263
12539991811,5cyclictest2547-21seq22:47:483
12539991811,5cyclictest21072-21seq20:44:593
12539991811,5cyclictest18759-21seq19:24:593
12539991811,5cyclictest11519-21grep20:20:233
12539991810,6cyclictest560-21dbus-daemon19:19:593
12539991810,6cyclictest22578-21cat23:50:253
12539991810,6cyclictest21026-21cut19:30:003
12539991810,6cyclictest21026-21cut19:30:003
12539991810,5cyclictest22660-21grep00:15:173
1253899189,6cyclictest8596-21grep20:15:172
1253899184,3cyclictest31794-21diskmemload23:09:312
1253899182,9cyclictest31794-21diskmemload23:35:132
1253899182,3cyclictest141rcu_preempt21:45:142
1253899182,3cyclictest141rcu_preempt21:45:142
1253899182,11cyclictest25610-21apt-get20:55:132
1253899182,11cyclictest13463-21apt-get21:20:122
1253899181,8cyclictest8502-21ssh21:15:272
12538991813,4cyclictest52850irq/140-enp3s0-21:30:522
12538991812,4cyclictest17839-21pmu-power19:20:212
12538991811,5cyclictest27991-21expr21:00:102
12538991811,5cyclictest2028-21expr20:00:132
12538991811,5cyclictest19850-21uname20:40:192
12538991811,5cyclictest13281-21expr23:20:142
12538991811,5cyclictest13281-21expr23:20:132
12538991811,5cyclictest12862-21expr19:10:132
12538991810,5cyclictest28-21ksoftirqd/221:14:582
12538991810,5cyclictest2126-21apt-get00:00:142
1253799188,7cyclictest17038-21apt-get20:35:121
12537991811,4cyclictest3101-21sh22:48:211
1253799181,12cyclictest27351-21apt-get19:45:051
12537991810,6cyclictest560-21dbus-daemon00:30:231
12537991810,6cyclictest31188-21ls21:05:251
12537991810,5cyclictest4225-21expr20:05:141
1253699181,8cyclictest12-21ksoftirqd/020:42:020
12536991811,5cyclictest1249-21seq19:56:210
1253999179,5cyclictest24446-21pmu-power20:50:203
12539991711,4cyclictest13244-21grep20:25:163
12539991710,5cyclictest2046-21open_files23:35:233
1253899179,6cyclictest17584-21grep22:35:112
1253899179,6cyclictest17584-21grep22:35:102
1253899179,5cyclictest21822-21grep20:45:172
1253899177,8cyclictest0-21swapper/221:52:082
12538991712,3cyclictest636-21runrttasks19:50:262
12538991712,3cyclictest23471-21apt20:50:092
12538991711,4cyclictest31802-21awk21:09:592
12538991711,4cyclictest24562-21sed19:35:242
12538991711,4cyclictest23055-21latency_hist19:35:002
12538991711,4cyclictest11617-21sed20:20:232
12538991711,3cyclictest31794-21diskmemload00:14:002
1253799178,6cyclictest278-21systemd-journal20:25:011
12537991711,5cyclictest22-21ksoftirqd/120:25:001
12539991611,3cyclictest8217-21/usr/sbin/munin20:15:193
12539991611,3cyclictest32522-21fschecks_time19:55:163
12539991610,4cyclictest1988-21seq20:00:133
12538991610,4cyclictest0-21swapper/219:25:252
12538991610,4cyclictest0-21swapper/219:25:242
12537991611,3cyclictest17055-21cstates19:20:121
12537991611,3cyclictest14874-21/usr/sbin/munin20:30:151
1253899159,4cyclictest25437-21/usr/sbin/munin19:40:182
1253799157,6cyclictest21716-21fschecks_count20:45:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional