You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 22:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Dec 27, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10408824760,8sleep20-21swapper/207:05:252
10373322240,20sleep00-21swapper/007:05:180
1114212800,4sleep00-21swapper/007:25:150
1158532790,4sleep10-21swapper/107:37:311
197399780,68rtkit-daemon0-21swapper/107:09:591
1605602730,5sleep3160561-21cpuspeed_turbos09:45:153
197399620,9rtkit-daemon1972-21rtkit-daemon09:55:230
197399610,9rtkit-daemon1972-21rtkit-daemon10:35:140
10330226142,13sleep30-21swapper/307:05:033
197399580,9rtkit-daemon1972-21rtkit-daemon08:26:050
197399550,8rtkit-daemon1972-21rtkit-daemon07:57:030
197399540,9rtkit-daemon1972-21rtkit-daemon08:32:281
104925995240,9cyclictest34-21ksoftirqd/211:55:582
197399500,7rtkit-daemon1972-21rtkit-daemon07:49:171
197399500,6rtkit-daemon1972-21rtkit-daemon08:45:570
197399500,6rtkit-daemon1972-21rtkit-daemon08:45:560
104925995025,21cyclictest151231-40lsb_release09:17:072
104921994926,18cyclictest150453-21irqstats09:15:201
1882192480,7sleep2188160-21munin-run11:05:022
1624832480,6sleep013-21ksoftirqd/009:50:140
104927994625,17cyclictest216717-21/usr/sbin/munin12:25:243
10491799460,43cyclictest1972-21rtkit-daemon07:38:210
1628892450,5sleep3411rcuc/309:50:253
10492799454,16cyclictest150838-21sensors09:15:273
104927994524,16cyclictest126091-21/usr/sbin/munin08:05:313
10492799444,10cyclictest131906-40lsb_release08:23:063
10492799444,10cyclictest131906-40lsb_release08:23:063
10492599441,9cyclictest924-21NetworkManager12:35:022
104917994427,7cyclictest1137-21runrttasks11:16:510
1401492430,7chrt0-21swapper/108:45:281
1401492430,7chrt0-21swapper/108:45:271
10492799435,23cyclictest42-21ksoftirqd/312:38:463
10492799434,17cyclictest135352-21taskset08:32:453
10492799432,19cyclictest2069-21gnome-shell07:42:593
1058622420,6chrt105861-21ntpq07:10:242
10492799427,11cyclictest42-21ksoftirqd/310:10:003
10492799426,16cyclictest121977-21df07:55:153
10492799425,17cyclictest141749-21ntp_states08:50:243
10492599425,8cyclictest34-21ksoftirqd/210:23:082
10492599423,5cyclictest141rcu_preempt12:25:002
104917994224,6cyclictest13-21ksoftirqd/007:20:020
2049222410,7sleep3411rcuc/311:50:143
2049222410,7sleep3411rcuc/311:50:143
1138202410,6sleep30-21swapper/307:30:293
10492799417,17cyclictest42-21ksoftirqd/311:29:593
10492799417,11cyclictest42-21ksoftirqd/307:37:303
10492799416,16cyclictest42-21ksoftirqd/310:27:123
10492799415,12cyclictest42-21ksoftirqd/309:15:013
10492799414,11cyclictest42-21ksoftirqd/310:20:283
10492799414,11cyclictest182602-21taskset10:45:573
10492799414,11cyclictest182602-21taskset10:45:573
10492799414,10cyclictest109710-21df07:20:173
104927994133,5cyclictest411rcuc/307:45:303
104927994119,19cyclictest106618-21unin-run07:14:593
10492799411,15cyclictest158268-21sshd09:38:293
104927994110,12cyclictest42-21ksoftirqd/310:10:433
104927994110,11cyclictest42-21ksoftirqd/310:55:013
104925994127,8cyclictest34-21ksoftirqd/209:25:012
104925994122,13cyclictest438-20systemd-journal08:25:002
104925994122,13cyclictest438-20systemd-journal08:25:002
104925994117,12cyclictest2080-21gdbus08:05:542
104925994113,16cyclictest34-21ksoftirqd/209:10:152
104917994122,14cyclictest13-21ksoftirqd/012:00:200
197399400,32rtkit-daemon4497-21PK-Backend08:21:041
197399400,32rtkit-daemon4497-21PK-Backend08:21:031
1839252400,7chrt0-21swapper/110:50:241
104927994031,6cyclictest411rcuc/307:50:243
104927994017,11cyclictest207451-21systemd-udevd11:55:573
10492799401,11cyclictest111579-21/usr/sbin/munin07:25:203
104925994031,6cyclictest924-21NetworkManager10:50:182
104925994023,12cyclictest168791-21/usr/sbin/munin10:10:012
104925994012,5cyclictest141rcu_preempt10:33:292
104925994012,15cyclictest34-21ksoftirqd/207:25:012
104917994024,6cyclictest13-21ksoftirqd/010:10:300
104917994023,12cyclictest153474-21awk09:25:110
104917994022,6cyclictest13-21ksoftirqd/012:15:020
104917994013,23cyclictest13-21ksoftirqd/009:45:010
10492799397,10cyclictest42-21ksoftirqd/312:35:013
10492799396,10cyclictest42-21ksoftirqd/308:35:113
10492799394,19cyclictest128225-21taskset08:14:053
10492799393,10cyclictest180173-21if_enp2s010:40:203
10492799393,10cyclictest138404-21sensors_temp08:40:273
104927993915,21cyclictest1137-21runrttasks07:17:223
104927993914,7cyclictest42-21ksoftirqd/310:35:163
10492599395,15cyclictest163880-21cpu09:55:132
104925993928,8cyclictest34-21ksoftirqd/207:48:152
104925993924,11cyclictest144184-21cron08:59:592
104925993923,12cyclictest116085-21cron07:40:002
104925993914,10cyclictest34-21ksoftirqd/208:35:102
104925993911,24cyclictest2193-21gdbus08:19:572
104917993923,6cyclictest13-21ksoftirqd/009:20:130
104917993923,6cyclictest13-21ksoftirqd/008:30:150
104917993922,6cyclictest13-21ksoftirqd/008:15:020
104917993922,6cyclictest13-21ksoftirqd/007:15:000
104917993922,5cyclictest13-21ksoftirqd/008:40:300
104917993921,6cyclictest13-21ksoftirqd/012:10:020
104917993912,6cyclictest141rcu_preempt12:17:280
104917993912,6cyclictest141rcu_preempt10:02:070
104917993910,11cyclictest13-21ksoftirqd/008:05:520
1981942380,4sleep2924-21NetworkManager11:30:212
1118282380,7chrt4497-21PK-Backend07:25:231
10492799387,9cyclictest42-21ksoftirqd/309:06:303
10492799387,9cyclictest42-21ksoftirqd/308:57:093
10492799386,9cyclictest42-21ksoftirqd/308:45:123
10492799386,9cyclictest42-21ksoftirqd/308:45:113
104927993830,5cyclictest411rcuc/311:08:143
10492799382,11cyclictest195068-21taskset11:21:313
104927993821,12cyclictest177060-21seq10:30:273
104927993818,17cyclictest42-21ksoftirqd/309:04:453
104927993816,8cyclictest42-21ksoftirqd/312:05:053
10492799381,12cyclictest198902-21taskset11:32:253
10492599386,5cyclictest141rcu_preempt10:04:082
104925993828,6cyclictest1054-21snmpd11:06:352
104925993822,12cyclictest976-21rs:main2
104925993815,20cyclictest34-21ksoftirqd/210:20:012
104921993823,11cyclictest128019-21sshd08:11:291
104921993821,12cyclictest438-20systemd-journal10:34:591
104917993823,5cyclictest13-21ksoftirqd/007:35:010
104917993822,6cyclictest13-21ksoftirqd/009:35:030
104917993815,7cyclictest924-21NetworkManager10:34:280
104917993811,6cyclictest141rcu_preempt09:10:590
10492799377,16cyclictest42-21ksoftirqd/311:14:473
10492799377,10cyclictest42-21ksoftirqd/311:49:483
10492799376,8cyclictest152617-21python309:20:273
104927993723,10cyclictest42-21ksoftirqd/308:00:273
10492799372,10cyclictest0-21swapper/309:34:273
104927993720,12cyclictest438-20systemd-journal09:25:293
10492799371,11cyclictest209350-21taskset12:02:353
10492799371,11cyclictest165022-21taskset09:56:193
104925993727,6cyclictest34-21ksoftirqd/209:50:122
104925993727,6cyclictest106830-21munin-node07:15:022
104925993722,11cyclictest923-21dbus-daemon11:15:012
104925993721,12cyclictest34-21ksoftirqd/208:32:092
104925993721,12cyclictest159382-21ntp_kernel_err09:40:262
104925993720,13cyclictest1036-21gdbus07:29:572
104925993715,5cyclictest141rcu_preempt07:42:582
104925993712,9cyclictest34-21ksoftirqd/211:45:332
104925993711,6cyclictest1327962cat08:25:192
10492599371,10cyclictest127945-21awk08:10:312
10491799374,29cyclictest194697-21grep11:20:290
104917993721,6cyclictest13-21ksoftirqd/010:45:020
104917993721,6cyclictest13-21ksoftirqd/008:50:140
104917993721,12cyclictest146818-21sed09:05:210
104917993720,6cyclictest13-21ksoftirqd/011:35:000
104917993719,6cyclictest13-21ksoftirqd/010:20:020
104917993716,16cyclictest124021-21cut08:00:230
104917993711,5cyclictest141rcu_preempt11:36:260
10491799370,34cyclictest1972-21rtkit-daemon09:35:580
197399360,28rtkit-daemon1941-21systemd09:55:171
10492799366,8cyclictest42-21ksoftirqd/311:17:013
10492799366,16cyclictest42-21ksoftirqd/310:57:223
10492799363,10cyclictest42-21ksoftirqd/312:20:153
10492799362,13cyclictest187531-21python311:00:293
104927993620,11cyclictest438-20systemd-journal12:15:293
104927993618,12cyclictest438-20systemd-journal08:20:013
10492599369,14cyclictest142270-21taskset08:52:392
104925993626,6cyclictest1081-21ntpd12:15:252
104925993625,7cyclictest34-21ksoftirqd/212:00:162
104925993624,9cyclictest34-21ksoftirqd/212:35:152
104925993620,12cyclictest193025-21sensors_temp11:15:292
104925993619,13cyclictest34-21ksoftirqd/209:08:592
104925993619,13cyclictest137903-21df08:40:152
104925993618,14cyclictest160543-21grep09:45:152
104921993620,11cyclictest438-20systemd-journal10:10:561
10492199361,31cyclictest215678-21needreboot12:20:241
10492199361,14cyclictest4497-21PK-Backend09:45:441
10492199361,10cyclictest0-21swapper/107:13:321
10491799363,17cyclictest150346-21df_inode09:15:150
104917993626,6cyclictest13-21ksoftirqd/007:45:140
104917993622,10cyclictest204310-21sh11:50:010
10492799355,9cyclictest172281-21munin-run10:20:013
10492799354,10cyclictest42-21ksoftirqd/310:00:133
104927993521,10cyclictest200061-21strings11:35:223
10492799351,15cyclictest132342-21awk08:25:103
10492599359,21cyclictest154934-21sh09:30:002
104925993523,8cyclictest34-21ksoftirqd/208:50:002
104925993523,8cyclictest34-21ksoftirqd/208:50:002
104925993521,10cyclictest34-21ksoftirqd/209:40:012
104925993519,12cyclictest196664-21sshd11:25:292
104925993519,12cyclictest194875-21sed11:20:312
104925993519,12cyclictest113824-21cat07:30:292
104925993518,13cyclictest123082-21/usr/sbin/munin08:00:012
104925993517,14cyclictest209924-21cat12:05:142
10492199351,30cyclictest4497-21PK-Backend09:25:171
104921993512,11cyclictest126202-21systemd-udevd08:05:541
10492199351,13cyclictest4497-21PK-Backend10:37:261
10492199351,13cyclictest4497-21PK-Backend10:00:571
104917993524,8cyclictest13-21ksoftirqd/011:55:580
104917993520,11cyclictest161109-21awk09:45:230
104917993519,5cyclictest13-21ksoftirqd/010:25:230
104917993519,13cyclictest13-21ksoftirqd/011:00:020
104917993518,5cyclictest13-21ksoftirqd/012:25:020
104917993517,6cyclictest13-21ksoftirqd/012:35:140
10492799341,28cyclictest212485-21sed12:10:293
10492599347,6cyclictest141rcu_preempt07:50:282
104925993421,9cyclictest34-21ksoftirqd/209:30:272
10492599341,9cyclictest924-21NetworkManager10:35:202
10492599341,9cyclictest0-21swapper/209:01:312
104921993420,10cyclictest204351-21ls11:50:021
10492199341,9cyclictest4497-21PK-Backend11:02:251
10492199341,9cyclictest4497-21PK-Backend08:18:191
104921993419,11cyclictest120203-21expr07:50:141
10492199341,8cyclictest0-21swapper/111:07:121
104921993418,12cyclictest186181-21sshd10:57:291
104921993417,13cyclictest205327-21ntp_states11:50:241
104921993417,13cyclictest205327-21ntp_states11:50:241
10492199341,13cyclictest4497-21PK-Backend12:30:521
10492199341,13cyclictest4497-21PK-Backend10:16:511
10492199341,10cyclictest4497-21PK-Backend11:38:041
10491799348,8cyclictest13-21ksoftirqd/008:16:500
104917993424,5cyclictest13-21ksoftirqd/008:20:200
104917993424,5cyclictest13-21ksoftirqd/008:20:190
104917993419,6cyclictest13-21ksoftirqd/011:45:020
104917993419,11cyclictest110853-21munin-run07:25:000
104917993417,5cyclictest13-21ksoftirqd/007:40:220
104927993320,10cyclictest42-21ksoftirqd/311:40:133
10492799331,28cyclictest159486-21cut09:40:293
10492599339,5cyclictest141rcu_preempt10:29:482
104925993322,7cyclictest179931-21grep10:40:142
104925993320,8cyclictest2069-21gnome-shell10:58:092
104925993318,11cyclictest201524-21idleruntime11:40:182
104925993317,13cyclictest34-21ksoftirqd/208:04:562
104925993317,12cyclictest217954-21seq12:26:572
104925993317,12cyclictest211785-21expr12:10:132
10492599331,16cyclictest924-21NetworkManager10:45:292
10492599331,16cyclictest924-21NetworkManager10:45:292
104921993322,6cyclictest26-21ksoftirqd/109:12:121
10492199331,9cyclictest4497-21PK-Backend12:13:331
10492199331,9cyclictest4497-21PK-Backend11:28:331
10492199331,9cyclictest4497-21PK-Backend10:46:121
10492199331,9cyclictest4497-21PK-Backend10:46:121
104921993318,11cyclictest175534-21wc10:25:301
104921993318,11cyclictest174419-21cut10:25:021
104921993317,12cyclictest163582-21cron09:55:011
10492199331,4cyclictest4497-21PK-Backend07:19:531
104921993314,16cyclictest195388-21unin-run11:25:011
10492199331,12cyclictest4497-21PK-Backend09:34:371
104917993323,5cyclictest13-21ksoftirqd/012:35:010
104917993323,5cyclictest13-21ksoftirqd/011:05:270
104917993322,6cyclictest13-21ksoftirqd/012:29:590
104917993318,11cyclictest144377-21sed09:00:020
10492599329,10cyclictest34-21ksoftirqd/211:50:122
10492599329,10cyclictest34-21ksoftirqd/211:50:112
104925993217,11cyclictest169454-21which10:10:202
104921993225,4cyclictest4497-21PK-Backend09:39:391
10492199321,8cyclictest4497-21PK-Backend12:37:161
104921993216,12cyclictest207120-21wc11:55:261
10492199321,4cyclictest4497-21PK-Backend12:05:241
10492199321,4cyclictest4497-21PK-Backend08:25:151
10492199321,10cyclictest4497-21PK-Backend07:31:021
10491799327,8cyclictest13-21ksoftirqd/009:00:450
104917993223,5cyclictest188124-21idleruntime-cro11:05:010
104917993222,5cyclictest13-21ksoftirqd/007:50:210
104917993221,7cyclictest183952-21sh10:50:260
104917993217,11cyclictest923-21dbus-daemon10:49:560
104917993217,11cyclictest923-21dbus-daemon10:49:560
104917993217,11cyclictest196494-21sed11:25:250
104917993217,10cyclictest135934-21tr08:35:120
104917993213,15cyclictest13-21ksoftirqd/010:05:140
10491799321,27cyclictest4497-21PK-Backend11:50:040
10491799321,27cyclictest4497-21PK-Backend11:50:040
104921993124,4cyclictest4497-21PK-Backend11:30:221
104921993116,11cyclictest923-21dbus-daemon07:45:011
10492199311,5cyclictest4497-21PK-Backend07:56:471
10492199311,4cyclictest4497-21PK-Backend09:41:141
10492199311,4cyclictest4497-21PK-Backend08:56:311
10492199311,27cyclictest4497-21PK-Backend11:10:221
104917993113,6cyclictest141rcu_preempt11:12:090
104917993110,6cyclictest141rcu_preempt10:25:000
104921993023,4cyclictest4497-21PK-Backend11:15:031
104921993023,4cyclictest4497-21PK-Backend09:05:171
104921993023,4cyclictest4497-21PK-Backend09:03:431
104921993020,5cyclictest26-21ksoftirqd/108:40:201
104921993017,9cyclictest213833-21grep12:15:221
104921993017,10cyclictest209552-21dump-pmu-power12:05:001
10492199301,10cyclictest4497-21PK-Backend09:20:141
10492199295,5cyclictest141rcu_preempt10:05:291
104921992918,8cyclictest4497-21PK-Backend08:54:261
104921992916,8cyclictest438-20systemd-journal07:20:191
10492199291,14cyclictest4497-21PK-Backend12:25:371
10492199291,10cyclictest4497-21PK-Backend11:43:151
10492199286,19cyclictest124797-21sh08:05:001
104921992818,7cyclictest4497-21PK-Backend10:41:131
10492199281,12cyclictest4497-21PK-Backend08:36:341
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional