You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 23:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Jun 18, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
219152242182,49sleep00-21swapper/007:05:290
219372238164,24sleep20-21swapper/207:05:462
219452205168,24sleep30-21swapper/307:05:513
220602203166,25sleep10-21swapper/107:07:231
215982540,2sleep20-21swapper/210:45:192
215982540,2sleep20-21swapper/210:45:192
307112500,1sleep30-21swapper/311:05:233
19122500,4sleep21904-21munin-run12:30:022
272762470,4sleep12235499cyclictest09:45:141
2234899346,13cyclictest16674-21munin-run11:50:010
2234899337,13cyclictest30221-21munin-run08:40:010
9862320,4sleep10-21swapper/112:25:231
323232320,3sleep02234899cyclictest09:55:250
22360993011,10cyclictest0-21swapper/210:00:012
2236099292,6cyclictest0-21swapper/211:17:182
2236099291,5cyclictest0-21swapper/209:51:382
2236099271,7cyclictest0-21swapper/211:10:442
2236099271,5cyclictest0-21swapper/210:03:562
2234899279,6cyclictest6698-21if_err_enp2s011:25:180
2234899270,25cyclictest0-21swapper/011:11:280
2236599268,3cyclictest27054-21munin-run11:00:023
2236099261,6cyclictest0-21swapper/210:41:572
2236099261,5cyclictest0-21swapper/209:25:372
2236099261,4cyclictest0-21swapper/212:03:492
22354992622,3cyclictest0-21swapper/111:11:271
22348992620,3cyclictest10562-21sed09:05:250
2234899261,17cyclictest0-21swapper/011:55:110
2234899261,17cyclictest0-21swapper/011:55:110
2234899261,13cyclictest0-21swapper/010:19:030
2234899260,23cyclictest0-21swapper/011:20:140
2234899260,12cyclictest23642-21idleruntime-cro08:25:000
2236599251,9cyclictest0-21swapper/312:15:213
2236099251,7cyclictest0-21swapper/212:36:552
2236099251,4cyclictest0-21swapper/211:46:492
2236099251,4cyclictest0-21swapper/210:14:472
22354992512,8cyclictest0-21swapper/110:13:401
2235499251,13cyclictest87450irq/127-enp1s0-11:23:431
22354992511,12cyclictest1204-21kworker/u8:311:53:111
2235499251,10cyclictest170272sh08:10:011
2234899258,9cyclictest0-21swapper/010:30:020
2234899258,10cyclictest22531-21/usr/sbin/munin12:00:240
22348992521,3cyclictest0-21swapper/008:15:190
22348992521,2cyclictest30088-21grep09:50:240
22348992513,2cyclictest0-21swapper/010:12:560
2234899251,22cyclictest0-21swapper/010:39:200
2234899251,16cyclictest0-21swapper/011:33:420
2234899251,14cyclictest0-21swapper/009:46:420
126382250,4sleep123-21ksoftirqd/111:40:111
2236599241,5cyclictest0-21swapper/311:30:193
2236099241,8cyclictest0-21swapper/210:35:252
2236099241,2cyclictest0-21swapper/211:20:132
22360992411,2cyclictest30-21rcuc/210:15:252
22354992413,2cyclictest27481-21expr11:00:111
2235499240,3cyclictest0-21swapper/111:46:071
22348992418,3cyclictest17259-21/usr/sbin/munin08:10:220
2234899241,18cyclictest0-21swapper/009:26:250
2234899241,17cyclictest0-21swapper/007:23:030
2234899241,13cyclictest0-21swapper/011:40:010
2234899241,11cyclictest23710-21kernelversion10:50:170
22365992310,3cyclictest361-21cat07:30:163
2236099235,7cyclictest0-21swapper/212:20:032
2236099233,9cyclictest2259650kworker/u8:011:42:272
2236099231,8cyclictest0-21swapper/212:11:072
2236099231,6cyclictest0-21swapper/211:05:272
2236099231,6cyclictest0-21swapper/210:08:482
2236099231,5cyclictest0-21swapper/210:33:392
2235499236,4cyclictest23950-21ntp_states10:50:201
2235499231,20cyclictest0-21swapper/107:35:161
22354992311,7cyclictest0-21swapper/109:20:031
2234899236,3cyclictest0-21swapper/010:01:160
22348992319,3cyclictest0-21swapper/012:23:410
22348992319,3cyclictest0-21swapper/012:10:190
22348992312,6cyclictest0-21swapper/011:05:230
2234899231,15cyclictest0-21swapper/009:15:170
2236599222,14cyclictest87350irq/126-enp1s0-09:25:253
2236599221,7cyclictest0-21swapper/311:02:363
2236599221,6cyclictest87350irq/126-enp1s0-12:05:063
2236599221,6cyclictest0-21swapper/309:57:543
2236599221,5cyclictest0-21swapper/309:41:523
2236099228,3cyclictest32-21ksoftirqd/211:25:252
2236099221,7cyclictest0-21swapper/209:37:492
2236099221,7cyclictest0-21swapper/209:37:492
2236099221,6cyclictest22596-21kworker/u8:011:37:172
2236099221,6cyclictest0-21swapper/209:45:012
2236099221,6cyclictest0-21swapper/209:33:282
2236099221,5cyclictest0-21swapper/211:57:122
2236099221,5cyclictest0-21swapper/211:57:122
2236099221,5cyclictest0-21swapper/211:03:132
2236099221,11cyclictest24805-21df09:40:122
2235499224,15cyclictest0-21swapper/111:00:011
2235499224,10cyclictest25645-21ps09:40:231
2235499221,4cyclictest0-21swapper/110:46:161
2235499221,4cyclictest0-21swapper/110:46:161
2235499221,4cyclictest0-21swapper/110:25:161
2235499221,3cyclictest30714-21cat08:40:111
2235499221,3cyclictest25410-21diskstats07:15:041
22354992212,6cyclictest0-21swapper/111:37:531
2235499221,18cyclictest21333-21smart_sda09:30:231
2235499221,18cyclictest1614-21expr11:15:101
2234899228,9cyclictest0-21swapper/007:30:360
2234899228,2cyclictest11-21rcuc/009:30:170
2234899227,5cyclictest0-21swapper/009:39:550
2234899227,5cyclictest0-21swapper/009:39:550
2234899227,10cyclictest0-21swapper/007:25:240
2234899223,10cyclictest87350irq/126-enp1s0-12:20:010
2234899221,3cyclictest18827-21df_inode10:40:130
22348992212,6cyclictest0-21swapper/011:01:370
2234899221,15cyclictest0-21swapper/012:36:040
22348992211,3cyclictest0-21swapper/010:45:170
22348992211,3cyclictest0-21swapper/010:45:170
2234899221,11cyclictest0-21swapper/009:10:420
2234899221,10cyclictest6018-21sshd07:44:530
2234899221,10cyclictest11950-21switchtime07:55:240
2234899221,10cyclictest0-21swapper/010:57:310
2234899221,10cyclictest0-21swapper/010:05:120
282192210,2sleep3401ktimersoftd/309:45:243
2236599218,3cyclictest19476-21needreboot10:40:193
2236599218,3cyclictest0-21swapper/309:04:463
2236599217,3cyclictest30919-21kernelversion12:20:163
2236599211,5cyclictest0-21swapper/311:22:253
2236599211,12cyclictest3104-21/usr/sbin/munin10:05:173
22365992111,2cyclictest1281-21awk11:15:023
2236099219,2cyclictest30-21rcuc/210:20:242
2236099213,4cyclictest11407-21kworker/u8:210:53:032
2236099212,5cyclictest790-21in:imjournal09:15:172
2236099211,7cyclictest6291-21cpu07:45:122
2236099211,7cyclictest25981-21cat08:30:022
2236099211,7cyclictest25981-21cat08:30:022
2236099211,7cyclictest11056-21diskmemload10:30:012
2236099211,7cyclictest0-21swapper/210:58:162
2236099211,7cyclictest0-21swapper/209:24:562
2236099211,7cyclictest0-21swapper/209:12:072
2236099211,6cyclictest9057-21cat09:05:012
22354992117,3cyclictest0-21swapper/108:29:471
22354992117,3cyclictest0-21swapper/108:29:471
22354992116,3cyclictest0-21swapper/109:50:581
22354992112,3cyclictest6255-21df_inode11:25:131
2235499211,18cyclictest0-21swapper/107:20:111
2235499211,17cyclictest31552-21fschecks_time09:55:161
2235499211,17cyclictest0-21swapper/108:55:121
2235499211,11cyclictest0-21swapper/107:25:141
22354992110,2cyclictest0-21swapper/109:10:111
2235499210,19cyclictest0-21swapper/112:20:201
2234899219,8cyclictest0-21swapper/011:16:350
2234899217,3cyclictest25642-21ps09:40:230
22348992118,2cyclictest0-21swapper/011:50:180
22348992117,3cyclictest0-21swapper/009:21:150
22348992117,3cyclictest0-21swapper/007:48:380
2234899211,14cyclictest0-21swapper/010:22:190
2234899211,12cyclictest0-21swapper/012:25:440
2234899211,11cyclictest0-21swapper/010:25:110
2234899210,6cyclictest13651-21pmu-power11:40:220
2236599208,3cyclictest5469-21grep12:35:213
2236599205,4cyclictest0-21swapper/312:29:543
2236599201,4cyclictest0-21swapper/310:54:253
2236599201,4cyclictest0-21swapper/310:38:513
22365992012,3cyclictest0-21swapper/311:27:093
2236599201,11cyclictest27643-21/usr/sbin/munin07:20:253
2236599201,11cyclictest2479-21/usr/sbin/munin08:50:143
2236599201,11cyclictest14703-21/usr/sbin/munin11:45:173
2236599201,10cyclictest30579-21tr08:40:123
2236099209,2cyclictest30-21rcuc/207:55:152
2236099206,3cyclictest87350irq/126-enp1s0-08:05:122
2236099205,4cyclictest9091-21/usr/sbin/munin11:30:192
2236099204,4cyclictest8564-1kworker/u9:112:06:222
2236099204,4cyclictest29732-21cat07:25:012
2236099202,5cyclictest29832-21users08:35:252
2236099201,7cyclictest4918-21wc07:40:182
2236099201,6cyclictest28747-21grep12:15:202
2236099201,6cyclictest20836-21timerandwakeup08:15:252
2236099201,6cyclictest0-21swapper/212:30:262
2235499207,8cyclictest0-21swapper/111:07:161
2235499206,8cyclictest0-21swapper/109:37:061
2235499206,8cyclictest0-21swapper/109:37:051
2235499204,9cyclictest0-21swapper/110:32:391
22354992016,3cyclictest0-21swapper/108:45:141
22354992016,3cyclictest0-21swapper/107:44:331
22354992015,3cyclictest29416-21grep08:35:201
2235499201,4cyclictest2882-21if_enp2s012:30:161
22354992014,3cyclictest5075-21dump-pmu-power10:10:021
2235499201,17cyclictest11682-21awk10:25:011
2235499201,17cyclictest0-21swapper/108:50:521
2235499201,16cyclictest21525-21latency_hist08:20:021
2235499201,14cyclictest0-21swapper/110:43:291
2235499200,5cyclictest0-21swapper/109:15:221
2235499200,4cyclictest0-21swapper/110:15:121
2235499200,3cyclictest0-21swapper/111:55:171
2235499200,3cyclictest0-21swapper/111:55:171
2235499200,14cyclictest0-21swapper/112:00:241
2234899201,9cyclictest0-21swapper/008:00:130
22348992017,2cyclictest0-21swapper/008:55:110
22348992017,2cyclictest0-21swapper/007:10:210
2234899201,6cyclictest0-21swapper/012:30:470
2234899201,6cyclictest0-21swapper/008:51:570
2234899201,5cyclictest7011-21cpuspeed_turbos09:00:100
2234899200,6cyclictest0-21swapper/012:06:410
2234899200,10cyclictest0-21swapper/008:42:570
2236599194,5cyclictest20704-21/usr/sbin/munin08:15:223
2236599192,5cyclictest1630750kworker/u8:111:50:263
2236599191,9cyclictest0-21swapper/307:55:023
2236599191,6cyclictest0-21swapper/308:45:243
2236599191,4cyclictest0-21swapper/310:33:563
22365991914,3cyclictest16014-21grep08:05:183
2236599191,10cyclictest808-21snmpd11:18:163
2236599191,10cyclictest25708-21/usr/sbin/munin12:10:283
2236599191,10cyclictest25445-21/usr/sbin/munin07:15:183
2236599191,10cyclictest23856-21/usr/sbin/munin08:25:243
2236599191,10cyclictest23856-21/usr/sbin/munin08:25:243
2236599191,10cyclictest19099-21/usr/sbin/munin11:55:153
2236599191,10cyclictest19099-21/usr/sbin/munin11:55:153
2236099195,3cyclictest87350irq/126-enp1s0-11:50:132
2236099195,3cyclictest303042grep07:25:152
2236099194,3cyclictest87350irq/126-enp1s0-08:45:202
2236099192,6cyclictest0-21swapper/208:20:172
2236099192,5cyclictest30-21rcuc/207:30:222
2236099191,7cyclictest0-21swapper/207:35:032
2236099191,6cyclictest31940-21unixbench_multi08:40:252
2236099191,6cyclictest0-21swapper/207:10:202
2236099191,5cyclictest27414-21latency_hist07:20:022
2236099191,5cyclictest26406-21entropy08:30:132
22360991913,3cyclictest17907-21head08:10:162
2235499195,4cyclictest0-21swapper/110:35:371
2235499195,3cyclictest184262proc_pri08:10:221
22354991915,3cyclictest0-21swapper/108:24:581
22354991914,3cyclictest8791-21hddtemp_smartct11:30:161
22354991914,3cyclictest7787-21memory09:00:161
2235499191,3cyclictest23857-21diskstats12:05:151
2235499191,2cyclictest0-21swapper/112:16:061
2235499191,2cyclictest0-21swapper/107:51:241
2235499191,17cyclictest0-21swapper/108:00:331
2235499191,16cyclictest0-21swapper/109:06:301
2235499191,16cyclictest0-21swapper/108:30:591
2235499191,16cyclictest0-21swapper/107:30:591
2235499191,16cyclictest0-21swapper/107:10:081
22354991910,2cyclictest0-21swapper/112:11:491
2235499190,5cyclictest0-21swapper/112:36:231
2234899191,5cyclictest23856-21/usr/sbin/munin08:25:110
2234899191,5cyclictest23856-21/usr/sbin/munin08:25:110
2234899191,17cyclictest0-21swapper/008:05:140
114732191,15sleep111574-21memory07:55:191
230432180,5sleep339-21rcuc/308:20:243
2236599188,3cyclictest12509-21/usr/sbin/munin11:40:173
2236599187,3cyclictest808-21snmpd08:37:483
2236599186,3cyclictest23846-21switchtime07:10:223
2236599186,3cyclictest23784-21tail09:35:253
2236599186,3cyclictest23784-21tail09:35:243
2236599185,5cyclictest14080-21grep09:15:163
2236599184,3cyclictest87650irq/129-enp1s0-12:30:253
2236599182,7cyclictest0-21swapper/312:00:213
2236599181,7cyclictest17324-21ls08:10:113
2236599181,7cyclictest0-21swapper/310:47:033
2236599181,7cyclictest0-21swapper/310:47:033
2236599181,7cyclictest0-21swapper/310:21:043
2236599181,7cyclictest0-21swapper/309:54:363
2236599181,15cyclictest0-21swapper/307:41:093
2236599181,10cyclictest839-21lldpd07:27:013
2236599180,5cyclictest0-21swapper/309:31:503
2236099183,3cyclictest13529-21hddtemp_smartct08:00:152
2236099181,7cyclictest0-21swapper/208:55:202
2236099181,5cyclictest9611-21fschecks_time09:05:142
2236099181,4cyclictest0-21swapper/207:50:162
2236099180,7cyclictest0-21swapper/208:50:132
22354991815,2cyclictest0-21swapper/109:25:171
22354991812,3cyclictest1208-21df10:00:131
2235499181,16cyclictest0-21swapper/107:46:111
2234899184,2cyclictest0-21swapper/007:35:170
2234899181,8cyclictest0-21swapper/007:16:480
2234899181,4cyclictest10099-21sshd07:50:490
2234899181,14cyclictest530-21sshd08:45:120
2234899180,4cyclictest0-21swapper/008:30:110
2236599176,3cyclictest87350irq/126-enp1s0-11:35:223
2236599176,3cyclictest8465-21/usr/sbin/munin07:50:213
2236599175,3cyclictest0-21swapper/310:10:123
2236599174,4cyclictest3736-21sshd07:38:473
2236599171,9cyclictest0-21swapper/307:45:133
2236599171,7cyclictest853-21/usr/sbin/munin10:00:123
2236599171,7cyclictest26048-21/usr/sbin/munin08:30:243
2236599171,7cyclictest11898-21/usr/sbin/munin10:25:173
2236599171,7cyclictest0-21swapper/309:10:463
2236599171,6cyclictest0-21swapper/310:15:183
2236599171,6cyclictest0-21swapper/309:05:253
2236599171,5cyclictest0-21swapper/309:21:083
2236599171,4cyclictest808-21snmpd08:57:523
2236599171,13cyclictest14408-21grep08:00:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional