You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 16:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 06, 2025 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
217262248164,71sleep10-21swapper/107:07:011
218162236162,22sleep30-21swapper/307:08:113
216642217168,37sleep20-21swapper/207:06:132
219392203166,25sleep00-21swapper/007:09:440
2193921610,5sleep32206999cyclictest08:20:163
22064991081,105cyclictest0-21swapper/209:49:182
164662570,2sleep10-21swapper/111:50:011
316772500,2sleep20-21swapper/208:40:262
196432480,4sleep339-21rcuc/309:30:013
293882470,1sleep30-21swapper/308:35:253
293882470,1sleep30-21swapper/308:35:253
2206999421,11cyclictest5015-21sed12:35:363
22064994239,2cyclictest0-21swapper/210:27:562
22069994119,19cyclictest0-21swapper/307:54:093
22064994036,3cyclictest0-21swapper/210:50:562
29592380,3sleep10-21swapper/112:30:231
2205299378,14cyclictest29313-21munin-run07:25:010
2205299378,13cyclictest6078-21munin-run12:40:010
167592360,7sleep011-21rcuc/008:10:020
22069993420,3cyclictest0-21swapper/308:59:073
2206999341,19cyclictest0-21swapper/310:38:513
2206999330,2cyclictest86550irq/126-enp1s0-09:52:503
22069993220,7cyclictest0-21swapper/307:28:553
22069993215,3cyclictest0-21swapper/307:43:373
2205299329,11cyclictest29950-21awk08:40:010
2205299329,11cyclictest29950-21awk08:40:010
2206999313,2cyclictest86750irq/128-enp1s0-11:44:483
22069993115,3cyclictest0-21swapper/308:12:173
22069993113,15cyclictest0-21swapper/308:00:413
2206999311,14cyclictest0-21swapper/310:18:223
2206999310,10cyclictest0-21swapper/307:17:083
200412310,4sleep02205299cyclictest09:30:180
243282290,4sleep341-21ksoftirqd/307:15:013
2206999296,13cyclictest28664-21strings12:15:203
2206999291,1cyclictest0-21swapper/310:34:493
22069992911,3cyclictest15742-21df_inode09:20:143
2206999290,11cyclictest0-21swapper/311:08:293
316562280,6sleep230-21rcuc/207:30:012
22069992820,3cyclictest0-21swapper/307:23:053
2206999281,13cyclictest7827-21fgrep09:00:263
2206999281,13cyclictest0-21swapper/307:46:053
2206999281,12cyclictest0-21swapper/309:36:383
22069992810,5cyclictest10058-21sensors_temp09:05:263
2206999280,8cyclictest0-21swapper/310:43:193
2205299285,11cyclictest26818-21idleruntime-cro11:00:010
2206999279,8cyclictest0-21swapper/308:31:213
2206999278,9cyclictest0-21swapper/308:07:473
2206999271,16cyclictest0-21swapper/308:40:133
22069992711,4cyclictest31775-21df11:10:133
2206999271,14cyclictest0-21swapper/311:20:183
2206999271,13cyclictest9454-21users11:30:283
2206999271,12cyclictest0-21swapper/311:39:103
2206999271,11cyclictest2220-21users10:00:283
2206999270,11cyclictest0-21swapper/307:36:113
2205799272,4cyclictest11426-21ntp_states07:55:241
2205799271,23cyclictest0-21swapper/112:21:101
2206999269,3cyclictest12972-21munin-run09:15:023
2206999263,7cyclictest86850irq/129-enp1s0-09:31:523
2206999261,18cyclictest0-21swapper/310:53:413
2205799261,17cyclictest14237-21munin-run11:45:011
22052992611,3cyclictest0-21swapper/007:35:260
2206999257,15cyclictest0-21swapper/310:25:163
2206999256,7cyclictest0-21swapper/312:05:043
2206999251,9cyclictest0-21swapper/308:26:373
22069992512,2cyclictest1911-21runrttasks08:49:273
22069992511,2cyclictest0-21swapper/311:25:273
2205799252,3cyclictest5445-21munin-run11:25:021
2205799250,18cyclictest0-21swapper/112:15:441
2205299251,6cyclictest0-21swapper/009:27:040
5292240,3sleep1533-21unixbench_singl11:10:271
2206999247,3cyclictest0-21swapper/307:55:573
2206999246,3cyclictest0-21swapper/309:57:103
2206999241,3cyclictest0-21swapper/311:51:383
2206999241,3cyclictest0-21swapper/311:51:383
2206999241,21cyclictest0-21swapper/310:08:123
2206999241,20cyclictest0-21swapper/312:04:023
2206999241,10cyclictest0-21swapper/310:13:263
22069992410,12cyclictest0-21swapper/308:15:473
2205799241,20cyclictest10734-21hddtemp_smartct11:35:181
2205299241,16cyclictest0-21swapper/010:10:190
2205299241,12cyclictest4173-21munin-run08:55:020
31912231,18sleep13395-21cut10:05:171
2206999238,3cyclictest2825-21load12:30:223
2206999238,3cyclictest10772-21diskmemload11:15:073
2206999231,15cyclictest0-21swapper/309:15:243
2206999230,1cyclictest0-21swapper/312:20:283
22057992317,3cyclictest7509-21memory09:00:211
22057992317,3cyclictest23702-21ntp_states10:50:241
22057992315,3cyclictest0-21swapper/108:22:001
22057992313,3cyclictest0-21swapper/107:40:221
2205799231,19cyclictest8508-21diskstats07:50:151
2205799231,19cyclictest28412-21ntp_states07:20:221
2205799231,17cyclictest0-21swapper/109:43:431
2205799231,17cyclictest0-21swapper/109:43:431
2205799231,16cyclictest0-21swapper/110:26:051
2205799231,16cyclictest0-21swapper/107:11:201
2205799231,11cyclictest18627-21latency_hist11:55:021
2205799231,11cyclictest18627-21latency_hist11:55:021
2205799230,13cyclictest21163-21cpuspeed_turbos12:00:151
2205299233,11cyclictest23585-21cpu08:25:140
2206999226,3cyclictest0-21swapper/312:12:063
2206999221,2cyclictest20621-40dnf11:57:323
2206999221,14cyclictest0-21swapper/308:52:533
2206999221,10cyclictest0-21swapper/307:31:173
2206999221,10cyclictest0-21swapper/307:31:173
2206999220,8cyclictest0-21swapper/309:42:243
2206999220,8cyclictest0-21swapper/309:42:243
2206999220,2cyclictest86750irq/128-enp1s0-11:45:223
2205799222,11cyclictest6754-21ntp_states11:25:371
2205799221,3cyclictest26119-21tail12:10:151
22057992212,3cyclictest27658-21smart_sda09:45:251
2205799221,18cyclictest20519-21sshd11:56:121
2205799221,15cyclictest0-21swapper/108:46:361
2205799221,15cyclictest0-21swapper/108:30:111
2205799221,15cyclictest0-21swapper/108:06:471
2205799221,13cyclictest0-21swapper/112:25:021
22057992210,9cyclictest0-21swapper/108:35:151
22057992210,9cyclictest0-21swapper/108:35:151
2205799220,6cyclictest0-21swapper/109:52:541
2205299223,2cyclictest17569-21grep08:10:160
2205299221,4cyclictest0-21swapper/011:30:010
2205299221,11cyclictest0-21swapper/007:45:270
2206999216,3cyclictest0-21swapper/310:48:133
2206999216,3cyclictest0-21swapper/310:48:133
2206999215,6cyclictest0-21swapper/312:29:123
2206999211,10cyclictest0-21swapper/311:03:493
2206499217,6cyclictest12756-21grep11:40:162
2206499214,6cyclictest0-21swapper/209:22:442
22064992110,3cyclictest21092-21hddtemp_smartct10:45:152
22064992110,3cyclictest21092-21hddtemp_smartct10:45:152
2205799219,4cyclictest18577-21diskstats10:40:171
2205799219,3cyclictest656-21cpuspeed_turbos10:00:141
2205799219,3cyclictest25159-21users08:25:281
2205799212,10cyclictest23623-21diskstats12:05:161
22057992116,3cyclictest9415-21grep11:30:271
2205799211,18cyclictest0-21swapper/110:35:281
2205799211,18cyclictest0-21swapper/107:25:151
2205799211,15cyclictest0-21swapper/110:20:471
2205799211,15cyclictest0-21swapper/110:14:401
2205799211,15cyclictest0-21swapper/109:39:031
2205799211,15cyclictest0-21swapper/109:24:541
2205799211,15cyclictest0-21swapper/108:50:591
2205799211,15cyclictest0-21swapper/108:14:101
2205799211,15cyclictest0-21swapper/107:30:091
2205799211,15cyclictest0-21swapper/107:30:091
2205799211,11cyclictest8302-21ntp_states10:15:221
2205799211,10cyclictest0-21swapper/109:15:141
2205799210,15cyclictest0-21swapper/111:01:011
2205799210,15cyclictest0-21swapper/107:47:231
2205299216,9cyclictest0-21swapper/011:01:360
2205299212,16cyclictest12330-21munin-run08:00:020
2205299211,9cyclictest10772-21diskmemload11:05:150
2205299211,9cyclictest0-21swapper/007:17:380
22052992111,3cyclictest5457-21awk11:25:020
2205299211,12cyclictest0-21swapper/012:17:420
2205299211,10cyclictest0-21swapper/009:20:180
2206999201,13cyclictest0-21swapper/310:23:103
2206999201,10cyclictest10772-21diskmemload10:55:093
2205799209,3cyclictest13550-21head08:00:211
2205799202,15cyclictest31138-21diskstats09:55:141
22057992016,3cyclictest0-21swapper/109:28:241
2205799201,5cyclictest0-21swapper/112:39:091
22057992013,2cyclictest0-21swapper/108:56:191
22057992013,2cyclictest0-21swapper/108:16:481
22057992012,3cyclictest0-21swapper/110:55:121
2205799201,18cyclictest0-21swapper/109:09:521
2205799201,17cyclictest0-21swapper/111:06:061
2205799201,15cyclictest0-21swapper/110:30:581
2205799201,15cyclictest0-21swapper/107:36:161
2205799201,14cyclictest0-21swapper/110:48:541
2205799201,14cyclictest0-21swapper/110:48:541
2205799201,14cyclictest0-21swapper/109:31:501
2205799201,14cyclictest0-21swapper/108:41:521
2205799201,10cyclictest0-21swapper/111:18:391
2205299201,8cyclictest0-21swapper/008:42:010
2205299201,2cyclictest14679-21sed09:15:270
2205299201,13cyclictest0-21swapper/010:30:230
2205299201,13cyclictest0-21swapper/008:30:260
2205299201,11cyclictest24197-21/usr/sbin/munin09:40:220
2205299201,11cyclictest24197-21/usr/sbin/munin09:40:220
2205299201,11cyclictest23444-21irqstats10:50:200
2205299201,11cyclictest13491-21gunzip11:40:250
2205299201,10cyclictest26779-21chrt09:45:160
2205299201,10cyclictest12544-21memory10:25:210
2206499193,3cyclictest0-21swapper/211:53:562
2206499193,3cyclictest0-21swapper/211:53:562
2206499191,8cyclictest581-21systemd-journal07:25:002
22064991912,4cyclictest2242-21gltestperf07:35:172
2206499191,10cyclictest20769-21runrttasks11:59:232
22057991915,3cyclictest0-21swapper/109:12:481
2205799191,2cyclictest0-21swapper/107:15:171
2205299199,2cyclictest22195-21kernelversion08:20:190
2205299198,5cyclictest4095-21kworker/0:309:08:490
2205299195,5cyclictest6595-21/usr/sbin/munin09:00:170
2205299193,6cyclictest10889-21kernelversion11:35:200
2205299192,8cyclictest18859-21/usr/sbin/munin11:55:170
2205299192,10cyclictest20790-21sshd08:15:440
2205299191,10cyclictest3755-21memory10:05:200
2205299190,4cyclictest0-21swapper/009:35:150
186712191,2sleep20-21swapper/208:11:442
95772181,2sleep00-21swapper/011:31:510
60562181,2sleep00-21swapper/008:56:110
277142181,3sleep20-21swapper/208:34:532
22069991813,3cyclictest27312-21memory09:45:203
2206499187,3cyclictest5222-21cstates10:10:132
2206499186,3cyclictest12050irq/123-ahci[0010:40:002
2206499183,10cyclictest1266-21sendmail09:56:332
2206499181,8cyclictest15390-21kworker/2:211:27:372
2206499181,8cyclictest14889-21ntpq10:30:222
2206499181,7cyclictest0-21swapper/210:05:292
2206499181,6cyclictest0-21swapper/212:21:432
22064991813,3cyclictest26207-21ntp_states07:15:212
22064991813,3cyclictest18308-21cat10:40:142
22064991813,3cyclictest15418-21expr08:05:162
2206499181,10cyclictest784-21NetworkManager11:10:352
2206499180,3cyclictest0-21swapper/209:17:272
2205299184,6cyclictest25477-21/usr/sbin/munin12:10:170
2205299183,9cyclictest11-21rcuc/008:00:260
2205299181,9cyclictest21174-21if_enp2s010:45:170
2205299181,9cyclictest21174-21if_enp2s010:45:170
2205299181,9cyclictest0-21swapper/010:20:240
2205299181,8cyclictest30782-21/usr/sbin/munin07:25:240
2205299181,8cyclictest0-21swapper/012:31:410
2205299181,7cyclictest1024-21turbostat11:15:020
2205299181,6cyclictest0-21swapper/012:23:380
2205299181,6cyclictest0-21swapper/012:05:150
2205299181,6cyclictest0-21swapper/011:49:020
2205299181,6cyclictest0-21swapper/011:15:240
2205299181,5cyclictest0-21swapper/010:36:480
2205299181,4cyclictest30454-21sshd09:53:450
2205299181,4cyclictest21398-21df12:00:160
22052991814,3cyclictest0-21swapper/012:26:290
22052991813,3cyclictest19771-21grep10:40:280
22052991812,3cyclictest7926-21cat10:15:170
22052991812,3cyclictest22414-21tr07:10:130
2205299181,10cyclictest0-21swapper/010:00:280
18972181,2sleep00-21swapper/008:49:170
13112181,2sleep20-21swapper/207:34:592
13112181,2sleep20-21swapper/207:34:592
116312180,1sleep20-21swapper/207:55:282
98512171,2sleep20-21swapper/211:35:012
54932171,2sleep00-21swapper/007:42:010
304432171,2sleep20-21swapper/209:53:372
25892171,2sleep20-21swapper/210:04:532
2206499171,7cyclictest27813-21cat11:00:192
2206499171,7cyclictest0-21swapper/212:35:222
2206499171,6cyclictest0-21swapper/209:40:132
2206499171,6cyclictest0-21swapper/209:40:132
2206499171,6cyclictest0-21swapper/209:35:522
2206499171,6cyclictest0-21swapper/209:27:532
2206499171,5cyclictest0-21swapper/212:15:122
2206499171,5cyclictest0-21swapper/211:30:532
2206499171,5cyclictest0-21swapper/210:58:292
2206499171,3cyclictest0-21swapper/208:00:132
22064991712,3cyclictest8820-21users10:15:262
22064991712,3cyclictest29239-21/usr/sbin/munin11:05:132
22064991712,3cyclictest2820-21load12:30:212
22064991712,3cyclictest24277-21netstat12:05:222
22064991712,3cyclictest22522-21awk08:20:232
22064991712,3cyclictest21023-21sshd08:18:432
22064991712,3cyclictest16428-21idleruntime-cro11:50:002
22064991712,3cyclictest12071-21ntp_states09:10:222
22064991711,3cyclictest8631-21grep07:50:172
22064991711,3cyclictest699-21sh12:25:242
22064991711,3cyclictest23853-21cat08:25:142
22064991710,4cyclictest28085-21tr08:35:122
22064991710,4cyclictest28085-21tr08:35:122
22064991710,4cyclictest10137-21ls10:20:172
2205299175,3cyclictest22552-1kworker/u9:109:55:030
2205299171,7cyclictest8161-21/usr/sbin/munin07:50:240
2205299171,6cyclictest0-21swapper/011:50:410
2205299171,6cyclictest0-21swapper/011:50:410
2205299171,6cyclictest0-21swapper/009:10:140
2206499161,5cyclictest0-21swapper/212:11:252
2206499161,5cyclictest0-21swapper/211:23:082
2206499161,2cyclictest0-21swapper/207:14:202
2206499161,14cyclictest0-21swapper/209:07:012
2206499161,14cyclictest0-21swapper/209:03:372
2206499161,14cyclictest0-21swapper/208:47:302
22064991611,3cyclictest6479-21cut07:45:152
22064991611,3cyclictest21689-21grep12:00:182
2205299161,6cyclictest0-21swapper/007:30:200
2205299161,6cyclictest0-21swapper/007:30:200
2206499156,7cyclictest1923-21gltestperf11:15:152
2206499151,5cyclictest0-21swapper/208:55:222
2206499151,4cyclictest0-21swapper/209:30:162
22064991510,3cyclictest4127-21forks07:40:152
22064991510,3cyclictest2478-21cat08:50:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional