You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 12:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Mar 19, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3032322580,4sleep339-21rcuc/319:05:263
304312235164,59sleep10-21swapper/119:06:181
307002230161,57sleep00-21swapper/019:09:470
305822219152,54sleep20-21swapper/219:08:152
406021470,3sleep33082499cyclictest23:00:163
3253721320,4sleep33082499cyclictest21:38:053
30824991070,2cyclictest0-21swapper/323:55:413
30824997572,2cyclictest0-21swapper/323:42:483
30824997471,2cyclictest0-21swapper/321:49:423
30824996865,2cyclictest0-21swapper/322:30:093
130082610,3sleep113011-21latency_hist20:55:011
187132590,1sleep318715-21grep21:05:223
27612540,2sleep230-21rcuc/222:55:252
30824995351,1cyclictest0-21swapper/320:15:133
30824995350,2cyclictest0-21swapper/320:30:143
126022520,2sleep30-21swapper/323:20:153
59922490,1sleep10-21swapper/121:50:191
219222480,1sleep30-21swapper/321:15:023
30824994138,2cyclictest0-21swapper/300:32:413
3080799355,13cyclictest6937-21idleruntime-cro21:55:020
3081399341,25cyclictest0-21swapper/119:51:371
3080799338,12cyclictest3811-21munin-run00:15:020
228162330,4sleep121-21rcuc/122:30:121
30824993229,2cyclictest0-21swapper/319:49:523
3081399321,23cyclictest0-21swapper/119:22:251
30824993128,2cyclictest0-21swapper/319:33:343
42182300,3sleep230-21rcuc/200:15:122
3081399301,13cyclictest0-21swapper/119:48:521
3081399301,10cyclictest0-21swapper/120:18:311
3080799303,13cyclictest17321-21munin-run21:05:000
30807993011,10cyclictest19510-21diskmemload21:34:530
3081399291,12cyclictest0-21swapper/123:42:281
3081399291,11cyclictest0-21swapper/120:37:351
3081399291,11cyclictest0-21swapper/119:27:141
3081399281,18cyclictest7497-21df_inode21:55:131
3081399281,12cyclictest0-21swapper/120:10:451
3081399281,11cyclictest0-21swapper/119:17:311
3081399280,21cyclictest0-21swapper/122:56:131
3081399271,9cyclictest0-21swapper/123:51:331
3081399271,6cyclictest0-21swapper/123:36:141
3081399271,6cyclictest0-21swapper/123:36:141
3081399271,12cyclictest0-21swapper/120:45:311
3081399271,11cyclictest0-21swapper/120:26:311
3081399271,11cyclictest0-21swapper/120:01:151
3081399271,11cyclictest0-21swapper/119:35:361
3081399271,11cyclictest0-21swapper/100:25:371
3081399271,10cyclictest15392-21seq22:10:451
3081399271,10cyclictest0-21swapper/123:05:271
3081399270,12cyclictest0-21swapper/100:14:351
3081399270,11cyclictest0-21swapper/122:16:471
3081399270,10cyclictest0-21swapper/123:24:131
3080799279,15cyclictest15123-21perf21:00:000
3080799274,10cyclictest2766-21/usr/sbin/munin21:45:160
3080799271,10cyclictest10267-21load22:00:180
3081399267,10cyclictest11749-21cut00:30:221
3081399261,9cyclictest0-21swapper/119:55:231
3081399261,12cyclictest0-21swapper/119:33:361
3081399261,10cyclictest0-21swapper/123:03:571
3081399260,12cyclictest0-21swapper/123:29:051
3081399260,12cyclictest0-21swapper/121:14:211
3081399260,12cyclictest0-21swapper/121:14:201
3081399260,12cyclictest0-21swapper/120:05:051
3081399260,11cyclictest0-21swapper/121:17:081
3081399260,10cyclictest0-21swapper/121:45:291
3081399251,9cyclictest0-21swapper/123:16:451
3081399251,11cyclictest0-21swapper/120:44:311
3081399251,10cyclictest0-21swapper/120:31:211
3081399251,10cyclictest0-21swapper/120:24:151
3081399251,10cyclictest0-21swapper/100:09:091
30813992510,12cyclictest23-21ksoftirqd/122:00:331
3081399250,8cyclictest0-21swapper/121:40:451
3081399250,13cyclictest0-21swapper/123:49:321
3081399250,13cyclictest0-21swapper/122:25:411
3081399250,13cyclictest0-21swapper/122:25:411
3081399250,12cyclictest0-21swapper/100:20:591
3081399250,11cyclictest0-21swapper/123:33:101
3081399250,10cyclictest0-21swapper/121:21:521
3080799253,10cyclictest20768-21awk23:40:010
3080799253,10cyclictest20768-21awk23:40:010
3080799252,8cyclictest28296-21latency_hist21:30:020
3080799252,11cyclictest8424-21/usr/sbin/munin00:25:210
3080799252,11cyclictest1622-21awk00:10:020
3082499242,11cyclictest991-21sshd19:15:023
30824992417,4cyclictest10496-21tr19:35:123
3081399244,3cyclictest0-21swapper/100:35:171
3081399241,8cyclictest833-21runrttasks22:07:411
3081399241,11cyclictest0-21swapper/121:00:111
3081399241,10cyclictest0-21swapper/121:25:021
3081399240,11cyclictest0-21swapper/122:51:091
3081399240,10cyclictest0-21swapper/100:00:091
3080799245,11cyclictest12529-21/usr/sbin/munin19:40:220
3080799244,12cyclictest26438-21ntp_states23:50:200
3080799243,5cyclictest21936-21/usr/sbin/munin21:15:190
3081999231,7cyclictest0-21swapper/223:22:352
3081999231,11cyclictest0-21swapper/220:19:522
30813992317,3cyclictest30059-21memory22:45:191
3081399231,10cyclictest0-21swapper/119:41:001
3080799236,3cyclictest0-21swapper/023:44:310
3080799234,7cyclictest32311-21/usr/sbin/munin20:25:190
3080799233,8cyclictest4744-21timerandwakeup23:00:240
3080799233,6cyclictest0-21swapper/023:55:160
3080799231,11cyclictest812-21snmpd20:54:340
3080799231,11cyclictest30722-21/usr/sbin/munin21:35:120
3080799230,11cyclictest9771-21perf23:15:000
3082499220,19cyclictest10395-21grep22:00:193
30819992211,3cyclictest32090-21munin-run20:25:012
3081399221,8cyclictest29357-21taskset23:57:551
3081399221,4cyclictest20008-21sshd22:23:371
3081399221,15cyclictest0-21swapper/121:34:451
3081399221,10cyclictest0-21swapper/119:14:011
3081399220,9cyclictest0-21swapper/122:40:291
3080799228,2cyclictest13498-21hddtemp_smartct00:35:160
3080799225,8cyclictest7157-21/usr/sbin/munin21:55:150
56972210,5sleep232-21ksoftirqd/223:05:132
3082499219,4cyclictest21017-21switchtime21:10:253
3082499219,4cyclictest21017-21switchtime21:10:253
3082499212,11cyclictest24133-21/usr/sbin/munin21:20:153
3082499212,10cyclictest9-21rcu_preempt21:25:153
3082499211,8cyclictest14727-21sh23:25:143
3082499211,5cyclictest0-21swapper/321:43:153
3082499211,14cyclictest0-21swapper/321:00:193
3081999211,6cyclictest0-21swapper/222:01:332
3081399215,3cyclictest0-21swapper/120:55:031
3080799215,8cyclictest86750irq/128-enp1s0-19:19:120
3080799212,10cyclictest9982-21/usr/sbin/munin23:15:230
3080799212,10cyclictest14724-21/usr/sbin/munin19:45:190
3080799211,7cyclictest0-21swapper/023:25:130
3080799211,5cyclictest17465-21sort23:30:180
3080799211,11cyclictest5892-21sshd00:18:470
30807992110,7cyclictest10971-21diskstats00:30:140
3082499208,4cyclictest16877-21cat22:15:163
3082499204,9cyclictest41-21ksoftirqd/300:22:193
3082499201,11cyclictest23520-21diskstats23:45:133
3082499201,11cyclictest0-21swapper/322:05:143
3081999202,7cyclictest12308-21munin-run19:40:012
3081999202,15cyclictest20215-21cat23:35:262
3081999202,15cyclictest20215-21cat23:35:262
3081999201,7cyclictest0-21swapper/220:11:442
3081999201,5cyclictest0-21swapper/222:26:322
3081999201,5cyclictest0-21swapper/222:26:322
30819992013,4cyclictest3914-21cut23:00:152
3081999201,10cyclictest14676-21memory22:10:182
3081399206,3cyclictest0-21swapper/122:39:431
3081399206,3cyclictest0-21swapper/100:15:511
30813992016,3cyclictest0-21swapper/121:36:481
3081399200,9cyclictest0-21swapper/123:13:091
3081399200,10cyclictest0-21swapper/121:05:271
3080799208,5cyclictest86750irq/128-enp1s0-00:00:410
3080799205,2cyclictest4352-21/usr/sbin/munin20:35:210
3080799203,8cyclictest2687-21grep22:55:250
3080799203,8cyclictest0-21swapper/023:05:250
3080799203,5cyclictest17541-21/usr/sbin/munin21:05:250
3080799201,8cyclictest12531-21diskstats23:20:130
30807992014,4cyclictest13900-21tr22:10:110
30807992011,5cyclictest12626-21ntp_states22:05:210
30807992011,4cyclictest9699-21sed19:30:250
3080799201,13cyclictest0-21swapper/022:51:330
3080799201,10cyclictest21316-21/usr/sbin/munin20:00:220
3080799201,10cyclictest18159-21/usr/sbin/munin22:20:170
3080799200,10cyclictest0-21swapper/020:40:090
245402201,3sleep324578-21ntpq20:05:203
222322201,16sleep221316-21/usr/sbin/munin20:00:202
3082499198,3cyclictest86550irq/126-enp1s0-00:05:233
3082499198,3cyclictest19510-21diskmemload21:55:033
3082499198,2cyclictest14221-21sshd19:41:373
3082499192,14cyclictest0-21swapper/300:25:123
3082499191,8cyclictest0-21swapper/322:58:463
30824991915,2cyclictest41-21ksoftirqd/323:30:263
30824991913,4cyclictest10103-21tr23:15:113
30824991912,3cyclictest0-21swapper/321:50:063
3082499191,12cyclictest86550irq/126-enp1s0-22:20:173
3082499191,11cyclictest836-21lldpd20:00:343
3082499191,11cyclictest0-21swapper/323:35:273
3082499191,11cyclictest0-21swapper/323:35:273
3082499191,10cyclictest4032-21/usr/sbin/munin00:15:113
3082499191,10cyclictest1834-21/usr/sbin/munin00:10:213
3082499191,10cyclictest0-21swapper/322:28:023
3082499191,10cyclictest0-21swapper/322:28:023
3082499191,10cyclictest0-21swapper/322:11:293
3081999197,4cyclictest86550irq/126-enp1s0-00:00:122
3081999191,4cyclictest0-21swapper/200:13:532
3081999191,3cyclictest0-21swapper/221:28:142
30819991913,4cyclictest27101-21tr22:40:112
30819991912,4cyclictest11235-21head00:30:152
3081999191,10cyclictest879-21sshd22:54:372
3081999191,10cyclictest20611-21cat21:10:192
3081999191,10cyclictest20611-21cat21:10:192
3080799197,3cyclictest86750irq/128-enp1s0-23:45:140
3080799194,12cyclictest25300-21grep21:20:230
3080799192,3cyclictest17334-21tune2fs19:50:150
3080799191,6cyclictest0-21swapper/021:11:250
3080799191,6cyclictest0-21swapper/021:11:250
3080799191,5cyclictest31362-21which20:20:220
3080799191,5cyclictest19977-21sed19:55:180
30807991911,4cyclictest1633-21ntp_states21:40:210
3080799191,10cyclictest2152-21/usr/sbin/munin20:30:180
3080799190,11cyclictest6812-21fw_packets00:20:150
3082499187,8cyclictest41-21ksoftirqd/319:15:013
3082499181,7cyclictest29806-21sed22:45:163
3082499181,6cyclictest0-21swapper/323:50:203
30824991813,3cyclictest446-21gltestperf20:25:143
30824991812,3cyclictest18392-21unixbench-2d19:50:253
3082499181,15cyclictest0-21swapper/320:55:273
3082499181,11cyclictest0-21swapper/323:06:143
3082499181,10cyclictest0-21swapper/321:30:213
3082499180,7cyclictest0-21swapper/300:00:193
3081999186,3cyclictest25484-21crond20:10:002
3081999181,9cyclictest0-21swapper/221:16:152
3081999181,7cyclictest0-21swapper/223:10:272
3081999181,5cyclictest0-21swapper/223:43:142
3081999181,4cyclictest0-21swapper/222:15:352
3081999181,3cyclictest0-21swapper/223:32:012
3081999181,3cyclictest0-21swapper/222:09:402
3081999181,3cyclictest0-21swapper/221:54:332
3081999181,3cyclictest0-21swapper/200:29:482
30819991813,3cyclictest15048-21sed23:25:162
30819991812,4cyclictest11523-21grep23:15:252
30819991812,3cyclictest15215-21fschecks_time19:45:142
3081999181,15cyclictest12074-21sed20:50:232
3081999181,11cyclictest0-21swapper/200:06:402
3081999180,4cyclictest0-21swapper/219:50:182
3080799182,2cyclictest86550irq/126-enp1s0-22:45:010
3080799181,6cyclictest0-21swapper/020:10:120
3080799181,5cyclictest0-21swapper/022:37:530
3080799181,14cyclictest8741-21/usr/sbin/munin20:45:240
3080799181,11cyclictest0-21swapper/022:16:070
3080799181,10cyclictest0-21swapper/022:25:250
3080799181,10cyclictest0-21swapper/022:25:250
242802181,2sleep20-21swapper/222:31:592
3082499177,2cyclictest0-21swapper/320:38:043
3082499176,3cyclictest86550irq/126-enp1s0-20:10:133
3082499176,3cyclictest19116-21/usr/sbin/munin19:55:193
3082499175,4cyclictest812-21snmpd22:52:403
3082499173,3cyclictest12050irq/123-ahci[0000:40:013
3082499172,7cyclictest27297-21diskstats22:40:133
3082499171,5cyclictest0-21swapper/320:20:113
3082499171,4cyclictest0-21swapper/323:10:133
30824991712,3cyclictest7607-21sshd20:40:183
3081999178,2cyclictest24894-21awk21:20:172
3081999173,8cyclictest0-21swapper/221:37:222
3081999173,2cyclictest86750irq/128-enp1s0-21:47:482
3081999172,6cyclictest19116-21/usr/sbin/munin19:55:262
3081999171,7cyclictest18654-21pmu-power21:05:222
3081999171,7cyclictest0-21swapper/200:38:392
3081999171,6cyclictest31126-21cpuspeed_turbos19:10:102
3081999171,6cyclictest0-21swapper/223:56:422
3081999171,5cyclictest7648-21sed00:20:242
3081999171,4cyclictest0-21swapper/222:35:122
3081999171,3cyclictest0-21swapper/222:46:282
3081999171,3cyclictest0-21swapper/222:24:502
30819991713,2cyclictest4352-21/usr/sbin/munin20:35:262
3081999171,2cyclictest0-21swapper/219:20:142
30819991712,3cyclictest8731-21grep21:55:242
3081999171,13cyclictest1638-21ntp_states21:40:202
30819991711,3cyclictest13757-21head20:55:162
3081999171,11cyclictest0-21swapper/221:32:462
3080799173,5cyclictest86750irq/128-enp1s0-19:10:120
3080799173,2cyclictest23519-21/usr/sbin/munin20:05:230
3080799173,11cyclictest3739-21/usr/sbin/munin19:20:120
3080799172,3cyclictest5936-21/usr/sbin/munin19:25:140
3080799171,4cyclictest22555-21/usr/sbin/munin22:30:150
3082499162,7cyclictest41-21ksoftirqd/319:27:173
3082499161,7cyclictest0-21swapper/320:50:173
3082499161,3cyclictest0-21swapper/322:35:213
3082499161,2cyclictest0-21swapper/319:21:583
3081999162,11cyclictest6051-21tr19:25:122
3081999161,6cyclictest32415-21cpuspeed_turbos20:25:122
3081999161,6cyclictest23847-21hddtemp_smartct23:45:152
3081999161,5cyclictest13785-21sensors19:40:232
30819991612,3cyclictest0-21swapper/223:50:452
30819991611,3cyclictest9332-21gltestperf20:45:152
30819991611,3cyclictest6735-21cat20:40:102
30819991611,3cyclictest15989-21head21:00:152
30819991610,3cyclictest969-21netstat19:15:022
3080799163,2cyclictest86750irq/128-enp1s0-22:45:170
3080799161,14cyclictest0-21swapper/020:16:550
30807991610,3cyclictest10628-21expr19:35:120
30824991510,3cyclictest8926-21ls20:45:123
3081999156,7cyclictest0-21swapper/220:30:032
30819991510,3cyclictest8136-21/usr/sbin/munin19:30:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional