You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 16:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Wed Mar 11, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
82602208134,46sleep10-21swapper/107:06:001
85232182107,24sleep30-21swapper/307:09:243
83892181107,25sleep00-21swapper/007:07:400
8241217695,23sleep20-21swapper/207:05:452
2773421580,3sleep3867299cyclictest07:50:163
41182590,2sleep10-21swapper/109:25:001
257962580,3sleep325795-21sendmail_mailqu11:25:233
136912500,4sleep339-21rcuc/311:00:113
198432460,2sleep230-21rcuc/207:30:252
867299431,33cyclictest0-21swapper/309:27:083
867299431,21cyclictest0-21swapper/311:20:463
867299421,21cyclictest16819-21hddtemp_smartct07:25:173
8672994027,8cyclictest0-21swapper/308:23:383
171422400,2sleep20-21swapper/212:20:162
8672993926,3cyclictest0-21swapper/308:09:223
867299390,24cyclictest0-21swapper/310:03:233
867299361,18cyclictest23136-21expr12:35:123
867299351,28cyclictest0-21swapper/311:50:293
867299351,27cyclictest0-21swapper/311:11:063
867299351,27cyclictest0-21swapper/309:47:063
867299351,21cyclictest0-21swapper/310:46:523
867299351,20cyclictest0-21swapper/309:24:063
867299341,26cyclictest0-21swapper/310:29:513
867299341,22cyclictest0-21swapper/307:35:353
867299331,30cyclictest0-21swapper/307:56:383
867299331,19cyclictest0-21swapper/310:53:303
867299331,19cyclictest0-21swapper/310:53:303
867299331,18cyclictest0-21swapper/311:18:163
867299330,21cyclictest0-21swapper/312:33:143
867299321,22cyclictest0-21swapper/308:35:093
867299321,21cyclictest0-21swapper/307:44:383
867299321,21cyclictest0-21swapper/307:32:513
867299321,20cyclictest0-21swapper/309:44:123
867299321,20cyclictest0-21swapper/309:44:123
867299321,19cyclictest0-21swapper/312:16:103
867299321,19cyclictest0-21swapper/312:12:303
867299321,13cyclictest0-21swapper/310:24:533
867299320,21cyclictest0-21swapper/308:31:423
867299320,21cyclictest0-21swapper/308:31:413
867299320,17cyclictest0-21swapper/308:26:123
865699328,11cyclictest12136-21awk08:30:010
867299311,20cyclictest0-21swapper/312:20:483
867299311,20cyclictest0-21swapper/307:15:363
867299311,19cyclictest0-21swapper/309:03:463
867299311,19cyclictest0-21swapper/308:12:023
867299310,20cyclictest0-21swapper/310:16:533
867299301,24cyclictest0-21swapper/310:05:533
867299301,20cyclictest0-21swapper/309:15:503
867299301,19cyclictest0-21swapper/310:13:553
867299300,19cyclictest0-21swapper/309:57:003
867299300,13cyclictest0-21swapper/312:05:163
865699304,12cyclictest20322-21awk07:35:010
8656993011,16cyclictest21535-21munin-run10:05:010
867299291,18cyclictest0-21swapper/312:01:023
867299291,18cyclictest0-21swapper/312:01:023
867299291,18cyclictest0-21swapper/309:37:223
867299291,13cyclictest0-21swapper/309:11:543
867299290,21cyclictest0-21swapper/309:54:463
867299290,19cyclictest0-21swapper/308:01:403
867299290,13cyclictest0-21swapper/311:55:513
867299281,25cyclictest0-21swapper/310:55:523
867299281,19cyclictest0-21swapper/311:45:573
867299281,19cyclictest0-21swapper/311:05:423
867299281,17cyclictest0-21swapper/311:33:463
867299281,12cyclictest0-21swapper/310:43:303
867299280,11cyclictest29705-21diskmemload11:35:373
865699281,23cyclictest11-21rcuc/007:50:190
865699281,21cyclictest0-21swapper/012:04:200
865699281,21cyclictest0-21swapper/012:04:200
867299271,18cyclictest0-21swapper/312:26:343
867299271,13cyclictest0-21swapper/310:35:403
867299271,12cyclictest0-21swapper/309:33:363
867299271,11cyclictest0-21swapper/311:40:093
867299271,11cyclictest0-21swapper/307:22:513
867299271,11cyclictest0-21swapper/307:22:513
866299271,12cyclictest0-21swapper/109:40:161
866299271,12cyclictest0-21swapper/109:40:161
865699271,2cyclictest0-21swapper/009:10:330
865699271,13cyclictest16164-21munin-run12:20:000
865699271,13cyclictest11767-21munin-run12:10:000
865699271,10cyclictest9224-21sed09:35:150
867299260,10cyclictest0-21swapper/307:47:223
8667992612,9cyclictest0-21swapper/212:00:272
8667992612,9cyclictest0-21swapper/212:00:272
865699261,22cyclictest27135-21needreboot10:15:180
865699261,21cyclictest14331-21fschecks_count07:20:150
865699261,21cyclictest14331-21fschecks_count07:20:150
867299250,17cyclictest0-21swapper/308:45:373
866799255,9cyclictest26839-21cat11:30:102
866799255,10cyclictest780-21in:imjournal08:40:212
8667992511,3cyclictest15391-21seq11:03:052
865699253,10cyclictest28117-21kworker/0:007:15:150
865699251,21cyclictest81ktimersoftd/010:55:270
865699251,21cyclictest0-21swapper/011:17:430
865699251,21cyclictest0-21swapper/010:08:000
865699251,12cyclictest17748-21munin-run11:10:010
865699251,11cyclictest6973-21/usr/sbin/munin10:45:120
865699251,11cyclictest27535-21cat11:30:160
865699251,11cyclictest12963-21/usr/sbin/munin09:45:210
867299248,9cyclictest0-21swapper/307:11:223
867299241,11cyclictest0-21swapper/310:31:223
8667992415,5cyclictest22506-21timerandwakeup08:50:232
8667992412,3cyclictest30931-21awk11:40:002
866299246,3cyclictest10461-21kworker/u8:009:39:391
865699249,4cyclictest28117-21kworker/0:008:55:140
865699243,10cyclictest7-21ksoftirqd/012:25:090
865699241,3cyclictest88150irq/126-enp1s0-09:55:550
865699241,21cyclictest28117-21kworker/0:007:10:250
865699241,21cyclictest0-21swapper/011:00:160
865699241,20cyclictest11-21rcuc/008:00:180
865699241,20cyclictest0-21swapper/009:27:180
865699241,19cyclictest1861-21grep10:30:230
865699241,11cyclictest23953-21/usr/sbin/munin10:10:120
865699241,10cyclictest9175-21/usr/sbin/munin10:50:230
865699241,10cyclictest9175-21/usr/sbin/munin10:50:230
865699241,10cyclictest28117-21kworker/0:011:45:190
867299231,14cyclictest0-21swapper/308:40:273
867299231,11cyclictest0-21swapper/309:05:123
866799236,9cyclictest88150irq/126-enp1s0-08:35:182
866799236,3cyclictest88450irq/129-enp1s0-10:55:272
866799235,10cyclictest780-21in:imjournal10:27:382
866299231,8cyclictest23346-21users10:05:261
866299231,6cyclictest0-21swapper/111:40:171
865699238,2cyclictest28117-21kworker/0:008:45:240
865699237,4cyclictest30513-21sed11:35:250
865699232,10cyclictest7-21ksoftirqd/012:20:250
865699231,20cyclictest0-21swapper/010:27:410
865699231,19cyclictest81ktimersoftd/011:10:180
865699231,11cyclictest0-21swapper/011:25:150
865699231,10cyclictest2937-21/usr/sbin/munin10:35:140
865699231,10cyclictest28476-21grep10:20:110
865699231,10cyclictest0-21swapper/009:30:190
8672992212,6cyclictest22478-21timerandwakeup08:50:223
866799228,2cyclictest26907-21awk07:50:012
866799223,10cyclictest16487-21df_inode07:25:162
866799222,5cyclictest10974-21munin-node07:15:012
866799222,11cyclictest3525-21df11:50:152
866799221,15cyclictest0-21swapper/207:15:502
866799221,12cyclictest0-21swapper/209:36:142
866799221,11cyclictest4511-21df_inode09:25:142
866799221,11cyclictest31253-21cpuspeed_turbos11:40:132
866799221,11cyclictest22672-21memory10:05:182
866299229,11cyclictest23-21ksoftirqd/107:45:121
866299224,3cyclictest0-21swapper/110:40:181
866299222,11cyclictest26487-21df_inode10:15:121
866299221,13cyclictest20624-21ntp_states10:00:211
8662992210,3cyclictest4628-21cat10:40:011
866299220,6cyclictest0-21swapper/110:23:321
866299220,18cyclictest0-21swapper/111:51:061
865699228,3cyclictest3187-21/usr/sbin/munin11:50:240
865699227,3cyclictest11-21rcuc/011:55:150
865699226,3cyclictest8717-21if_enp2s008:20:160
865699225,6cyclictest0-21swapper/009:42:590
865699225,6cyclictest0-21swapper/009:42:590
865699224,7cyclictest88150irq/126-enp1s0-09:51:070
865699222,7cyclictest0-21swapper/011:40:170
865699221,9cyclictest24921-21/usr/sbin/munin07:45:150
865699221,9cyclictest0-21swapper/008:09:560
865699221,7cyclictest32132-21/usr/sbin/munin09:15:160
865699221,7cyclictest19108-1kworker/u9:009:24:250
865699221,10cyclictest0-21swapper/012:33:010
867299210,13cyclictest0-21swapper/308:15:063
866799215,4cyclictest9063-21/usr/sbin/munin08:20:212
866799215,4cyclictest88450irq/129-enp1s0-11:45:122
866799215,3cyclictest25607-21ntp_kernel_pll_11:25:192
866799214,9cyclictest1378-21cpu08:05:122
866799213,10cyclictest17323-21unixbench_singl11:05:282
866799211,7cyclictest0-21swapper/211:19:032
866799211,6cyclictest19278-21df_inode08:45:132
866799211,11cyclictest27461-21df07:50:142
866799211,11cyclictest22535-21/usr/sbin/munin11:20:122
866799211,11cyclictest13303-21df_inode09:45:132
866799211,10cyclictest16154-21latency_hist12:20:022
866299216,3cyclictest8718-21if_enp2s008:20:151
866299213,8cyclictest0-21swapper/112:22:521
866299212,8cyclictest0-21swapper/112:28:241
866299211,11cyclictest32242-21if_enp1s008:00:161
866299211,11cyclictest19047-21ntp_states11:10:221
865699217,3cyclictest14547-21/usr/sbin/munin08:35:200
865699216,3cyclictest88350irq/128-enp1s0-10:40:180
865699211,6cyclictest16144-21/usr/sbin/munin07:25:240
865699210,19cyclictest0-21swapper/009:05:140
867299201,11cyclictest0-21swapper/308:59:463
866799205,6cyclictest768-21rngd08:00:372
866799204,9cyclictest0-21swapper/210:39:532
866799204,8cyclictest3721-21sshd09:22:212
866799204,8cyclictest0-21swapper/207:41:072
866799201,7cyclictest32257-21expr09:15:142
866799201,7cyclictest0-21swapper/212:08:342
866799201,6cyclictest19042-21ntp_states11:10:212
866799201,6cyclictest0-21swapper/210:15:132
866799201,6cyclictest0-21swapper/209:30:032
8667992016,3cyclictest0-21swapper/210:49:212
866799201,5cyclictest6869-21grep11:55:242
866799201,5cyclictest0-21swapper/210:40:182
866799201,12cyclictest0-21swapper/209:41:382
866799201,12cyclictest0-21swapper/209:41:382
866799201,10cyclictest25526-21/usr/sbin/munin09:00:192
866799201,10cyclictest23349-21bc08:55:122
866799201,10cyclictest16882-21chrt09:51:582
866799201,10cyclictest0-21swapper/210:10:192
866299207,4cyclictest1980-21needreboot11:45:191
866299206,12cyclictest87852chrt10:48:051
866299201,8cyclictest0-21swapper/109:50:541
866299201,14cyclictest0-21swapper/109:10:421
866299201,11cyclictest12692-21df_inode08:30:131
866299201,11cyclictest12692-21df_inode08:30:131
8662992010,2cyclictest88450irq/129-enp1s0-11:55:591
865699201,7cyclictest3555-21/usr/sbin/munin08:10:200
865699201,12cyclictest0-21swapper/012:10:140
865699201,10cyclictest496-21jbd2/dm-0-808:44:130
865699200,18cyclictest0-21swapper/011:20:170
866799193,8cyclictest10850-21sshd10:51:202
866799193,8cyclictest10850-21sshd10:51:202
866799192,8cyclictest3555-21/usr/sbin/munin08:10:172
866799192,4cyclictest23323-21diskstats12:35:132
866799191,9cyclictest13388-21sshd08:30:192
866799191,9cyclictest13388-21sshd08:30:192
866799191,7cyclictest29705-21diskmemload09:55:172
866799191,6cyclictest29705-21diskmemload10:21:262
866799191,5cyclictest30308-21grep07:55:202
866799191,5cyclictest12126-21cpuspeed_turbos12:10:122
866799191,5cyclictest0-21swapper/210:30:252
866799191,4cyclictest0-21swapper/210:00:242
866299199,4cyclictest30970-21kworker/u8:208:40:061
866299195,3cyclictest7237-21latency09:30:171
866299191,17cyclictest0-21swapper/111:05:561
866299191,12cyclictest0-21swapper/112:34:501
866299191,10cyclictest21514-21needreboot07:35:201
866299191,10cyclictest15550-21latency_hist11:05:011
866299191,10cyclictest15064-21latency12:15:171
866299191,10cyclictest14862-21/usr/sbin/munin07:20:191
866299191,10cyclictest14862-21/usr/sbin/munin07:20:191
866299191,10cyclictest11764-21cpu07:15:111
866299190,3cyclictest0-21swapper/110:13:161
866299190,14cyclictest0-21swapper/110:59:571
865699195,4cyclictest0-21swapper/012:35:330
865699191,7cyclictest0-21swapper/008:18:040
865699191,16cyclictest14327-21munin-run08:35:010
865699191,16cyclictest14327-21munin-run08:35:010
865699191,15cyclictest28117-21kworker/0:007:35:150
866799185,4cyclictest30343-21cut09:10:152
866799184,3cyclictest29216-21unixbench-2d09:05:282
866799183,4cyclictest32-21ksoftirqd/207:35:132
866799182,4cyclictest7725-21munin-run08:20:002
866799181,5cyclictest20139-21grep12:25:252
866799180,6cyclictest15739-21sshd07:22:542
866799180,6cyclictest15739-21sshd07:22:542
866299189,2cyclictest4621-21fschecks_count09:25:141
866299188,2cyclictest18344-21/usr/sbin/munin07:30:111
866299187,3cyclictest780-21in:imjournal08:18:371
866299186,3cyclictest23-21ksoftirqd/112:10:181
866299185,3cyclictest88350irq/128-enp1s0-08:05:151
866299184,4cyclictest13951-21needreboot09:45:201
866299181,7cyclictest0-21swapper/108:38:001
866299181,4cyclictest580-21systemd-journal09:00:201
866299181,4cyclictest0-21swapper/110:30:171
866299181,3cyclictest9864-21wc12:05:111
866299181,3cyclictest30394-21sort11:35:241
866299181,3cyclictest29713-21perf09:10:021
866299181,2cyclictest22134-21idleruntime-cro11:20:021
8662991812,3cyclictest22048-21memory08:50:171
866299181,14cyclictest31288-21munin-run08:00:021
866299181,10cyclictest0-21swapper/110:25:201
866299180,9cyclictest0-21swapper/107:10:121
866299180,5cyclictest0-21swapper/111:33:161
865699181,5cyclictest29318-21/usr/sbin/munin07:55:150
866799171,6cyclictest0-21swapper/208:26:122
866799171,5cyclictest0-21swapper/212:30:042
866299177,4cyclictest7993-21/usr/sbin/munin12:00:131
866299177,4cyclictest7993-21/usr/sbin/munin12:00:131
866299175,3cyclictest11708-21grep08:25:251
866299173,12cyclictest9629-21fschecks_count10:50:121
866299173,12cyclictest9629-21fschecks_count10:50:121
866299173,11cyclictest17522-21cstates09:55:111
866299171,8cyclictest25115-21perl11:25:151
866299171,7cyclictest23720-21fschecks_count08:55:151
866299171,3cyclictest22448-21cat11:20:121
866299171,3cyclictest0-21swapper/109:15:181
8662991712,3cyclictest16481-21df07:25:151
866299171,13cyclictest27702-21perl07:50:161
8662991711,3cyclictest21060-21cut08:50:021
866299171,10cyclictest0-21swapper/112:37:051
865699171,14cyclictest0-21swapper/007:42:360
866299168,3cyclictest0-21swapper/107:40:021
866299161,2cyclictest0-21swapper/108:10:151
865699161,3cyclictest0-21swapper/009:00:150
865699151,11cyclictest21605-21grep08:50:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional