You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 17:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Dec 07, 2025 12:44:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2974839213457,72sleep30-21swapper/307:06:563
297500428756,10sleep10-21swapper/107:09:031
297494027956,6sleep00-21swapper/007:08:120
297481827964,9sleep20-21swapper/207:06:392
33922812480,2sleep00-21swapper/012:15:170
33104042331,9sleep03104654-21diskmemload11:23:060
31661362331,14sleep03166365-21df_inode09:50:140
31661362331,14sleep03166365-21df_inode09:50:130
32237202291,23sleep03220295-21/usr/sbin/munin10:25:210
34062602260,4sleep134-21ksoftirqd/112:24:041
2975213992621,3cyclictest0-21swapper/309:10:003
2975213992621,2cyclictest3432157-21turbostat12:40:013
2975213992614,8cyclictest3398627-21turbostat12:20:003
297521399252,15cyclictest3304157-21taskset11:20:113
297521399252,15cyclictest3250216-21taskset10:45:003
2975213992422,1cyclictest0-21swapper/309:00:003
2975213992422,1cyclictest0-21swapper/308:45:003
2975213992421,1cyclictest0-21swapper/307:38:143
297521399241,19cyclictest3142853-21turbostat09:39:553
297521399241,17cyclictest3078743-21taskset08:45:143
297521399232,14cyclictest3279448-21taskset11:03:203
33133982221,14sleep03315152-21cut11:25:180
297521399222,13cyclictest3421127-21taskset12:31:193
2975213992220,1cyclictest0-21swapper/308:30:023
2975213992218,2cyclictest0-21swapper/310:50:013
2975213992217,2cyclictest0-21swapper/309:45:253
297521399221,18cyclictest3218669-21taskset10:22:473
297521399221,15cyclictest3374618-21sleep12:09:563
31576652219,8sleep13104654-21diskmemload09:44:331
297521399212,12cyclictest3355878-21taskset11:50:343
2975213992117,2cyclictest0-21swapper/310:25:193
2975213992117,2cyclictest0-21swapper/310:09:193
2975213992117,2cyclictest0-21swapper/307:55:013
2975213992116,3cyclictest0-21swapper/309:55:233
2975213992112,5cyclictest15950irq/132-ahci[0000:00:17.0]09:28:063
297521399211,15cyclictest3284774-21chrt11:05:203
297521399210,13cyclictest3263901-21taskset10:52:293
32330402201,2sleep10-21swapper/110:31:161
31276192202,12sleep13127862-21latency_hist09:25:021
31183692201,2sleep00-21swapper/009:17:180
297521399205,12cyclictest0-21swapper/311:30:173
297521399202,13cyclictest3271490-21taskset10:57:313
297521399202,12cyclictest3294108-21taskset11:12:053
2975213992018,1cyclictest0-21swapper/309:10:223
2975213992017,1cyclictest0-21swapper/310:12:553
2975213992017,1cyclictest0-21swapper/308:50:153
2975213992016,2cyclictest0-21swapper/312:20:203
2975213992011,5cyclictest32346872sleep310:34:223
297521399201,14cyclictest3117664-21chrt09:16:003
297521399192,5cyclictest0-21swapper/309:40:093
297521399192,12cyclictest3415207-21taskset12:29:523
297521399192,12cyclictest3373415-21taskset12:03:093
297521399192,12cyclictest3127516-21taskset09:24:503
2975213991916,1cyclictest0-21swapper/309:50:243
2975213991916,1cyclictest0-21swapper/309:50:233
2975213991916,1cyclictest0-21swapper/307:10:213
2975213991915,2cyclictest0-21swapper/311:45:133
2975213991913,4cyclictest0-21swapper/309:31:213
297521399191,15cyclictest3388389-21chrt12:11:193
297521399191,15cyclictest3210958-21taskset10:17:493
297521399191,14cyclictest0-21swapper/311:15:363
297521399190,16cyclictest66550irq/131-enp1s0-TxRx-311:55:413
297521399190,16cyclictest66550irq/131-enp1s0-TxRx-310:00:033
297521399190,11cyclictest3331523-21taskset11:35:313
297521399190,11cyclictest3331523-21taskset11:35:313
34303432181,5sleep13104654-21diskmemload12:37:291
33982042181,5sleep13104654-21diskmemload12:19:241
32883632181,2sleep00-21swapper/011:10:110
32413792181,11sleep00-21swapper/010:37:480
32186232181,3sleep20-21swapper/210:22:442
32033982181,3sleep20-21swapper/210:13:052
31416042181,4sleep13104654-21diskmemload09:32:391
297521399182,11cyclictest3317413-21taskset11:26:573
2975213991815,1cyclictest0-21swapper/311:40:133
2975213991815,1cyclictest0-21swapper/310:35:253
2975213991815,1cyclictest0-21swapper/308:05:173
2975213991815,1cyclictest0-21swapper/307:25:163
2975213991815,1cyclictest0-21swapper/307:15:173
33901382177,2sleep2171rcu_preempt12:14:192
33565852171,4sleep23104654-21diskmemload11:51:422
33397482170,9sleep03339758-21ssh11:40:510
33264332170,3sleep00-21swapper/011:34:320
33032492171,2sleep10-21swapper/111:19:391
32936042171,4sleep13104654-21diskmemload11:11:081
32704412171,3sleep00-21swapper/010:55:360
32111362171,2sleep10-21swapper/110:18:091
31867892171,4sleep23104654-21diskmemload10:01:192
2975213991713,2cyclictest0-21swapper/308:20:213
297521399171,13cyclictest0-21swapper/308:25:233
34240552161,2sleep00-21swapper/012:35:100
34150292161,3sleep03104654-21diskmemload12:29:370
34123162161,3sleep13104654-21diskmemload12:25:461
33978692161,3sleep20-21swapper/212:18:532
33892632161,3sleep00-21swapper/012:12:580
33743412161,9sleep23104654-21diskmemload12:04:372
33438742161,10sleep23345472-21tr11:45:172
33178392161,4sleep23104654-21diskmemload11:27:422
33027282161,9sleep23104654-21diskmemload11:18:442
32563652161,3sleep13104654-21diskmemload10:47:511
32495162160,2sleep00-21swapper/010:43:500
32411612161,3sleep23241166-21ssh10:37:182
31873192160,3sleep10-21swapper/110:02:141
30771822161,11sleep13077267-21ps08:40:251
30731172161,4sleep03073349-21df_inode08:40:160
297521399161,9cyclictest3030226-21taskset08:00:023
2975213991613,1cyclictest0-21swapper/308:10:143
297521399160,13cyclictest0-21swapper/307:45:193
297521399160,13cyclictest0-21swapper/307:40:193
33823642151,9sleep03382374-21ssh12:09:360
33732142151,8sleep03104654-21diskmemload12:02:520
33413042151,9sleep13341306-21ssh11:43:201
33397702151,2sleep20-21swapper/211:40:532
32846602151,8sleep23284773-21ntpq11:05:212
32416832151,3sleep13104654-21diskmemload10:38:251
32274282151,2sleep20-21swapper/210:29:482
32184952151,9sleep03218505-21ssh10:22:300
31892942151,2sleep00-21swapper/010:05:130
31886392150,2sleep00-21swapper/010:04:280
31655962150,8sleep00-21swapper/009:49:530
31056062150,10sleep03106333-21cut09:10:170
30616522151,9sleep03061889-21latency_hist08:30:010
29879662151,4sleep22987985-21apt-get07:20:132
297521399152,10cyclictest3029634-21taskset07:55:563
2975213991512,2cyclictest0-21swapper/308:15:123
2975213991511,2cyclictest0-21swapper/308:35:213
297521399151,11cyclictest0-21swapper/309:00:133
297521399151,10cyclictest2992039-21chrt07:20:213
2975213991510,2cyclictest0-21swapper/307:30:203
33892722143,1sleep1171rcu_preempt12:12:591
33790682141,8sleep13379122-21ntpq12:05:201
33726182140,2sleep13104654-21diskmemload12:01:571
32798352141,8sleep23279844-21ssh11:04:122
32557632140,2sleep00-21swapper/010:46:140
31965572141,1sleep20-21swapper/210:09:442
31712572141,7sleep23104654-21diskmemload09:51:042
31712572141,7sleep23104654-21diskmemload09:51:032
31483092140,9sleep23148318-21ssh09:35:562
30372632141,8sleep13039065-21cut08:05:171
29938562146,4sleep02995129-21kthreadcore07:25:170
34282042131,5sleep23428263-21ntpq12:35:212
34160062131,5sleep10-21swapper/112:30:141
32705932131,7sleep13104654-21diskmemload10:55:531
32650222131,8sleep13265031-21sh10:54:491
32431862130,8sleep13243242-21tune2fs10:40:141
31713722131,6sleep13104654-21diskmemload09:51:151
31713722131,6sleep13104654-21diskmemload09:51:151
31576042131,6sleep23104654-21diskmemload09:44:262
31488082131,6sleep13104654-21diskmemload09:36:511
31198992131,7sleep23120397-21apt-key09:20:012
31131252131,7sleep23113804-21kthreadcore09:15:162
30940692131,6sleep03094088-21apt-get09:00:110
30927832131,8sleep23092832-21ntp_kernel_pll_08:55:202
33797942121,2sleep23379804-21ssh12:05:292
32862472120,1sleep10-21swapper/111:06:521
32339852121,5sleep00-21swapper/010:32:560
32195632121,5sleep10-21swapper/110:24:231
32047862121,6sleep03204843-21grep10:15:110
32029212121,5sleep10-21swapper/110:12:161
32027942120,7sleep03104654-21diskmemload10:12:020
31636452121,7sleep23163656-21sh09:46:272
31461202121,5sleep03147164-21memory09:35:180
31113922121,5sleep20-21swapper/209:13:272
31102912122,2sleep10-21swapper/109:11:271
31001702121,7sleep03100672-21cut09:05:150
30998992127,2sleep20-21swapper/209:05:132
30689022120,8sleep03070641-21grep08:35:180
30448922120,1sleep10-21swapper/108:10:221
30108232121,6sleep23012615-21kthreadcore07:40:172
29973462122,5sleep12997400-21ntpq07:25:221
34234372111,4sleep03104654-21diskmemload12:34:380
34044232111,4sleep23104654-21diskmemload12:21:252
33648142111,4sleep23104654-21diskmemload11:57:042
33614032111,4sleep03104654-21diskmemload11:55:170
33589772111,4sleep13104654-21diskmemload11:55:061
33571502110,1sleep00-21swapper/011:52:370
33496962111,4sleep13104654-21diskmemload11:48:571
33319892110,4sleep03104654-21diskmemload11:35:460
33319892110,4sleep03104654-21diskmemload11:35:450
33319342111,4sleep13104654-21diskmemload11:35:391
33319342111,4sleep13104654-21diskmemload11:35:381
33274042110,4sleep20-21swapper/211:35:162
33274042110,4sleep20-21swapper/211:35:152
33172892111,6sleep13104654-21diskmemload11:26:451
33097202111,4sleep13104654-21diskmemload11:21:551
33041302111,4sleep23304152-21apt-get11:20:112
32879982111,3sleep00-21swapper/011:09:590
32796322111,4sleep03104654-21diskmemload11:03:420
32785842111,3sleep10-21swapper/111:01:461
32713312111,3sleep23271334-21ssh10:57:152
32632622111,4sleep03104654-21diskmemload10:51:240
32628282111,4sleep23104654-21diskmemload10:50:372
32473912111,6sleep23247624-21unixbench_multi10:40:272
32250952111,4sleep13104654-21diskmemload10:25:341
32112152110,1sleep20-21swapper/210:18:182
31947382111,3sleep10-21swapper/110:06:341
31809922111,4sleep20-21swapper/209:59:532
31807912111,4sleep13104654-21diskmemload09:59:321
31805252111,3sleep00-21swapper/009:59:030
31582892111,4sleep13158313-21apt-get09:45:131
31407492111,4sleep03104654-21diskmemload09:31:020
31279692111,4sleep13104654-21diskmemload09:25:131
31261322111,3sleep00-21swapper/009:22:160
31192992111,4sleep13104654-21diskmemload09:18:591
30846772111,6sleep13086414-21kthreadcore08:50:191
30513822111,5sleep13051401-21apt-get08:20:171
30249692111,5sleep23024988-21apt-get07:55:122
30132382110,1sleep10-21swapper/107:40:201
30087902111,4sleep23008753-31find07:38:172
29826702111,6sleep02982865-21cstates07:15:140
34210212101,2sleep20-21swapper/212:31:112
34125652101,3sleep20-21swapper/212:26:072
34044952101,3sleep00-21swapper/012:21:320
33509772101,5sleep13351004-21apt-get11:50:141
33480812101,2sleep00-21swapper/011:46:150
33248222101,2sleep20-21swapper/211:31:382
33234312101,5sleep13323480-21gcc11:30:201
33012572101,3sleep03104654-21diskmemload11:16:000
32898432101,4sleep23104654-21diskmemload11:10:252
32571702100,1sleep20-21swapper/210:49:212
32330772101,2sleep20-21swapper/210:31:212
31578792101,3sleep00-21swapper/009:44:570
31415182100,3sleep20-21swapper/209:32:272
31336982101,3sleep03104654-21diskmemload09:27:080
31335612101,3sleep20-21swapper/209:26:542
30996402101,4sleep13099898-21df09:05:121
30928872100,6sleep03092970-21pmu-power08:55:220
30926642101,4sleep13092777-21ntpq08:55:191
30838362100,6sleep23083892-21fschecks_count08:50:162
30823922101,5sleep23082468-21awk08:45:222
30605642101,4sleep23060653-21ntpq08:25:222
29924372101,4sleep12992654-21users07:20:281
3079824291,4sleep13081612-21cut08:45:171
3062467291,4sleep03062511-21tune2fs08:30:140
3043950291,3sleep23044743-21sed08:10:202
3031640291,4sleep23033386-21tr08:00:182
3024708291,4sleep23024735-21munin-run07:55:002
3008808290,4sleep087-21kswapd007:38:270
3084107280,4sleep03084676-21kthreadcore08:50:180
3078899281,3sleep03079823-21cut08:45:160
3071623281,3sleep23071763-21ntp_192.168.11408:35:202
3062191281,2sleep23062210-21apt-get08:30:112
3056082280,1sleep20-21swapper/208:21:142
3049998280,1sleep20-21swapper/208:15:202
3049866281,4sleep13049997-21uname08:15:191
3040831281,2sleep00-21swapper/008:10:130
3029924280,1sleep00-21swapper/007:59:400
3015291281,3sleep03016959-21kthreadcore07:45:170
3003542280,4sleep23003675-21wc07:35:002
2990106281,3sleep00-21swapper/007:20:190
2984841281,4sleep12986494-21wc07:15:201
2980377280,1sleep20-21swapper/207:14:482
2980224280,1sleep10-21swapper/107:12:501
2977097281,3sleep02977439-21kthreadcore07:10:170
3094091270,4sleep10-21swapper/109:00:121
3072672270,1sleep10-21swapper/108:38:431
3056933271,2sleep10-21swapper/108:25:171
3056192270,1sleep00-21swapper/008:22:350
3035800271,3sleep23035971-21gltestperf08:05:142
3030239270,2sleep10-21swapper/108:00:121
3008533270,1sleep10-21swapper/107:36:161
2998840271,3sleep12998941-21fschecks_time07:30:141
3099003260,1sleep20-21swapper/209:04:012
3077748260,1sleep20-21swapper/208:40:572
3067120260,1sleep10-21swapper/108:33:471
3050142260,2sleep00-21swapper/008:15:230
3036084261,2sleep00-21swapper/008:05:170
3035052260,1sleep00-21swapper/008:02:310
3029715260,4sleep10-21swapper/107:57:001
3024018260,2sleep00-21swapper/007:50:250
3019270260,1sleep10-21swapper/107:47:181
2999939261,2sleep00-21swapper/007:30:170
2998260260,1sleep20-21swapper/207:29:402
2982945260,2sleep22983157-21smartctl07:15:172
3014796250,1sleep20-21swapper/207:45:152
3014237250,1sleep00-21swapper/007:45:000
3020035240,1sleep10-21swapper/107:50:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional