You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-09 - 21:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Mon Mar 09, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
529005212855,69sleep30-21swapper/307:05:413
92021821000,2sleep30-21swapper/311:35:493
52837828863,14sleep20-21swapper/207:05:202
52921928253,12sleep10-21swapper/107:08:241
52917027956,6sleep00-21swapper/007:07:470
52947199251,20cyclictest0-21swapper/312:30:233
7898962240,4sleep0221rcuc/010:21:570
529471992319,2cyclictest0-21swapper/310:10:173
9040542221,10sleep2904067-21sh11:27:352
89531522110,7sleep0895329-21ssh11:22:360
5896292211,6sleep0589670-21ntpq08:00:210
52947199214,13cyclictest650429-21taskset09:00:143
52947199210,18cyclictest0-21swapper/311:11:213
7791952201,8sleep2779294-21ntpq10:15:192
52947199205,12cyclictest0-21swapper/309:44:543
529471992017,1cyclictest0-21swapper/310:20:183
52947199201,15cyclictest66950irq/132-enp1s0-TxRx-311:50:563
52947199201,15cyclictest0-21swapper/310:55:153
529471992011,5cyclictest0-21swapper/309:30:113
52947199201,15cyclictest0-21swapper/307:36:393
52947199201,15cyclictest0-21swapper/307:15:113
52947199201,14cyclictest0-21swapper/311:15:223
52947199191,16cyclictest0-21swapper/312:35:173
52947199191,15cyclictest0-21swapper/310:42:223
52947199191,14cyclictest0-21swapper/311:22:563
52947199191,14cyclictest0-21swapper/310:36:193
52947199191,14cyclictest0-21swapper/310:25:223
52947199191,14cyclictest0-21swapper/309:50:253
52947199191,14cyclictest0-21swapper/309:10:193
52947199190,2cyclictest0-21swapper/309:18:503
52947199190,16cyclictest66950irq/132-enp1s0-TxRx-311:40:183
9238152181,3sleep00-21swapper/011:40:120
8786602181,3sleep00-21swapper/011:13:420
5429282181,12sleep0543018-21tune2fs07:20:130
52947199188,6cyclictest660251-21turbostat09:10:013
52947199184,11cyclictest0-21swapper/312:05:123
529471991812,4cyclictest0-21swapper/307:13:323
52947199181,14cyclictest0-21swapper/312:25:123
52947199181,14cyclictest0-21swapper/312:24:193
52947199181,14cyclictest0-21swapper/312:01:393
52947199181,14cyclictest0-21swapper/311:45:193
52947199181,14cyclictest0-21swapper/311:31:533
52947199181,14cyclictest0-21swapper/311:25:203
52947199181,14cyclictest0-21swapper/311:06:353
52947199181,14cyclictest0-21swapper/311:00:113
52947199181,14cyclictest0-21swapper/309:55:203
52947199181,14cyclictest0-21swapper/309:45:353
52947199181,14cyclictest0-21swapper/309:37:423
52947199181,14cyclictest0-21swapper/309:26:523
52947199181,14cyclictest0-21swapper/309:23:553
52947199180,15cyclictest0-21swapper/310:46:223
52947199180,14cyclictest66950irq/132-enp1s0-TxRx-310:33:403
52947199180,14cyclictest0-21swapper/312:15:113
52947199180,14cyclictest0-21swapper/310:02:063
9564492171,2sleep10-21swapper/111:57:321
9311022171,4sleep1660261-21diskmemload11:43:371
8420842170,7sleep0660261-21diskmemload10:51:370
7022922170,3sleep20-21swapper/209:32:222
52947199171,3cyclictest617246-21taskset08:25:323
52947199171,3cyclictest0-21swapper/311:55:113
529471991712,3cyclictest0-21swapper/312:10:183
52947199171,14cyclictest0-21swapper/310:15:003
52947199171,13cyclictest0-21swapper/310:50:053
52947199171,13cyclictest0-21swapper/310:05:033
52947199171,12cyclictest638927-21turbostat08:50:003
8876032161,2sleep00-21swapper/011:18:590
8695112161,10sleep1660261-21diskmemload11:08:101
8521822161,10sleep1660261-21diskmemload10:58:271
8012082161,3sleep20-21swapper/210:30:022
7642272161,3sleep10-21swapper/110:07:591
7585232168,4sleep2760157-21cut10:05:172
7008992161,3sleep00-21swapper/009:30:310
5641102160,3sleep0564269-21cat07:40:130
52947199161,12cyclictest627921-21chrt08:36:063
52947199161,12cyclictest627921-21chrt08:36:053
52947199161,10cyclictest0-21swapper/307:20:113
52947199160,13cyclictest0-21swapper/307:55:243
9904182151,4sleep2660261-21diskmemload12:16:092
9891352152,7sleep0989176-21ntpq12:15:190
9804052151,7sleep1980466-21ntpq12:10:191
9417632151,7sleep1941891-21irqstats11:50:151
9029672150,1sleep10-21swapper/111:26:091
8969832151,3sleep1660261-21diskmemload11:24:491
7577862155,6sleep0757995-21if_enp1s010:05:150
6936182151,3sleep1660261-21diskmemload09:27:281
529471991512,2cyclictest0-21swapper/308:50:143
529471991512,1cyclictest0-21swapper/308:00:153
52947199151,11cyclictest0-21swapper/307:45:113
52947199151,11cyclictest0-21swapper/307:30:103
52947199150,13cyclictest0-21swapper/308:55:143
52947199150,13cyclictest0-21swapper/308:30:123
52947199150,13cyclictest0-21swapper/308:10:133
52947199150,13cyclictest0-21swapper/308:05:183
52947199150,12cyclictest0-21swapper/308:20:163
52947199150,11cyclictest0-21swapper/307:50:103
10019162151,2sleep20-21swapper/212:24:232
9586072141,9sleep0958622-21ssh12:00:040
9554912140,2sleep20-21swapper/211:56:172
7814622141,2sleep00-21swapper/010:17:250
7303532141,8sleep1660261-21diskmemload09:49:441
52947199149,3cyclictest0-21swapper/307:25:103
529471991411,1cyclictest0-21swapper/307:40:173
52947199140,11cyclictest0-21swapper/308:40:183
52947199140,11cyclictest0-21swapper/308:15:133
9552602131,8sleep0955275-21ssh11:55:590
8866112130,1sleep1660261-21diskmemload11:17:401
8610662131,8sleep2861077-21ssh11:03:342
8508342130,5sleep20-21swapper/210:56:402
8436642131,3sleep2843675-21ssh10:53:432
8340802131,3sleep0834094-21ssh10:47:360
8263042131,7sleep1660261-21diskmemload10:43:521
8092932131,7sleep0660261-21diskmemload10:34:330
8078282131,6sleep1660261-21diskmemload10:32:371
7753972131,6sleep1775616-21hddtemp_smartct10:15:141
7725862131,7sleep00-21swapper/010:12:320
7711012131,3sleep1660261-21diskmemload10:10:331
6853132131,3sleep0685324-21ssh09:23:020
5375852137,3sleep1537701-21grep07:15:121
10208842131,8sleep11021713-21cut12:35:161
10098332131,7sleep2660261-21diskmemload12:28:162
9934182121,7sleep1993882-21packagekitd12:20:011
9921632122,4sleep1660261-21diskmemload12:18:291
9129182120,5sleep20-21swapper/211:32:442
8705542121,7sleep0870568-21ssh11:09:330
7975322121,7sleep2797544-21ssh10:25:292
7921992121,7sleep1792258-21unin-run10:25:001
7478652121,6sleep2660261-21diskmemload09:59:522
7438982121,7sleep1743964-21open_files09:55:221
7369132120,6sleep10-21swapper/109:51:531
7195642121,7sleep1719572-21ssh09:42:021
7194002121,7sleep0719411-21ssh09:41:500
6325432121,5sleep2632594-21ntpq08:40:192
6272332121,7sleep0627281-21ntpq08:35:200
6272332121,7sleep0627281-21ntpq08:35:200
6068052120,1sleep00-21swapper/008:20:010
6002882121,6sleep2600335-21ntpq08:10:212
10100622121,6sleep1660261-21diskmemload12:28:351
10070852121,7sleep01007159-21/usr/sbin/munin12:25:210
9945682112,4sleep00-21swapper/012:20:160
9757192111,7sleep1660261-21diskmemload12:09:511
9394682110,7sleep2939481-21rm11:48:072
9218782111,4sleep1660261-21diskmemload11:38:001
9202062111,4sleep2660261-21diskmemload11:35:492
9113692111,3sleep00-21swapper/011:30:410
8975192111,5sleep0897545-21apt-get11:25:120
8777632111,4sleep1660261-21diskmemload11:12:301
8579562111,5sleep1858004-21ntpq11:00:201
8528332110,2sleep00-21swapper/010:59:190
8412432110,6sleep1660261-21diskmemload10:50:301
8356232111,6sleep1660261-21diskmemload10:49:381
8349212110,2sleep2834935-21ssh10:48:422
8178002111,3sleep00-21swapper/010:39:140
8169982111,3sleep10-21swapper/110:38:111
8162022111,3sleep20-21swapper/210:37:082
7552412111,4sleep1660261-21diskmemload10:03:031
7483882112,3sleep2748410-21apt-get10:00:122
7378442111,4sleep00-21swapper/009:53:080
7374212111,3sleep20-21swapper/209:52:342
7280782110,8sleep2728091-21rm09:46:452
7277052110,1sleep00-21swapper/009:46:150
7096582111,6sleep1709671-21ssh09:35:331
7088842111,5sleep0708940-21ntpq09:35:200
6938492111,3sleep00-21swapper/009:27:470
6771702110,2sleep2677181-21ssh09:18:512
6677922110,1sleep00-21swapper/009:13:270
6652942111,5sleep1665552-21users09:10:281
6647822111,5sleep2664851-21ntpq09:10:212
6557272111,2sleep20-21swapper/209:05:162
6459492111,7sleep1647697-21tr08:55:171
6432582112,4sleep2643305-21ntpq08:50:212
6348232111,6sleep2636510-21cut08:45:162
6248392111,6sleep1626630-21cut08:35:171
6248392111,6sleep1626630-21cut08:35:171
5911962111,5sleep2591405-21hddtemp_smartct08:05:122
5696132111,5sleep2569865-21df07:45:122
5469672111,7sleep2546969-21grep07:20:232
5299302111,6sleep2530045-21fschecks_time07:10:132
9831202101,4sleep2660261-21diskmemload12:13:052
9732022101,3sleep20-21swapper/212:06:312
9727662101,3sleep00-21swapper/012:05:570
9666002101,3sleep20-21swapper/212:04:232
9494952101,4sleep0660261-21diskmemload11:54:510
9488502101,3sleep20-21swapper/211:54:012
9367722101,4sleep0936836-21ntpq11:45:210
9279962101,3sleep20-21swapper/211:40:222
9143862101,4sleep1660261-21diskmemload11:34:391
8948342101,6sleep2894845-21sh11:21:582
8805522101,5sleep2880663-21irqstats11:15:162
8772202101,4sleep2660261-21diskmemload11:11:472
8701322101,3sleep20-21swapper/211:08:592
8587922100,1sleep00-21swapper/011:00:350
8256822100,2sleep00-21swapper/010:43:030
8246982101,4sleep2660261-21diskmemload10:41:452
8002452101,2sleep10-21swapper/110:29:051
7925872101,4sleep0792611-21apt-get10:25:110
7561462101,4sleep0660261-21diskmemload10:04:150
7197782101,2sleep20-21swapper/209:42:202
7050142101,4sleep2705235-21hddtemp_smartct09:35:142
7033452101,4sleep10-21swapper/109:33:451
6949062100,1sleep20-21swapper/209:29:092
6849642101,4sleep1660261-21diskmemload09:22:341
6839222101,2sleep20-21swapper/209:21:112
6440822101,2sleep10-21swapper/108:53:071
6288112101,4sleep1629021-21hddtemp_smartct08:40:141
6056482101,4sleep1605699-21ntpq08:15:211
6015662101,4sleep0601585-21apt-get08:15:120
5914792101,5sleep0592883-21kthreadcore08:05:140
5856072101,4sleep1585863-21df08:00:141
5841372101,4sleep1584278-21ntpq07:55:171
5832152101,4sleep0584136-21memory07:55:160
5807842101,6sleep2581460-21kthreadcore07:55:142
5693282101,4sleep2569353-21munin-run07:45:002
5594142101,4sleep0561102-21cut07:35:160
5573992101,4sleep1557457-21ntpq07:30:201
5373262101,4sleep0537584-21df_inode07:15:120
10266522101,4sleep0660261-21diskmemload12:37:240
10164452101,2sleep20-21swapper/212:30:292
10114762101,5sleep1660261-21diskmemload12:30:071
976101291,2sleep00-21swapper/012:10:020
958623291,4sleep1958634-21ssh12:00:051
938897291,3sleep1660261-21diskmemload11:47:221
921456291,2sleep00-21swapper/011:37:270
790883291,3sleep2660261-21diskmemload10:23:172
772915290,4sleep2660261-21diskmemload10:12:592
745361291,4sleep0745373-21sh09:56:330
674908291,4sleep0660261-21diskmemload09:15:520
669596291,4sleep1660261-21diskmemload09:15:081
654028290,5sleep10-21swapper/109:00:221
648518291,4sleep0648569-21gcc08:55:200
633844291,2sleep10-21swapper/108:45:111
617858290,3sleep20-21swapper/208:30:092
617405291,1sleep00-21swapper/008:27:340
607766291,4sleep2609201-21grep08:20:152
607194291,3sleep1607213-21apt-get08:20:101
601304291,4sleep1601319-21latency_hist08:15:011
579027291,4sleep1579084-21/usr/sbin/munin07:50:201
574962291,4sleep00-21swapper/007:50:110
563672291,4sleep1563717-40tracker-extract07:39:461
548409291,4sleep0548525-21irqstats07:25:140
534691291,2sleep00-21swapper/007:14:360
1024810290,1sleep20-21swapper/212:35:262
1011518291,3sleep00-21swapper/012:30:100
649847280,3sleep2649866-21apt-get09:00:112
618137281,3sleep0618161-21/usr/sbin/munin08:30:110
612558281,3sleep2612713-21/usr/sbin/munin08:25:122
612536281,2sleep1612555-21apt-get08:25:111
565252281,3sleep1566352-40gst-plugin-scan07:40:171
562470280,4sleep2562643-21strings07:35:182
549629281,3sleep2551425-21kthreadcore07:25:162
655454271,3sleep00-21swapper/009:05:140
654764270,1sleep00-21swapper/009:03:090
649545270,1sleep20-21swapper/208:59:512
628211270,1sleep20-21swapper/208:39:472
628211270,1sleep20-21swapper/208:39:472
601182270,1sleep00-21swapper/008:13:320
592886271,2sleep10-21swapper/108:05:141
585864270,1sleep20-21swapper/208:00:152
569591270,2sleep10-21swapper/107:45:111
558270270,1sleep20-21swapper/207:34:022
553308270,4sleep00-21swapper/007:30:110
547563270,1sleep10-21swapper/107:23:271
535251271,3sleep2536511-21cat07:15:022
659927260,4sleep10-21swapper/109:06:071
603774260,3sleep2421rcuc/208:15:182
579622260,1sleep20-21swapper/207:50:372
574552260,4sleep00-21swapper/007:48:270
553024260,3sleep1553137-21sed07:30:001
531077261,2sleep10-21swapper/107:10:161
643367250,1sleep00-21swapper/008:50:230
621767250,2sleep10-21swapper/108:30:151
634169240,1sleep00-21swapper/008:45:140
628235240,1sleep00-21swapper/008:40:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional