You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 04:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Wed Nov 26, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
127572212956,68sleep30-21swapper/319:09:003
12747728454,9sleep10-21swapper/119:07:451
12734828152,9sleep20-21swapper/219:06:102
12751628052,19sleep00-21swapper/019:08:160
127780993530,2cyclictest0-21swapper/323:25:143
5784022272,17sleep1574452-21/usr/sbin/munin00:30:191
127780992724,1cyclictest0-21swapper/322:50:193
127780992724,1cyclictest0-21swapper/320:35:003
127780992521,2cyclictest0-21swapper/323:45:213
127780992515,5cyclictest2968862sleep321:33:533
12778099251,18cyclictest187868-21taskset20:00:253
12778099251,18cyclictest187868-21taskset20:00:253
127780992418,3cyclictest0-21swapper/322:22:293
127780992411,9cyclictest288887-21taskset21:28:473
12778099232,13cyclictest424614-21taskset22:55:063
12778099231,11cyclictest581061-21taskset00:34:043
127780992310,10cyclictest321074-21turbostat21:54:553
4058832221,3sleep10-21swapper/122:41:271
12778099222,15cyclictest589029-21turbostat00:39:593
127780992214,4cyclictest4085162sleep322:45:093
12778099221,16cyclictest460559-21chrt23:15:233
12778099212,13cyclictest570984-21taskset00:25:183
12778099212,13cyclictest397800-21taskset22:36:423
12778099211,16cyclictest0-21swapper/320:35:143
12778099211,15cyclictest556774-21chrt00:15:313
12778099211,15cyclictest384633-21chrt22:30:123
5095482205,2sleep1321rcuc/123:46:321
12778099202,13cyclictest438126-21taskset23:01:323
12778099202,13cyclictest438126-21taskset23:01:313
12778099202,13cyclictest280631-21taskset21:23:143
127780992016,2cyclictest0-21swapper/322:40:163
127780992016,2cyclictest0-21swapper/319:44:593
12778099201,15cyclictest0-21swapper/319:25:213
12778099201,14cyclictest0-21swapper/300:21:163
5888832191,7sleep2257700-21diskmemload00:39:492
5785492191,5sleep2578597-21ntpq00:30:202
12778099192,12cyclictest542844-21taskset00:08:263
12778099192,12cyclictest343123-21taskset22:02:183
12778099192,12cyclictest343123-21taskset22:02:183
12778099192,12cyclictest318554-21taskset21:45:433
127780991916,1cyclictest0-21swapper/322:05:283
127780991916,1cyclictest0-21swapper/321:15:123
127780991916,1cyclictest0-21swapper/320:55:173
127780991916,1cyclictest0-21swapper/319:30:243
127780991915,2cyclictest0-21swapper/322:15:163
127780991914,3cyclictest66550irq/131-enp1s0-TxRx-323:30:113
12778099191,14cyclictest0-21swapper/323:20:303
12778099191,13cyclictest452881-21chrt23:10:253
12778099191,13cyclictest264287-21taskset21:12:393
5290042181,12sleep1530555-21tr00:00:201
4391642181,4sleep2257700-21diskmemload23:03:212
4391642181,4sleep2257700-21diskmemload23:03:202
3125842181,5sleep2257700-21diskmemload21:43:552
12778099189,6cyclictest173059-21taskset19:50:113
12778099186,9cyclictest66550irq/131-enp1s0-TxRx-322:10:283
127780991815,1cyclictest0-21swapper/322:25:153
127780991815,1cyclictest0-21swapper/320:40:163
127780991815,1cyclictest0-21swapper/319:55:133
127780991813,3cyclictest0-21swapper/300:00:203
12778099181,15cyclictest0-21swapper/323:43:063
12778099181,14cyclictest446334-21chrt23:07:133
12778099181,13cyclictest0-21swapper/323:55:133
12778099181,13cyclictest0-21swapper/321:36:523
12778099181,13cyclictest0-21swapper/300:10:043
12778099180,15cyclictest204367-21taskset20:19:343
5733372171,10sleep2257700-21diskmemload00:28:262
5036882171,4sleep1257700-21diskmemload23:45:001
4144742171,2sleep00-21swapper/022:47:100
3821832171,9sleep1257700-21diskmemload22:27:031
3813352171,11sleep2381345-21ssh22:25:392
3591462171,4sleep0257700-21diskmemload22:12:560
3523772171,2sleep00-21swapper/022:09:420
3438852171,4sleep1257700-21diskmemload22:03:391
3438852171,4sleep1257700-21diskmemload22:03:381
3271552171,3sleep20-21swapper/221:52:012
2735652171,10sleep2257700-21diskmemload21:19:492
12778099177,7cyclictest247397-21chrt21:00:013
127780991713,2cyclictest0-21swapper/320:50:173
12778099171,12cyclictest493275-21taskset23:35:563
12778099171,12cyclictest311376-21taskset21:41:503
12778099171,11cyclictest133045-21sleep19:15:003
12778099170,14cyclictest0-21swapper/323:50:063
5514182161,10sleep2257700-21diskmemload00:14:482
5262412161,10sleep2526248-21ssh23:57:392
4405422161,10sleep0440747-21cstates23:05:140
3906812160,2sleep20-21swapper/222:33:062
3674992160,3sleep10-21swapper/122:18:491
3667692160,2sleep00-21swapper/022:17:320
3432222161,3sleep0257700-21diskmemload22:02:290
3432222161,3sleep0257700-21diskmemload22:02:280
3028182161,2sleep10-21swapper/121:35:431
2968962161,4sleep0296909-21ssh21:33:550
12778099161,9cyclictest136016-21taskset19:15:123
12778099161,3cyclictest0-21swapper/321:55:213
12778099161,3cyclictest0-21swapper/321:55:213
127780991613,1cyclictest0-21swapper/320:05:153
127780991612,2cyclictest0-21swapper/321:05:243
12778099161,11cyclictest172816-21turbostat19:49:593
12778099161,10cyclictest204683-21chrt20:20:093
12778099160,11cyclictest194143-21taskset20:10:153
5424082151,9sleep2257700-21diskmemload00:07:382
4528132152,7sleep2452880-21cat23:10:252
4246292151,7sleep10-21swapper/122:55:091
4157952151,9sleep2257700-21diskmemload22:49:212
4077502151,9sleep0407762-21ssh22:44:210
3991412151,6sleep2257700-21diskmemload22:38:582
3195262151,2sleep10-21swapper/121:47:221
2648942151,3sleep1257700-21diskmemload21:13:401
2157772151,9sleep2215987-21hddtemp_smartct20:30:152
1744262151,9sleep0176166-21cut19:50:180
12778099151,4cyclictest145898-21sleep19:25:003
127780991511,2cyclictest0-21swapper/320:25:153
127780991511,2cyclictest0-21swapper/319:35:133
12778099151,11cyclictest66550irq/131-enp1s0-TxRx-320:47:323
5875832140,1sleep10-21swapper/100:37:111
5362682143,5sleep0536292-21apt-get00:05:110
5352912141,4sleep0535293-21ssh00:04:040
5195682140,1sleep10-21swapper/123:54:511
5035632141,3sleep0503572-21ssh23:44:480
4138282141,2sleep1257700-21diskmemload22:46:071
3987822141,2sleep10-21swapper/122:38:211
3275062141,9sleep1327517-21ssh21:52:361
2953982141,8sleep1295412-21ssh21:31:231
2871722141,7sleep10-21swapper/121:26:011
1372612141,8sleep1139037-21cut19:15:151
5647152131,8sleep1564720-21ssh00:21:011
5583522131,3sleep0558357-21ssh00:18:140
5355282131,6sleep20-21swapper/200:04:262
5173202131,8sleep0517332-21ssh23:51:170
4868632131,8sleep1257700-21diskmemload23:33:491
4702032131,8sleep2257700-21diskmemload23:22:162
4622332131,5sleep10-21swapper/123:17:221
4541932131,6sleep1257700-21diskmemload23:12:141
3513952131,6sleep2257700-21diskmemload22:07:592
3438562131,5sleep2257700-21diskmemload22:03:362
3438562131,5sleep2257700-21diskmemload22:03:352
3208422131,7sleep2320852-21ssh21:49:352
1395912131,2sleep00-21swapper/019:15:180
5568812120,1sleep20-21swapper/200:15:432
5500302121,4sleep1257700-21diskmemload00:12:231
5180052121,3sleep2518018-21ssh23:52:222
5002162121,7sleep2500259-21ntpq23:40:202
4941432121,5sleep0257700-21diskmemload23:37:250
4640962121,4sleep1257700-21diskmemload23:20:021
4463232125,1sleep20-21swapper/223:07:122
4228942121,5sleep00-21swapper/022:52:390
3827372121,6sleep0382752-21ssh22:27:580
3578422121,4sleep1257700-21diskmemload22:10:401
3123342121,6sleep0257700-21diskmemload21:43:290
3046782121,4sleep0257700-21diskmemload21:38:590
2801292121,3sleep10-21swapper/121:22:301
2800702120,1sleep00-21swapper/021:22:240
2666452122,4sleep1266725-21irqstats21:15:151
2639422121,6sleep20-21swapper/221:12:072
2621872121,6sleep0262245-21ntpq21:10:220
2479712121,6sleep2248014-21kthreadcore21:00:172
2065882121,7sleep2208390-21kthreadcore20:20:172
1414962121,7sleep1143166-21kthreadcore19:20:171
5867382111,4sleep0257700-21diskmemload00:35:380
5794372111,3sleep0257700-21diskmemload00:30:490
5670322111,4sleep0567053-21apt-get00:25:120
5644952111,2sleep2564501-21ssh00:20:342
5582552111,4sleep1257700-21diskmemload00:18:031
5489172111,3sleep00-21swapper/000:10:290
5252852111,5sleep1257700-21diskmemload23:56:061
5242782111,4sleep0257700-21diskmemload23:55:210
5038002111,4sleep2257700-21diskmemload23:45:012
4852102111,3sleep00-21swapper/023:31:010
4800202111,3sleep20-21swapper/223:30:052
4791492111,3sleep10-21swapper/123:29:041
4789302111,4sleep2257700-21diskmemload23:28:422
4775102111,4sleep0257700-21diskmemload23:26:150
4717192111,4sleep0257700-21diskmemload23:24:510
4614972111,4sleep2257700-21diskmemload23:16:072
4532512111,3sleep00-21swapper/023:10:400
4407482111,4sleep1257700-21diskmemload23:05:151
4245552111,4sleep2257700-21diskmemload22:55:022
4221052111,3sleep20-21swapper/222:51:262
4219662111,4sleep1257700-21diskmemload22:51:131
4078192111,4sleep2257700-21diskmemload22:44:282
3893542111,4sleep1257700-21diskmemload22:30:531
3848552111,4sleep0257700-21diskmemload22:30:150
3744732111,4sleep0257700-21diskmemload22:22:220
3743622111,3sleep1257700-21diskmemload22:22:071
3586082111,4sleep2257700-21diskmemload22:12:012
3507902110,1sleep10-21swapper/122:06:581
3351522111,6sleep2335162-21ssh21:57:222
3351522111,6sleep2335162-21ssh21:57:212
3348522110,1sleep00-21swapper/021:56:480
3348522110,1sleep00-21swapper/021:56:470
3193432111,4sleep0257700-21diskmemload21:47:050
3113072111,4sleep1257700-21diskmemload21:41:461
2956102111,3sleep2257700-21diskmemload21:31:452
2878022111,4sleep2257700-21diskmemload21:27:032
2798012111,4sleep2257700-21diskmemload21:21:582
2722192111,3sleep00-21swapper/021:17:300
2386832111,6sleep1238941-21kthreadcore20:50:191
2219652111,6sleep0223621-21kthreadcore20:35:180
1827602110,2sleep00-21swapper/019:55:260
1822772111,5sleep1182328-21ntpq19:55:191
1662362112,4sleep0166274-21ntpq19:40:200
5731712101,2sleep10-21swapper/100:28:051
5661192101,2sleep00-21swapper/000:23:550
5413752100,1sleep10-21swapper/100:05:521
4937782101,3sleep20-21swapper/223:36:522
4919002101,4sleep1257700-21diskmemload23:35:181
4400692101,2sleep10-21swapper/123:04:511
4400692101,2sleep10-21swapper/123:04:511
4333632101,5sleep0434920-21cut23:00:180
4333632101,5sleep0434920-21cut23:00:170
4288202101,5sleep0428877-21ntpq22:55:210
3973062101,3sleep00-21swapper/022:35:560
3742832101,3sleep2257700-21diskmemload22:22:002
3352712101,5sleep1257700-21diskmemload21:57:331
3352712101,5sleep1257700-21diskmemload21:57:321
3040772101,3sleep2257700-21diskmemload21:37:572
1993902101,4sleep2199409-21apt-get20:15:132
1941212100,4sleep2194140-21apt-get20:10:152
1889312101,4sleep0189134-21diskstats20:05:130
1840982100,2sleep10-21swapper/120:00:171
1840982100,2sleep10-21swapper/120:00:161
1628522100,6sleep1164465-21cut19:40:171
1330132101,4sleep2133040-21perf19:15:002
509623290,2sleep00-21swapper/023:46:390
461308291,2sleep00-21swapper/023:15:440
366070291,2sleep20-21swapper/222:16:212
326804291,3sleep00-21swapper/021:51:270
287534291,2sleep00-21swapper/021:26:360
242425291,4sleep0242554-21grep20:55:140
226850290,1sleep00-21swapper/020:40:160
209958291,4sleep1210216-21df_inode20:25:131
208672291,4sleep1208721-21ntpq20:20:201
199094290,1sleep00-21swapper/020:14:340
167201290,3sleep20-21swapper/219:44:152
160787290,2sleep1171rcu_preempt19:35:231
146645291,4sleep1146919-21cat19:25:171
145173291,4sleep2145318-21ls19:20:252
128522291,4sleep0129964-21cut19:10:180
254010281,3sleep0255734-21kthreadcore21:05:160
247695281,3sleep0247819-21perl21:00:160
242707281,3sleep2243653-21kthreadcore20:55:162
237463281,3sleep0238681-21kthreadcore20:50:180
233072281,3sleep1233693-21kthreadcore20:45:171
231646280,3sleep0231664-21apt-get20:45:120
225448281,4sleep2221022-21/usr/sbin/munin20:35:292
223622281,3sleep1224910-21memory20:35:191
214035281,3sleep20-21swapper/220:25:232
209937281,2sleep00-21swapper/020:25:120
204685281,2sleep00-21swapper/020:20:110
201359281,3sleep0203173-21kthreadcore20:15:180
188384280,1sleep00-21swapper/020:02:280
188384280,1sleep00-21swapper/020:02:270
182330281,2sleep20-21swapper/219:55:202
171836281,3sleep1171885-21ntpq19:45:211
171171280,1sleep20-21swapper/219:45:182
161683281,3sleep2162153-21awk19:40:012
158119281,3sleep0158724-21kthreadcore19:35:180
156244280,1sleep20-21swapper/219:33:122
226922271,2sleep10-21swapper/120:40:171
226347270,1sleep20-21swapper/220:40:092
178068270,1sleep10-21swapper/119:55:001
177724270,4sleep20-21swapper/219:50:402
156050270,1sleep00-21swapper/019:30:460
140878270,2sleep00-21swapper/019:20:120
257537260,1sleep20-21swapper/221:08:022
257478260,4sleep10-21swapper/121:07:171
251273260,1sleep10-21swapper/121:00:211
242139261,2sleep10-21swapper/120:55:111
241584260,1sleep20-21swapper/220:51:202
236370260,4sleep20-21swapper/220:46:422
220468260,1sleep00-21swapper/020:34:470
215662261,2sleep10-21swapper/120:30:141
194105260,1sleep10-21swapper/120:10:021
193655260,1sleep20-21swapper/220:07:272
193547260,4sleep10-21swapper/120:06:061
184170261,2sleep20-21swapper/220:00:182
184170261,2sleep20-21swapper/220:00:172
168364260,2sleep00-21swapper/019:45:160
150772260,1sleep00-21swapper/019:25:400
146920261,2sleep20-21swapper/219:25:182
203784250,1sleep10-21swapper/120:15:281
153670250,2sleep10-21swapper/119:30:191
140260250,1sleep20-21swapper/219:15:482
128115250,1sleep10-21swapper/119:10:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional