You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 12:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Fri Nov 28, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
95977122510,2sleep2959762-21ntpq23:55:202
554345212265,53sleep30-21swapper/319:06:373
55452329074,9sleep10-21swapper/119:08:511
55449128150,9sleep20-21swapper/219:08:282
55434727759,13sleep00-21swapper/019:06:380
7271422400,3sleep0221rcuc/021:35:190
7670022280,3sleep00-21swapper/022:00:120
8538642272,8sleep2853923-21ntp_states22:50:222
554760992723,2cyclictest0-21swapper/322:35:003
554760992623,1cyclictest0-21swapper/323:45:003
55476099243,14cyclictest915475-21taskset23:30:113
55476099242,16cyclictest833015-21taskset22:40:123
554760992420,2cyclictest0-21swapper/322:05:193
554760992419,3cyclictest0-21swapper/323:35:163
10169682241,11sleep11017185-21sort00:35:011
55476099231,19cyclictest707964-21taskset21:24:273
55476099222,13cyclictest979234-21taskset00:10:003
554760992219,1cyclictest0-21swapper/320:43:123
554760992218,2cyclictest0-21swapper/319:40:243
554760992218,2cyclictest0-21swapper/300:10:183
554760992217,3cyclictest0-21swapper/322:35:193
55476099212,13cyclictest970165-21taskset00:03:193
55476099212,13cyclictest913275-21taskset23:27:183
554760992116,2cyclictest0-21swapper/322:13:353
55476099211,15cyclictest749910-21sleep21:50:003
55476099211,15cyclictest717004-21chrt21:30:123
55476099211,15cyclictest715795-21chrt21:28:483
9989212201,5sleep0999015-21ntpq00:20:200
55476099202,13cyclictest806286-21taskset22:22:363
55476099201,9cyclictest960949-21taskset23:56:173
554760992015,2cyclictest0-21swapper/322:50:223
55476099201,15cyclictest946579-21chrt23:48:393
55476099201,15cyclictest0-21swapper/322:58:173
55476099201,14cyclictest0-21swapper/323:08:423
55476099201,14cyclictest0-21swapper/322:45:223
554760992010,7cyclictest993638-21taskset00:18:223
554760992010,7cyclictest993638-21taskset00:18:223
6616992191,14sleep2661746-21/usr/sbin/munin20:45:232
55476099192,12cyclictest954544-21taskset23:53:443
55476099192,12cyclictest814126-21taskset22:27:013
55476099192,12cyclictest791437-21taskset22:15:103
55476099192,12cyclictest770852-21taskset22:00:193
55476099192,12cyclictest738573-21taskset21:40:473
55476099192,12cyclictest699497-21taskset21:19:103
55476099192,12cyclictest1007421-21taskset00:25:363
554760991915,2cyclictest0-21swapper/319:15:003
554760991912,5cyclictest66550irq/131-enp1s0-TxRx-300:34:363
55476099191,14cyclictest898719-21chrt23:20:093
55476099191,14cyclictest898719-21chrt23:20:093
55476099191,14cyclictest763572-21chrt21:55:503
55476099191,14cyclictest0-21swapper/323:15:123
55476099191,14cyclictest0-21swapper/300:39:273
55476099190,11cyclictest755117-21taskset21:50:333
9383612181,5sleep1683773-21diskmemload23:43:211
9070772181,12sleep287-21kswapd023:25:022
8375182181,11sleep0837567-21switchtime22:40:260
8074662181,8sleep1807477-21ssh22:24:231
7982262181,2sleep10-21swapper/122:18:181
7325802181,3sleep10-21swapper/121:39:091
6160652181,7sleep0617681-21tr20:05:170
55476099189,5cyclictest628608-21taskset20:15:173
554760991815,1cyclictest0-21swapper/320:35:163
554760991814,2cyclictest0-21swapper/323:00:513
554760991814,2cyclictest0-21swapper/300:20:173
554760991812,3cyclictest640765-21taskset20:25:233
55476099181,14cyclictest732870-21chrt21:39:343
55476099181,12cyclictest882317-21taskset23:10:113
55476099180,14cyclictest684154-21chrt21:10:123
9359102171,4sleep2935968-21ntpq23:40:212
9232842171,10sleep2923342-21munin-run23:35:012
8735042171,4sleep00-21swapper/023:04:350
8648912171,4sleep1683773-21diskmemload22:58:561
8311662171,4sleep1683773-21diskmemload22:37:461
8305852170,7sleep2683773-21diskmemload22:36:552
8155242171,9sleep0683773-21diskmemload22:29:090
7731952171,3sleep1683773-21diskmemload22:02:591
7667092171,11sleep2766740-21ssh22:00:112
7006342171,10sleep1700763-21diskstats21:20:141
6913112171,4sleep1683773-21diskmemload21:14:081
6889572171,2sleep00-21swapper/021:10:300
554760991714,1cyclictest0-21swapper/319:30:263
55476099171,13cyclictest0-21swapper/320:00:273
9856432161,2sleep20-21swapper/200:12:532
8222592160,1sleep20-21swapper/222:31:542
8133762161,10sleep2683773-21diskmemload22:25:532
7638912161,3sleep0683773-21diskmemload21:56:200
7497752161,2sleep10-21swapper/121:49:521
7243972161,4sleep1683773-21diskmemload21:34:161
7067652161,4sleep0683773-21diskmemload21:22:410
55476099164,9cyclictest572323-21chrt19:21:233
55476099161,9cyclictest583417-21taskset19:35:053
554760991613,1cyclictest0-21swapper/319:50:283
55476099161,12cyclictest646743-21chrt20:33:173
55476099161,12cyclictest0-21swapper/319:45:183
55476099161,12cyclictest0-21swapper/319:45:183
55476099160,13cyclictest0-21swapper/321:05:243
55476099160,13cyclictest0-21swapper/319:59:403
55476099160,10cyclictest662616-21taskset20:49:563
9921992151,3sleep0683773-21diskmemload00:15:550
9921992151,3sleep0683773-21diskmemload00:15:550
9424762151,8sleep2943713-21memory23:45:192
9196182151,9sleep0919682-21ntpq23:30:210
9059222151,3sleep1683773-21diskmemload23:24:061
9059222151,3sleep1683773-21diskmemload23:24:051
8808842151,2sleep10-21swapper/123:08:221
8557692151,8sleep1683773-21diskmemload22:52:261
8471452150,3sleep10-21swapper/122:46:441
8153182151,3sleep10-21swapper/122:28:481
7772572150,3sleep20-21swapper/222:05:202
7563502151,9sleep2683773-21diskmemload21:52:222
7462322151,9sleep0746300-21ssh21:45:220
554760991512,1cyclictest0-21swapper/320:50:153
55476099151,11cyclictest572896-21chrt19:25:113
55476099150,13cyclictest0-21swapper/321:00:203
55476099150,13cyclictest0-21swapper/320:55:263
55476099150,13cyclictest0-21swapper/320:20:173
55476099150,12cyclictest0-21swapper/319:15:153
55476099150,11cyclictest0-21swapper/320:05:163
10152572151,3sleep01015267-21ssh00:31:230
9476272141,8sleep1683773-21diskmemload23:50:031
9293482141,7sleep0683773-21diskmemload23:36:430
9196842143,6sleep1919741-21cut23:30:221
8963122141,9sleep0896326-21ssh23:16:540
8420392141,8sleep2842442-21kthreadcore22:45:172
7967612141,9sleep0796770-21sh22:16:090
7814562141,8sleep0781469-21ssh22:07:590
7653412140,2sleep20-21swapper/221:58:312
7228972141,2sleep00-21swapper/021:32:030
6995972140,2sleep00-21swapper/021:19:200
5992902141,9sleep2599338-21munin-run19:50:002
5992902141,9sleep2599338-21munin-run19:50:002
5884572141,8sleep0588682-21apt-key19:40:000
554760991410,2cyclictest0-21swapper/320:10:213
9122332130,1sleep10-21swapper/123:25:451
7957372131,6sleep2683773-21diskmemload22:15:222
7902262131,6sleep1683773-21diskmemload22:13:451
7386812130,8sleep1683773-21diskmemload21:40:581
7229192131,2sleep20-21swapper/221:32:052
7146202131,7sleep1683773-21diskmemload21:27:031
6740472131,8sleep1675366-21grep21:00:211
6421342131,8sleep1642345-21sh20:30:151
6054452136,3sleep0605964-21cut19:55:150
5632192131,8sleep2564921-21cut19:15:172
9944652121,5sleep20-21swapper/200:19:542
9944652121,5sleep20-21swapper/200:19:542
9837542121,3sleep1983881-21sendmail_mailqu00:10:231
9703612121,4sleep0683773-21diskmemload00:03:410
9630912121,5sleep1683773-21diskmemload23:59:531
9545352121,7sleep2683773-21diskmemload23:53:442
9374012121,5sleep0683773-21diskmemload23:41:440
9046532121,3sleep00-21swapper/023:22:110
9046532121,3sleep00-21swapper/023:22:110
8987062121,5sleep2683773-21diskmemload23:20:092
8987062121,5sleep2683773-21diskmemload23:20:092
8247542121,5sleep00-21swapper/022:35:130
7874622121,7sleep0683773-21diskmemload22:10:230
7481332121,5sleep2683773-21diskmemload21:47:242
7405592126,3sleep0740571-21rm21:43:450
6963882121,7sleep2696454-21sh21:15:212
6753682121,3sleep20-21swapper/221:00:222
6521262121,6sleep0652197-21munin-run20:40:000
5781902121,5sleep2578442-21df19:30:122
10075722121,4sleep1683773-21diskmemload00:25:551
10002922121,7sleep11000300-21ssh00:21:301
9990752111,5sleep2999127-21ntpq00:20:222
9921312111,3sleep10-21swapper/100:15:481
9921312111,3sleep10-21swapper/100:15:481
9770662111,4sleep0683773-21diskmemload00:06:450
9760322111,6sleep10-21swapper/100:05:261
9706522111,3sleep2683773-21diskmemload00:04:152
9675652111,5sleep1967642-21memory00:00:221
9618112111,4sleep0683773-21diskmemload23:57:430
9539482111,6sleep0953958-21ssh23:52:460
9438622111,4sleep0683773-21diskmemload23:45:210
9408372111,5sleep1942474-21grep23:45:181
9287392111,6sleep1683773-21diskmemload23:35:471
8823632111,3sleep10-21swapper/123:10:141
8811452111,4sleep2683773-21diskmemload23:08:462
8709332111,3sleep2683773-21diskmemload23:00:372
8633712111,3sleep2683773-21diskmemload22:56:362
8562522111,4sleep0683773-21diskmemload22:53:090
8487022111,3sleep0683773-21diskmemload22:49:040
8384702111,3sleep10-21swapper/122:41:231
8381402111,3sleep20-21swapper/222:40:522
8068972111,4sleep2683773-21diskmemload22:23:322
7896342111,2sleep20-21swapper/222:12:542
7809712111,4sleep1683773-21diskmemload22:07:161
7636802111,3sleep10-21swapper/121:56:011
7579552111,3sleep0683773-21diskmemload21:54:440
7554962111,3sleep10-21swapper/121:51:081
7411642111,3sleep20-21swapper/221:44:412
7302582111,6sleep2730270-21sh21:35:382
7159352111,5sleep0683773-21diskmemload21:29:020
7153462111,4sleep2683773-21diskmemload21:28:082
6993902110,1sleep10-21swapper/121:19:011
6630232111,4sleep1663170-21diskstats20:50:151
5728742110,2sleep20-21swapper/219:25:112
10227442111,4sleep2683773-21diskmemload00:36:462
10225032111,3sleep0683773-21diskmemload00:36:160
10070012111,3sleep2683773-21diskmemload00:25:272
9777802101,3sleep20-21swapper/200:07:522
9238202101,4sleep2923946-21diskstats23:35:122
9124392101,3sleep00-21swapper/023:26:040
8966382101,3sleep1683773-21diskmemload23:17:241
8895122101,3sleep00-21swapper/023:14:020
8823082101,3sleep2683773-21diskmemload23:10:112
8793352101,3sleep0683773-21diskmemload23:06:000
8699812101,4sleep1683773-21diskmemload23:00:191
8577782101,3sleep0683773-21diskmemload22:55:090
8240592101,2sleep00-21swapper/022:34:330
8215942100,2sleep10-21swapper/122:30:531
7999612101,3sleep00-21swapper/022:20:050
6898872101,5sleep2689897-21rm21:11:562
6739832101,2sleep00-21swapper/021:00:200
6681432100,4sleep00-21swapper/020:55:030
6629052101,5sleep0663021-21tr20:50:140
6525732101,4sleep1652696-21diskstats20:40:141
6511552101,4sleep2651204-21ntpq20:35:212
6249712101,5sleep2624975-21mailstats20:10:242
6142182101,4sleep0614268-21ntpq20:00:200
5874382100,2sleep10-21swapper/119:35:201
5621932101,4sleep1562212-21apt-get19:15:121
10229822101,3sleep10-21swapper/100:37:131
10157022101,3sleep2683773-21diskmemload00:32:212
985417291,2sleep00-21swapper/000:12:270
891088291,3sleep20-21swapper/223:15:162
704665290,2sleep20-21swapper/221:20:222
672210291,4sleep2672264-21ntpq20:55:212
661645291,4sleep1661692-21ntpq20:45:221
658206291,4sleep0659474-21grep20:45:170
652978294,2sleep0654161-21kthreadcore20:40:160
640358291,4sleep2640525-21grep20:25:192
625959291,3sleep1625977-21apt-get20:15:121
617682291,4sleep1615462-21/usr/sbin/munin20:05:181
601004291,4sleep1602766-21grep19:50:191
594256291,3sleep1594275-21apt-get19:45:101
594256291,3sleep1594275-21apt-get19:45:101
588687291,3sleep1588472-21apt-get19:40:011
572286290,2sleep10-21swapper/119:20:531
559271291,4sleep0559544-21grep19:10:260
558920291,4sleep2558966-21ntpq19:10:202
1002588291,3sleep01002614-21apt-get00:25:130
682718281,3sleep0682769-21ntpq21:05:210
668154280,3sleep10-21swapper/120:55:121
641839280,1sleep20-21swapper/220:30:082
635419281,3sleep01-21systemd20:20:240
631255280,3sleep1631274-21apt-get20:20:131
625429280,1sleep10-21swapper/120:11:401
608939282,2sleep134-21ksoftirqd/119:55:201
600119281,3sleep0601001-21kthreadcore19:50:180
593301281,4sleep0593344-21grep19:40:240
578168280,2sleep10-21swapper/119:30:111
577700280,1sleep00-21swapper/019:27:260
569399281,3sleep00-21swapper/019:20:200
566114281,3sleep0566254-21/usr/sbin/munin19:15:190
558829281,3sleep10-21swapper/119:10:191
683425270,1sleep10-21swapper/121:05:501
680818271,3sleep2682536-21latency21:05:192
647140270,2sleep10-21swapper/120:35:121
637122271,2sleep0638535-21kthreadcore20:25:170
636164270,1sleep20-21swapper/220:23:272
630963270,4sleep00-21swapper/020:19:560
626916271,2sleep20-21swapper/220:15:172
625655270,1sleep00-21swapper/020:14:310
615159271,1sleep20-21swapper/220:04:312
615075270,1sleep10-21swapper/120:03:261
603903271,2sleep20-21swapper/219:50:282
572605270,1sleep20-21swapper/219:24:552
667549260,1sleep20-21swapper/220:50:402
657092260,1sleep20-21swapper/220:41:302
620154260,1sleep20-21swapper/220:06:192
609891260,4sleep20-21swapper/219:59:502
599271260,1sleep00-21swapper/019:49:460
599271260,1sleep00-21swapper/019:49:460
584184261,2sleep20-21swapper/219:35:172
645929250,1sleep00-21swapper/020:30:220
641520250,1sleep10-21swapper/120:29:151
592897250,2sleep20-21swapper/219:40:182
578157250,1sleep00-21swapper/019:30:020
573167250,1sleep10-21swapper/119:25:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional