You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-08 - 00:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Fri Nov 07, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2268964215255,92sleep30-21swapper/307:06:493
226908228454,10sleep20-21swapper/207:08:222
226840028252,9sleep10-21swapper/107:05:241
226898127753,6sleep00-21swapper/007:07:030
2269336992421,1cyclictest0-21swapper/307:39:243
226933699232,13cyclictest2303547-21taskset07:40:123
24519572221,9sleep02452590-21grep09:45:150
226933699222,14cyclictest2432904-21taskset09:30:223
226933699221,17cyclictest2449340-21taskset09:42:003
226933699220,18cyclictest2504520-21taskset10:19:523
226933699213,14cyclictest2625950-21taskset11:36:373
2269336992118,1cyclictest0-21swapper/309:55:003
2269336992111,8cyclictest0-21swapper/312:00:113
226933699211,17cyclictest0-21swapper/309:12:303
226933699211,17cyclictest0-21swapper/309:12:303
226933699211,16cyclictest2565305-21taskset10:57:453
26359202201,9sleep22636109-21expr11:45:152
226933699208,8cyclictest2671579-21taskset12:06:033
2269336992017,1cyclictest0-21swapper/312:24:203
2269336992017,1cyclictest0-21swapper/311:05:213
2269336992017,1cyclictest0-21swapper/310:05:143
2269336992016,2cyclictest0-21swapper/309:55:173
226933699201,15cyclictest66550irq/131-enp1s0-TxRx-310:10:073
226933699201,15cyclictest2596095-21chrt11:18:333
26720292191,8sleep02398796-21diskmemload12:06:490
26193712191,3sleep20-21swapper/211:33:442
24354372191,2sleep10-21swapper/109:34:341
23697012190,7sleep02369702-21kthreadcore08:40:160
226933699198,7cyclictest2688684-21taskset12:17:423
226933699192,13cyclictest2634488-21taskset11:43:143
226933699192,12cyclictest2713793-21taskset12:32:573
2269336991915,2cyclictest0-21swapper/310:35:273
226933699191,15cyclictest0-21swapper/310:00:073
226933699191,14cyclictest2555779-21chrt10:50:183
226933699191,14cyclictest0-21swapper/311:00:253
226933699191,14cyclictest0-21swapper/310:23:093
226933699191,14cyclictest0-21swapper/309:46:493
226933699191,14cyclictest0-21swapper/309:20:133
226933699191,14cyclictest0-21swapper/309:16:433
226933699191,13cyclictest2398816-21sleep09:10:003
226933699191,13cyclictest0-21swapper/311:46:033
226933699191,11cyclictest2620083-21taskset11:35:013
226933699191,11cyclictest2291350-21taskset07:25:213
226933699190,15cyclictest2335279-21chrt08:09:343
226933699190,15cyclictest2335279-21chrt08:09:343
24748592181,4sleep12474938-21irqstats10:00:151
24659922181,2sleep00-21swapper/009:54:560
24328442182,8sleep22432903-21ntp_states09:30:222
226933699187,7cyclictest2313674-21taskset07:47:053
226933699184,11cyclictest66550irq/131-enp1s0-TxRx-310:45:073
226933699182,12cyclictest2704561-21taskset12:26:483
2269336991815,2cyclictest0-21swapper/311:10:243
2269336991815,2cyclictest0-21swapper/311:10:243
2269336991815,1cyclictest0-21swapper/311:50:233
2269336991815,1cyclictest0-21swapper/310:25:153
2269336991815,1cyclictest0-21swapper/307:55:173
2269336991813,2cyclictest66550irq/131-enp1s0-TxRx-310:32:313
226933699181,2cyclictest66550irq/131-enp1s0-TxRx-311:26:283
2269336991812,4cyclictest0-21swapper/312:10:363
226933699181,14cyclictest2657331-21chrt11:57:293
226933699181,14cyclictest0-21swapper/311:21:213
226933699181,14cyclictest0-21swapper/307:10:133
226933699181,13cyclictest0-21swapper/312:35:123
226933699181,13cyclictest0-21swapper/309:25:353
226933699181,13cyclictest0-21swapper/308:20:133
226933699181,10cyclictest2328902-21taskset08:00:203
27150022171,11sleep02715009-21ssh12:34:590
26245832171,10sleep02624643-21ntpq11:35:230
26189262178,2sleep02398796-21diskmemload11:32:580
25958652171,6sleep12595866-21sh11:18:081
25814422171,9sleep22398796-21diskmemload11:09:272
25348272171,9sleep12398796-21diskmemload10:39:041
226933699172,11cyclictest2441495-21taskset09:36:353
2269336991714,1cyclictest0-21swapper/308:35:193
226933699171,2cyclictest0-21swapper/307:55:003
226933699171,12cyclictest2292847-21chrt07:30:143
226933699170,14cyclictest2388302-21turbostat09:04:553
226933699170,14cyclictest0-21swapper/308:50:163
27205312161,9sleep12398796-21diskmemload12:35:571
27149912161,3sleep20-21swapper/212:34:572
25723902161,10sleep12572405-21ssh11:01:321
25433582161,9sleep02398796-21diskmemload10:45:000
24860622163,8sleep02486182-21ldconfig10:05:190
24205342161,10sleep22420546-21ssh09:24:452
23927282161,5sleep22392796-21systemctl09:00:242
22969582161,10sleep12297081-21mailstats07:30:251
2269336991614,1cyclictest0-21swapper/310:40:163
2269336991614,1cyclictest0-21swapper/308:40:173
2269336991613,1cyclictest0-21swapper/308:10:123
226933699161,10cyclictest2356251-21taskset08:25:403
226933699160,13cyclictest0-21swapper/308:30:213
226933699160,13cyclictest0-21swapper/307:20:193
226933699160,11cyclictest2281714-21chrt07:17:023
26427612150,10sleep12642767-21ssh11:48:501
25665312151,7sleep12566613-21munin-run11:00:001
25257592151,9sleep02525764-21ssh10:31:290
23828212155,4sleep014-21ksoftirqd/008:52:470
23030742151,10sleep22303143-40gst-plugin-scan07:39:242
226933699151,9cyclictest2388095-21taskset08:57:313
2269336991512,2cyclictest0-21swapper/308:15:213
226933699150,11cyclictest0-21swapper/308:45:193
26726182140,7sleep12398796-21diskmemload12:07:511
26502022141,9sleep02650213-21ssh11:52:580
25581192141,9sleep02558128-21sh10:53:380
25352062141,3sleep22535211-21ssh10:39:412
25197842141,6sleep12398796-21diskmemload10:29:331
25072822141,4sleep02508904-21cat10:20:190
24894782141,2sleep20-21swapper/210:10:102
24729082141,3sleep22472912-21ssh09:58:202
24426842141,7sleep12398796-21diskmemload09:38:511
22810402141,9sleep12281117-21awk07:15:221
26568542131,8sleep20-21swapper/211:56:332
26207062133,6sleep12620765-21grep11:35:161
25368082131,7sleep22538474-21kthreadcore10:40:162
25049452131,8sleep22504972-21apt-get10:20:132
25036332131,8sleep22503648-21ssh10:18:242
24876972131,3sleep12487702-21ssh10:07:131
24504862131,7sleep22450489-21ssh09:44:012
24071232131,8sleep02408080-21grep09:15:170
27044062121,4sleep12398796-21diskmemload12:26:351
26921982121,6sleep12693927-21cut12:20:171
26597682121,6sleep12661237-21kthreadcore12:00:171
26596922121,7sleep02659766-21kernelversion12:00:160
26479962121,6sleep12648057-21ntp_states11:50:231
26178552121,8sleep10-21swapper/111:31:061
26044382121,5sleep20-21swapper/211:24:502
25892302120,1sleep10-21swapper/111:14:591
25892302120,1sleep10-21swapper/111:14:591
25509252121,6sleep12398796-21diskmemload10:49:181
25431172121,8sleep10-21swapper/110:44:331
24804222121,3sleep22480432-21ssh10:02:372
24279452121,3sleep02427954-21ssh09:29:200
24119332121,3sleep20-21swapper/209:16:512
24051842121,5sleep12398796-21diskmemload09:12:491
24051842121,5sleep12398796-21diskmemload09:12:491
23898972123,4sleep02391647-21cut09:00:180
23021722121,7sleep02302234-21ntp_offset07:35:210
27213102111,3sleep20-21swapper/212:37:022
27206592111,3sleep00-21swapper/012:36:070
27132492111,4sleep12398796-21diskmemload12:32:121
27036882111,4sleep02398796-21diskmemload12:25:330
26887982111,4sleep12398796-21diskmemload12:17:531
26873212111,3sleep20-21swapper/212:15:372
26807792111,3sleep10-21swapper/112:13:401
26737452111,4sleep22398796-21diskmemload12:09:522
26583012110,4sleep12398796-21diskmemload11:59:131
26339642110,5sleep22398796-21diskmemload11:42:222
26090922110,7sleep12609161-21ssh11:25:211
25972182111,3sleep02597221-21sh11:20:030
25266482111,4sleep22398796-21diskmemload10:32:592
25244502112,4sleep12524498-21ntpq10:30:211
25179792111,2sleep02517988-21ssh10:26:320
24961142111,3sleep12398796-21diskmemload10:13:531
24736722111,4sleep12398796-21diskmemload09:59:391
24733882111,4sleep02398796-21diskmemload09:59:140
24584832111,4sleep12398796-21diskmemload09:50:011
24525912111,5sleep12454261-21tr09:45:161
24505462111,7sleep02450556-21ssh09:44:060
24204962111,3sleep12398796-21diskmemload09:24:401
23991602111,5sleep02399423-21df_inode09:10:120
23991602111,5sleep02399423-21df_inode09:10:120
23568542111,5sleep02356873-21apt-get08:30:120
23368682111,3sleep12338609-21kthreadcore08:10:171
23076682110,7sleep02307720-21/usr/sbin/munin07:40:210
22881222111,6sleep12289607-21ps07:25:201
22763442111,7sleep22276785-21munin-node07:15:032
26967282101,2sleep20-21swapper/212:22:152
26804592100,2sleep0171rcu_preempt12:13:070
26747302104,2sleep22675700-21kthreadcore12:10:162
26646272101,2sleep20-21swapper/212:01:382
26533822101,2sleep00-21swapper/011:55:190
26405122100,1sleep00-21swapper/011:45:290
26118412101,3sleep00-21swapper/011:29:170
26044252100,3sleep12398796-21diskmemload11:24:491
25870122101,3sleep02398796-21diskmemload11:10:370
25870122101,3sleep02398796-21diskmemload11:10:360
25809702101,4sleep02398796-21diskmemload11:08:370
25726872100,1sleep20-21swapper/211:02:012
25650452101,3sleep22398796-21diskmemload10:57:162
25597842101,5sleep02561004-21cut10:55:180
25565392101,2sleep20-21swapper/210:50:312
25537822101,5sleep12555567-21kthreadcore10:50:181
25503042100,1sleep00-21swapper/010:48:170
25340112101,3sleep02398796-21diskmemload10:37:140
25098922100,1sleep10-21swapper/110:20:391
25044142100,3sleep10-21swapper/110:19:431
25042092101,5sleep02504220-21ssh10:19:220
24943752101,3sleep02398796-21diskmemload10:10:360
24874392100,3sleep20-21swapper/210:06:262
24797962101,3sleep00-21swapper/010:01:340
24569772100,3sleep22398796-21diskmemload09:47:132
24291592101,5sleep02429251-21tr09:30:160
24251582101,4sleep12425196-21ntpq09:25:201
24189942100,3sleep00-21swapper/009:21:020
24052942100,1sleep20-21swapper/209:13:302
24052942100,1sleep20-21swapper/209:13:292
23938002101,3sleep22393819-21apt-get09:05:132
23876012101,5sleep22387718-21sed08:55:272
23873072101,4sleep12387353-21ntpq08:55:221
23465312101,5sleep12346675-21gltestperf08:20:151
23462692101,4sleep02346530-21df_inode08:20:140
23351472101,6sleep2802-21snmpd08:07:532
23351472101,6sleep2802-21snmpd08:07:522
23250992101,4sleep12325354-21df08:00:141
23250772101,4sleep02325096-21apt-get08:00:140
22968102101,5sleep22296849-21ldconfig07:30:222
22861792101,5sleep22286268-21ntpq07:20:202
2704928290,1sleep20-21swapper/212:27:202
2691250290,5sleep02398796-21diskmemload12:20:160
2689852290,1sleep00-21swapper/012:19:200
2635526291,4sleep0567-21dbus-daemon11:45:010
2635051291,2sleep12398796-21diskmemload11:44:121
2626028291,3sleep22398796-21diskmemload11:36:452
2596086291,2sleep20-21swapper/211:18:332
2582690291,4sleep22582880-21tr11:10:192
2582690291,4sleep22582880-21tr11:10:192
2578788291,4sleep12578849-21ntpq11:05:211
2571782291,4sleep02571791-21ssh11:00:310
2549396291,4sleep22549405-21sh10:46:542
2512490291,3sleep22512519-21apt-get10:25:142
2458714291,3sleep22458742-21apt-get09:50:102
2440272291,3sleep22440332-21ntpq09:35:232
2436085290,4sleep02436117-21ssh09:35:130
2412055291,5sleep10-21swapper/109:17:021
2392847291,4sleep12393040-21cat09:00:261
2383260291,3sleep02383279-21apt-get08:55:130
2382010291,4sleep12382056-21ntpq08:50:211
2376730291,4sleep12376782-21gcc08:45:201
2362105291,3sleep02362124-21apt-get08:35:130
2352856291,4sleep02354597-21tr08:25:190
2335578291,3sleep02335597-21apt-get08:10:120
2327938291,4sleep22328901-21wc08:00:202
2319791291,3sleep12320049-21df_inode07:55:141
2318484291,3sleep02318525-21ntpq07:50:200
2308879291,4sleep02309131-21df07:45:110
2303525291,2sleep20-21swapper/207:40:122
2303189291,3sleep12303142-31find07:39:311
2269802291,3sleep02270015-21hddtemp_smartct07:10:160
2269511291,3sleep12269530-21apt-get07:10:131
2648816281,3sleep22648825-21ssh11:50:472
2610099281,3sleep22610107-21ssh11:26:032
2594965280,1sleep00-21swapper/011:16:270
2449299281,3sleep12449304-21ssh09:41:551
2421054281,3sleep22421121-21ssh09:25:132
2371383281,3sleep22371428-21ntpq08:40:202
2366400281,3sleep2766-21sshd08:35:272
2335208280,1sleep00-21swapper/008:08:400
2335208280,1sleep00-21swapper/008:08:400
2320329281,3sleep00-21swapper/007:55:170
2308597281,3sleep1567-21dbus-daemon07:45:001
2287070280,1sleep10-21swapper/107:22:271
2283736281,3sleep02285515-21kthreadcore07:20:180
2278230281,3sleep02279994-21grep07:15:170
2398786271,2sleep10-21swapper/109:10:011
2377442270,1sleep00-21swapper/008:46:430
2363174271,3sleep12364897-21grep08:35:181
2342463270,3sleep02344244-21kthreadcore08:15:170
2324536270,1sleep20-21swapper/207:56:472
2319272270,1sleep10-21swapper/107:52:041
2309132270,3sleep12309176-21tune2fs07:45:121
2398096261,2sleep00-21swapper/009:05:270
2382700260,1sleep20-21swapper/208:51:182
2377351260,4sleep20-21swapper/208:45:332
2371244261,2sleep10-21swapper/108:40:191
2361151261,2sleep20-21swapper/208:30:262
2357944261,2sleep10-21swapper/108:30:171
2351000260,4sleep20-21swapper/208:21:202
2345703260,1sleep20-21swapper/208:16:462
2345049260,3sleep10-21swapper/108:15:221
2340394260,4sleep20-21swapper/208:12:062
2334433261,2sleep12334520-21/usr/sbin/munin08:05:241
2334433261,2sleep12334520-21/usr/sbin/munin08:05:241
2314421260,2sleep20-21swapper/207:50:022
2313683260,1sleep20-21swapper/207:47:132
2297136262,1sleep00-21swapper/007:30:270
2289613261,2sleep20-21swapper/207:25:212
2287541260,1sleep00-21swapper/007:25:150
2274429260,1sleep20-21swapper/207:13:192
2356372250,3sleep10-21swapper/108:27:161
2356358250,1sleep20-21swapper/208:27:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional