You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 15:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Mon Nov 24, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3084873212061,54sleep30-21swapper/307:06:233
317692921070,2sleep10-21swapper/108:30:201
308490728958,10sleep20-21swapper/207:06:502
308505228756,9sleep10-21swapper/107:08:401
308492027854,6sleep00-21swapper/007:06:590
3085479992921,4cyclictest3452469-21taskset11:50:013
308547999271,21cyclictest3362923-21chrt10:50:083
308547999257,13cyclictest3467519-21sleep12:04:553
3085479992520,2cyclictest0-21swapper/311:00:173
34950492242,14sleep13495172-21ntpq12:15:171
308547999242,17cyclictest3444398-21sleep11:45:003
308547999242,16cyclictest3215089-21turbostat09:10:003
3085479992421,1cyclictest0-21swapper/308:15:213
3085479992419,3cyclictest0-21swapper/311:35:163
3085479992419,2cyclictest0-21swapper/312:10:193
34575672231,6sleep13457573-21sh11:50:481
3085479992319,2cyclictest0-21swapper/309:40:183
308547999231,8cyclictest3295217-21taskset10:04:063
308547999231,18cyclictest3512004-21taskset12:27:023
34907762221,2sleep00-21swapper/012:15:010
308547999223,13cyclictest3475928-21taskset12:05:133
308547999222,14cyclictest3399495-21taskset11:14:013
308547999222,13cyclictest3316146-21taskset10:16:213
3085479992219,1cyclictest0-21swapper/310:40:213
3085479992217,2cyclictest0-21swapper/311:25:143
308547999221,16cyclictest3444734-21chrt11:45:123
308547999212,13cyclictest3527187-21taskset12:36:523
308547999212,13cyclictest3521201-21taskset12:34:553
308547999212,13cyclictest3414766-21taskset11:24:563
3085479992118,1cyclictest0-21swapper/309:10:153
3085479992118,1cyclictest0-21swapper/308:50:213
3085479992118,1cyclictest0-21swapper/308:40:173
308547999211,3cyclictest0-21swapper/307:34:533
308547999211,15cyclictest3465900-21chrt11:57:033
308547999211,14cyclictest3423131-21chrt11:30:133
34752242201,4sleep13475236-21ssh12:04:561
33184032201,7sleep23318430-21/usr/sbin/munin10:20:132
3085479992018,1cyclictest0-21swapper/309:18:343
3085479992016,2cyclictest0-21swapper/309:50:183
3085479992016,2cyclictest0-21swapper/309:46:223
3085479992016,2cyclictest0-21swapper/309:46:223
3085479992015,2cyclictest0-21swapper/309:20:163
308547999201,14cyclictest3249991-21chrt09:32:153
34432392191,3sleep03215074-21diskmemload11:42:060
34269642191,8sleep23427027-21ld11:30:212
33837862191,2sleep03215074-21diskmemload11:02:170
31196102191,13sleep23119783-21expr07:40:122
308547999192,13cyclictest3318431-21taskset10:20:123
308547999191,15cyclictest0-21swapper/310:05:053
308547999191,14cyclictest3405131-21chrt11:15:303
308547999191,14cyclictest0-21swapper/310:30:153
308547999191,13cyclictest3124807-21turbostat07:45:003
308547999190,16cyclictest0-21swapper/312:21:353
34755422181,3sleep03215074-21diskmemload12:05:030
34366342181,4sleep13215074-21diskmemload11:38:411
32871922181,3sleep13215074-21diskmemload09:57:461
308547999182,12cyclictest3346386-21taskset10:37:253
308547999182,12cyclictest3287167-21taskset09:57:443
3085479991815,1cyclictest0-21swapper/310:55:143
3085479991815,1cyclictest0-21swapper/308:25:173
308547999181,14cyclictest0-21swapper/312:16:233
308547999181,14cyclictest0-21swapper/311:05:023
308547999181,14cyclictest0-21swapper/308:10:113
308547999181,13cyclictest0-21swapper/310:25:113
308547999181,13cyclictest0-21swapper/309:36:573
308547999181,12cyclictest0-21swapper/309:25:153
308547999181,10cyclictest3303641-21taskset10:10:153
3085479991810,6cyclictest3177891-21turbostat08:35:003
3085479991810,5cyclictest33553452sleep310:45:123
3085479991810,5cyclictest33553452sleep310:45:123
34904862171,3sleep10-21swapper/112:14:341
34508452171,3sleep00-21swapper/011:48:070
34212472171,4sleep23215074-21diskmemload11:27:482
34071532171,3sleep20-21swapper/211:19:432
33312622171,3sleep10-21swapper/110:27:181
32600592171,8sleep23260497-21kthreadcore09:40:162
308547999178,5cyclictest3140603-21turbostat08:00:003
3085479991715,1cyclictest0-21swapper/308:20:173
3085479991713,2cyclictest0-21swapper/307:20:133
308547999171,13cyclictest0-21swapper/307:25:213
308547999171,11cyclictest3146146-21chrt08:05:133
308547999171,10cyclictest3090426-21taskset07:10:263
34752722161,10sleep23475325-21systemd-run12:05:012
33457762161,3sleep13215074-21diskmemload10:36:081
32646072161,4sleep13215074-21diskmemload09:41:081
32511882161,10sleep03251198-21ssh09:34:450
31675782161,10sleep23167746-21expr08:25:132
308547999161,10cyclictest3209710-21taskset09:03:213
308547999161,10cyclictest3183258-21taskset08:40:003
308547999160,13cyclictest0-21swapper/307:50:173
308547999160,13cyclictest0-21swapper/307:45:153
33379982151,2sleep10-21swapper/110:30:351
33096182155,4sleep2431ktimers/210:13:162
32871382151,7sleep03215074-21diskmemload09:57:420
32781362151,9sleep03278189-21ntpq09:50:210
32418812151,7sleep13215074-21diskmemload09:25:361
32360922151,7sleep134-21ksoftirqd/109:23:391
308547999151,3cyclictest0-21swapper/308:55:113
308547999151,3cyclictest0-21swapper/308:55:113
3085479991512,1cyclictest0-21swapper/307:35:263
308547999150,13cyclictest0-21swapper/308:45:163
308547999150,13cyclictest0-21swapper/307:15:253
308547999150,12cyclictest0-21swapper/308:00:133
35257572143,6sleep13525818-21ntpq12:35:221
35210682141,4sleep23521071-21ssh12:34:432
35129742141,4sleep03512986-21ssh12:29:000
34892582141,7sleep03215074-21diskmemload12:12:200
34597792141,7sleep03215074-21diskmemload11:55:000
34433312141,8sleep23215074-21diskmemload11:42:162
34079022141,8sleep13407963-21/usr/sbin/munin11:20:141
33983882141,7sleep23215074-21diskmemload11:10:372
33850122141,8sleep13385022-21ssh11:04:261
33464502140,3sleep23215074-21diskmemload10:37:332
33010862141,6sleep03215074-21diskmemload10:06:200
32948832141,6sleep03215074-21diskmemload10:03:240
32548442141,6sleep23255899-21memory09:35:202
32429852142,7sleep014-21ksoftirqd/009:27:560
32278232141,7sleep23215074-21diskmemload09:16:482
31831432141,9sleep23183257-21date08:40:002
31041702141,8sleep13104478-21tr07:25:151
30901572141,8sleep23090425-21sed07:10:262
35221962132,6sleep03523533-21cut12:35:170
34756642132,5sleep23475927-21df12:05:122
34586182131,5sleep23215074-21diskmemload11:52:462
34508232131,8sleep13450826-21ssh11:48:041
34072982131,6sleep03407337-21munin-run11:20:010
33851012131,2sleep20-21swapper/211:04:352
33242712131,7sleep13324282-21ssh10:23:131
32890292131,4sleep13289204-21/usr/sbin/munin10:00:161
32726562131,6sleep13215074-21diskmemload09:48:051
32726562131,6sleep13215074-21diskmemload09:48:051
32493582131,5sleep10-21swapper/109:31:011
32276212130,2sleep10-21swapper/109:15:541
32087082131,1sleep00-21swapper/009:00:180
32045422131,7sleep23204558-21sendmail09:00:002
32045422131,7sleep23204558-21sendmail09:00:002
31663302131,8sleep03166400-21/usr/sbin/munin08:20:260
31371022131,8sleep03138880-21kthreadcore07:55:170
31190952130,1sleep10-21swapper/107:38:381
30988942131,7sleep03098964-21kthreadcore07:20:140
35111692121,6sleep10-21swapper/112:25:281
34988342121,5sleep23215074-21diskmemload12:20:082
34884502121,7sleep23215074-21diskmemload12:10:582
34665422121,4sleep23215074-21diskmemload11:58:062
34292012121,4sleep13215074-21diskmemload11:33:211
34204042120,1sleep00-21swapper/011:26:180
34185302121,7sleep13419147-21missed_timers11:25:181
34076102121,5sleep00-21swapper/011:20:050
34071422121,3sleep10-21swapper/111:19:411
33910392120,1sleep20-21swapper/211:06:002
33763992121,4sleep03215074-21diskmemload10:57:320
33691122121,2sleep23369123-21ssh10:53:092
33543272121,4sleep13215074-21diskmemload10:43:501
33392082121,5sleep00-21swapper/010:32:420
33380352121,4sleep23215074-21diskmemload10:30:402
33169932126,3sleep13317004-21sh10:18:201
33168292121,4sleep23215074-21diskmemload10:17:572
33101212121,5sleep00-21swapper/010:14:140
32882312121,4sleep23215074-21diskmemload09:59:572
32805032121,4sleep13215074-21diskmemload09:54:071
32724502120,6sleep23215074-21diskmemload09:47:422
32724502120,6sleep23215074-21diskmemload09:47:412
32213972122,5sleep13221248-21kworker/u16:1+xprtiod09:13:101
31979322121,5sleep13197980-21ntpq08:50:171
31875342121,7sleep23187592-21pmu-power08:40:242
35274712111,3sleep20-21swapper/212:37:232
35191892111,3sleep00-21swapper/012:31:240
35191062111,4sleep13215074-21diskmemload12:31:171
35120882111,6sleep23512090-21sh12:27:142
34988562112,4sleep13498882-21apt-get12:20:111
34963272111,4sleep23215074-21diskmemload12:16:112
34809012111,3sleep10-21swapper/112:06:121
34748592110,1sleep00-21swapper/012:04:180
34665362111,4sleep13215074-21diskmemload11:58:061
34601162111,3sleep00-21swapper/011:55:120
34487422111,4sleep23215074-21diskmemload11:45:202
34427212111,4sleep13215074-21diskmemload11:41:101
34371832111,3sleep20-21swapper/211:39:392
34292662111,4sleep03215074-21diskmemload11:33:280
34140432111,4sleep23215074-21diskmemload11:23:272
33998102111,3sleep03215074-21diskmemload11:14:390
33928452111,3sleep00-21swapper/011:09:220
33769782111,4sleep13215074-21diskmemload10:58:521
33756152111,4sleep23215074-21diskmemload10:56:052
33679312111,3sleep03215074-21diskmemload10:50:420
33672842111,3sleep13215074-21diskmemload10:50:231
33610202111,3sleep00-21swapper/010:46:530
33610202111,3sleep00-21swapper/010:46:530
33555212111,4sleep13355654-21diskstats10:45:141
33555212111,4sleep13355654-21diskstats10:45:141
33478642111,6sleep03347875-21find10:40:120
33467612110,1sleep00-21swapper/010:38:360
33326572110,1sleep20-21swapper/210:29:522
33296272111,4sleep03329674-21ntpq10:25:210
33236962111,3sleep00-21swapper/010:22:040
33176952111,6sleep03317699-21turbostat.cron10:19:540
33039352114,3sleep1331ktimers/110:10:181
33012502111,3sleep10-21swapper/110:06:361
32935012111,4sleep23215074-21diskmemload10:00:332
32658662111,3sleep00-21swapper/009:44:200
32570792111,3sleep10-21swapper/109:36:111
32427392111,6sleep23215074-21diskmemload09:27:302
32347192111,4sleep23215074-21diskmemload09:21:002
32212552111,4sleep03215074-21diskmemload09:12:400
32204222111,3sleep20-21swapper/209:10:542
31768752111,4sleep03176921-21ntpq08:30:200
31501232111,4sleep23150172-21ntpq08:05:202
31290832111,5sleep13129140-21ntpq07:45:201
31195832111,5sleep13119258-31find07:40:071
31192782111,4sleep03119258-31find07:39:580
30993002110,7sleep13100422-21kthreadcore07:20:151
35039182101,3sleep03215074-21diskmemload12:20:580
34363652101,3sleep00-21swapper/011:38:130
33977052101,2sleep10-21swapper/111:10:241
33910302101,3sleep13215074-21diskmemload11:05:581
33620612101,3sleep20-21swapper/210:49:042
33620612101,3sleep20-21swapper/210:49:042
33541792101,2sleep20-21swapper/210:43:302
32958952101,3sleep23215074-21diskmemload10:05:102
32805112101,2sleep20-21swapper/209:54:082
32723252101,2sleep00-21swapper/009:47:240
32723252101,2sleep00-21swapper/009:47:240
32577722100,4sleep03215074-21diskmemload09:37:460
32494092101,2sleep20-21swapper/209:31:072
32333702100,3sleep00-21swapper/009:20:180
32284392101,3sleep00-21swapper/009:18:020
32048482101,4sleep23205106-21df_inode09:00:122
31834532101,4sleep13183709-21df08:40:151
31572092101,4sleep23157281-21irqstats08:15:142
31250362101,4sleep03125055-21apt-get07:45:110
3182163291,4sleep03182209-21ntpq08:35:210
3160862290,2sleep10-21swapper/108:15:211
3158527291,4sleep03160349-21cut08:15:170
3156411290,1sleep00-21swapper/008:14:290
3150231291,4sleep03150295-21cut08:05:230
3149978291,3sleep13150122-21ntpq08:05:201
3119306291,4sleep23119525-21cut07:39:592
3109490291,4sleep13110972-21kthreadcore07:30:171
3183431281,2sleep00-21swapper/008:40:140
3134632281,3sleep13134804-21timerwakeupswit07:50:251
3119587281,2sleep00-21swapper/007:40:100
3098636281,3sleep20-21swapper/207:20:122
3091164281,3sleep13092341-21kthreadcore07:15:011
3090813281,3sleep03091162-21ntpq07:15:010
3210099271,2sleep23210118-21apt-get09:05:132
3204826270,2sleep10-21swapper/109:00:101
3199029271,2sleep03199493-21packagekitd08:55:010
3199029271,2sleep03199493-21packagekitd08:55:010
3194537271,2sleep20-21swapper/208:50:142
3193570270,1sleep00-21swapper/008:48:160
3193527270,1sleep20-21swapper/208:47:422
3182964270,1sleep10-21swapper/108:37:421
3176362271,3sleep23176805-21needreboot08:30:182
3172029271,3sleep03172216-21vmstat08:25:270
3167556270,2sleep10-21swapper/108:25:111
3165794270,2sleep13165967-21strings08:20:191
3141489271,2sleep20-21swapper/208:00:162
3140877270,2sleep10-21swapper/108:00:111
3140594271,2sleep21-21systemd07:59:592
3108602270,1sleep20-21swapper/207:29:302
3085752271,3sleep00-21swapper/007:10:120
3214977260,1sleep10-21swapper/109:08:571
3214732260,4sleep00-21swapper/009:05:470
3162030260,2sleep20-21swapper/208:20:122
3155353260,1sleep20-21swapper/208:10:182
3145741260,1sleep00-21swapper/008:03:270
3140420260,4sleep10-21swapper/107:57:471
3130329261,2sleep00-21swapper/007:50:120
3130300260,4sleep20-21swapper/207:50:062
3113694260,1sleep00-21swapper/007:32:130
3098044260,4sleep20-21swapper/207:19:362
3085207260,2sleep13085703-21packagekitd07:10:011
3198769250,1sleep00-21swapper/008:52:210
3127779250,1sleep20-21swapper/207:45:162
3110976250,1sleep20-21swapper/207:30:182
3103936251,1sleep00-21swapper/007:25:150
3199497240,1sleep10-21swapper/108:55:021
3199497240,1sleep10-21swapper/108:55:011
3189050240,1sleep10-21swapper/108:45:151
3151133240,1sleep10-21swapper/108:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional