You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 09:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri Nov 28, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3810831212357,61sleep30-21swapper/307:06:413
381074929282,5sleep20-21swapper/207:05:402
381094628352,20sleep00-21swapper/007:08:100
381087427963,10sleep10-21swapper/107:07:151
3811201993530,2cyclictest0-21swapper/309:55:153
38127652320,3sleep2421rcuc/207:10:162
381120199271,23cyclictest0-21swapper/311:28:173
381120199265,3cyclictest0-21swapper/310:25:133
381120199251,2cyclictest0-21swapper/311:52:483
39477132241,4sleep00-21swapper/009:14:350
381120199242,16cyclictest4133522-21taskset11:19:343
3811201992421,1cyclictest0-21swapper/309:30:143
3811201992420,2cyclictest0-21swapper/312:05:183
3811201992419,2cyclictest0-21swapper/310:40:163
3811201992415,4cyclictest41919592sleep311:57:073
3811201992412,9cyclictest4043878-21chrt10:19:013
3811201992412,9cyclictest4043878-21chrt10:19:003
381120199241,19cyclictest4177141-21taskset11:47:203
381120199241,12cyclictest0-21swapper/309:38:163
381120199232,15cyclictest4020770-21taskset10:02:443
381120199232,14cyclictest23780-21taskset12:15:143
3811201992319,2cyclictest0-21swapper/310:20:153
3811201992319,2cyclictest0-21swapper/310:20:143
3811201992318,3cyclictest0-21swapper/312:25:213
3811201992318,3cyclictest0-21swapper/311:00:003
381120199231,5cyclictest0-21swapper/311:30:123
381120199231,4cyclictest0-21swapper/312:35:153
381120199231,4cyclictest0-21swapper/309:45:123
381120199231,14cyclictest18558-21taskset12:10:163
41767002221,6sleep10-21swapper/111:46:261
381120199222,14cyclictest4193817-21taskset12:00:023
381120199222,14cyclictest4162717-21taskset11:37:583
3811201992220,1cyclictest0-21swapper/307:15:003
3811201992219,1cyclictest0-21swapper/311:10:203
3811201992219,1cyclictest0-21swapper/310:11:583
3811201992219,1cyclictest0-21swapper/307:55:003
381120199221,5cyclictest0-21swapper/311:05:123
381120199221,4cyclictest0-21swapper/311:00:123
41119172211,4sleep13940511-21diskmemload11:03:361
381120199212,14cyclictest3953631-21taskset09:16:263
381120199211,5cyclictest0-21swapper/310:50:133
381120199211,5cyclictest0-21swapper/309:10:103
381120199211,4cyclictest0-21swapper/312:20:123
381120199211,4cyclictest0-21swapper/310:35:093
381120199211,4cyclictest0-21swapper/310:30:133
3811201992113,5cyclictest39607802sleep309:20:533
3811201992113,5cyclictest39607802sleep309:20:523
381120199211,15cyclictest3940534-21sleep09:09:593
381120199210,2cyclictest0-21swapper/309:04:553
381120199210,11cyclictest0-21swapper/308:24:553
39911282201,13sleep23940511-21diskmemload09:42:202
38548242201,9sleep23854875-21ld07:45:202
381120199202,5cyclictest0-21swapper/312:30:143
381120199202,12cyclictest4027769-21taskset10:06:453
381120199201,5cyclictest0-21swapper/311:20:113
381120199201,14cyclictest3968545-21chrt09:26:203
288022201,5sleep10-21swapper/112:16:441
65252191,5sleep20-21swapper/212:03:272
40587162191,3sleep20-21swapper/210:28:072
3811201991917,1cyclictest0-21swapper/307:17:393
381120199191,15cyclictest3990254-21taskset09:40:313
381120199191,14cyclictest4087960-21taskset10:45:503
381120199191,14cyclictest0-21swapper/308:55:113
381120199191,13cyclictest4164454-21chrt11:40:133
381120199190,9cyclictest0-21swapper/308:36:023
40667102181,2sleep10-21swapper/110:33:461
40350942188,5sleep04035107-21sh10:11:240
381120199188,2cyclictest0-21swapper/307:20:153
3811201991814,2cyclictest0-21swapper/307:36:553
381120199181,14cyclictest4005653-21chrt09:52:303
381120199181,13cyclictest3861391-21chrt07:55:113
381120199180,9cyclictest0-21swapper/307:29:313
41916582171,4sleep13940511-21diskmemload11:56:281
41093762171,4sleep23940511-21diskmemload11:00:212
40793572171,11sleep14079411-21ntp_kernel_pll_10:40:191
40647562171,10sleep23940511-21diskmemload10:30:262
40362392171,10sleep14036254-21sshd10:13:371
40289292171,4sleep23940511-21diskmemload10:08:582
39632192170,11sleep00-21swapper/009:25:100
3811201991714,1cyclictest0-21swapper/307:30:153
3811201991712,2cyclictest0-21swapper/308:45:163
381120199171,12cyclictest3882254-21turbostat08:15:003
381120199171,11cyclictest3871296-21chrt08:00:313
381120199171,10cyclictest3887308-21taskset08:17:073
593172161,4sleep23940511-21diskmemload12:37:312
41171012161,10sleep14117167-21ssh11:05:221
40500612160,2sleep00-21swapper/010:21:140
40500612160,2sleep00-21swapper/010:21:140
40216322161,2sleep00-21swapper/010:04:220
40129562161,3sleep23940511-21diskmemload09:57:202
39978272161,3sleep23940511-21diskmemload09:46:252
39545042161,3sleep03940511-21diskmemload09:18:050
3811201991614,1cyclictest0-21swapper/307:40:183
381120199161,3cyclictest0-21swapper/308:25:113
3811201991612,2cyclictest0-21swapper/308:05:193
381120199160,13cyclictest0-21swapper/308:40:143
375982162,2sleep0171rcu_preempt12:24:130
41912132151,3sleep23940511-21diskmemload11:55:342
41710672151,9sleep04171117-21munin-run11:45:000
41333832151,9sleep23940511-21diskmemload11:19:222
41278392151,9sleep14128510-21grep11:15:151
41095762151,7sleep04109628-21ntpq11:00:230
40835922151,9sleep24084902-21cut10:45:182
39830112150,2sleep00-21swapper/009:35:520
39776792151,2sleep10-21swapper/109:34:571
39452562155,7sleep13945409-21timerwakeupswit09:10:241
381120199151,8cyclictest3898312-21taskset08:30:013
381120199150,12cyclictest0-21swapper/308:50:253
381120199150,12cyclictest0-21swapper/307:45:123
323462151,9sleep134092-21grep12:20:171
206992151,10sleep120711-21rm12:11:111
41769512140,2sleep00-21swapper/011:46:570
41484202141,7sleep13940511-21diskmemload11:29:341
41456592141,9sleep03940511-21diskmemload11:25:240
41346212141,8sleep04135479-21kthreadcore11:20:150
40413842141,7sleep20-21swapper/210:15:202
40413842141,7sleep20-21swapper/210:15:192
39913912140,2sleep00-21swapper/009:43:220
39624902140,7sleep244-21ksoftirqd/209:24:122
39624902140,7sleep244-21ksoftirqd/209:24:122
295322141,8sleep00-21swapper/012:18:040
595522131,5sleep03940511-21diskmemload12:37:560
513582134,3sleep03811194-21cyclictest12:32:030
41619622131,8sleep14161972-21ssh11:36:361
41205472131,8sleep04120669-21ssh11:10:130
40968822131,5sleep23940511-21diskmemload10:53:392
40501802131,5sleep23940511-21diskmemload10:21:292
40501802131,5sleep23940511-21diskmemload10:21:282
40496762131,5sleep13940511-21diskmemload10:20:311
40496762131,5sleep13940511-21diskmemload10:20:311
40211532131,8sleep14021163-21ssh10:03:271
39692742131,5sleep13940511-21diskmemload09:27:511
39483022131,5sleep13940511-21diskmemload09:15:131
41703482121,5sleep10-21swapper/111:43:011
41700932121,3sleep20-21swapper/211:42:262
41554512123,5sleep24133010-21kworker/u16:3+nfsiod11:33:282
41455402121,5sleep24145596-21ntpq11:25:232
41408722121,4sleep13940511-21diskmemload11:24:451
41401132120,8sleep24140123-21ssh11:22:582
41268482121,4sleep13940511-21diskmemload11:14:431
40945032121,5sleep14094557-21ntpq10:50:221
40819042120,5sleep20-21swapper/210:44:092
40742182121,4sleep13940511-21diskmemload10:38:561
40670512121,4sleep03940511-21diskmemload10:34:290
40595432121,4sleep13940511-21diskmemload10:29:491
40423862121,4sleep13940511-21diskmemload10:16:021
40423862121,4sleep13940511-21diskmemload10:16:021
40339612123,4sleep24034015-21ntpq10:10:222
40209212121,3sleep20-21swapper/210:03:002
40072572121,5sleep04007279-21apt-get09:55:120
40057512121,4sleep13940511-21diskmemload09:52:411
39777112121,3sleep03977738-21perf09:35:000
39760332121,4sleep23940511-21diskmemload09:31:412
39469762121,4sleep23940511-21diskmemload09:12:572
39404982121,7sleep13940513-21munin-plugin-st09:09:591
39085612125,3sleep03908961-21date08:40:010
38759242122,5sleep23875973-21ntpq08:05:202
38551542121,7sleep03855376-21unixbench_multi07:45:260
38295652122,5sleep13829584-21apt-get07:25:111
369152121,5sleep23940511-21diskmemload12:22:512
60782111,4sleep13940511-21diskmemload12:02:411
51392111,4sleep03940511-21diskmemload12:01:040
457502110,1sleep20-21swapper/212:30:122
439612111,4sleep13940511-21diskmemload12:26:591
41862382111,4sleep03940511-21diskmemload11:55:030
41852162110,4sleep03940511-21diskmemload11:53:430
41838202111,4sleep13940511-21diskmemload11:50:551
41836352111,3sleep20-21swapper/211:50:362
41763032111,4sleep23940511-21diskmemload11:45:432
41617522111,6sleep23940511-21diskmemload11:36:102
41565642111,3sleep03940511-21diskmemload11:35:100
41532422111,4sleep13940511-21diskmemload11:30:221
41267602111,3sleep20-21swapper/211:14:282
41193672111,3sleep03940511-21diskmemload11:08:290
41036092111,4sleep23940511-21diskmemload10:56:492
41033802111,3sleep13940511-21diskmemload10:56:231
41029882111,4sleep03940511-21diskmemload10:55:420
40952402111,3sleep00-21swapper/010:50:370
40888382111,3sleep13940511-21diskmemload10:47:341
40877912111,6sleep04087802-21ssh10:45:310
40800172111,3sleep00-21swapper/010:40:260
40748712111,6sleep2412173-21cron10:40:012
40421902111,4sleep03940511-21diskmemload10:15:410
40421902111,4sleep03940511-21diskmemload10:15:410
40282012111,4sleep03940511-21diskmemload10:07:350
40142872111,4sleep13940511-21diskmemload09:59:511
40054262111,3sleep03940511-21diskmemload09:52:040
39984792111,4sleep13940511-21diskmemload09:47:361
39915852111,4sleep13940511-21diskmemload09:43:471
39846852111,3sleep20-21swapper/209:39:122
39554222111,4sleep03940511-21diskmemload09:20:010
39554222111,4sleep03940511-21diskmemload09:20:010
39213922111,5sleep23923190-21cut08:50:172
39144062114,3sleep13914427-21dpkg08:45:111
38970892111,5sleep23897138-21ntpq08:25:202
38769242111,6sleep13876965-21systemd-run08:10:001
38722042111,5sleep03872413-21hddtemp_smartct08:05:140
38602022111,6sleep03860269-21pmu-power07:50:240
38164312110,6sleep03816500-21systemd-run07:15:000
212082110,1sleep00-21swapper/012:12:020
128472111,4sleep23940511-21diskmemload12:06:002
588532100,3sleep13940511-21diskmemload12:36:491
438722101,3sleep23940511-21diskmemload12:26:492
41548882101,2sleep00-21swapper/011:32:220
41327432101,3sleep00-21swapper/011:17:140
41188582100,3sleep20-21swapper/211:07:312
40745322100,1sleep00-21swapper/010:39:260
40583812101,2sleep00-21swapper/010:27:230
40222202101,3sleep13940511-21diskmemload10:05:081
40067132101,2sleep20-21swapper/209:54:342
39982082101,3sleep03940511-21diskmemload09:47:070
39821132101,4sleep13982164-21ntpq09:35:221
39687072101,2sleep20-21swapper/209:26:412
39560462101,5sleep13956249-21expr09:20:141
39560462101,5sleep13956249-21expr09:20:141
39524332101,3sleep20-21swapper/209:15:222
39392792101,5sleep23939439-21tail09:05:202
39303392101,4sleep13930555-21diskstats09:00:141
39252612100,1sleep20-21swapper/208:55:142
39194372101,5sleep23919528-21cut08:50:012
39131182100,2sleep20-21swapper/208:40:192
39041652101,5sleep23905602-21cut08:35:142
38930972101,4sleep13893116-21apt-get08:25:111
38918522101,4sleep03891902-21ntpq08:20:220
38671672101,5sleep23867236-21/usr/sbin/munin08:00:152
38613692101,4sleep23861388-21apt-get07:55:102
38599822101,4sleep13860103-21ntpq07:50:201
38449752100,5sleep23844935-31find07:36:582
38385792101,4sleep03838646-21memory07:30:190
282152101,3sleep23940511-21diskmemload12:15:422
207122101,3sleep23940511-21diskmemload12:11:122
139692101,4sleep013981-21ssh12:08:020
118732101,4sleep111927-21ntpq12:05:221
51360290,3sleep10-21swapper/112:32:041
43246291,2sleep00-21swapper/012:25:370
3936143291,4sleep03937770-21grep09:05:190
3929981291,4sleep0606-21rs:main0
3918103291,4sleep03918393-21cut08:45:180
3903593291,3sleep13903612-21apt-get08:35:091
3846046291,4sleep03846491-21grep07:40:180
3845535291,3sleep13845787-21df07:40:141
3844803290,1sleep10-21swapper/107:36:051
3828989290,3sleep00-21swapper/007:21:010
3824330291,3sleep13824580-21df_inode07:20:151
3811950291,4sleep172-21kcompactd007:10:171
3935077281,3sleep2802-21snmpd09:02:312
3877182281,4sleep20-21swapper/208:10:092
3834814281,3sleep13835068-21df07:30:131
3823888280,1sleep20-21swapper/207:18:302
3929750270,1sleep10-21swapper/108:57:061
3924565272,3sleep1331ktimers/108:53:121
3920152271,2sleep03920268-21irqstats08:50:150
3913860271,1sleep10-21swapper/108:41:261
3903021270,1sleep00-21swapper/008:31:040
3897987270,1sleep00-21swapper/008:29:060
3887199270,1sleep00-21swapper/008:15:440
3866257270,1sleep00-21swapper/007:58:280
3850228270,1sleep20-21swapper/207:40:432
3816501271,3sleep13816439-21apt-get07:15:011
3930839260,2sleep00-21swapper/009:00:170
3902108260,2sleep23902281-21strings08:30:172
3892141261,2sleep10-21swapper/108:20:261
3888115260,3sleep20-21swapper/208:20:152
3878787261,2sleep00-21swapper/008:10:150
3871312260,1sleep10-21swapper/108:00:421
3856655260,3sleep2421rcuc/207:50:172
3840636261,2sleep00-21swapper/007:35:160
3834430260,1sleep00-21swapper/007:28:460
3824581261,2sleep20-21swapper/207:20:152
3823995260,1sleep00-21swapper/007:19:490
3913844250,1sleep00-21swapper/008:41:130
3887433250,3sleep10-21swapper/108:18:411
3882491250,3sleep20-21swapper/208:15:022
3861746250,1sleep10-21swapper/107:55:131
3898488240,1sleep10-21swapper/108:30:121
3877468240,1sleep10-21swapper/108:10:121
3866782240,1sleep00-21swapper/008:00:130
3850823240,1sleep10-21swapper/107:45:071
3834530240,1sleep20-21swapper/207:30:012
3829587240,1sleep20-21swapper/207:25:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional