You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 21:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Nov 24, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
155592242164,25sleep20-21swapper/207:08:052
154242241163,25sleep00-21swapper/007:06:170
156332217144,25sleep30-21swapper/307:09:033
155382204167,24sleep10-21swapper/107:07:481
15807998986,2cyclictest0-21swapper/210:51:482
15807997067,2cyclictest0-21swapper/212:30:242
181762500,2sleep10-21swapper/108:25:191
46802410,4sleep21580799cyclictest10:20:262
1579399355,14cyclictest23440-21munin-run12:20:010
1580099312,6cyclictest7304-21diskstats09:15:151
1581299302,19cyclictest86550irq/126-enp1s0-11:50:183
15807993026,3cyclictest0-21swapper/209:21:082
15807993018,5cyclictest18083-21kworker/2:211:25:302
1580099301,23cyclictest0-21swapper/109:14:001
15807992918,3cyclictest21736-21kworker/2:209:47:222
15807992913,3cyclictest18083-21kworker/2:211:50:222
1580099291,17cyclictest8729-21pmu-power10:30:241
15800992815,3cyclictest848-21sssd_nss09:45:011
15800992815,3cyclictest848-21sssd_nss09:45:011
1580099281,3cyclictest0-21swapper/110:47:251
1579399280,13cyclictest2332-21munin-run09:05:020
15807992718,2cyclictest21736-21kworker/2:208:55:482
15807992715,3cyclictest18083-21kworker/2:212:39:362
15807992714,2cyclictest21736-21kworker/2:210:00:252
1580099272,22cyclictest11376-21memory11:50:201
15800992713,4cyclictest0-21swapper/112:00:541
1580099271,18cyclictest9220-21cpuspeed_turbos09:20:121
15807992614,2cyclictest21736-21kworker/2:207:15:212
15807992611,3cyclictest18083-21kworker/2:212:28:562
1580799261,11cyclictest18083-21kworker/2:212:23:362
1580099264,19cyclictest24576-21grep08:40:191
15800992615,3cyclictest0-21swapper/108:16:511
1580099261,22cyclictest6079-21latency_hist08:00:021
1580099261,22cyclictest17719-21latency_hist09:40:011
1580099261,21cyclictest18136-21ntp_states12:05:211
1580099261,16cyclictest0-21swapper/110:44:461
1580799252,21cyclictest18083-21kworker/2:211:13:282
15807992515,2cyclictest21736-21kworker/2:209:30:042
15807992515,2cyclictest21736-21kworker/2:207:48:242
15807992511,9cyclictest30944-21kworker/2:010:48:202
15807992511,3cyclictest30944-21kworker/2:010:26:282
1580099252,20cyclictest27086-21df10:00:151
1580099252,15cyclictest0-21swapper/107:43:461
1580099252,15cyclictest0-21swapper/107:30:341
1580099252,10cyclictest23669-21/usr/sbin/munin12:20:221
1580099251,6cyclictest10032-21kworker/u8:311:13:381
15800992514,3cyclictest0-21swapper/111:04:581
15800992514,1cyclictest0-21swapper/108:55:201
1580099251,3cyclictest27904-21wc12:30:021
1580099251,16cyclictest0-21swapper/107:23:551
1580099251,15cyclictest0-21swapper/110:53:081
1580099251,15cyclictest0-21swapper/110:21:531
1580099250,16cyclictest0-21swapper/107:10:131
1580799247,5cyclictest21736-21kworker/2:208:00:142
15807992413,6cyclictest18083-21kworker/2:211:02:582
15807992412,3cyclictest30944-21kworker/2:010:10:232
1580799241,12cyclictest18083-21kworker/2:212:12:342
15807992410,3cyclictest21736-21kworker/2:208:12:342
1580099242,10cyclictest2910-21diskstats09:05:161
1580099241,21cyclictest0-21swapper/111:30:021
1580099241,16cyclictest0-21swapper/107:17:411
1580099241,15cyclictest0-21swapper/110:56:331
1580099241,15cyclictest0-21swapper/110:17:371
1580099241,15cyclictest0-21swapper/109:28:251
1580099241,15cyclictest0-21swapper/108:52:271
1580099241,15cyclictest0-21swapper/108:36:261
1580099241,15cyclictest0-21swapper/108:36:261
1580099241,15cyclictest0-21swapper/108:06:501
1580099241,15cyclictest0-21swapper/108:00:581
1580099241,15cyclictest0-21swapper/107:28:081
1580099241,12cyclictest12682-21latency_hist08:15:021
1580099241,11cyclictest22537-21ntp_states12:15:211
1580099241,11cyclictest21355-21chrt08:33:261
1580099241,10cyclictest4541-21diskmemload10:10:231
1580099241,10cyclictest0-21swapper/109:34:181
1579399241,5cyclictest0-21swapper/009:49:310
1581299231,6cyclictest0-21swapper/311:47:023
1581299231,16cyclictest0-21swapper/307:25:243
1580799239,3cyclictest18083-21kworker/2:210:55:122
15807992312,2cyclictest21736-21kworker/2:208:37:482
15807992312,2cyclictest21736-21kworker/2:208:37:482
1580099238,3cyclictest26343-21cat08:45:141
1580099233,10cyclictest0-21swapper/109:45:181
1580099231,7cyclictest0-21swapper/107:50:171
15800992316,2cyclictest0-21swapper/111:22:021
15800992313,3cyclictest0-21swapper/109:52:261
15800992313,2cyclictest0-21swapper/110:26:201
15800992312,3cyclictest0-21swapper/110:37:231
15800992312,3cyclictest0-21swapper/109:00:541
1580099231,19cyclictest29657-21users12:30:271
1580099231,15cyclictest0-21swapper/109:55:281
1580099231,15cyclictest0-21swapper/107:48:381
1580099231,11cyclictest27838-21grep11:15:151
1580099231,10cyclictest13826-21ps11:55:241
1580099231,10cyclictest13826-21ps11:55:241
1579399237,6cyclictest0-21swapper/010:10:400
1579399231,11cyclictest27180-21perf11:15:020
88952221,3sleep39075-21grep10:30:263
1581299229,2cyclictest27741-21needreboot10:00:213
1581299228,2cyclictest29794-21sed08:50:233
1581299221,8cyclictest17086-21egrep12:05:113
1581299221,14cyclictest0-21swapper/312:36:193
1581299221,13cyclictest4541-21diskmemload10:35:223
1580799229,2cyclictest86750irq/128-enp1s0-08:27:322
1580799225,3cyclictest32-21ksoftirqd/207:14:442
1580799225,3cyclictest21736-21kworker/2:207:54:522
1580799222,9cyclictest18083-21kworker/2:211:40:562
1580799221,5cyclictest0-21swapper/210:41:582
1580799221,19cyclictest18083-21kworker/2:211:19:142
15807992211,3cyclictest6096-21awk08:00:012
15807992211,3cyclictest21736-21kworker/2:209:12:442
15807992210,3cyclictest21736-21kworker/2:209:39:022
15807992210,3cyclictest21736-21kworker/2:209:29:562
15800992212,3cyclictest0-21swapper/108:22:561
1580099221,19cyclictest0-21swapper/111:46:201
1580099221,19cyclictest0-21swapper/111:05:281
1580099221,19cyclictest0-21swapper/107:39:371
1580099221,14cyclictest0-21swapper/110:09:441
1580099221,10cyclictest19460-21cstates12:10:151
1579399226,7cyclictest0-21swapper/011:30:540
1579399226,6cyclictest0-21swapper/012:12:120
1581299217,4cyclictest0-21swapper/309:40:203
1581299217,4cyclictest0-21swapper/309:40:203
1581299217,3cyclictest28859-21ps07:35:223
1581299217,3cyclictest24629-21runrttasks11:05:533
1581299215,7cyclictest4541-21diskmemload11:28:323
1581299214,10cyclictest41-21ksoftirqd/307:45:173
1581299213,9cyclictest13575-21users10:40:273
1581299213,10cyclictest28877-21sed11:15:293
1581299212,6cyclictest6079-21latency_hist08:00:013
1581299211,7cyclictest5997-21latency_hist11:40:003
1581299211,7cyclictest28067-21/usr/sbin/munin12:30:163
1581299211,7cyclictest27105-21needreboot08:45:203
15812992117,3cyclictest0-21swapper/311:34:293
1581299211,5cyclictest25228-21smartctl09:55:163
1581299211,5cyclictest0-21swapper/311:03:413
1581299211,13cyclictest0-21swapper/310:26:593
1581299211,11cyclictest23942-21users09:50:273
1581299211,11cyclictest18233-21memory08:25:193
1580799219,2cyclictest86650irq/127-enp1s0-11:05:322
1580799218,4cyclictest21736-21kworker/2:207:37:402
1580799216,3cyclictest32-21ksoftirqd/207:43:482
1580799216,3cyclictest21736-21kworker/2:208:45:142
1580799213,6cyclictest22552-1kworker/u9:112:05:112
1580799213,3cyclictest86650irq/127-enp1s0-08:08:462
15807992118,2cyclictest21736-21kworker/2:208:50:582
15800992116,3cyclictest0-21swapper/111:42:201
1580099211,19cyclictest0-21swapper/112:37:461
1580099211,19cyclictest0-21swapper/111:26:141
1579399218,7cyclictest86550irq/126-enp1s0-10:00:410
1579399217,4cyclictest0-21swapper/012:32:470
15793992112,7cyclictest21266-21kworker/u8:208:09:520
1579399211,11cyclictest6046-21/usr/sbin/munin11:40:350
1579399211,11cyclictest25621-21ntp_states09:55:220
1579399211,11cyclictest21087-21/usr/sbin/munin07:20:140
15793992110,3cyclictest32240-21munin-run12:40:020
58322200,4sleep011-21rcuc/009:10:210
1581299209,5cyclictest20829-21head12:10:283
1581299209,3cyclictest86850irq/129-enp1s0-09:10:253
1581299209,3cyclictest21192-21sshd08:31:203
1581299206,6cyclictest0-21swapper/310:54:093
1581299205,3cyclictest4256-21ntp_states10:20:203
1581299204,9cyclictest39-21rcuc/310:05:143
1581299204,3cyclictest3402-21sed07:50:253
1581299201,8cyclictest0-21swapper/309:01:053
1581299201,7cyclictest19739-21kernelversion07:15:193
1581299201,6cyclictest17716-21turbostat.cron09:40:003
1581299201,5cyclictest0-21swapper/312:23:343
15812992014,3cyclictest15098-21memory10:45:193
1580799205,9cyclictest30-21rcuc/208:41:482
1580799205,6cyclictest2902-21/usr/sbin/munin11:30:252
1580799205,3cyclictest21736-21kworker/2:209:17:362
1580799204,8cyclictest32-21ksoftirqd/207:27:422
1580799200,10cyclictest30-21rcuc/209:08:122
1580099201,2cyclictest0-21swapper/111:36:071
1579399202,3cyclictest12796-21irqstats10:40:190
1579399201,9cyclictest0-21swapper/009:08:210
1579399201,5cyclictest0-21swapper/010:07:390
15793992013,4cyclictest31144-21gltestperf08:55:160
15793992013,4cyclictest10232-21ntp_states09:20:260
1579399201,11cyclictest3823-21/usr/sbin/munin11:35:150
1579399201,10cyclictest30657-21sshd11:20:220
1581299198,2cyclictest0-21swapper/308:40:173
1581299193,9cyclictest0-21swapper/310:15:593
1581299192,9cyclictest11352-21/usr/sbin/munin09:25:153
1581299191,9cyclictest13472-21kernelversion11:55:183
1581299191,9cyclictest13472-21kernelversion11:55:183
1581299191,9cyclictest10347-21ps09:20:383
1581299191,6cyclictest20968-21kernelversion09:45:193
1581299191,6cyclictest18929-21df_inode10:55:123
1581299191,6cyclictest11794-21/usr/sbin/munin08:10:203
1581299191,6cyclictest10487-21latency_hist08:10:013
1581299191,5cyclictest29926-21df11:20:133
1581299191,5cyclictest16192-21ntp_states08:20:223
15812991915,2cyclictest31013-21cut07:40:213
1581299191,3cyclictest0-21swapper/310:10:173
15812991913,3cyclictest8288-21idleruntime-cro08:05:013
1581299191,15cyclictest23130-21runrttasks12:16:123
1581299191,11cyclictest0-21swapper/309:15:193
1581299191,10cyclictest4179-21sshd09:05:373
1581299191,10cyclictest22788-21NetworkManager07:21:383
1580799199,2cyclictest86850irq/129-enp1s0-12:15:162
1580799199,2cyclictest86650irq/127-enp1s0-11:36:202
1580799198,3cyclictest21736-21kworker/2:209:42:102
1580799198,3cyclictest21736-21kworker/2:209:42:102
1580799197,3cyclictest18083-21kworker/2:211:24:342
1580799196,3cyclictest21736-21kworker/2:209:00:562
1580799195,8cyclictest30944-21kworker/2:010:34:122
1580799194,9cyclictest21736-21kworker/2:207:30:202
1580799194,8cyclictest21736-21kworker/2:208:23:382
1580799192,16cyclictest18083-21kworker/2:211:46:282
1580799191,8cyclictest30-21rcuc/207:20:462
15807991912,3cyclictest21736-21kworker/2:208:30:262
1579399198,3cyclictest17953-21/usr/sbin/munin09:40:190
1579399198,3cyclictest17953-21/usr/sbin/munin09:40:190
1579399197,3cyclictest0-21swapper/012:20:460
1579399197,3cyclictest0-21swapper/010:47:430
1579399196,5cyclictest0-21swapper/011:26:040
1579399195,3cyclictest0-21swapper/008:35:120
1579399195,3cyclictest0-21swapper/008:35:120
1579399194,5cyclictest0-21swapper/011:02:030
1579399191,7cyclictest0-21swapper/011:55:130
1579399191,7cyclictest0-21swapper/011:55:130
1579399191,6cyclictest0-21swapper/009:27:400
15793991914,3cyclictest14891-21munin-run08:20:010
15793991913,3cyclictest18588-21/usr/sbin/munin10:55:230
1579399191,10cyclictest7592-21/usr/sbin/munin10:30:230
1579399191,10cyclictest6313-21/usr/sbin/munin08:00:240
1579399191,10cyclictest14634-21snmpd08:13:100
1579399191,10cyclictest0-21swapper/010:52:590
1579399191,10cyclictest0-21swapper/010:29:070
1579399191,10cyclictest0-21swapper/010:20:210
1579399191,10cyclictest0-21swapper/009:30:080
1579399191,10cyclictest0-21swapper/009:19:100
121622191,2sleep00-21swapper/011:51:410
256952181,2sleep00-21swapper/008:42:420
1581299187,3cyclictest26780-21memory12:25:193
1581299184,3cyclictest12050irq/123-ahci[0007:15:013
1581299183,9cyclictest0-21swapper/312:00:113
1581299183,3cyclictest14171-21/usr/sbin/munin08:15:233
1581299181,5cyclictest23683-21latency_hist08:40:013
1581299181,5cyclictest23683-21latency_hist08:40:013
1581299181,5cyclictest0-21swapper/311:40:193
1581299181,4cyclictest32029-21tail08:55:313
1581299181,4cyclictest26535-21/usr/sbin/munin11:10:263
1581299181,4cyclictest14616-21sshd09:30:213
1580799188,2cyclictest86750irq/128-enp1s0-10:08:312
1580799188,2cyclictest21736-21kworker/2:209:59:142
1580799186,3cyclictest86850irq/129-enp1s0-08:19:382
1580799184,9cyclictest32-21ksoftirqd/210:36:282
1580799181,8cyclictest13696-21awk11:55:222
1580799181,8cyclictest13696-21awk11:55:222
1580799181,7cyclictest0-21swapper/212:03:252
1580799181,6cyclictest0-21swapper/209:50:032
15807991812,3cyclictest1651-21cut10:15:132
1579399186,3cyclictest9337-21ntp_states11:45:230
1579399182,6cyclictest0-21swapper/010:19:200
1579399182,13cyclictest23462-21ls09:50:240
1579399181,6cyclictest0-21swapper/012:01:160
1579399181,5cyclictest20687-21ps08:30:230
1579399181,3cyclictest27208-21awk12:25:260
1579399181,11cyclictest0-21swapper/010:36:330
294682171,2sleep00-21swapper/007:37:460
230772171,3sleep00-21swapper/007:25:020
1579399175,4cyclictest0-21swapper/009:39:270
1579399171,8cyclictest26123-21cut07:30:170
1579399171,6cyclictest18384-21ntp_states08:25:220
15793991713,3cyclictest0-21swapper/008:20:250
1579399171,14cyclictest0-21swapper/008:51:010
1579399171,14cyclictest0-21swapper/007:15:200
15793991711,3cyclictest27680-21cat11:15:130
15793991710,5cyclictest0-21swapper/011:05:250
15812991611,3cyclictest25727-21cat07:30:123
1579399168,6cyclictest21266-21kworker/u8:207:45:340
1579399164,10cyclictest86550irq/126-enp1s0-07:10:120
1579399161,9cyclictest0-21swapper/012:06:100
15793991612,3cyclictest0-21swapper/008:46:440
15793991611,3cyclictest4928-21irqstats07:55:190
15793991611,3cyclictest30165-21expr07:40:130
15793991611,3cyclictest2583-21expr07:50:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional