You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-30 - 12:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Aug 30, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13654213085,39sleep20-21swapper/219:05:272
14251210996,9sleep10-21swapper/119:08:391
14101210592,9sleep30-21swapper/319:06:453
1359027857,16sleep00-21swapper/019:05:250
14443992920,1cyclictest31853-21apache221:40:133
1442999250,21cyclictest0-21swapper/023:30:140
1443999230,23cyclictest0-21swapper/221:51:232
1443999230,21cyclictest0-21swapper/219:55:132
14433992318,2cyclictest31113-21cron23:45:001
1442999230,2cyclictest0-21swapper/023:09:060
1443399220,21cyclictest0-21swapper/121:14:111
1444399210,20cyclictest0-21swapper/323:36:393
1443999210,21cyclictest0-21swapper/222:12:482
1443999210,20cyclictest0-21swapper/223:40:592
1444399191,17cyclictest0-21swapper/300:20:153
1443999191,14cyclictest0-21swapper/222:40:132
1444399180,1cyclictest0-21swapper/323:00:413
14433991816,1cyclictest7288-21diskmemload22:38:581
1443999178,5cyclictest224-21systemd-journal20:56:252
1443999170,17cyclictest0-21swapper/221:59:072
1443999170,0cyclictest0-21swapper/200:22:162
14443991615,1cyclictest0-21swapper/323:26:593
1443399169,3cyclictest10861-21awk20:10:001
1443399169,3cyclictest10861-21awk20:10:001
1442999166,5cyclictest12754-21munin-run22:00:000
14443991514,1cyclictest0-21swapper/323:31:383
14443991514,1cyclictest0-21swapper/322:06:583
1444399150,9cyclictest0-21swapper/321:21:123
1443999158,6cyclictest22587-21cut19:25:162
1443999157,4cyclictest224-21systemd-journal22:30:002
14439991513,1cyclictest0-21swapper/219:35:132
1443999150,1cyclictest0-21swapper/221:16:232
1443999150,15cyclictest0-21swapper/223:10:572
1443999150,13cyclictest0-21swapper/221:48:512
1443399154,4cyclictest18558-21sleep22:50:001
1442999150,1cyclictest0-21swapper/023:24:180
14443991414,0cyclictest0-21swapper/323:42:133
14443991413,1cyclictest0-21swapper/322:59:183
1443999148,5cyclictest32-21ksoftirqd/223:40:002
14439991411,2cyclictest31647-21kworker/u8:221:37:322
1443999140,2cyclictest0-21swapper/223:24:572
1443999140,13cyclictest0-21swapper/223:50:472
1443399142,2cyclictest141rcu_preempt00:06:091
14433991411,2cyclictest8077-21grep23:50:261
14433991410,3cyclictest25465-21sh22:55:011
14429991414,0cyclictest0-21swapper/021:05:140
14429991411,2cyclictest14531-21wc20:15:240
1442999140,6cyclictest0-21swapper/019:15:120
1442999140,14cyclictest0-21swapper/021:14:550
1444399139,2cyclictest4121-21grep22:15:243
1444399139,2cyclictest17284-21tr19:15:143
1444399139,2cyclictest0-21swapper/321:15:163
14443991312,1cyclictest0-21swapper/322:45:573
14443991310,2cyclictest9483-21cat22:20:133
14443991310,2cyclictest2904-21grep21:00:143
14443991310,2cyclictest20401-21grep00:00:143
14443991310,2cyclictest19881-21grep20:25:263
14443991310,2cyclictest14762-21wc20:15:263
1444399130,12cyclictest2998-21cat19:50:253
1443999138,1cyclictest10555-21cat23:30:122
1443999135,7cyclictest19678-21cpuspeed_turbos19:20:142
1443999135,6cyclictest15630-21systemctl20:20:002
14439991311,1cyclictest7288-21diskmemload00:34:082
1443999131,11cyclictest7288-21diskmemload00:03:572
14439991310,2cyclictest26032-21grep19:30:262
14439991310,2cyclictest14618-21tr22:00:232
14439991310,2cyclictest14618-21tr22:00:222
1443999130,1cyclictest0-21swapper/222:32:492
1443999130,12cyclictest10706-21grep00:15:272
1443999130,12cyclictest0-21swapper/223:45:262
1443999130,12cyclictest0-21swapper/223:45:262
1443399139,3cyclictest32403-21grep19:45:151
1443399139,2cyclictest14905-21sh00:20:001
14433991310,2cyclictest29516-21uname21:25:191
14433991310,2cyclictest11859-21tr23:30:211
1443399130,1cyclictest0-21swapper/123:03:361
1443399130,11cyclictest0-21swapper/122:34:591
1442999139,3cyclictest16060-21cut00:20:180
1442999139,3cyclictest1316-21grep19:50:000
1442999139,3cyclictest0-21swapper/020:30:140
14429991312,1cyclictest0-21swapper/022:01:430
14429991312,1cyclictest0-21swapper/022:01:430
14429991310,2cyclictest5107-21wc23:25:250
14429991310,2cyclictest3557-21cut22:15:190
14429991310,2cyclictest32175-21grep22:35:280
14429991310,2cyclictest29886-21cat22:35:000
14429991310,2cyclictest27168-21grep22:55:240
14429991310,2cyclictest22911-21idleruntime-cro22:30:000
14429991310,2cyclictest1452-21ps20:55:240
14429991310,2cyclictest13-21ksoftirqd/019:40:220
1442999130,12cyclictest0-21swapper/021:51:580
93022120,1sleep30-21swapper/323:51:353
1444399129,2cyclictest9011-21tr21:10:213
1444399129,2cyclictest8484-21perf20:05:003
1444399129,2cyclictest8265-21grep23:05:173
1444399129,2cyclictest5132-21grep21:30:283
1444399129,2cyclictest25308-21grep21:45:203
1444399129,2cyclictest24155-21cat19:30:003
1444399129,2cyclictest22771-21grep23:15:223
1444399129,2cyclictest21326-21cut20:30:173
1444399129,2cyclictest13526-21grep22:00:153
1444399129,2cyclictest13526-21grep22:00:153
1444399129,2cyclictest11845-21grep20:10:173
1444399128,3cyclictest28283-21wc19:35:283
1444399128,3cyclictest25918-21grep20:40:143
1444399128,3cyclictest22774-21sed00:25:193
1444399128,2cyclictest27770-21tr00:05:193
1444399125,6cyclictest1489-21sendmail-mta23:46:333
1444399125,6cyclictest1489-21sendmail-mta23:46:323
1444399122,6cyclictest16316-21apt-get22:25:143
14443991211,1cyclictest39-21ksoftirqd/322:11:213
1444399120,12cyclictest0-21swapper/321:58:093
1444399120,12cyclictest0-21swapper/321:51:383
1443999129,2cyclictest9275-21ssh22:20:132
1443999129,2cyclictest866-21grep19:45:242
1443999129,2cyclictest8069-21grep21:10:152
1443999129,2cyclictest4207-21grep23:25:172
1443999129,2cyclictest4036-21head21:00:222
1443999129,2cyclictest3220-21grep00:10:212
1443999129,2cyclictest29302-21tail21:25:182
1443999129,2cyclictest27772-21sed22:55:302
1443999129,2cyclictest21861-21tr21:20:152
1443999129,2cyclictest20930-21sort22:05:162
1443999129,2cyclictest17522sleep223:00:202
1443999129,2cyclictest1620-21cut19:50:142
1443999129,2cyclictest11600-21df20:10:152
1443999128,3cyclictest6593-21cut20:00:162
1443999128,3cyclictest32033-21taskset20:51:082
1443999128,2cyclictest2411-21tr00:35:162
1443999128,2cyclictest14941-21tr19:10:152
1443999127,3cyclictest14075-21tune2fs23:55:182
1443999124,5cyclictest224-21systemd-journal22:48:452
1443999121,2cyclictest224-21systemd-journal00:07:552
14439991210,1cyclictest0-21swapper/223:16:172
1443999120,12cyclictest0-21swapper/221:34:542
1443999120,11cyclictest3001-21grep22:15:142
1443399129,2cyclictest8054-21tail21:55:271
1443399129,2cyclictest5864-21tr22:40:181
1443399129,2cyclictest586-21wc23:45:221
1443399129,2cyclictest586-21wc23:45:221
1443399129,2cyclictest4633-21grep19:55:171
1443399129,2cyclictest29441-21grep23:20:201
1443399129,2cyclictest27828-21sed22:55:301
1443399129,2cyclictest27166-21sed22:10:001
1443399129,2cyclictest23754-21head20:35:181
1443399129,2cyclictest22405-21grep23:15:181
1443399129,2cyclictest18002-21cron20:25:011
1443399129,2cyclictest17174-21cat21:40:001
1443399129,2cyclictest16848-21wc00:20:251
1443399129,2cyclictest15450-21grep19:10:181
1443399129,2cyclictest14219-21head23:55:181
1443399124,4cyclictest224-21systemd-journal20:35:001
1443399120,11cyclictest17104-21cut19:15:121
1442999129,2cyclictest7573-21cron00:40:000
1442999129,2cyclictest4920-21ls21:05:010
1442999129,2cyclictest30430-21tr20:50:130
1442999129,2cyclictest28631-21sort20:45:190
1442999129,2cyclictest23781-21grep20:35:180
1442999129,2cyclictest22283-21cut23:15:170
1442999129,2cyclictest21881-21cut21:20:150
1442999129,2cyclictest20440-21grep00:00:140
1442999129,2cyclictest20025-21cat00:00:000
1442999129,2cyclictest17113-21ls20:20:220
1442999129,2cyclictest14894-21grep21:15:170
1442999123,3cyclictest31080-21perf21:50:000
1442999122,9cyclictest0-21swapper/019:55:140
1442999120,0cyclictest0-21swapper/023:12:060
1444399118,2cyclictest5081-21tr19:55:223
1444399118,2cyclictest32410-21grep19:45:163
1444399118,2cyclictest29426-21grep20:45:283
1444399118,2cyclictest29326-21cut21:25:183
1444399118,2cyclictest2665-21tr00:10:183
1444399118,2cyclictest26395-21ssh22:31:343
1444399118,2cyclictest24350-21wc20:35:233
1444399118,2cyclictest20228-21cut19:20:173
1444399118,2cyclictest19258-21sh23:59:233
1444399118,2cyclictest15154-21cut23:10:173
1444399118,2cyclictest13527-21ssh21:36:403
1444399118,2cyclictest12093-21ssh00:16:573
1444399117,3cyclictest2390-21ssh00:35:163
1444399115,2cyclictest224-21systemd-journal20:54:543
1444399110,1cyclictest0-21swapper/323:20:553
1444399110,10cyclictest32380-21ssh00:33:013
1444399110,0cyclictest0-21swapper/322:50:583
1443999119,1cyclictest5823-21cut21:05:172
1443999118,2cyclictest9989-21wc20:05:232
1443999118,2cyclictest9989-21wc20:05:222
1443999118,2cyclictest26535-21uname20:40:192
1443999118,2cyclictest25248-21cat20:40:002
1443999118,2cyclictest23821-21tail00:25:292
1443999118,2cyclictest20389-21ls20:30:002
1443999118,2cyclictest18711-21ntpq21:40:192
1443999118,2cyclictest17866-21taskset20:23:192
1443999118,2cyclictest11453-21ssh23:07:272
1443999117,3cyclictest17794-21head19:15:152
1443999110,3cyclictest224-21systemd-journal22:55:002
1443399118,2cyclictest9957-21cat21:35:001
1443399118,2cyclictest7934-21cat23:05:151
1443399118,2cyclictest7322-21grep21:10:001
1443399118,2cyclictest4753-21ssh00:36:211
1443399118,2cyclictest31875-21tr21:50:171
1443399118,2cyclictest31371-21ssh00:31:501
1443399118,2cyclictest28537-21grep20:45:191
1443399118,2cyclictest2723-21grep00:10:181
1443399118,2cyclictest2497-21tr22:15:121
1443399118,2cyclictest24804-21tr21:45:181
1443399118,2cyclictest21167-21grep00:00:181
1443399118,2cyclictest19598-21chrt23:35:291
1443399118,2cyclictest18751-21ntp_states21:40:201
1443399118,2cyclictest18518-21grep20:25:141
1443399118,2cyclictest17048-21ssh23:11:031
1443399118,2cyclictest16490-21sed21:15:301
1443399118,2cyclictest16456-21ssh22:01:411
1443399118,2cyclictest16456-21ssh22:01:411
1443399118,2cyclictest16360-21grep22:25:141
1443399118,2cyclictest10176-21sed23:30:001
1443399117,3cyclictest502-21rs:main1
1443399116,2cyclictest224-21systemd-journal22:25:001
1443399114,6cyclictest31374-21latency_hist19:45:001
1443399113,6cyclictest31846-21apache219:30:131
1443399110,10cyclictest0-21swapper/120:15:171
1442999119,1cyclictest24065-21sort19:30:000
1442999118,2cyclictest6387-21ssh21:31:390
1442999118,2cyclictest32626-21ssh00:33:170
1442999118,2cyclictest32440-21grep23:45:180
1442999118,2cyclictest32440-21grep23:45:180
1442999118,2cyclictest3207-21grep19:50:260
1442999118,2cyclictest31974-21ssh21:26:540
1442999118,2cyclictest29394-21grep22:10:280
1442999118,2cyclictest28428-21sed19:35:270
1442999118,2cyclictest27998-21sh00:05:200
1442999118,2cyclictest26512-21ssh22:09:390
1442999118,2cyclictest24760-21ssh23:40:150
1442999118,2cyclictest17530-21cut21:40:130
1442999118,2cyclictest14041-21grep22:45:260
1442999118,2cyclictest13064-21seq00:18:000
1442999118,2cyclictest12479-21ssh21:35:380
1442999118,2cyclictest10587-21rm23:52:510
1442999110,1cyclictest0-21swapper/023:00:160
1442999110,10cyclictest7450-21rm22:40:520
1442999110,10cyclictest26204-21tail20:40:170
1444399108,1cyclictest7179-21chrt21:08:453
1444399108,1cyclictest17231-21readlink20:20:233
1444399108,1cyclictest16552-21chrt19:12:093
1444399107,2cyclictest4841-21ssh22:40:023
1444399107,2cyclictest30811-21grep19:40:283
1444399107,2cyclictest25520-21grep19:30:193
1444399107,2cyclictest1094-21ssh22:36:403
1444399100,9cyclictest32600-21bc20:55:143
1444399100,9cyclictest10248-21sort20:05:263
1444399100,9cyclictest10248-21sort20:05:263
1443999108,1cyclictest29496-21wc19:40:172
1443999108,1cyclictest277742sleep220:45:122
1443999108,1cyclictest21327-21sort20:30:172
1443999107,2cyclictest30295-21ssh22:35:142
1443399108,2cyclictest28191-21sensors19:35:251
1443399108,2cyclictest21238-21chrt19:20:571
1443399108,1cyclictest23780-21ssh22:06:591
1443399108,1cyclictest1560-21basename19:50:111
1443399108,1cyclictest1348-21ntpq20:55:231
1443399107,2cyclictest30428-21grep20:50:121
1443399107,2cyclictest25810-21cat20:40:141
1443399107,2cyclictest22090-21tr19:25:131
1443399107,2cyclictest22090-21grep00:25:161
1443399107,2cyclictest11370-21tr20:10:161
1443399100,9cyclictest3929-21grep21:00:201
1442999108,1cyclictest8968-21grep20:05:150
1442999108,1cyclictest8968-21grep20:05:150
1442999108,1cyclictest224782sleep000:25:170
1442999108,1cyclictest18112-21cat23:35:140
1442999108,1cyclictest146842sleep019:10:140
1442999108,1cyclictest12204-21tr20:10:200
1442999107,2cyclictest9184-21ssh22:20:080
1442999107,2cyclictest4036-21wc00:10:280
1442999107,2cyclictest24245-21dpkg19:30:010
1442999107,2cyclictest20650-21ls19:20:220
1442999107,2cyclictest18783-21ssh22:50:010
1442999100,9cyclictest20325-21chrt20:29:230
144339997,1cyclictest8359-21taskset20:03:471
144299996,2cyclictest6493-21wget20:00:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional