2020-10-30 16:25:06 +08:00
# Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
2022-03-16 16:51:22 +08:00
GO ?= go
PWD := $( shell pwd )
2022-06-28 19:58:25 +08:00
GOPATH := $( shell $( GO) env GOPATH)
SHELL := /bin/bash
2022-03-16 16:51:22 +08:00
OBJPREFIX := "github.com/milvus-io/milvus/cmd/milvus"
2020-10-30 16:25:06 +08:00
INSTALL_PATH := $( PWD) /bin
LIBRARY_PATH := $( PWD) /lib
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
PGO_PATH := $( PWD) /configs/pgo
2022-02-09 14:27:46 +08:00
OS := $( shell uname -s)
mode = Release
2023-10-11 20:45:35 +08:00
use_disk_index = OFF
2023-10-12 19:25:35 +08:00
i f d e f d i s k _ i n d e x
use_disk_index = ${ disk_index }
2023-10-11 20:45:35 +08:00
e n d i f
use_asan = OFF
i f d e f U S E _ A S A N
use_asan = ${ USE_ASAN }
2023-01-29 10:59:49 +08:00
e n d i f
2023-10-11 20:45:35 +08:00
2023-12-18 12:04:42 +08:00
use_dynamic_simd = ON
2023-07-13 16:22:30 +08:00
i f d e f U S E _ D Y N A M I C _ S I M D
use_dynamic_simd = ${ USE_DYNAMIC_SIMD }
e n d i f
2024-02-20 19:16:52 +08:00
use_opendal = OFF
i f d e f U S E _ O P E N D A L
use_opendal = ${ USE_OPENDAL }
e n d i f
2023-09-05 17:39:48 +08:00
# golangci-lint
2023-11-21 10:04:21 +08:00
GOLANGCI_LINT_VERSION := 1.55.2
2023-09-05 17:39:48 +08:00
GOLANGCI_LINT_OUTPUT := $( shell $( INSTALL_PATH) /golangci-lint --version 2>/dev/null)
INSTALL_GOLANGCI_LINT := $( findstring $( GOLANGCI_LINT_VERSION) , $( GOLANGCI_LINT_OUTPUT) )
# mockery
MOCKERY_VERSION := 2.32.4
MOCKERY_OUTPUT := $( shell $( INSTALL_PATH) /mockery --version 2>/dev/null)
INSTALL_MOCKERY := $( findstring $( MOCKERY_VERSION) ,$( MOCKERY_OUTPUT) )
2023-09-21 09:45:27 +08:00
# gci
GCI_VERSION := 0.11.2
GCI_OUTPUT := $( shell $( INSTALL_PATH) /gci --version 2>/dev/null)
INSTALL_GCI := $( findstring $( GCI_VERSION) ,$( GCI_OUTPUT) )
# gofumpt
GOFUMPT_VERSION := 0.5.0
GOFUMPT_OUTPUT := $( shell $( INSTALL_PATH) /gofumpt --version 2>/dev/null)
INSTALL_GOFUMPT := $( findstring $( GOFUMPT_VERSION) ,$( GOFUMPT_OUTPUT) )
2024-03-22 10:29:06 +08:00
# gotestsum
GOTESTSUM_VERSION := 1.11.0
GOTESTSUM_OUTPUT := $( shell $( INSTALL_PATH) /gotestsum --version 2>/dev/null)
INSTALL_GOTESTSUM := $( findstring $( GOTESTSUM_VERSION) ,$( GOTESTSUM_OUTPUT) )
2020-10-30 16:25:06 +08:00
2023-09-22 09:59:26 +08:00
index_engine = knowhere
2022-10-16 20:49:27 +08:00
export GIT_BRANCH = master
2022-10-12 14:33:23 +08:00
2023-09-20 13:59:23 +08:00
i f e q ( $ { E N A B L E _ A Z U R E } , f a l s e )
AZURE_OPTION := -Z
e n d i f
2022-10-17 19:19:25 +08:00
milvus : build -cpp print -build -info
2022-10-12 14:33:23 +08:00
@echo "Building Milvus ..."
@source $( PWD) /scripts/setenv.sh && \
mkdir -p $( INSTALL_PATH) && go env -w CGO_ENABLED = "1" && \
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
GO111MODULE = on $( GO) build -pgo= $( PGO_PATH) /default.pgo -ldflags= " -r $$ {RPATH} -X ' $( OBJPREFIX) .BuildTags= $( BUILD_TAGS) ' -X ' $( OBJPREFIX) .BuildTime= $( BUILD_TIME) ' -X ' $( OBJPREFIX) .GitCommit= $( GIT_COMMIT) ' -X ' $( OBJPREFIX) .GoVersion= $( GO_VERSION) ' " \
2023-07-13 15:34:33 +08:00
-tags dynamic -o $( INSTALL_PATH) /milvus $( PWD) /cmd/main.go 1>/dev/null
2020-10-30 16:25:06 +08:00
2023-03-21 17:55:57 +08:00
milvus-gpu : build -cpp -gpu print -gpu -build -info
2023-03-14 23:21:56 +08:00
@echo "Building Milvus-gpu ..."
@source $( PWD) /scripts/setenv.sh && \
mkdir -p $( INSTALL_PATH) && go env -w CGO_ENABLED = "1" && \
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
GO111MODULE = on $( GO) build -pgo= $( PGO_PATH) /default.pgo -ldflags= " -r $$ {RPATH} -X ' $( OBJPREFIX) .BuildTags= $( BUILD_TAGS_GPU) ' -X ' $( OBJPREFIX) .BuildTime= $( BUILD_TIME) ' -X ' $( OBJPREFIX) .GitCommit= $( GIT_COMMIT) ' -X ' $( OBJPREFIX) .GoVersion= $( GO_VERSION) ' " \
2023-07-13 15:34:33 +08:00
-tags dynamic -o $( INSTALL_PATH) /milvus $( PWD) /cmd/main.go 1>/dev/null
2023-03-14 23:21:56 +08:00
2020-10-30 16:25:06 +08:00
get-build-deps :
@( env bash $( PWD) /scripts/install_deps.sh)
2021-12-07 17:59:02 +08:00
# attention: upgrade golangci-lint should also change Dockerfiles in build/docker/builder/cpu/<os>
2020-12-09 20:07:27 +08:00
getdeps :
2023-01-29 10:59:49 +08:00
@mkdir -p $( INSTALL_PATH)
2023-09-05 17:39:48 +08:00
@if [ -z " $( INSTALL_GOLANGCI_LINT) " ] ; then \
2023-11-21 10:04:21 +08:00
echo "Installing golangci-lint into ./bin/" && curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $( INSTALL_PATH) v${ GOLANGCI_LINT_VERSION } ; \
2023-09-05 17:39:48 +08:00
else \
echo " golangci-lint v@ $( GOLANGCI_LINT_VERSION) already installed " ; \
fi
@if [ -z " $( INSTALL_MOCKERY) " ] ; then \
echo " Installing mockery v $( MOCKERY_VERSION) to ./bin/ " && GOBIN = $( INSTALL_PATH) go install github.com/vektra/mockery/v2@v$( MOCKERY_VERSION) ; \
else \
echo " Mockery v $( MOCKERY_VERSION) already installed " ; \
2023-09-11 20:43:17 +08:00
fi
2024-03-22 10:29:06 +08:00
@if [ -z " $( INSTALL_GOTESTSUM) " ] ; then \
2024-07-03 14:44:13 +08:00
echo " Install gotestsum v $( GOTESTSUM_VERSION) to ./bin/ " && GOBIN = $( INSTALL_PATH) go install -ldflags= " -X 'gotest.tools/gotestsum/cmd.version= $( GOTESTSUM_VERSION) ' " gotest.tools/gotestsum@v$( GOTESTSUM_VERSION) ; \
2024-03-22 10:29:06 +08:00
else \
echo " gotestsum v $( GOTESTSUM_VERSION) already installed " ; \
fi
2020-12-09 20:07:27 +08:00
2020-12-31 17:25:45 +08:00
tools/bin/revive : tools /check /go .mod
cd tools/check; \
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
$( GO) build -pgo= $( PGO_PATH) /default.pgo -o ../bin/revive github.com/mgechev/revive
2020-12-31 17:25:45 +08:00
2020-11-23 19:57:05 +08:00
cppcheck :
2024-02-22 19:40:53 +08:00
@#( env bash ${ PWD } /scripts/core_build.sh -l)
@( env bash ${ PWD } /scripts/check_cpp_fmt.sh)
2020-11-18 10:07:05 +08:00
2020-10-30 16:25:06 +08:00
fmt :
2020-12-29 17:19:44 +08:00
i f d e f G O _ D I F F _ F I L E S
@echo " Running $@ check "
@GO111MODULE= on env bash $( PWD) /scripts/gofmt.sh $( GO_DIFF_FILES)
e l s e
2020-10-30 16:25:06 +08:00
@echo " Running $@ check "
2020-11-18 10:07:05 +08:00
@GO111MODULE= on env bash $( PWD) /scripts/gofmt.sh cmd/
@GO111MODULE= on env bash $( PWD) /scripts/gofmt.sh internal/
2023-02-07 19:24:30 +08:00
@GO111MODULE= on env bash $( PWD) /scripts/gofmt.sh tests/integration/
2020-12-03 19:00:11 +08:00
@GO111MODULE= on env bash $( PWD) /scripts/gofmt.sh tests/go/
2023-09-08 09:51:17 +08:00
@GO111MODULE= on env bash $( PWD) /scripts/gofmt.sh pkg/
2020-12-29 17:19:44 +08:00
e n d i f
2020-10-30 16:25:06 +08:00
2023-09-21 09:45:27 +08:00
lint-fix : getdeps
@mkdir -p $( INSTALL_PATH)
@if [ -z " $( INSTALL_GCI) " ] ; then \
echo " Installing gci v $( GCI_VERSION) to ./bin/ " && GOBIN = $( INSTALL_PATH) go install github.com/daixiang0/gci@v$( GCI_VERSION) ; \
else \
echo " gci v $( GCI_VERSION) already installed " ; \
fi
@if [ -z " $( INSTALL_GOFUMPT) " ] ; then \
echo " Installing gofumpt v $( GOFUMPT_VERSION) to ./bin/ " && GOBIN = $( INSTALL_PATH) go install mvdan.cc/gofumpt@v$( GOFUMPT_VERSION) ; \
else \
echo " gofumpt v $( GOFUMPT_VERSION) already installed " ; \
fi
@echo "Running gofumpt fix"
2023-09-26 10:05:25 +08:00
@$( INSTALL_PATH) /gofumpt -l -w internal/
@$( INSTALL_PATH) /gofumpt -l -w cmd/
@$( INSTALL_PATH) /gofumpt -l -w pkg/
2024-05-20 20:47:38 +08:00
@$( INSTALL_PATH) /gofumpt -l -w client/
2024-06-21 10:32:02 +08:00
@$( INSTALL_PATH) /gofumpt -l -w tests/go_client/
2023-09-26 10:05:25 +08:00
@$( INSTALL_PATH) /gofumpt -l -w tests/integration/
2023-09-21 09:45:27 +08:00
@echo "Running gci fix"
@$( INSTALL_PATH) /gci write cmd/ --skip-generated -s standard -s default -s "prefix(github.com/milvus-io)" --custom-order
@$( INSTALL_PATH) /gci write internal/ --skip-generated -s standard -s default -s "prefix(github.com/milvus-io)" --custom-order
@$( INSTALL_PATH) /gci write pkg/ --skip-generated -s standard -s default -s "prefix(github.com/milvus-io)" --custom-order
2024-05-20 20:47:38 +08:00
@$( INSTALL_PATH) /gci write client/ --skip-generated -s standard -s default -s "prefix(github.com/milvus-io)" --custom-order
2023-09-21 09:45:27 +08:00
@$( INSTALL_PATH) /gci write tests/ --skip-generated -s standard -s default -s "prefix(github.com/milvus-io)" --custom-order
@echo "Running golangci-lint auto-fix"
2024-05-20 20:47:38 +08:00
@source $( PWD) /scripts/setenv.sh && GO111MODULE = on $( INSTALL_PATH) /golangci-lint run --fix --timeout= 30m --config $( PWD) /.golangci.yml;
@source $( PWD) /scripts/setenv.sh && cd pkg && GO111MODULE = on $( INSTALL_PATH) /golangci-lint run --fix --timeout= 30m --config $( PWD) /.golangci.yml
@source $( PWD) /scripts/setenv.sh && cd client && GO111MODULE = on $( INSTALL_PATH) /golangci-lint run --fix --timeout= 30m --config $( PWD) /client/.golangci.yml
2020-12-31 17:25:45 +08:00
2020-10-30 16:25:06 +08:00
#TODO: Check code specifications by golangci-lint
2023-01-29 10:59:49 +08:00
static-check : getdeps
2020-10-30 16:25:06 +08:00
@echo " Running $@ check "
2024-06-21 10:32:02 +08:00
@echo "Start check core packages"
2024-06-09 22:55:53 +08:00
@source $( PWD) /scripts/setenv.sh && GO111MODULE = on $( INSTALL_PATH) /golangci-lint run --build-tags dynamic,test --timeout= 30m --config $( PWD) /.golangci.yml
2024-06-21 10:32:02 +08:00
@echo "Start check pkg package"
2024-06-09 22:55:53 +08:00
@source $( PWD) /scripts/setenv.sh && cd pkg && GO111MODULE = on $( INSTALL_PATH) /golangci-lint run --build-tags dynamic,test --timeout= 30m --config $( PWD) /.golangci.yml
2024-06-21 10:32:02 +08:00
@echo "Start check client package"
2024-05-20 20:47:38 +08:00
@source $( PWD) /scripts/setenv.sh && cd client && GO111MODULE = on $( INSTALL_PATH) /golangci-lint run --timeout= 30m --config $( PWD) /client/.golangci.yml
2024-06-21 10:32:02 +08:00
@echo "Start check go_client e2e package"
@source $( PWD) /scripts/setenv.sh && cd tests/go_client && GO111MODULE = on $( INSTALL_PATH) /golangci-lint run --timeout= 30m --config $( PWD) /client/.golangci.yml
2020-10-30 16:25:06 +08:00
2022-06-10 14:58:07 +08:00
verifiers : build -cpp getdeps cppcheck fmt static -check
2020-10-30 16:25:06 +08:00
2021-09-29 14:41:58 +08:00
# Build various components locally.
2021-04-09 14:07:03 +08:00
binlog :
@echo "Building binlog ..."
2022-07-21 19:48:27 +08:00
@source $( PWD) /scripts/setenv.sh && \
2022-10-13 20:57:23 +08:00
mkdir -p $( INSTALL_PATH) && go env -w CGO_ENABLED = "1" && \
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
GO111MODULE = on $( GO) build -pgo= $( PGO_PATH) /default.pgo -ldflags= " -r $$ {RPATH} " -o $( INSTALL_PATH) /binlog $( PWD) /cmd/tools/binlog/main.go 1>/dev/null
2021-02-20 10:15:37 +08:00
2022-10-12 11:37:23 +08:00
MIGRATION_PATH = $( PWD) /cmd/tools/migration
2022-10-17 15:07:25 +08:00
meta-migration :
2022-10-12 11:37:23 +08:00
@echo "Building migration tool ..."
2023-01-11 19:31:39 +08:00
@source $( PWD) /scripts/setenv.sh && \
mkdir -p $( INSTALL_PATH) && go env -w CGO_ENABLED = "1" && \
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
GO111MODULE = on $( GO) build -pgo= $( PGO_PATH) /default.pgo -ldflags= " -r $$ {RPATH} -X ' $( OBJPREFIX) .BuildTags= $( BUILD_TAGS) ' -X ' $( OBJPREFIX) .BuildTime= $( BUILD_TIME) ' -X ' $( OBJPREFIX) .GitCommit= $( GIT_COMMIT) ' -X ' $( OBJPREFIX) .GoVersion= $( GO_VERSION) ' " \
2023-07-13 15:34:33 +08:00
-tags dynamic -o $( INSTALL_PATH) /meta-migration $( MIGRATION_PATH) /main.go 1>/dev/null
2022-10-12 11:37:23 +08:00
2023-02-07 19:24:30 +08:00
INTERATION_PATH = $( PWD) /tests/integration
2024-03-22 10:29:06 +08:00
integration-test : getdeps
2023-02-07 19:24:30 +08:00
@echo "Building integration tests ..."
2024-03-22 10:29:06 +08:00
@( env bash $( PWD) /scripts/run_intergration_test.sh " $( INSTALL_PATH) /gotestsum -- " )
2023-02-07 19:24:30 +08:00
2021-06-22 11:34:04 +08:00
BUILD_TAGS = $( shell git describe --tags --always --dirty= "-dev" )
2023-03-20 19:03:57 +08:00
BUILD_TAGS_GPU = ${ BUILD_TAGS } -gpu
2022-02-09 14:27:46 +08:00
BUILD_TIME = $( shell date -u)
2021-06-22 11:34:04 +08:00
GIT_COMMIT = $( shell git rev-parse --short HEAD)
GO_VERSION = $( shell go version)
2023-02-10 14:24:32 +08:00
2021-12-15 10:21:10 +08:00
print-build-info :
2022-09-08 17:14:36 +08:00
$( shell git config --global --add safe.directory '*' )
2021-12-15 10:21:10 +08:00
@echo " Build Tag: $( BUILD_TAGS) "
@echo " Build Time: $( BUILD_TIME) "
@echo " Git Commit: $( GIT_COMMIT) "
@echo " Go Version: $( GO_VERSION) "
2023-03-21 17:55:57 +08:00
print-gpu-build-info :
$( shell git config --global --add safe.directory '*' )
@echo " Build Tag: $( BUILD_TAGS_GPU) "
@echo " Build Time: $( BUILD_TIME) "
@echo " Git Commit: $( GIT_COMMIT) "
@echo " Go Version: $( GO_VERSION) "
2021-04-15 09:28:46 +08:00
2022-10-19 20:45:32 +08:00
update-milvus-api : download -milvus -proto
@echo "Update milvus/api version ..."
2023-06-09 01:28:37 +08:00
@( env bash $( PWD) /scripts/update-api-version.sh $( PROTO_API_VERSION) )
2022-10-17 19:19:25 +08:00
2022-10-16 20:49:27 +08:00
download-milvus-proto :
@echo "Download milvus-proto repo ..."
@( env bash $( PWD) /scripts/download_milvus_proto.sh)
2022-10-12 14:33:23 +08:00
2023-04-21 14:16:33 +08:00
build-3rdparty :
@echo "Build 3rdparty ..."
2024-02-20 19:16:52 +08:00
@( env bash $( PWD) /scripts/3rdparty_build.sh -o ${ use_opendal } )
2023-04-21 14:16:33 +08:00
2023-12-22 12:00:43 +08:00
generated-proto-without-cpp : download -milvus -proto
@echo "Generate proto ..."
@mkdir -p ${ GOPATH } /bin
@which protoc-gen-go 1>/dev/null || ( echo "Installing protoc-gen-go" && cd /tmp && go install github.com/golang/protobuf/protoc-gen-go@v1.3.2)
@( env bash $( PWD) /scripts/generate_proto.sh)
2023-04-21 14:16:33 +08:00
generated-proto : download -milvus -proto build -3rdparty
@echo "Generate proto ..."
@mkdir -p ${ GOPATH } /bin
@which protoc-gen-go 1>/dev/null || ( echo "Installing protoc-gen-go" && cd /tmp && go install github.com/golang/protobuf/protoc-gen-go@v1.3.2)
@( env bash $( PWD) /scripts/generate_proto.sh)
build-cpp : generated -proto
2023-09-26 10:31:27 +08:00
@echo "Building Milvus cpp library ..."
2024-02-20 19:16:52 +08:00
@( env bash $( PWD) /scripts/core_build.sh -t ${ mode } -n ${ use_disk_index } -y ${ use_dynamic_simd } ${ AZURE_OPTION } -x ${ index_engine } -o ${ use_opendal } )
2022-03-16 16:51:22 +08:00
2023-04-21 14:16:33 +08:00
build-cpp-gpu : generated -proto
2023-09-20 13:59:23 +08:00
@echo "Building Milvus cpp gpu library ... "
2024-02-20 19:16:52 +08:00
@( env bash $( PWD) /scripts/core_build.sh -t ${ mode } -g -n ${ use_disk_index } -y ${ use_dynamic_simd } ${ AZURE_OPTION } -x ${ index_engine } -o ${ use_opendal } )
2023-03-14 23:21:56 +08:00
2023-04-21 14:16:33 +08:00
build-cpp-with-unittest : generated -proto
2023-09-20 13:59:23 +08:00
@echo "Building Milvus cpp library with unittest ... "
2024-02-20 19:16:52 +08:00
@( env bash $( PWD) /scripts/core_build.sh -t ${ mode } -u -n ${ use_disk_index } -y ${ use_dynamic_simd } ${ AZURE_OPTION } -x ${ index_engine } -o ${ use_opendal } )
2020-10-30 16:25:06 +08:00
2023-04-21 14:16:33 +08:00
build-cpp-with-coverage : generated -proto
2022-04-11 09:49:34 +08:00
@echo "Building Milvus cpp library with coverage and unittest ..."
2024-02-20 19:16:52 +08:00
@( env bash $( PWD) /scripts/core_build.sh -t ${ mode } -a ${ use_asan } -u -c -n ${ use_disk_index } -y ${ use_dynamic_simd } ${ AZURE_OPTION } -x ${ index_engine } -o ${ use_opendal } )
2022-04-11 09:49:34 +08:00
2023-04-21 14:16:33 +08:00
check-proto-product : generated -proto
@( env bash $( PWD) /scripts/check_proto_product.sh)
2022-04-11 09:49:34 +08:00
2021-09-29 14:41:58 +08:00
# Run the tests.
2020-10-31 16:30:20 +08:00
unittest : test -cpp test -go
2020-10-30 16:25:06 +08:00
2022-09-08 17:36:35 +08:00
test-util :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t util)
test-storage :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t storage)
test-allocator :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t allocator)
test-config :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t config)
test-tso :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t tso)
test-kv :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t kv)
test-mq :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t mq)
test-rootcoord :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t rootcoord)
2022-01-24 17:18:46 +08:00
test-indexnode :
@echo "Running go unittests..."
2022-09-08 17:36:35 +08:00
@( env bash $( PWD) /scripts/run_go_unittest.sh -t indexnode)
2022-01-24 17:18:46 +08:00
2022-10-08 15:38:58 +08:00
test-indexcoord :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t indexcoord)
2021-10-24 15:16:00 +08:00
test-proxy :
@echo "Running go unittests..."
2022-09-08 17:36:35 +08:00
@( env bash $( PWD) /scripts/run_go_unittest.sh -t proxy)
2021-10-24 15:16:00 +08:00
2022-03-15 21:51:21 +08:00
test-datacoord :
@echo "Running go unittests..."
2022-09-08 17:36:35 +08:00
@( env bash $( PWD) /scripts/run_go_unittest.sh -t datacoord)
2022-03-15 21:51:21 +08:00
2022-01-04 23:33:24 +08:00
test-datanode :
@echo "Running go unittests..."
2022-09-08 17:36:35 +08:00
@( env bash $( PWD) /scripts/run_go_unittest.sh -t datanode)
2021-10-24 15:16:00 +08:00
2022-03-08 17:39:58 +08:00
test-querynode :
@echo "Running go unittests..."
2022-09-08 17:36:35 +08:00
@( env bash $( PWD) /scripts/run_go_unittest.sh -t querynode)
2022-03-08 17:39:58 +08:00
2021-10-27 23:00:24 +08:00
test-querycoord :
@echo "Running go unittests..."
2022-09-08 17:36:35 +08:00
@( env bash $( PWD) /scripts/run_go_unittest.sh -t querycoord)
2021-10-27 23:00:24 +08:00
2022-09-08 17:36:35 +08:00
test-metastore :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t metastore)
2021-10-27 23:00:24 +08:00
2024-07-15 20:49:38 +08:00
test-streaming :
@echo "Running go unittests..."
@( env bash $( PWD) /scripts/run_go_unittest.sh -t streaming)
2021-07-16 14:52:21 +08:00
test-go : build -cpp -with -unittest
2020-10-30 16:25:06 +08:00
@echo "Running go unittests..."
2021-12-08 10:55:04 +08:00
@( env bash $( PWD) /scripts/run_go_unittest.sh)
2020-10-30 16:25:06 +08:00
test-cpp : build -cpp -with -unittest
2020-11-03 11:25:47 +08:00
@echo "Running cpp unittests..."
2021-12-08 10:55:04 +08:00
@( env bash $( PWD) /scripts/run_cpp_unittest.sh)
2020-10-30 16:25:06 +08:00
2021-09-29 14:41:58 +08:00
# Run code coverage.
2021-09-22 16:01:53 +08:00
codecov : codecov -go codecov -cpp
2021-09-18 14:19:51 +08:00
2021-09-22 16:01:53 +08:00
# Run codecov-go
2022-04-11 09:49:34 +08:00
codecov-go : build -cpp -with -coverage
2021-09-18 14:19:51 +08:00
@echo "Running go coverage..."
2021-04-24 15:10:24 +08:00
@( env bash $( PWD) /scripts/run_go_codecov.sh)
2024-03-27 15:29:10 +08:00
# Run codecov-go without build core again, used in github action
codecov-go-without-build : getdeps
@echo "Running go coverage..."
@( env bash $( PWD) /scripts/run_go_codecov.sh " $( INSTALL_PATH) /gotestsum -- " )
2021-09-22 16:01:53 +08:00
# Run codecov-cpp
2022-04-11 09:49:34 +08:00
codecov-cpp : build -cpp -with -coverage
2021-09-18 14:19:51 +08:00
@echo "Running cpp coverage..."
2021-09-17 14:21:50 +08:00
@( env bash $( PWD) /scripts/run_cpp_codecov.sh)
2021-09-29 14:41:58 +08:00
# Build each component and install binary to $GOPATH/bin.
2022-10-12 14:33:23 +08:00
install : milvus
2020-10-30 16:25:06 +08:00
@echo "Installing binary to './bin'"
2021-04-15 09:28:46 +08:00
@mkdir -p $( GOPATH) /bin && cp -f $( PWD) /bin/milvus $( GOPATH) /bin/milvus
2022-11-02 15:39:35 +08:00
@mkdir -p $( LIBRARY_PATH)
2022-11-23 10:39:11 +08:00
-cp -r -P $( PWD) /internal/core/output/lib/*.dylib* $( LIBRARY_PATH) 2>/dev/null
-cp -r -P $( PWD) /internal/core/output/lib/*.so* $( LIBRARY_PATH) 2>/dev/null
-cp -r -P $( PWD) /internal/core/output/lib64/*.so* $( LIBRARY_PATH) 2>/dev/null
2020-10-30 16:25:06 +08:00
@echo "Installation successful."
2023-03-20 19:03:57 +08:00
gpu-install : milvus -gpu
@echo "Installing binary to './bin'"
@mkdir -p $( GOPATH) /bin && cp -f $( PWD) /bin/milvus $( GOPATH) /bin/milvus
@mkdir -p $( LIBRARY_PATH)
-cp -r -P $( PWD) /internal/core/output/lib/*.dylib* $( LIBRARY_PATH) 2>/dev/null
-cp -r -P $( PWD) /internal/core/output/lib/*.so* $( LIBRARY_PATH) 2>/dev/null
-cp -r -P $( PWD) /internal/core/output/lib64/*.so* $( LIBRARY_PATH) 2>/dev/null
@echo "Installation successful."
2020-10-30 16:25:06 +08:00
clean :
@echo "Cleaning up all the generated files"
2021-01-11 18:35:54 +08:00
@rm -rf bin/
@rm -rf lib/
2021-09-08 16:32:00 +08:00
@rm -rf $( GOPATH) /bin/milvus
2022-02-10 17:03:48 +08:00
@rm -rf cmake_build
2022-05-26 22:58:03 +08:00
@rm -rf internal/core/output
2021-08-24 15:51:51 +08:00
2021-12-15 10:21:10 +08:00
milvus-tools : print -build -info
2021-08-24 15:51:51 +08:00
@echo "Building tools ..."
2024-07-25 18:05:46 +08:00
@. $( PWD) /scripts/setenv.sh && mkdir -p $( INSTALL_PATH) /tools && go env -w CGO_ENABLED = "1" && GO111MODULE = on $( GO) build \
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
-pgo= $( PGO_PATH) /default.pgo -ldflags= " -X 'main.BuildTags= $( BUILD_TAGS) ' -X 'main.BuildTime= $( BUILD_TIME) ' -X 'main.GitCommit= $( GIT_COMMIT) ' -X 'main.GoVersion= $( GO_VERSION) ' " \
2021-08-24 15:51:51 +08:00
-o $( INSTALL_PATH) /tools $( PWD) /cmd/tools/* 1>/dev/null
2021-12-28 09:52:10 +08:00
2022-06-24 21:12:15 +08:00
rpm-setup :
2021-12-28 09:52:10 +08:00
@echo "Setuping rpm env ...;"
2021-12-30 16:59:20 +08:00
@build/rpm/setup-env.sh
2021-12-28 09:52:10 +08:00
rpm : install
@echo "Note: run 'make rpm-setup' to setup build env for rpm builder"
@echo "Building rpm ...;"
@yum -y install rpm-build rpmdevtools wget
@rm -rf ~/rpmbuild/BUILD/*
@rpmdev-setuptree
@wget https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz && tar -xf etcd-v3.5.0-linux-amd64.tar.gz
@cp etcd-v3.5.0-linux-amd64/etcd bin/etcd
@wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio.RELEASE.2021-02-14T04-01-33Z -O bin/minio
@cp -r bin ~/rpmbuild/BUILD/
@cp -r lib ~/rpmbuild/BUILD/
@cp -r configs ~/rpmbuild/BUILD/
@cp -r build/rpm/services ~/rpmbuild/BUILD/
2021-12-30 16:59:20 +08:00
@QA_RPATHS= " $$ [ 0x001|0x0002|0x0020 ] " rpmbuild -ba ./build/rpm/milvus.spec
2022-09-23 10:50:51 +08:00
2023-09-04 21:19:48 +08:00
generate-mockery-types : getdeps
# RootCoord
$( INSTALL_PATH) /mockery --name= RootCoordComponent --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_rootcoord.go --with-expecter --structname= RootCoord
# Proxy
$( INSTALL_PATH) /mockery --name= ProxyComponent --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_proxy.go --with-expecter --structname= MockProxy
# QueryCoord
2023-09-11 20:43:17 +08:00
$( INSTALL_PATH) /mockery --name= QueryCoordComponent --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_querycoord.go --with-expecter --structname= MockQueryCoord
2023-09-04 21:19:48 +08:00
# QueryNode
2023-09-11 20:43:17 +08:00
$( INSTALL_PATH) /mockery --name= QueryNodeComponent --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_querynode.go --with-expecter --structname= MockQueryNode
2023-09-04 21:19:48 +08:00
# DataCoord
$( INSTALL_PATH) /mockery --name= DataCoordComponent --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_datacoord.go --with-expecter --structname= MockDataCoord
# DataNode
$( INSTALL_PATH) /mockery --name= DataNodeComponent --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_datanode.go --with-expecter --structname= MockDataNode
# IndexNode
$( INSTALL_PATH) /mockery --name= IndexNodeComponent --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_indexnode.go --with-expecter --structname= MockIndexNode
2023-09-26 09:57:25 +08:00
# Clients
$( INSTALL_PATH) /mockery --name= RootCoordClient --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_rootcoord_client.go --with-expecter --structname= MockRootCoordClient
$( INSTALL_PATH) /mockery --name= QueryCoordClient --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_querycoord_client.go --with-expecter --structname= MockQueryCoordClient
$( INSTALL_PATH) /mockery --name= DataCoordClient --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_datacoord_client.go --with-expecter --structname= MockDataCoordClient
$( INSTALL_PATH) /mockery --name= QueryNodeClient --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_querynode_client.go --with-expecter --structname= MockQueryNodeClient
$( INSTALL_PATH) /mockery --name= DataNodeClient --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_datanode_client.go --with-expecter --structname= MockDataNodeClient
$( INSTALL_PATH) /mockery --name= IndexNodeClient --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_indexnode_client.go --with-expecter --structname= MockIndexNodeClient
$( INSTALL_PATH) /mockery --name= ProxyClient --dir= $( PWD) /internal/types --output= $( PWD) /internal/mocks --filename= mock_proxy_client.go --with-expecter --structname= MockProxyClient
2023-09-04 21:19:48 +08:00
generate-mockery-rootcoord : getdeps
$( INSTALL_PATH) /mockery --name= IMetaTable --dir= $( PWD) /internal/rootcoord --output= $( PWD) /internal/rootcoord/mocks --filename= meta_table.go --with-expecter --outpkg= mockrootcoord
$( INSTALL_PATH) /mockery --name= GarbageCollector --dir= $( PWD) /internal/rootcoord --output= $( PWD) /internal/rootcoord/mocks --filename= garbage_collector.go --with-expecter --outpkg= mockrootcoord
generate-mockery-proxy : getdeps
$( INSTALL_PATH) /mockery --name= Cache --dir= $( PWD) /internal/proxy --output= $( PWD) /internal/proxy --filename= mock_cache.go --structname= MockCache --with-expecter --outpkg= proxy --inpackage
$( INSTALL_PATH) /mockery --name= timestampAllocatorInterface --dir= $( PWD) /internal/proxy --output= $( PWD) /internal/proxy --filename= mock_tso_test.go --structname= mockTimestampAllocator --with-expecter --outpkg= proxy --inpackage
$( INSTALL_PATH) /mockery --name= LBPolicy --dir= $( PWD) /internal/proxy --output= $( PWD) /internal/proxy --filename= mock_lb_policy.go --structname= MockLBPolicy --with-expecter --outpkg= proxy --inpackage
$( INSTALL_PATH) /mockery --name= LBBalancer --dir= $( PWD) /internal/proxy --output= $( PWD) /internal/proxy --filename= mock_lb_balancer.go --structname= MockLBBalancer --with-expecter --outpkg= proxy --inpackage
$( INSTALL_PATH) /mockery --name= shardClientMgr --dir= $( PWD) /internal/proxy --output= $( PWD) /internal/proxy --filename= mock_shardclient_manager.go --structname= MockShardClientManager --with-expecter --outpkg= proxy --inpackage
2023-09-12 10:19:17 +08:00
$( INSTALL_PATH) /mockery --name= channelsMgr --dir= $( PWD) /internal/proxy --output= $( PWD) /internal/proxy --filename= mock_channels_manager.go --structname= MockChannelsMgr --with-expecter --outpkg= proxy --inpackage
2023-09-04 21:19:48 +08:00
generate-mockery-querycoord : getdeps
$( INSTALL_PATH) /mockery --name= QueryNodeServer --dir= $( PWD) /internal/proto/querypb/ --output= $( PWD) /internal/querycoordv2/mocks --filename= mock_querynode.go --with-expecter --structname= MockQueryNodeServer
$( INSTALL_PATH) /mockery --name= Broker --dir= $( PWD) /internal/querycoordv2/meta --output= $( PWD) /internal/querycoordv2/meta --filename= mock_broker.go --with-expecter --structname= MockBroker --outpkg= meta
2024-05-17 10:11:37 +08:00
$( INSTALL_PATH) /mockery --name= TargetManagerInterface --dir= $( PWD) /internal/querycoordv2/meta --output= $( PWD) /internal/querycoordv2/meta --filename= mock_target_manager.go --with-expecter --structname= MockTargetManager --inpackage
2023-09-04 21:19:48 +08:00
$( INSTALL_PATH) /mockery --name= Scheduler --dir= $( PWD) /internal/querycoordv2/task --output= $( PWD) /internal/querycoordv2/task --filename= mock_scheduler.go --with-expecter --structname= MockScheduler --outpkg= task --inpackage
$( INSTALL_PATH) /mockery --name= Cluster --dir= $( PWD) /internal/querycoordv2/session --output= $( PWD) /internal/querycoordv2/session --filename= mock_cluster.go --with-expecter --structname= MockCluster --outpkg= session --inpackage
$( INSTALL_PATH) /mockery --name= Balance --dir= $( PWD) /internal/querycoordv2/balance --output= $( PWD) /internal/querycoordv2/balance --filename= mock_balancer.go --with-expecter --structname= MockBalancer --outpkg= balance --inpackage
$( INSTALL_PATH) /mockery --name= Controller --dir= $( PWD) /internal/querycoordv2/dist --output= $( PWD) /internal/querycoordv2/dist --filename= mock_controller.go --with-expecter --structname= MockController --outpkg= dist --inpackage
2023-09-05 14:13:49 +08:00
generate-mockery-querynode : getdeps build -cpp
@source $( PWD) /scripts/setenv.sh # setup PKG_CONFIG_PATH
2023-10-11 20:45:35 +08:00
$( INSTALL_PATH) /mockery --name= QueryHook --dir= $( PWD) /internal/querynodev2/optimizers --output= $( PWD) /internal/querynodev2/optimizers --filename= mock_query_hook.go --with-expecter --outpkg= optimizers --structname= MockQueryHook --inpackage
2023-09-04 21:19:48 +08:00
$( INSTALL_PATH) /mockery --name= Manager --dir= $( PWD) /internal/querynodev2/cluster --output= $( PWD) /internal/querynodev2/cluster --filename= mock_manager.go --with-expecter --outpkg= cluster --structname= MockManager --inpackage
$( INSTALL_PATH) /mockery --name= SegmentManager --dir= $( PWD) /internal/querynodev2/segments --output= $( PWD) /internal/querynodev2/segments --filename= mock_segment_manager.go --with-expecter --outpkg= segments --structname= MockSegmentManager --inpackage
$( INSTALL_PATH) /mockery --name= CollectionManager --dir= $( PWD) /internal/querynodev2/segments --output= $( PWD) /internal/querynodev2/segments --filename= mock_collection_manager.go --with-expecter --outpkg= segments --structname= MockCollectionManager --inpackage
$( INSTALL_PATH) /mockery --name= Loader --dir= $( PWD) /internal/querynodev2/segments --output= $( PWD) /internal/querynodev2/segments --filename= mock_loader.go --with-expecter --outpkg= segments --structname= MockLoader --inpackage
$( INSTALL_PATH) /mockery --name= Segment --dir= $( PWD) /internal/querynodev2/segments --output= $( PWD) /internal/querynodev2/segments --filename= mock_segment.go --with-expecter --outpkg= segments --structname= MockSegment --inpackage
$( INSTALL_PATH) /mockery --name= Worker --dir= $( PWD) /internal/querynodev2/cluster --output= $( PWD) /internal/querynodev2/cluster --filename= mock_worker.go --with-expecter --outpkg= worker --structname= MockWorker --inpackage
$( INSTALL_PATH) /mockery --name= ShardDelegator --dir= $( PWD) /internal/querynodev2/delegator/ --output= $( PWD) /internal/querynodev2/delegator/ --filename= mock_delegator.go --with-expecter --outpkg= delegator --structname= MockShardDelegator --inpackage
generate-mockery-datacoord : getdeps
$( INSTALL_PATH) /mockery --name= compactionPlanContext --dir= internal/datacoord --filename= mock_compaction_plan_context.go --output= internal/datacoord --structname= MockCompactionPlanContext --with-expecter --inpackage
$( INSTALL_PATH) /mockery --name= Handler --dir= internal/datacoord --filename= mock_handler.go --output= internal/datacoord --structname= NMockHandler --with-expecter --inpackage
$( INSTALL_PATH) /mockery --name= allocator --dir= internal/datacoord --filename= mock_allocator_test.go --output= internal/datacoord --structname= NMockAllocator --with-expecter --inpackage
2023-11-07 14:02:20 +08:00
$( INSTALL_PATH) /mockery --name= RWChannelStore --dir= internal/datacoord --filename= mock_channel_store.go --output= internal/datacoord --structname= MockRWChannelStore --with-expecter --inpackage
2023-11-21 09:26:22 +08:00
$( INSTALL_PATH) /mockery --name= IndexEngineVersionManager --dir= internal/datacoord --filename= mock_index_engine_version_manager.go --output= internal/datacoord --structname= MockVersionManager --with-expecter --inpackage
2023-11-23 17:30:25 +08:00
$( INSTALL_PATH) /mockery --name= TriggerManager --dir= internal/datacoord --filename= mock_trigger_manager.go --output= internal/datacoord --structname= MockTriggerManager --with-expecter --inpackage
2023-12-11 17:52:37 +08:00
$( INSTALL_PATH) /mockery --name= Cluster --dir= internal/datacoord --filename= mock_cluster.go --output= internal/datacoord --structname= MockCluster --with-expecter --inpackage
$( INSTALL_PATH) /mockery --name= SessionManager --dir= internal/datacoord --filename= mock_session_manager.go --output= internal/datacoord --structname= MockSessionManager --with-expecter --inpackage
2023-11-23 17:30:25 +08:00
$( INSTALL_PATH) /mockery --name= compactionPlanContext --dir= internal/datacoord --filename= mock_compaction_plan_context.go --output= internal/datacoord --structname= MockCompactionPlanContext --with-expecter --inpackage
2023-12-05 18:44:37 +08:00
$( INSTALL_PATH) /mockery --name= CompactionMeta --dir= internal/datacoord --filename= mock_compaction_meta.go --output= internal/datacoord --structname= MockCompactionMeta --with-expecter --inpackage
2023-12-25 19:24:46 +08:00
$( INSTALL_PATH) /mockery --name= ChannelManager --dir= internal/datacoord --filename= mock_channelmanager.go --output= internal/datacoord --structname= MockChannelManager --with-expecter --inpackage
2024-05-07 15:49:30 +08:00
$( INSTALL_PATH) /mockery --name= SubCluster --dir= internal/datacoord --filename= mock_subcluster.go --output= internal/datacoord --structname= MockSubCluster --with-expecter --inpackage
2024-04-12 16:01:19 +08:00
$( INSTALL_PATH) /mockery --name= Broker --dir= internal/datacoord/broker --filename= mock_coordinator_broker.go --output= internal/datacoord/broker --structname= MockBroker --with-expecter --inpackage
2024-06-06 17:37:51 +08:00
$( INSTALL_PATH) /mockery --name= WorkerManager --dir= internal/datacoord --filename= mock_worker_manager.go --output= internal/datacoord --structname= MockWorkerManager --with-expecter --inpackage
2024-07-26 10:15:46 +08:00
$( INSTALL_PATH) /mockery --name= Manager --dir= internal/datacoord --filename= mock_segment_manager.go --output= internal/datacoord --structname= MockManager --with-expecter --inpackage
2023-09-04 21:19:48 +08:00
generate-mockery-datanode : getdeps
$( INSTALL_PATH) /mockery --name= Allocator --dir= $( PWD) /internal/datanode/allocator --output= $( PWD) /internal/datanode/allocator --filename= mock_allocator.go --with-expecter --structname= MockAllocator --outpkg= allocator --inpackage
2023-10-11 17:03:34 +08:00
$( INSTALL_PATH) /mockery --name= Broker --dir= $( PWD) /internal/datanode/broker --output= $( PWD) /internal/datanode/broker/ --filename= mock_broker.go --with-expecter --structname= MockBroker --outpkg= broker --inpackage
2023-10-31 02:30:16 +08:00
$( INSTALL_PATH) /mockery --name= MetaCache --dir= $( PWD) /internal/datanode/metacache --output= $( PWD) /internal/datanode/metacache --filename= mock_meta_cache.go --with-expecter --structname= MockMetaCache --outpkg= metacache --inpackage
2023-11-04 12:10:17 +08:00
$( INSTALL_PATH) /mockery --name= SyncManager --dir= $( PWD) /internal/datanode/syncmgr --output= $( PWD) /internal/datanode/syncmgr --filename= mock_sync_manager.go --with-expecter --structname= MockSyncManager --outpkg= syncmgr --inpackage
2023-12-26 10:40:47 +08:00
$( INSTALL_PATH) /mockery --name= MetaWriter --dir= $( PWD) /internal/datanode/syncmgr --output= $( PWD) /internal/datanode/syncmgr --filename= mock_meta_writer.go --with-expecter --structname= MockMetaWriter --outpkg= syncmgr --inpackage
2024-01-05 21:42:47 +08:00
$( INSTALL_PATH) /mockery --name= Serializer --dir= $( PWD) /internal/datanode/syncmgr --output= $( PWD) /internal/datanode/syncmgr --filename= mock_serializer.go --with-expecter --structname= MockSerializer --outpkg= syncmgr --inpackage
2024-02-05 11:33:43 +08:00
$( INSTALL_PATH) /mockery --name= Task --dir= $( PWD) /internal/datanode/syncmgr --output= $( PWD) /internal/datanode/syncmgr --filename= mock_task.go --with-expecter --structname= MockTask --outpkg= syncmgr --inpackage
2023-11-15 15:24:18 +08:00
$( INSTALL_PATH) /mockery --name= WriteBuffer --dir= $( PWD) /internal/datanode/writebuffer --output= $( PWD) /internal/datanode/writebuffer --filename= mock_write_buffer.go --with-expecter --structname= MockWriteBuffer --outpkg= writebuffer --inpackage
2023-11-16 00:22:20 +08:00
$( INSTALL_PATH) /mockery --name= BufferManager --dir= $( PWD) /internal/datanode/writebuffer --output= $( PWD) /internal/datanode/writebuffer --filename= mock_mananger.go --with-expecter --structname= MockBufferManager --outpkg= writebuffer --inpackage
2023-11-30 14:30:28 +08:00
$( INSTALL_PATH) /mockery --name= BinlogIO --dir= $( PWD) /internal/datanode/io --output= $( PWD) /internal/datanode/io --filename= mock_binlogio.go --with-expecter --structname= MockBinlogIO --outpkg= io --inpackage
2024-07-01 14:46:07 +08:00
$( INSTALL_PATH) /mockery --name= FlowgraphManager --dir= $( PWD) /internal/datanode/pipeline --output= $( PWD) /internal/datanode/pipeline --filename= mock_fgmanager.go --with-expecter --structname= MockFlowgraphManager --outpkg= pipeline --inpackage
$( INSTALL_PATH) /mockery --name= ChannelManager --dir= $( PWD) /internal/datanode/channel --output= $( PWD) /internal/datanode/channel --filename= mock_channelmanager.go --with-expecter --structname= MockChannelManager --outpkg= channel --inpackage
2024-05-23 09:53:40 +08:00
$( INSTALL_PATH) /mockery --name= Compactor --dir= $( PWD) /internal/datanode/compaction --output= $( PWD) /internal/datanode/compaction --filename= mock_compactor.go --with-expecter --structname= MockCompactor --outpkg= compaction --inpackage
2023-09-04 21:19:48 +08:00
generate-mockery-metastore : getdeps
2023-09-11 20:43:17 +08:00
$( INSTALL_PATH) /mockery --name= RootCoordCatalog --dir= $( PWD) /internal/metastore --output= $( PWD) /internal/metastore/mocks --filename= mock_rootcoord_catalog.go --with-expecter --structname= RootCoordCatalog --outpkg= mocks
$( INSTALL_PATH) /mockery --name= DataCoordCatalog --dir= $( PWD) /internal/metastore --output= $( PWD) /internal/metastore/mocks --filename= mock_datacoord_catalog.go --with-expecter --structname= DataCoordCatalog --outpkg= mocks
$( INSTALL_PATH) /mockery --name= QueryCoordCatalog --dir= $( PWD) /internal/metastore --output= $( PWD) /internal/metastore/mocks --filename= mock_querycoord_catalog.go --with-expecter --structname= QueryCoordCatalog --outpkg= mocks
2023-09-04 21:19:48 +08:00
generate-mockery-utils : getdeps
# dependency.Factory
$( INSTALL_PATH) /mockery --name= Factory --dir= internal/util/dependency --output= internal/util/dependency --filename= mock_factory.go --with-expecter --structname= MockFactory --inpackage
# tso.Allocator
$( INSTALL_PATH) /mockery --name= Allocator --dir= internal/tso --output= internal/tso/mocks --filename= allocator.go --with-expecter --structname= Allocator --outpkg= mocktso
2023-11-03 10:14:16 +08:00
$( INSTALL_PATH) /mockery --name= SessionInterface --dir= $( PWD) /internal/util/sessionutil --output= $( PWD) /internal/util/sessionutil --filename= mock_session.go --with-expecter --structname= MockSession --inpackage
2023-12-13 10:46:44 +08:00
$( INSTALL_PATH) /mockery --name= GrpcClient --dir= $( PWD) /internal/util/grpcclient --output= $( PWD) /internal/mocks --filename= mock_grpc_client.go --with-expecter --structname= MockGrpcClient
2023-12-20 19:22:42 +08:00
# proxy_client_manager.go
$( INSTALL_PATH) /mockery --name= ProxyClientManagerInterface --dir= $( PWD) /internal/util/proxyutil --output= $( PWD) /internal/util/proxyutil --filename= mock_proxy_client_manager.go --with-expecter --structname= MockProxyClientManager --inpackage
$( INSTALL_PATH) /mockery --name= ProxyWatcherInterface --dir= $( PWD) /internal/util/proxyutil --output= $( PWD) /internal/util/proxyutil --filename= mock_proxy_watcher.go --with-expecter --structname= MockProxyWatcher --inpackage
2023-09-04 21:19:48 +08:00
generate-mockery-kv : getdeps
$( INSTALL_PATH) /mockery --name= TxnKV --dir= $( PWD) /internal/kv --output= $( PWD) /internal/kv/mocks --filename= txn_kv.go --with-expecter
$( INSTALL_PATH) /mockery --name= MetaKv --dir= $( PWD) /internal/kv --output= $( PWD) /internal/kv/mocks --filename= meta_kv.go --with-expecter
$( INSTALL_PATH) /mockery --name= WatchKV --dir= $( PWD) /internal/kv --output= $( PWD) /internal/kv/mocks --filename= watch_kv.go --with-expecter
$( INSTALL_PATH) /mockery --name= SnapShotKV --dir= $( PWD) /internal/kv --output= $( PWD) /internal/kv/mocks --filename= snapshot_kv.go --with-expecter
2023-09-27 10:21:26 +08:00
$( INSTALL_PATH) /mockery --name= Predicate --dir= $( PWD) /internal/kv/predicates --output= $( PWD) /internal/kv/predicates --filename= mock_predicate.go --with-expecter --inpackage
2023-09-04 21:19:48 +08:00
2023-11-29 20:52:27 +08:00
generate-mockery-chunk-manager : getdeps
$( INSTALL_PATH) /mockery --name= ChunkManager --dir= $( PWD) /internal/storage --output= $( PWD) /internal/mocks --filename= mock_chunk_manager.go --with-expecter
2023-09-04 21:19:48 +08:00
generate-mockery-pkg :
$( MAKE) -C pkg generate-mockery
2024-07-15 20:49:38 +08:00
generate-mockery-internal :
$( INSTALL_PATH) /mockery --config $( PWD) /internal/.mockery.yaml
2024-06-11 10:38:01 +08:00
2024-07-15 20:49:38 +08:00
generate-mockery : generate -mockery -types generate -mockery -kv generate -mockery -rootcoord generate -mockery -proxy generate -mockery -querycoord generate -mockery -querynode generate -mockery -datacoord generate -mockery -pkg generate -mockery -internal
2023-09-11 20:43:17 +08:00
2024-05-10 14:31:37 +08:00
generate-yaml : milvus -tools
@echo "Updating milvus config yaml"
@$( PWD) /bin/tools/config gen-yaml && mv milvus.yaml configs/milvus.yaml
2024-03-25 15:51:09 +08:00
MMAP_MIGRATION_PATH = $( PWD) /cmd/tools/migration/mmap/tool
mmap-migration :
@echo "Building migration tool ..."
@source $( PWD) /scripts/setenv.sh && \
mkdir -p $( INSTALL_PATH) && go env -w CGO_ENABLED = "1" && \
Upgrade go from 1.20 to 1.21 (#33047)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]
issue: https://github.com/milvus-io/milvus/issues/32982
# Background
Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. According to
[Go 1.21 Release Notes](https://tip.golang.org/doc/go1.21), the big
difference of Go 1.21 is enabling Profile-Guided Optimization by
default, which can improve performance by around 2-14%. Here are the
summary steps of PGO:
1. Build Initial Binary (Without PGO)
2. Deploying the Production Environment
3. Run the program and collect Performance Analysis Data (CPU pprof)
4. Analyze the Collected Data and Select a Performance Profile for PGO
5. Place the Performance Analysis File in the Main Package Directory and
Name It default.pgo
6. go build Detects the default.pgo File and Enables PGO
7. Build and Release the Updated Binary (With PGO)
8. Iterate and Repeat the Above Steps
<img width="657" alt="Screenshot 2024-05-14 at 15 57 01"
src="https://github.com/milvus-io/milvus/assets/167743503/b08d4300-0be1-44dc-801f-ce681dabc581">
# What does this PR do
There are three experiments, search benchmark by Zilliz test platform,
search benchmark by open-source
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file),
and search benchmark with PGO. We do both search benchmarks by Zilliz
test platform and by VectorDBBench to reduce reliance on a single
experimental result. Besides, we validate the performance enhancement
with PGO.
## Search Benchmark Report by Zilliz Test Platform
An upgrade to Go 1.21 was conducted on a Milvus Standalone server,
equipped with 16 CPUs and 64GB of memory. The search performance was
evaluated using a 1 million entry local dataset with an L2 metric type
in a 768-dimensional space. The system was tested for concurrent
searches with 50 concurrent tasks for 1 hour, each with a 20-second
interval. The reason for using one server rather than two servers to
compare is to guarantee the same data source and same segment state
after compaction.
Test Sequence:
1. Go 1.20 Initial Run: Insert data, build index, load index, and
search.
2. Go 1.20 Rebuild: Rebuild the index with the same dataset, load index,
and search.
3. Go 1.21 Load: Upload to Go 1.21 within the server. Then load the
index from the second run, and search.
4. Go 1.21 Rebuild: Rebuild the index with the same dataset, load index,
and search.
Search Metrics:
| Metric | Go 1.20 | Go 1.20 Rebuild Index | Go 1.21 | Go 1.21 Rebuild
Index |
|----------------------------|------------------|-----------------|------------------|-----------------|
| `search requests` | 10,942,683 | 16,131,726 | 16,200,887 | 16,331,052
|
| `search fails` | 0 | 0 | 0 | 0 |
| `search RT_avg` (ms) | 16.44 | 11.15 | 11.11 | 11.02 |
| `search RT_min` (ms) | 1.30 | 1.28 | 1.31 | 1.26 |
| `search RT_max` (ms) | 446.61 | 233.22 | 235.90 | 147.93 |
| `search TP50` (ms) | 11.74 | 10.46 | 10.43 | 10.35 |
| `search TP99` (ms) | 92.30 | 25.76 | 25.36 | 25.23 |
| `search RPS` | 3,039 | 4,481 | 4,500 | 4,536 |
### Key Findings
The benchmark tests reveal that the index build time with Go 1.20 at
340.39 ms and Go 1.21 at 337.60 ms demonstrated negligible performance
variance in index construction. However, Go 1.21 offers slightly better
performance in search operations compared to Go 1.20, with improvements
in handling concurrent tasks and reducing response times.
## Search Benchmark Report By VectorDb Bench
Follow
[VectorDBBench](https://github.com/zilliztech/VectorDBBench?tab=readme-ov-file)
to create a VectorDb Bench test for Go 1.20 and Go 1.21. We test the
search performance with Go 1.20 and Go 1.21 (without PGO) on the Milvus
Standalone system. The tests were conducted using the Cohere dataset
with 1 million entries in a 768-dimensional space, utilizing the COSINE
metric type.
Search Metrics:
Metric | Go 1.20 | Go 1.21 without PGO
-- | -- | --
Load Duration (seconds) | 1195.95 | 976.37
Queries Per Second (QPS) | 841.62 | 875.89
99th Percentile Serial Latency (seconds) | 0.0047 | 0.0076
Recall | 0.9487 | 0.9489
### Key Findings
Go 1.21 indicates faster index loading times and larger search QPS
handling.
## PGO Performance Test
Milvus has already added
[net/http/pprof](https://pkg.go.dev/net/http/pprof) in the metrics. So
we can curl the CPU profile directly by running
`curl -o default.pgo
"http://${MILVUS_SERVER_IP}:${MILVUS_SERVER_PORT}/debug/pprof/profile?seconds=${TIME_SECOND}"`
to collect the profile as the default.pgo during the first search. Then
I build Milvus with PGO and use the same index to run the search again.
The result is as below:
Search Metrics
| Metric | Go 1.21 Without PGO | Go 1.21 With PGO | Change (%) |
|---------------------------------------------|------------------|-----------------|------------|
| `search Requests` | 2,644,583 | 2,837,726 | +7.30% |
| `search Fails` | 0 | 0 | N/A |
| `search RT_avg` (ms) | 11.34 | 10.57 | -6.78% |
| `search RT_min` (ms) | 1.39 | 1.32 | -5.18% |
| `search RT_max` (ms) | 349.72 | 143.72 | -58.91% |
| `search TP50` (ms) | 10.57 | 9.93 | -6.05% |
| `search TP99` (ms) | 26.14 | 24.16 | -7.56% |
| `search RPS` | 4,407 | 4,729 | +7.30% |
### Key Findings
PGO led to a notable enhancement in search performance, particularly in
reducing the maximum response time by 58% and increasing the search QPS
by 7.3%.
### Further Analysis
Generate a diff flame graphs between two CPU profiles by running `go
tool pprof -http=:8000 -diff_base nopgo.pgo pgo.pgo -normalize`
<img width="1894" alt="goprofiling"
src="https://github.com/milvus-io/milvus/assets/167743503/ab9e91eb-95c7-4963-acd9-d1c3c73ee010">
Further insight of HnswIndexNode and Milvus Search Handler
<img width="1906" alt="hnsw"
src="https://github.com/milvus-io/milvus/assets/167743503/a04cf4a0-7c97-4451-b3cf-98afc20a0b05">
<img width="1873" alt="search_handler"
src="https://github.com/milvus-io/milvus/assets/167743503/5f4d3982-18dd-4115-8e76-460f7f534c7f">
After applying PGO to the Milvus server, the CPU utilization of the
faiss::fvec_L2 function has decreased. This optimization significantly
enhances the performance of the
[HnswIndexNode::Search::searchKnn](https://github.com/zilliztech/knowhere/blob/e0c9c41aa22d8f6e6761a0a54460e4573de15bfe/src/index/hnsw/hnsw.cc#L203)
method, which is frequently invoked by Knowhere during high-concurrency
searches. As the explanation from Go release notes, the function might
be more aggressively inlined by Go compiler during the second build with
the CPU profiling collected from the first run. As a result, the search
handler efficiency within Milvus DataNode has improved, allowing the
server to process a higher number of search queries per second (QPS).
# Conclusion
The combination of Go 1.21 and PGO has led to substantial enhancements
in search performance for Milvus server, particularly in terms of search
QPS and response times, making it more efficient for handling
high-concurrency search operations.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-22 13:21:39 +08:00
GO111MODULE = on $( GO) build -pgo= $( PGO_PATH) /default.pgo -ldflags= " -r $$ {RPATH} -X ' $( OBJPREFIX) .BuildTags= $( BUILD_TAGS) ' -X ' $( OBJPREFIX) .BuildTime= $( BUILD_TIME) ' -X ' $( OBJPREFIX) .GitCommit= $( GIT_COMMIT) ' -X ' $( OBJPREFIX) .GoVersion= $( GO_VERSION) ' " \
2024-05-23 09:53:40 +08:00
-tags dynamic -o $( INSTALL_PATH) /mmap-migration $( MMAP_MIGRATION_PATH) /main.go 1>/dev/null
