jaime
919e96ac22
enhance: add IT for rate limit using db properties ( #35930 )
...
issue: #35929
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:37:04 +08:00
SimFG
ea36d13ab0
feat: add static view for the expr ( #35887 )
...
- issue: #35886
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-04 11:09:04 +08:00
nico
40d34f711e
test: update test cases ( #35867 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-04 10:37:03 +08:00
Abdullah Ahmed
cc02dc0a55
fix: Handle Input/Output Errors in vsnprintf and snprintf ( #35898 )
...
Fix for Issue: #35897
2024-09-04 08:15:04 +08:00
congqixia
69b1eea6f2
enhance: [GoSDK] Add load option for field partial load ( #35920 )
...
Related to #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-03 19:27:04 +08:00
foxspy
9da86529a7
enhance: Add disk filemananger parallel load control to reduce the memory consumption ( #35281 )
...
issue: #35280
add parallel control to limit the memory consumption during index file
loading
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-09-03 18:01:03 +08:00
Zhen Ye
f68df9a11e
fix: SkipIndex cause segment fault ( #35907 )
...
issue: #35882
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-03 17:15:03 +08:00
ThreadDao
f068729a26
test: fix repeated random partition name ( #35805 )
...
/kind improvement
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-09-03 17:13:12 +08:00
wt
cb49b32358
test: add bitmap index cases ( #35909 )
...
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-03 16:49:03 +08:00
zhagnlu
74048ce34f
fix:rename mmap file path to avoid directory conflict ( #35810 )
...
#35784
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-03 16:05:03 +08:00
wei liu
c84ea5465c
fix: Fix some replicas don't participate in the query after the failure recovery ( #35850 )
...
issue: #35846
querycoord will notify proxy to update shard leader cache after
delegator location changes, but during querynode's failure recovery,
some delegator may become unserviceable due to lacking of segments, and
back to serviceable after segment loaded, so we also need to notify
proxy to invalidate shard leader cache when delegator serviceable state
changes.
This PR will maintain querynode's serviceable state during heartbeat,
and notify proxy to invalidate shard leader cache if serviceable state
changes.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-03 15:39:03 +08:00
yihao.dai
6130a85444
enhance: Remove bf from streaming node ( #35902 )
...
Remove bf from streaming node:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/33285 ,
https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-03 14:17:02 +08:00
zhagnlu
325f1987d9
enhance: rewrite index params for compatibility ( #35788 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 19:41:03 +08:00
congqixia
3698c53a72
enhance: Check load fields for previous loaded collection ( #35905 )
...
Related to #35415
This PR make querycoord report error when load request tries to update
load fields list, which is currently not supported.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 18:39:03 +08:00
Chun Han
4641fd9195
enhance: make search groupby stop when reaching topk groups ( #35814 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-02 18:25:03 +08:00
zhuwenxing
57422cb2ed
test: add array inverted index function test ( #35874 )
...
/kind improvement
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-02 17:49:02 +08:00
Zhen Ye
b2eb9fe2a7
fix: memory leak in unittest and open the USE_ASAN option when build unittest ( #35855 )
...
issue: #35854
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-02 15:59:04 +08:00
yihao.dai
6fd33285e1
fix: Fix compile error ( #35901 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-02 14:50:35 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
congqixia
9d80137698
fix: Check clustering key skip load behavior ( #35865 )
...
feature issue: #35415
See also #35861
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 11:17:02 +08:00
zhagnlu
576ac2bbed
fix: Fix the reference to a variable after it has been moved ( #35875 )
...
#35607
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 10:05:02 +08:00
Jiquan Long
5ea2454fdf
feat: tantivy tokenizer binding ( #35801 )
...
fix : #35800
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-01 17:13:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
yihao.dai
2e090b2426
enhance: Refine flusher channel management ( #35870 )
...
Change the ChannelTask to ChannelLifetime, only removing it upon
unregistering.
issue: https://github.com/milvus-io/milvus/issues/33285
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:07:10 +08:00
yihao.dai
ef451f5e1f
enhance: Add build-go target ( #35844 )
...
Add a `build-go` target to the Makefile that only compiles Go.
issue: https://github.com/milvus-io/milvus/issues/35611
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:05:09 +08:00
Yinzuo Jiang
407fc933e7
fix: bump bytedance/sonic to v1.12.2 to fix compilation error with go 1.23.0 ( #35879 )
...
fixes : #35878
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-09-01 13:43:01 +08:00
SimFG
311f860676
enhance: support to drop the role which is related the privilege list ( #35727 )
...
- issue: #35545
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 15:17:00 +08:00
yellow-shine
1e3e34d149
enhance: [ci] organize values file in the form of milvus deployment options ( #35832 )
...
https://github.com/milvus-io/milvus/issues/35842
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-30 13:07:15 +08:00
yellow-shine
ac736ad472
enhance: friendly for docker cache ( #35789 )
...
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-30 13:05:34 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
yihao.dai
1c3abd1aea
fix: Fix unstable TestFlusher ut ( #35831 )
...
issue: https://github.com/milvus-io/milvus/issues/35830
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-30 10:25:01 +08:00
Zhen Ye
f9be01ce89
fix: mockery config ( #35834 )
...
issue: #33285
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-30 09:49:00 +08:00
SimFG
07bb071aaf
enhance: add the unmarshal method for the rbac msg ( #35692 )
...
/kind improvement
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 07:23:00 +08:00
zhagnlu
671112d17b
enhance: add more info to hybrid index log ( #35808 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-29 21:07:04 +08:00
smellthemoon
a3f2f044d6
fix: not set nullable when stream writer write headers ( #35799 )
...
#35802
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 20:59:00 +08:00
zhenshan.cao
d10aa4626f
enhance: [skip e2e] add make run-test-cpp with support for filter gtest ( #35829 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-29 20:03:02 +08:00
congqixia
4373c6994f
fix: [GoSDK] reset cache after UsingDatabase ( #35638 )
...
Related to milvus-io/milvus-sdk-go#809
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 19:41:02 +08:00
yihao.dai
69265978bf
fix: Fix arrow go client ( #35819 )
...
issue: https://github.com/milvus-io/milvus/issues/35662
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 17:53:01 +08:00
congqixia
3c8941a88e
enhance: Print log only when rate limit updates ( #35806 )
...
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.
This PR make the log be printed only when rate limit is updated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 15:07:00 +08:00
congqixia
09ef3f1b4f
fix: Make sure querycoord observers started once ( #35811 )
...
Related to #35809
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 14:45:00 +08:00
Zhen Ye
70bc0e4538
enhance: add skip 3rdparty env var ( #35736 )
...
issue: #35611
- call `conan install` is too slow for developer, add option to skip it.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-29 14:21:01 +08:00
Patrick Weizhi Xu
b3089b5bdc
feat: support range search pagination retains order ( #35738 )
...
issue: #35464
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-29 14:09:00 +08:00
wei liu
73be0ba941
enhance: avoid the coexistence of the old coordinator and the new node/proxy ( #35720 )
...
issue: #35719
In standalone mode, block the start process until the new coordinator is
active to avoid the coexistence of the old coordinator and the new
node/proxy
1. In the start/restart process, the new coordinator will become active
immediately and will not be blocked
2. In the rolling upgrade process, the new coordinator will not be
active until the old coordinator is down, and it will be blocked
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-29 13:55:01 +08:00
zhuwenxing
e34b0c43a1
enhance: add GitHub Actions to delete spam issues comments ( #35807 )
...
/kind improvement
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-08-29 13:19:00 +08:00
jaime
b0ac04d104
enhance: set database properties to restrict read access ( #35745 )
...
issue: #35744
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-29 13:17:01 +08:00
smellthemoon
b51b4a2838
fix: try get not exist file after upgrade ( #35740 )
...
https://github.com/milvus-io/milvus/issues/35741
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 11:09:01 +08:00
congqixia
86691656f3
enhance: Change frequent balancer debug log to rated one ( #35749 )
...
"skip balance" log is too frequent in debug level. This PR changes it
into rated on.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:07:00 +08:00
congqixia
985d84d3ee
enhance: Add skip load validation for create collection task ( #35737 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:05:08 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
yellow-shine
ae53ce381b
delete pvc when finished ( #35783 )
...
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-29 09:57:06 +08:00
